First, getting a heap dump. This used to be something of a pain, but Java 6 now includes the jmap -dump command. This allows you to (relatively) easily extract a heap dump from a running JVM.

  % pid=$(cat /some/where/pid.txt)
  % jmap=$JAVA_HOME/bin/jmap
  % sudo $jmap -F -dump:live,format=b,file=mla-post-cache-clear.hprof $pid

That takes a little while (a minute or two) to run, and ended up with a 500Mb file. It’s best to bring it back to your workstation in order to look through. Thankfully it compresses quite nicely.

  % time rsync -avz server:mla-post-cache-clear.hprof .
  receiving file list ... done
  mla-post-cache-clear.hprof
 
  sent 42 bytes  received 72988157 bytes  3105880.81 bytes/sec
  total size is 580115453  speedup is 7.95
  rsync -avz server:mla-post-cache-clear.hprof .  8.42s user 2.85s system 47% cpu 23.643 total

Now things get interesting. We can fire up visualvm. Inside visualvm, do File → Load… and point it at the heap dump file we just obtained. You should end up with something like this:

Now this is interesting, but not that useful. If you click on the classes button, you see this:

What’s really interesting about that is the byte[] line. Those instances are only 2% of objects, but are taking up 67% of memory! If you double click on that line, you get taken through to the instances view.

There are three panels here:

1. The left hand view is a list of all the instances, ordered by size. In this case the top instances are the same size, which is quite suspicious.
2. The top right hand view is a breakdown of the fields in the object. Not that useful for a byte[].
3. The bottom right hand view is the useful one — it lists all references to this instance. Here it’s showing that thi byte[] is linked to a ByteArrayOutputStream.

Now, you can expand the references to that byte[] to find out where it’s being referenced. But there’s a really nifty trick. If you right-click, you get an option to “Show Nearest GC Root”. And this is what it shows (after a little while).

This isn’t the simplest thing to interpret. But, it does show that the HttpServletResponseBufferingWrapper is holding the byte[] itself. Whilst that servlet response is held by a TraxTransformer. That sounds like a good place to begin investigating. Why is TraxTransformer holding on to ServletResponses?