Heap Dump Analysis
As part of the investigation into cocoon memory usage, I had to try and understand what was going on inside the JVM. The best way to do that is a heap dump. The general idea is to dump out the contents of the JVM’s memory into a file, then analyse it using a tool.
First, getting a heap dump. This used to be something of a pain, but Java 6 now includes the jmap -dump command. This allows you to (relatively) easily extract a heap dump from a running JVM.
% pid=$(cat /some/where/pid.txt) % jmap=$JAVA_HOME/bin/jmap % sudo $jmap -F -dump:live,format=b,file=mla-post-cache-clear.hprof $pid
That takes a little while (a minute or two) to run, and ended up with a 500Mb file. It’s best to bring it back to your workstation in order to look through. Thankfully it compresses quite nicely.
% time rsync -avz server:mla-post-cache-clear.hprof . receiving file list ... done mla-post-cache-clear.hprof sent 42 bytes received 72988157 bytes 3105880.81 bytes/sec total size is 580115453 speedup is 7.95 rsync -avz server:mla-post-cache-clear.hprof . 8.42s user 2.85s system 47% cpu 23.643 total
Now things get interesting. We can fire up visualvm. Inside visualvm, do File → Load… and point it at the heap dump file we just obtained. You should end up with something like this:
Now this is interesting, but not that useful. If you click on the classes button, you see this:
What’s really interesting about that is the
byte line. Those instances are only 2% of objects, but are taking up 67% of memory! If you double click on that line, you get taken through to the instances view.
There are three panels here:
- The left hand view is a list of all the instances, ordered by size. In this case the top instances are the same size, which is quite suspicious.
- The top right hand view is a breakdown of the fields in the object. Not that useful for a
- The bottom right hand view is the useful one — it lists all references to this instance. Here it’s showing that thi
byteis linked to a ByteArrayOutputStream.
Now, you can expand the references to that
byte to find out where it’s being referenced. But there’s a really nifty trick. If you right-click, you get an option to “Show Nearest GC Root”. And this is what it shows (after a little while).
This isn’t the simplest thing to interpret. But, it does show that the HttpServletResponseBufferingWrapper is holding the
byte itself. Whilst that servlet response is held by a TraxTransformer. That sounds like a good place to begin investigating. Why is TraxTransformer holding on to ServletResponses?