EvilRob.org -> Weblog

Sysadmin Field Notes

The memory leak that ate a week of my life.

April 30, 2006

Where have I been? Debugging a gnarly memory leak in our just-released production application (not in our code, but in another part of the system).

I wouldn't have even thought it possible to leak 2-3MB of memory per request. Yes, that's not a typo, that's per request. How? Through a combination of an xsl transform retrieval method that the caller thought was caching underneath but wasn't, some old untouched xsl, and some weird behavior with script blocks in proprietary script blocks from MS in XSL files.

So we were leaking assemblies like a sieve. Thirty or so per request, as seen by the perfmon counters for bytes in loader heap and current assemblies (under CLR loading and CLR memory).

Remember to always check your bytes in loader heap when you have a CLR memory leak, as well as the normal heaps. Otherwise you won't see the growth, but you'll see the private bytes growing, and assume that it's a native leak.

I also found it interesting that the CLR has 3 generations instead of Java's 2. It makes watching for heap growth in the aggregate counter (# of bytes in all heaps) a lot more difficult. I recommend looking at the behavior of each heap individually, it makes the behavior a lot clearer.

Anyway, problem found. We may have a slow leak as well, but first up was definitely to stop this bleeding.

Posted by rmeyer at 8:14 AM

This is Rob Meyer's weblog, a weblog focused on software development and system administration based on 10 years of experience. Want to explore further? You can find out more me or see the rest of my website.

Wondering if I've written on something in particular? Try searching:

You might want to take a look at some of the more requested postings (as judged by incoming traffic):

Want more? Subscribe to this site or contact me at rob at big dis dot com.

See my writings on:


Powered by Movable Type | Technorati Profile