Have you tried connecting with jConsole that comes with JDK 1.6, going to MBeans/com.sun.management/HotSpotDiagnostic/Operations/dumpHeap?
Once you've dumped heap, you can use jhat (also with JDK 1.6) to track down what's holding references to objects. It may be something as simple as adding a destroy() method to a servlet to properly uninitialize things between deployments.
I know almost nothing about GlassFish, but we've had memory issues on other J2EE type projects, and it ended up being Log4J/Commons-logging/or some other framework type thing that needs to be unitialized between deployments (ie LogManager.shutdown())
[Message sent by forum member 'acaldwel' (acaldwel)]
http://forums.java.net/jive/thread.jspa?messageID=226425