Well, the instructions say that you need -Xmx256m just to unjar the system and install it. But a 256Mb Java heap is > 256MB of real RAM. On my Mac I was able to extract the jar using -Xmx76m.
However, I also know that once it is installed, GF doesn't need that much RAM to actually run.
It also sounds like the 256MB limit on your zone is a hard limit, that is 256MB max including any swap space (which is unfortunate).
You may try running the install on a different system. even a different OS. The initial part of the install looks like little more than a extraction of the jar. So, once you've done that early phase, you can transfer the glassfish directory up to you zone, then continue with the setup.
Just make sure you use the correct archive for your Solaris. So, you'd use the Solaris archive and run the jar on, say, Windows. Then transfer the glassfish directory up, and once there you'd run lib/ant/bin/ant -f setup.xml.
Also, of course, what else is running in your zone? Perhaps there's a process or two you can kill while this is happening (maybe you have a lot of buffers allocated to your DB, but it's not needed for initial install).
Finally, just FYI, Java programs HATE memory pressure by the host system. There's few things more unhappy than a Java program that's swapped out. Not that it sounds like you have much swap anyway, so be cautious of your memory consumption on the system. Glassfish can be a bit heavy handed. If you don't need the EJB parts of Glassfish, I suggest running Jetty, or even Tomcat, instead. Jetty I know is a miser when it comes to memory.
GFv3 will fix a lot of this, but it's a way out yet.
[Message sent by forum member 'whartung' (whartung)]
http://forums.java.net/jive/thread.jspa?messageID=267853