The xmx/xms values are per instance so it would be possible to run 6 instances of xmx=512MB on an 8GB ram server (remember operating system and other apps such as databases would obviously require some of the remaining ram). I seem to recall a JVM limit of 2GB on 32 bit machines so there is a maximum in some circumstances (others will probably correct me of the exact details). It would seem a waste in having a 4 core 6GB server hosting a single GF instance when you could have more running. The allocated memory will depend on many things including connection pool sizes, number of web applications, processing threads etc. so there is no hard and fast rule. Smaller apps will probably only require an instance with 512MB memory. You will need to experiment and test to configure the instance to your requirements (connecting with JConsole to your running instance and viewing memory usage is a great start). You shouldn't assume that you need to allocate 4GB per instance.
Don't underestimate what Glassfish can do. As an example, we are hosting 5 public web apps per instance (with 512MB allocated per instance, each having 5 connection pools - one per web app - with 16 max connections each). The instances have been running for over 360 days and we are running four such GF V2.1 instances on a 4GB server (which is also running the MySQL database). Load per web app is not high but reliability and stability is superb!
[Message sent by forum member 'ggierer']
http://forums.java.net/jive/thread.jspa?messageID=481627