You might look at
1. Turning on the security manager and tuning the security.policy file.
2. verify your applications, in your IDE and on deployment. applications that are not strictly valid JEE apps won't behave as well. (note that the verify doesn't check everything) Google for EJB programming restrictions to get more details on this.
3. consider separate thread pools for each application
http://blogs.sun.com/binublog/entry/monitoring_in_glassfish If you have one bad application it can starve the others of threads.
4. decide which resources and connection pools should be shared between the apps and which should be isolated. for example if all the applications are programmed to use the same jdbc connection pool in the application assembly phase (
http://java.sun.com/javaee/5/docs/tutorial/doc/bnaca.html) you can change this in the deployment descriptors and application server specific deployment descriptors. I have had applications a few times which acquired connections but never released them, which block other application using that pool)
5. additional domains are a good isolation feature. each domain will have its own jvm, ports, threads, etc. generally you want to do this to group related applications or provide 'bigger' applications with there own space.
[Message sent by forum member 'radix_zero']
http://forums.java.net/jive/thread.jspa?messageID=474193