The issue you are having is at a JDK level. Take a look in server.log
The exact commandline that the JDK is started with is written to the log every time the server starts. Is "-Xrs" in there?
Try it without running as a service -- after all the main thing the Service does is simply
asadmin start-domain
-- does it work when you start the server manually?
[Message sent by forum member 'bnevins' (byron.nevins_at_sun.com)]
http://forums.java.net/jive/thread.jspa?messageID=386229