I had modified domain.xml to include the -Xrs, which didn't work, thinking whatever appservService.exe does uses the JVM parameters in there. What worked for me was modifying <GF_HOME>/lib/processLauncher.xml, and adding:
<process name="as9-server">
...
<sysproperty key="-Xrs"/>
...
and now logging off the console doesn't kill the process.
[Message sent by forum member 'jshevland' (jshevland)]
http://forums.java.net/jive/thread.jspa?messageID=243041