Basically, the approach you're taking won't work at all from the server, as those APIs you reference are used for a program running on a Windows machine to launch a local program.
What you need to do is somehow convince the users Web browser to launch the program, which has nothing to do with Glassfish (I don't know how to do that either).
You might be able to use a Java Applet to pull this off, but you will run afoul of the security infrastructure that's essentially designed, among others things, to keep you from doing what you want to do. But with a singed applet, it might be possible.
Alas, that's a topic for the Applet folks, as it's not related to Glassfish either.
Finally, another way it could be done is to have a little Java program already running on the users computer (it can even live in the toolbar), and that program can either poll or simply listen on a JMS topic for a message from the server to start up a program, and then the server can post a message to the topic whenever it wants to start up the program. GF can be used to host those JMS topics, but the program is a generic, Java SE program.
[Message sent by forum member 'whartung' (whartung)]
http://forums.java.net/jive/thread.jspa?messageID=295191