I can reproduce the problem with an extension library JAR in ${domainDir}/lib/ext.
The __system.jnlp refers to the extension JAR so that, if the client needs it, it's available. But that means that Java Web Start will try to download it regardless.
So there are two issues - one is the unnecessary downloading of extension JARs that are not used by a given client. That's a convenience/performance issue. The other is that the server is not able to respond to the request from Java Web Start to download the extension, which is a functionality problem.
I have opened Issue 11713 (
https://glassfish.dev.java.net/issues/show_bug.cgi?id=11713) to track this.
[Message sent by forum member 'tjquinn']
http://forums.java.net/jive/thread.jspa?messageID=393164