My binary custom realm was compiled against appserv-rt.jar from Glassfish v2. It was then being used against a DIGEST auth-type as I mentioned. The combination of the two gave rise to the confusing error message mentioned above.
1) I did not mean to imply that backwards compatibility was broken for http-BASIC realms
2) Building custom realms for Glassfish is *way* too hard.
Honestly, you guys have to confront this front-and-center. There are tons of posts on this forum showing people running into the same problems over and over again.
Part of the problem is that there is no clear documentation saying: "here is how you do this for Glassfish v2, and here is how you do this for Glassfish v3". There are many different versions of the documentation floating around and users have no way of knowing which one is the correct one. Worse yet, the Glassfish v3 documentation does not say anything about how to build DIGEST custom realms so I assumed (incorrectly) that it was the same as building BASIC custom realms. The documentation should explicitly discuss this.
Furthermore, there should be more verbose and clearer debugging output for troubleshooting these problems. For example, if web.xml refers to a non-existant Realm then Glassfish should output an error message front-and-center saying "Could not find Realm <name>. This realm must show up in /domains/<your domain>/config/domain.xml under <auth-realm name="..."/>"
Messages like that would go *a long way* in helping users. You have to describe what operation failed and what could be done to fix it. You have to tell users *exactly* the key you're looking at. Just my 2 cents.
Gili
[Message sent by forum member 'cowwoc' (cowwoc)]
http://forums.java.net/jive/thread.jspa?messageID=321287