Make sure that the domain.xml has only one auth-realm entry for Certificates and the name of the Realm should be "certificate" (you can specify your classname over there).
<auth-realm classname="MyCertificateRealm" name="certificate">
<property name="assign-groups" value="xyz" />
</auth-realm>
There is a Bug filed GlassFish for this.
[Message sent by forum member 'kumarjayanti' (kumarjayanti)]
http://forums.java.net/jive/thread.jspa?messageID=225927