In v3 most jars are in the modules directory. The lib directory only contains jars that may be used by some old code.
The class com.sun.enterprise.security.auth.realm.IASRealm is in security.jar. The manifest in appserv-rt.jar references this jar, so the file can be accessed including this jar too.
The NoClassDefFoundError you report seems strange, since this class is an internal class used for many purposes. I believe there is some problem with your installation. Please reinstall and try again.
[Message sent by forum member 'nasradu8' (Sudarsan.Sridhar_at_sun.com)]
http://forums.java.net/jive/thread.jspa?messageID=385796