I had the same problem trying to install GlassFish v2ur2 on Mac OS X 10.5 (Leopard). I also encountered the permission problem with .asadmintruststore in my home directory (which is used by keytool). When I examined the attributes of the file, I realized that it was created with "root" permission. So I tried using "sudo" and specifying my root password for both the Ant setup as well as asadmin to start the server:
sudo lib/ant/bin/ant -f setup.xml
sudo asadmin start-domain domain1
Perhaps, I should consider changing the .asadmintruststore permissions, but this worked for now. It sounds like several others have had similar problems with both Mac OS X and Linux platforms. Give this a try.
[Message sent by forum member 'dpw67' (dpw67)]
http://forums.java.net/jive/thread.jspa?messageID=286278