I was having the same problem as the original poster and though some time has passed, I am posting this for the benefit of the next person who may be having this problem.
My problem was that I used keytool to import a certificate that was generated from a private key NOT in the keystore. Glassfish (Sun Java System Application Server 9.0, to be exact) stopped responding to web requests and the solution required having the certificate reissued and editing domain.xml manually to replace the reference (the "alias") to the old certificate and replace it with the name of the alias of the new certificate.
Now first, I want to say that my limited experience with Glassfish has been great, and I'm planning to keep using it by all means.
I installed Glassfish, which went extremely smoothly and with the management web interface, was really easy to configure. It took minutes as opposed to the normal hours of download/install/fix that usually happens (I do not do this often enough.)
My problems started when I tried to install an SSL certificate that I had obtained for a plain Apache configuration.
I used keytool to import the existing certificate, and when I restarted Glassfish, and tried to access the secure site, it never completed the response and left the browser hanging. Ditto with the web interface - it was no longer usable.
I saw the same exceptions in the logfile that the first poster did. It seemed that the private key had to be in the keystore along with the certificate. But the keytool app does not provide a means of inserting the private key into the keystore.
My certificate was a (relatively) inexpensive SSL123 cert from Thawte, which will let you "start over' if you need to. For free, you can have them reissue a certificate (which invalidates the old one) from their website.
So I followed various instructions I found for generating a key in the keystore Glassfish would use, generating a certificate request, using Thawte's service to reissue my cert, approved it, downloaded it afte waiting about a half hour, and installed it with keytool.
(I must point out that this was not as easy as it sounds. You have to get the commands exactly right because when you don't, the error messages often are for problems unrelated to what really went wrong. Also, I would advise everyone to be super careful when cutting and pasting the cert as I spent 8 hours on this particular problem, with at least three of them caused by an extra empty line at the top of the certificate file (I had not noticed it when pasting.) Thawte warns to eliminate extraneous whitespace at the end of each line after pasting. The bottom line is, make sure the alias (basically, a name so you can easily identify the certificate lately and which Glassfish uses to refer to it) is the same AND specified with the -alias parameter every time; make sure you're using the same keystore you generated the request from (the documentation in one case made it seem otherwise), and go over the certificate file with a fine tooth comb to be sure there is no extra whitespace or newline preceding the header line, at the beginning or end of each line, or following the footer.)
Once you have the key installed, you might try using keytool to delete the old key (hope you used a different alias for the reissued key than the key that did not work!). Not that it matters much but if you go poking around the keystore later you might wonder why it's there and whether or not you still need it - or worse, you might try to use it and lock up your app server again.
Finally, you need to get the app server out of its locked state. Restarting it did not help me. Perhaps deleting the old key from the keystore would help; I did not try this. What I did was edit the domain.xml file manually, searched for and found the alias for the old key, and replaced it with the alias for the new key.
Then I used the command line app server admin tool (asadmin, found in the bin directory of the app server's root) to restart the app server, and everything worked.
To summarize:
- Don't try to import a certificate whose private key isn't in the keystore.
- Make very sure that the certificate has no extraneous whitespace.
- Be sure you understand what the "alias" is (just a simple name that the app server uses to find the cert in the keystore) and make sure you include it when using any keytool commands.
- Do not trust the keytool or openssl command error text to be relevant to the problem you're having.
Let me tell you, while I was trying to figure this out, it was a real pain in the keystore.
=========
Some extra stuff for the benefit of other people who might have similar issues:
keytool gave me this error when I tried to import my newly reissued certificate. I spent hours going down the wrong track because it was not relevant to the problem (an extra blank line at the beginning of the certificate, that had found its way in while I was copying it from the Thawte website into a text editor):
keytool error: java.lang.Exception: Input not an X.509 certificate
Others have report that keystore will generate the same error if the alias is not specified with the -alias keyword in the keytool command.
This led me to mistakenly believe that somehow I had to convert my PKCS #7 certificate into an X.509 certificate (not knowing if that was even possible, since I don't spend much time in this space.) openssl gave me errors like this when encountering my certificate with the extra carriage return at the beginning:
unable to load PKCS7 object
14492:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:644:Expecting: PKCS7
and
unable to load PKCS7 object
14480:error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1296:
14480:error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:380:Type=PKCS7
[Message sent by forum member 'sean_sheedy' (sean_sheedy)]
http://forums.java.net/jive/thread.jspa?messageID=248156