dev@glassfish.java.net

Re: Instance instance xxx is not reachable]

From: Nandini Ektare <Nandini.Ektare_at_Sun.COM>
Date: Tue, 26 Sep 2006 12:07:05 -0700

Sreenivas Munnangi wrote:
> Sreenivas Munnangi wrote:
>
>> Nandini Ektare wrote:
>>
>>> Jothir
>>>
>>> Not sure if my following suggestion reached you. (I remember there
>>> was a thread of discussion and I assume Sreeni has already made you
>>> aware of various suggestions in that thread.)
>>> The only known environment change that happened between b13 and b15
>>> was that of JDK minor version. So you might want to try with older
>>> version.
>>>
>>> "One more important thing... JDK build version changed between the
>>> two builds (switched to 1.5.0_08 on 8/29/06 so builds after that are
>>> using this b15, b16..)... I have not found any specific fixes
>>> related to java.io.ObjectOutputStream (see stack trace below)... but
>>> it will be good to check against the old JDK version to rule out
>>> this variable. JDK performance fixes had to be picked up... So this
>>> change could be a regression due to that.
>>>
>>> [#|2006-09-12T12:36:04.046+0530|WARNING|sun-appserver-ee9.1|javax.ee.enterprise.system.tools.admin|_ThreadID=52;_ThreadName=event-handler-2;_RequestID=3ea01d2b-1c34-429c-b32d-d4d8cc24dbb3;|Instance:
>>> [instance3] not reachable
>>> com.sun.enterprise.admin.servermgmt.InstanceException:
>>> java.io.UTFDataFormatException
>>> at
>>> com.sun.enterprise.ee.admin.proxy.InstanceProxy.mangleInvokeException(InstanceProxy.java:150)
>>>
>>> at
>>> com.sun.enterprise.ee.admin.proxy.InstanceProxy.invoke(InstanceProxy.java:176)
>>>
>>> at $Proxy23.forwardEvent(Unknown Source)
>>> at
>>> com.sun.enterprise.ee.admin.event.EndPointHandler.run(EndPointHandler.java:88)
>>>
>>> at java.lang.Thread.run(Thread.java:595)
>>> Caused by: java.io.UTFDataFormatException
>>> at
>>> java.io.ObjectOutputStream$BlockDataOutputStream.writeUTF(ObjectOutputStream.java:1969)
>>>
>>> at
>>> java.io.ObjectOutputStream$BlockDataOutputStream.writeUTF(ObjectOutputStream.java:1812)
>>>
>>> at java.io.ObjectOutputStream.writeUTF(ObjectOutputStream.java:816)
>>> at
>>> com.sun.enterprise.config.serverbeans.Domain.writeObject(Domain.java:606)
>>>
>>> at sun.reflect.GeneratedMethodAccessor145.invoke(Unknown Source)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>
>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>> at
>>> java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:917)
>>> at
>>> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1339)
>>>
>>> at
>>> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
>>>
>>
>>
>> Nandini, I don't think the sole cause for "Instance instance1 is not
>> reachable" is java.io.UTFDataFormatException. Because we have seen
>> this error even without java.io.UTFDataFormatException. Seems we need
>> some way of reproducing this consistently to better understand the
>> issue.
So you are saying we have tried with earlier versions? Reason to double
check on this is that two of us reported that we cannot reproduce the
bug on b16 but both were using an older JDK version. There were no
changes related to this from admin and deployment side(deployment team
confirmed this) between these builds. So in all probability it is to do
with the setup/environment.

About consistently reproducing this, I though Jothir reported deployment
always fails for a cluster.

Thanks
Nandini
>>
> Also this seems an old issue
> http://forum.sun.com/jive/thread.jspa?forumID=114&threadID=64882
>
>>>
>>> Thanks
>>> Nandini
>>>
>>>
>>> jothir ganesan wrote:
>>>
>>>> Yes Kedar,
>>>>
>>>> I am stuck with this issue for 3 builds starting from build 16. It
>>>> is reproducible on Linux too. Please see bug#6467623.
>>>> Could you please explain what the EE QL is?
>>>>
>>>> Thanks
>>>> Jothir
>>>>
>>>> Kedar Mhaswade wrote:
>>>>
>>>>> Also, can you try the same thing on an older build? (b16, b17 for
>>>>> instance)?
>>>>>
>>>>> Thanks,
>>>>> Kedar
>>>>>
>>>>> Kedar Mhaswade wrote:
>>>>>
>>>>>> Hi Jothir,
>>>>>>
>>>>>> I am seeing these errors while running the EE QL on Solaris.
>>>>>> Let us investigate ...
>>>>>> The tinderbox appears to be all green, however.
>>>>>>
>>>>>> Regards,
>>>>>> Kedar
>>>>>>
>>>>>> jothir ganesan wrote:
>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> Could anyone please let me know the possible reasons for
>>>>>>> "EEADM0068: Instance instance xxx is not reachable" ? The
>>>>>>> exception shows up when trying to deploy an app on the cluster
>>>>>>> and the app fails to get deployed...
>>>>>>>
>>>>>>> I have the following setup:
>>>>>>>
>>>>>>> 3 nodeagents on 3 different machines with each nodeagent having
>>>>>>> one instance.
>>>>>>> Domain is started and the cluster,created on DAS, with the three
>>>>>>> instances - running
>>>>>>> Nodeagents- running
>>>>>>>
>>>>>>> Other configuration did:
>>>>>>> - hadb nodes running and ha-cluster is configured.
>>>>>>> - Access log buffer size ->
>>>>>>> cluster1-config.http-service.virtual-server.server.property.accessLogBufferSize
>>>>>>> set to 10 for cluster config
>>>>>>> - cluster1-config.http-service.keep-alive.timeout-in-seconds set
>>>>>>> to 0.
>>>>>>> - Some jdbc resources and connection pools created.
>>>>>>>
>>>>>>>
>>>>>>> I have my nodeagents and instances up and running.
>>>>>>> Command start-cluster executed successfully.
>>>>>>> bash-3.00# asadmin list-instances --passwordfile passwordfile
>>>>>>> instance1 running
>>>>>>> instance2 running
>>>>>>> instance3 running
>>>>>>> Command list-instances executed successfully.
>>>>>>> bash-3.00# /space/91/appserver/bin/asadmin deploy --host
>>>>>>> localhost --port 4848 --user admin --passwordfile passwordfile
>>>>>>> --target cluster1
>>>>>>> /space/91/sift/agent/file-repository/70/Apps/SampleApplications/Clusterjsp1/clu
>>>>>>> sterjsp1.war
>>>>>>> Command deploy executed successfully.
>>>>>>> bash-3.00# asadmin list-node-agents --passwordfile passwordfile
>>>>>>> sqe-agent1 running
>>>>>>> sqe-agent2 running
>>>>>>> sqe-agent3 running
>>>>>>> Command list-node-agents executed successfully.
>>>>>>>
>>>>>>> But still when I try to deploy the app, I could see :
>>>>>>> [#|2006-09-26T16:03:20.722+0530|WARNING|sun-appserver-ee9.1|javax.ee.enterprise
>>>>>>> .system.tools.admin|_ThreadID=198;_ThreadName=event-handler-0;instance1;_Request
>>>>>>> ID=4c5551da-d066-451e-a7a9-ee36c5a8ab47;|EEADM0068: Instance
>>>>>>> instance1 is not reachable.|#]
>>>>>>>
>>>>>>> [#|2006-09-26T16:03:20.739+0530|WARNING|sun-appserver-ee9.1|javax.ee.enterprise
>>>>>>> .system.tools.admin|_ThreadID=199;_ThreadName=event-handler-1;instance2;_Request
>>>>>>> ID=3c7c1df3-a47f-44a9-a418-e506bab400bf;|EEADM0068: Instance
>>>>>>> instance2 is not reachable.|#]
>>>>>>>
>>>>>>> [#|2006-09-26T16:03:20.759+0530|WARNING|sun-appserver-ee9.1|javax.ee.enterprise
>>>>>>> .system.tools.admin|_ThreadID=200;_ThreadName=event-handler-2;instance3;_Request
>>>>>>> ID=da9f94fd-c35e-417d-ba3a-41c0cecc91c5;|EEADM0068: Instance
>>>>>>> instance3 is not reachable.|#]
>>>>>>>
>>>>>>> What else do I need to check?
>>>>>>>
>>>>>>> Thanks
>>>>>>> Jothir
>>>>>>>