dev@glassfish.java.net

Re: Instability of QuickLook tests

From: Ken Cavanaugh <Ken.Cavanaugh_at_Sun.COM>
Date: Sun, 04 Oct 2009 11:48:19 -0700

On Oct 4, 2009, at 11:43 AM, Ken Cavanaugh wrote:

>
> On Oct 4, 2009, at 11:34 AM, Byron Nevins wrote:
>
>> Ken,
>>
>> How can I check out the gmbal source? I can't find a working svn
>> or hg URL from the website...
>
> Go to http://kenai.com/projects/gmbal/sources, which lists the hg
> URL. https://hg.kenai.com/hg/gmbal~master
> is the one I always use; if it's not working for you today, it must
> be a Kenai site problem.
> hg pull should work for anyone; hg push requires developer role in
> the Gmbal project.

Note that I have not yet released gmbal b016 to maven.

By the way, from all of the discussion, it looks the problem I
have is most likely because I have not been re-installing GFv3 between
runs of the QL.
My normal procedure is:

update my GFv3 working copy
mvn -U install
run a script that installs the newly build GFv3 and runs the QL
(sanity check)
copy my new modules (e.g. CORBA or Gmbal) into the modules directory
run a script that re-runs the QL (using the same install as before,
since I just updated the modules directory)

I do this because I want to verify that QLs are functioning correctly
before I try my updates.
Unfortunately this seems to CAUSE the exceptions I've mentioned
previously.

I'll modify my procedures to re-install before every QL run, but
that's a little less convenient.

Ken.

>
> Ken.
>>
>>
>> Ken Cavanaugh wrote:
>>>
>>> Darani and I have been running the QLs on multiple machines in
>>> order to
>>> verify the functioning of ORB b030 (GFv3 is currently using ORB
>>> b029),
>>> and every time we run the tests we get different results. I've
>>> had the same
>>> problems with trying to integrate Gmbal b016.
>>>
>>> The main tests that do this in the QL appear to be
>>> iterateAllSanityCheck (an
>>> AMX test), testAMXComplianceMonitorFailureCount, and deleteListener.
>>> Over 4 QL runs, I've observed the following:
>>> Run 1: passed with Gmbal b015
>>> Run 2: all 3 tests failed with Gmbal b016
>>> Run 3: with CORBA b030, testAMXComplianceMonitorFailureCount failed
>>> The AMX compliance monitor complained about the following:
>>> Attribute 'Enabled' failed for amx:pp=/domain/configs/
>>> config[server-config]/java-config,type=profiler
>>> Attribute 'JvmOptions' failed for amx:pp=/domain/configs/
>>> config[server-config]/java-config,type=profiler
>>> Attribute 'NativeLibraryPath' failed for amx:pp=/domain/configs/
>>> config[server-config]/java-config,type=profiler
>>> Attribute 'Classpath' failed for amx:pp=/domain/configs/
>>> config[server-config]/java-config,type=profiler
>>> Attribute 'Property' failed for amx:pp=/domain/configs/
>>> config[server-config]/java-config,type=profiler
>>> amx:pp=/domain/configs/config[server-config]/java-
>>> config,type=profiler
>>> General test failure in validateAMXConfig:
>>> java.lang.NullPointerException: "null"
>>> org
>>> .glassfish
>>> .admin.amx.core.AMXValidator.validateAMXConfig(AMXValidator.java:
>>> 650)
>>> org
>>> .glassfish.admin.amx.core.AMXValidator._validate(AMXValidator.java:
>>> 618)
>>> org
>>> .glassfish.admin.amx.core.AMXValidator.validate(AMXValidator.java:
>>> 1213)
>>> org.glassfish.admin.amx.impl.mbean.ComplianceMonitor
>>> $ValidatorThread.doRun(ComplianceMonitor.java:235)
>>> org.glassfish.admin.amx.impl.mbean.ComplianceMonitor
>>> $ValidatorThread.run(ComplianceMonitor.java:206)
>>> Run 4: I re-ran the same software as in run 3. This time all 3
>>> tests failed.
>>> There were several Gmbal failures in the log (unregister
>>> failures). These were all caused by InstanceNotFoundExceptions on
>>> a number of MBeans:
>>> amx:pp=/mon/server-mon[server],type=bean-method-
>>> mon,name=remoteview/HelloBean/bean-methods/hello
>>> amx:pp=/mon/server-mon[server],type=bean-method-
>>> mon,name=remoteview/HelloBean/bean-methods/asyncCancel-int
>>> amx:pp=/mon/server-mon[server],type=bean-method-
>>> mon,name=remoteview/HelloBean/bean-methods/asyncThrowException-
>>> java.lang.String
>>> amx:pp=/mon/server-mon[server],type=bean-method-
>>> mon,name=remoteview/HelloBean/bean-methods/throwException-
>>> java.lang.String
>>> amx:pp=/mon/server-mon[server],type=bean-method-
>>> mon,name=remoteview/HelloBean/bean-methods/asyncBlock-int
>>> etc. for a total of 153 MBeans: each unregister failure happened
>>> twice in a row.
>>> Now, I initially assumed that the QL failures I observed in Run 2
>>> were do to interactions between
>>> a new Gmbal feature (root parent monitoring) and the rest of GFv3
>>> monitoring. But the same tests
>>> seem to fail almost randomly in other situations. Particularly
>>> disturbing is the fact the Run 3 and Run 4
>>> shared completely IDENTICAL app server code, and exhibited
>>> different failures. I also highly doubt that
>>> the ORB has anything to do with the observed (and unpredictable)
>>> failures. Note that very similar failures
>>> are seen both with Gmbal b016 and with CORBA b030. That
>>> coincidence and the lack of repeatability lead
>>> me to wonder about the stability of the QLs, and whether the
>>> failures on Gmbal b016 have anything at all to do
>>> with the recent Gmbal changes.
>>>
>>> This is a big problem. Darani and I have been unable to complete
>>> any CORBA or Gmbal integrations recently,
>>> because we keep seeing different failures. Darani had b30 passing
>>> ALL tests at one point, then when she attempted
>>> to verify one final time to complete the integration, similar
>>> failures occurred.
>>>
>>> We need help to understand why we see this level of instability in
>>> QL runs. Without fixing this, we
>>> are unable to integrate either CORBA or Gmbal, and we are nearly
>>> out of time for a number
>>> of important fixes (many of which are ready to go) before hard
>>> code freeze.
>>>
>>> Thanks,
>>>
>>> Ken.
>>> --------------------------------------------------------------------- To
>>> unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net For
>>> additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>> --------------------------------------------------------------------- To
>> unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net For
>> additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>