dev@glassfish.java.net

Re: Instability of QuickLook tests

From: Marina Vatkina <Marina.Vatkina_at_Sun.COM>
Date: Sun, 04 Oct 2009 14:33:05 -0700

Ken,

I usually do 'mvn clean' before updating the ws. to avoid left behind class
files after any major changes.

Regards,
-marina

Ken Cavanaugh wrote:
>
> On Oct 4, 2009, at 11:43 AM, Ken Cavanaugh wrote:
>
>>
>> On Oct 4, 2009, at 11:34 AM, Byron Nevins wrote:
>>
>>> Ken,
>>>
>>> How can I check out the gmbal source? I can't find a working svn or
>>> hg URL from the website...
>>
>>
>> Go to http://kenai.com/projects/gmbal/sources, which lists the hg URL.
>> https://hg.kenai.com/hg/gmbal~master
>> is the one I always use; if it's not working for you today, it must be
>> a Kenai site problem.
>> hg pull should work for anyone; hg push requires developer role in the
>> Gmbal project.
>
>
> Note that I have not yet released gmbal b016 to maven.
>
> By the way, from all of the discussion, it looks the problem I
> have is most likely because I have not been re-installing GFv3 between
> runs of the QL.
> My normal procedure is:
>
> * update my GFv3 working copy
> * mvn -U install
> * run a script that installs the newly build GFv3 and runs the QL
> (sanity check)
> * copy my new modules (e.g. CORBA or Gmbal) into the modules directory
> * run a script that re-runs the QL (using the same install as
> before, since I just updated the modules directory)
>
>
> I do this because I want to verify that QLs are functioning correctly
> before I try my updates.
> Unfortunately this seems to CAUSE the exceptions I've mentioned previously.
>
> I'll modify my procedures to re-install before every QL run, but that's
> a little less convenient.
>
> Ken.
>
>>
>> Ken.
>>
>>>
>>>
>>> Ken Cavanaugh wrote:
>>>
>>>> Darani and I have been running the QLs on multiple machines in order to
>>>> verify the functioning of ORB b030 (GFv3 is currently using ORB b029),
>>>> and every time we run the tests we get different results. I've had
>>>> the same
>>>> problems with trying to integrate Gmbal b016.
>>>>
>>>> The main tests that do this in the QL appear to be
>>>> iterateAllSanityCheck (an
>>>> AMX test), testAMXComplianceMonitorFailureCount, and deleteListener.
>>>> Over 4 QL runs, I've observed the following:
>>>>
>>>> * Run 1: passed with Gmbal b015
>>>> * Run 2: all 3 tests failed with Gmbal b016
>>>> * Run 3: with CORBA b030, testAMXComplianceMonitorFailureCount
>>>> failed
>>>> o The AMX compliance monitor complained about the following:
>>>> + Attribute 'Enabled' failed for
>>>> amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
>>>> + Attribute 'JvmOptions' failed for
>>>> amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
>>>> + Attribute 'NativeLibraryPath' failed for
>>>> amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
>>>> + Attribute 'Classpath' failed for
>>>> amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
>>>> + Attribute 'Property' failed for
>>>> amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
>>>> + amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
>>>> General test failure in validateAMXConfig:
>>>> java.lang.NullPointerException: "null"
>>>> org.glassfish.admin.amx.core.AMXValidator.validateAMXConfig(AMXValidator.java:650)
>>>> org.glassfish.admin.amx.core.AMXValidator._validate(AMXValidator.java:618)
>>>> org.glassfish.admin.amx.core.AMXValidator.validate(AMXValidator.java:1213)
>>>> org.glassfish.admin.amx.impl.mbean.ComplianceMonitor$ValidatorThread.doRun(ComplianceMonitor.java:235)
>>>> org.glassfish.admin.amx.impl.mbean.ComplianceMonitor$ValidatorThread.run(ComplianceMonitor.java:206)
>>>> * Run 4: I re-ran the same software as in run 3. This time all
>>>> 3 tests failed.
>>>> + There were several Gmbal failures in the log
>>>> (unregister failures). These were all caused by
>>>> InstanceNotFoundExceptions on a number of MBeans:
>>>> # amx:pp=/mon/server-mon[server],type=bean-method-mon,name=remoteview/HelloBean/bean-methods/hello
>>>> # amx:pp=/mon/server-mon[server],type=bean-method-mon,name=remoteview/HelloBean/bean-methods/asyncCancel-int
>>>> # amx:pp=/mon/server-mon[server],type=bean-method-mon,name=remoteview/HelloBean/bean-methods/asyncThrowException-java.lang.String
>>>> # amx:pp=/mon/server-mon[server],type=bean-method-mon,name=remoteview/HelloBean/bean-methods/throwException-java.lang.String
>>>> # amx:pp=/mon/server-mon[server],type=bean-method-mon,name=remoteview/HelloBean/bean-methods/asyncBlock-int
>>>> # etc. for a total of 153 MBeans: each
>>>> unregister failure happened twice in a row.
>>>>
>>>> Now, I initially assumed that the QL failures I observed in Run 2
>>>> were do to interactions between
>>>> a new Gmbal feature (root parent monitoring) and the rest of GFv3
>>>> monitoring. But the same tests
>>>> seem to fail almost randomly in other situations. Particularly
>>>> disturbing is the fact the Run 3 and Run 4
>>>> shared completely IDENTICAL app server code, and exhibited different
>>>> failures. I also highly doubt that
>>>> the ORB has anything to do with the observed (and unpredictable)
>>>> failures. Note that very similar failures
>>>> are seen both with Gmbal b016 and with CORBA b030. That coincidence
>>>> and the lack of repeatability lead
>>>> me to wonder about the stability of the QLs, and whether the
>>>> failures on Gmbal b016 have anything at all to do
>>>> with the recent Gmbal changes.
>>>>
>>>> This is a big problem. Darani and I have been unable to complete
>>>> any CORBA or Gmbal integrations recently,
>>>> because we keep seeing different failures. Darani had b30 passing
>>>> ALL tests at one point, then when she attempted
>>>> to verify one final time to complete the integration, similar
>>>> failures occurred.
>>>>
>>>> We need help to understand why we see this level of instability in
>>>> QL runs. Without fixing this, we
>>>> are unable to integrate either CORBA or Gmbal, and we are nearly out
>>>> of time for a number
>>>> of important fixes (many of which are ready to go) before hard code
>>>> freeze.
>>>>
>>>> Thanks,
>>>>
>>>> Ken.
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>> <mailto:dev-unsubscribe_at_glassfish.dev.java.net> For additional
>>>> commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>> <mailto:dev-help_at_glassfish.dev.java.net>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>> <mailto:dev-unsubscribe_at_glassfish.dev.java.net> For additional
>>> commands, e-mail: dev-help_at_glassfish.dev.java.net
>>> <mailto:dev-help_at_glassfish.dev.java.net>
>>
>>
>