From: Byron Nevins <Byron.Nevins_at_Sun.COM>
Date: Sun, 04 Oct 2009 11:34:17 -0700

Ken,

How can I check out the gmbal source?  I can't find a working svn or hg URL from the website...


Ken Cavanaugh wrote:

Darani and I have been running the QLs on multiple machines in order to
verify the functioning of ORB b030 (GFv3 is currently using ORB b029),
and every time we run the tests we get different results.  I've had the same
problems with trying to integrate Gmbal b016.

The main tests that do this in the QL appear to be iterateAllSanityCheck (an
AMX test), testAMXComplianceMonitorFailureCount, and deleteListener.
Over 4 QL runs, I've observed the following:

• Run 1: passed with Gmbal b015
• Run 2: all 3 tests failed with Gmbal b016
• Run 3: with CORBA b030, testAMXComplianceMonitorFailureCount failed
• The AMX compliance monitor complained about the following:
• Attribute 'Enabled' failed for amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
• Attribute 'JvmOptions' failed for amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
• Attribute 'NativeLibraryPath' failed for amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
• Attribute 'Classpath' failed for amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
• Attribute 'Property' failed for amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
  
• amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
  General test failure in validateAMXConfig:
  java.lang.NullPointerException: "null"
  org.glassfish.admin.amx.core.AMXValidator.validateAMXConfig(AMXValidator.java:650)
  org.glassfish.admin.amx.core.AMXValidator._validate(AMXValidator.java:618)
  org.glassfish.admin.amx.core.AMXValidator.validate(AMXValidator.java:1213)
  org.glassfish.admin.amx.impl.mbean.ComplianceMonitor$ValidatorThread.doRun(ComplianceMonitor.java:235)
  org.glassfish.admin.amx.impl.mbean.ComplianceMonitor$ValidatorThread.run(ComplianceMonitor.java:206)
  
• Run 4: I re-ran the same software as in run 3.  This time all 3 tests failed.
• There were several Gmbal failures in the log (unregister failures). These were all caused by InstanceNotFoundExceptions on a number of MBeans:
• amx:pp=/mon/server-mon[server],type=bean-method-mon,name=remoteview/HelloBean/bean-methods/hello
• amx:pp=/mon/server-mon[server],type=bean-method-mon,name=remoteview/HelloBean/bean-methods/asyncCancel-int
• amx:pp=/mon/server-mon[server],type=bean-method-mon,name=remoteview/HelloBean/bean-methods/asyncThrowException-java.lang.String
• amx:pp=/mon/server-mon[server],type=bean-method-mon,name=remoteview/HelloBean/bean-methods/throwException-java.lang.String
• amx:pp=/mon/server-mon[server],type=bean-method-mon,name=remoteview/HelloBean/bean-methods/asyncBlock-int
• etc. for a total of 153 MBeans: each unregister failure happened twice in a row.

Now, I initially assumed that the QL failures I observed in Run 2 were do to interactions between
a new Gmbal feature (root parent monitoring) and the rest of GFv3 monitoring.  But the same tests
seem to fail almost randomly in other situations.  Particularly disturbing is the fact the Run 3 and Run 4
shared completely IDENTICAL app server code, and exhibited different failures.  I also highly doubt that
the ORB has anything to do with the observed (and unpredictable) failures.  Note that very similar failures
are seen both with Gmbal b016 and with CORBA b030.  That coincidence and the lack of repeatability lead
me to wonder about the stability of the QLs, and whether the failures on Gmbal b016 have anything at all to do
with the recent Gmbal changes.

This is a big problem.  Darani and I have been unable to complete any CORBA or Gmbal integrations recently,
because we keep seeing different failures.  Darani had b30 passing ALL tests at one point, then when she attempted
to verify one final time to complete the integration, similar failures occurred.

We need help to understand why we see this level of instability in QL runs.  Without fixing this, we
are unable to integrate either CORBA or Gmbal, and we are nearly out of time for a number
of important fixes (many of which are ready to go) before hard code freeze.

Thanks,

Ken.
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@glassfish.dev.java.net For additional commands, e-mail: dev-help@glassfish.dev.java.net