I tried to reproduce the issue with the promoted b66
(
http://download.java.net/glassfish/v3/promoted/glassfish-v3-b66.zip)
and here are what I found:
* Run QL first time in web distribution profile and all passed.
* Run QL again and the amx or admin tests start having errors
* To isolate the problem, I deleted the b66 and unzip again. Run amx
test individually multiple times and tests passed:
o quicklook/amx % ant
-Dglassfish.home=/export/home/ming/v3/b66/glassfishv3/glassfish
all
* Start domain and derby. Run admin tests individually. The tests
passed first time and started failed when ran again.
o quicklook/admin % ant
-Dglassfish.home=/export/home/ming/v3/b66/glassfishv3/glassfish
all
The stack trace on server side is clean:
http://javaweb.sfbay/~mzh777/v3/ql/b66/server.log
But there are exceptions on client side. Can someone on admin team take
a look for the error?
Thanks,
Ming
To turn on the the verbose mode for testng
quicklook/admin % svn diff
Index: testng.xml
===================================================================
--- testng.xml (revision 32234)
+++ testng.xml (working copy)
@@ -35,7 +35,7 @@
-->
<!DOCTYPE suite SYSTEM "
http://testng.org/testng-1.0.dtd" >
-<suite name="QuickLookTests" verbose="2" >
+<suite name="QuickLookTests" verbose="4" >
and here are the output on console:
runtest-impl-xml:
[echo] =============Starting TestNG test at ..//classes/test from
testng.xml ============
[testng] [Parser] Running:
[testng] /export/home/ming/v3/quicklook/admin/testng.xml
[testng] [RunInfo] Adding method selector:
org.testng.internal.XmlMethodSelector_at_ef2c60 priority: 10
[testng] [TestClass] Creating TestClass for [ClassImpl
test.admin.JdbcConnectionPoolTests]
[testng] [TestClass] Adding method
test.admin.JdbcConnectionPoolTests.createPool() on TestClass class
test.admin.JdbcConnectionPoolTests
[testng] [TestClass] Adding method
test.admin.JdbcConnectionPoolTests.ensureCreatedPoolExists() on
TestClass class test.admin.JdbcConnectionPoolTests
[testng] [TestClass] Adding method
test.admin.JdbcConnectionPoolTests.deletedPoolDoesNotExist() on
TestClass class test.admin.JdbcConnectionPoolTests
[testng] [TestClass] Adding method
test.admin.JdbcConnectionPoolTests.deletePool() on TestClass class
test.admin.JdbcConnectionPoolTests
[testng] [TestClass] Adding method
test.admin.JdbcConnectionPoolTests.pingPool() on TestClass class
test.admin.JdbcConnectionPoolTests
[testng] [XmlMethodSelector] Including method
test.admin.setUpEnvironment()
[testng] [XmlMethodSelector] Including method
test.admin.setupEnvironment()
[testng] [MethodInheritance]
test.admin.JdbcConnectionPoolTests.setupEnvironment(java.lang.String)
DEPENDS ON test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
java.lang.String, java.lang.String)
[testng] [TestClass] Creating TestClass for [ClassImpl
test.admin.JvmOptionTests]
[testng] [TestClass] Adding method
test.admin.JvmOptionTests.deletedJoeDoesNotExist() on TestClass class
test.admin.JvmOptionTests
[testng] [TestClass] Adding method
test.admin.JvmOptionTests.deleteJoe() on TestClass class
test.admin.JvmOptionTests
[testng] [TestClass] Adding method
test.admin.JvmOptionTests.ensureCreatedJoeExists() on TestClass class
test.admin.JvmOptionTests
[testng] [TestClass] Adding method
test.admin.JvmOptionTests.createJoe() on TestClass class
test.admin.JvmOptionTests
[testng] [TestClass] Creating TestClass for [ClassImpl
test.admin.WebtierTests]
[testng] [TestClass] Adding method
test.admin.WebtierTests.deleteListener() on TestClass class
test.admin.WebtierTests
[testng] [TestClass] Adding method
test.admin.WebtierTests.ensureDeletedListenerDoesNotExist() on TestClass
class test.admin.WebtierTests
[testng] [TestClass] Adding method
test.admin.WebtierTests.createListenerWithOldParam() on TestClass class
test.admin.WebtierTests
[testng] [TestClass] Adding method
test.admin.WebtierTests.ensureCreatedListenerExists() on TestClass class
test.admin.WebtierTests
[testng] [TestClass] Adding method
test.admin.WebtierTests.createListener() on TestClass class
test.admin.WebtierTests
[testng] [XmlMethodSelector] Including method test.admin.setup()
[testng] [MethodInheritance] test.admin.WebtierTests.setup() DEPENDS
ON test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
java.lang.String, java.lang.String)
[testng] [MethodInheritance]
test.admin.JdbcConnectionPoolTests.setupEnvironment(java.lang.String)
DEPENDS ON test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
java.lang.String, java.lang.String)
[testng] [MethodInheritance] test.admin.WebtierTests.setup() DEPENDS
ON test.admin.JdbcConnectionPoolTests.setupEnvironment(java.lang.String)
[testng] [XmlMethodSelector] Including method test.admin.createPool()
[testng] [XmlMethodSelector] Including method
test.admin.ensureCreatedPoolExists()
[testng] [XmlMethodSelector] Including method
test.admin.deletedPoolDoesNotExist()
[testng] [XmlMethodSelector] Including method test.admin.deletePool()
[testng] [XmlMethodSelector] Including method test.admin.pingPool()
[testng] [XmlMethodSelector] Including method
test.admin.deletedJoeDoesNotExist()
[testng] [XmlMethodSelector] Including method test.admin.deleteJoe()
[testng] [XmlMethodSelector] Including method
test.admin.ensureCreatedJoeExists()
[testng] [XmlMethodSelector] Including method test.admin.createJoe()
[testng] [XmlMethodSelector] Including method test.admin.deleteListener()
[testng] [XmlMethodSelector] Including method
test.admin.ensureDeletedListenerDoesNotExist()
[testng] [XmlMethodSelector] Including method
test.admin.createListenerWithOldParam()
[testng] [XmlMethodSelector] Including method
test.admin.ensureCreatedListenerExists()
[testng] [XmlMethodSelector] Including method test.admin.createListener()
[testng] [SuiteRunner] Created 1 TestRunners
[testng] [TestRunner] Running test asadmin_tests on 3 classes,
included groups:[] excluded groups:[]
[testng] [TestClass]
[testng] ======
[testng] TESTCLASS: test.admin.JdbcConnectionPoolTests
[testng] [TestClass] BeforeClass :
test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
java.lang.String, java.lang.String)
[testng] [TestClass] BeforeClass :
test.admin.JdbcConnectionPoolTests.setupEnvironment(java.lang.String)
[testng] [TestClass] Test :
test.admin.JdbcConnectionPoolTests.createPool()
[testng] [TestClass] Test :
test.admin.JdbcConnectionPoolTests.ensureCreatedPoolExists()
[testng] [TestClass] Test :
test.admin.JdbcConnectionPoolTests.deletedPoolDoesNotExist()
[testng] [TestClass] Test :
test.admin.JdbcConnectionPoolTests.deletePool()
[testng] [TestClass] Test :
test.admin.JdbcConnectionPoolTests.pingPool()
[testng] [TestClass]
[testng] ======
[testng] [TestClass]
[testng] ======
[testng] TESTCLASS: test.admin.JvmOptionTests
[testng] [TestClass] BeforeClass :
test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
java.lang.String, java.lang.String)
[testng] [TestClass] Test :
test.admin.JvmOptionTests.deletedJoeDoesNotExist()
[testng] [TestClass] Test :
test.admin.JvmOptionTests.deleteJoe()
[testng] [TestClass] Test :
test.admin.JvmOptionTests.ensureCreatedJoeExists()
[testng] [TestClass] Test :
test.admin.JvmOptionTests.createJoe()
[testng] [TestClass]
[testng] ======
[testng] [TestClass]
[testng] ======
[testng] TESTCLASS: test.admin.WebtierTests
[testng] [TestClass] BeforeClass :
test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
java.lang.String, java.lang.String)
[testng] [TestClass] BeforeClass : test.admin.WebtierTests.setup()
[testng] [TestClass] Test :
test.admin.WebtierTests.deleteListener()
[testng] [TestClass] Test :
test.admin.WebtierTests.ensureDeletedListenerDoesNotExist()
[testng] [TestClass] Test :
test.admin.WebtierTests.createListenerWithOldParam()
[testng] [TestClass] Test :
test.admin.WebtierTests.ensureCreatedListenerExists()
[testng] [TestClass] Test :
test.admin.WebtierTests.createListener()
[testng] [TestClass]
[testng] ======
[testng] [TestRunner] WILL BE RUN IN RANDOM ORDER:
[testng] [TestRunner]
test.admin.WebtierTests.createListenerWithOldParam()
[testng] [TestRunner] WILL BE RUN SEQUENTIALLY:
[testng] [TestRunner] test.admin.WebtierTests.createListener()
[testng] [TestRunner] test.admin.JdbcConnectionPoolTests.createPool()
[testng] [TestRunner] test.admin.JvmOptionTests.createJoe()
[testng] [TestRunner]
test.admin.JvmOptionTests.ensureCreatedJoeExists()
[testng] [TestRunner] test.admin.JvmOptionTests.deleteJoe()
[testng] [TestRunner]
test.admin.WebtierTests.ensureCreatedListenerExists()
[testng] [TestRunner] test.admin.WebtierTests.deleteListener()
[testng] [TestRunner]
test.admin.WebtierTests.ensureDeletedListenerDoesNotExist()
[testng] [TestRunner] test.admin.JdbcConnectionPoolTests.pingPool()
[testng] [TestRunner]
test.admin.JvmOptionTests.deletedJoeDoesNotExist()
[testng] [TestRunner]
test.admin.JdbcConnectionPoolTests.ensureCreatedPoolExists()
[testng] [TestRunner] test.admin.JdbcConnectionPoolTests.deletePool()
[testng] [TestRunner]
test.admin.JdbcConnectionPoolTests.deletedPoolDoesNotExist()
[testng] [TestRunner] ====
[testng] [TestRunner] ===
[testng] [TestRunner] Found 2 applicable methods
[testng] [Invoker 20469344] Invoking @BeforeClass
test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
java.lang.String, java.lang.String)
[testng] [Invoker 20469344] Invoking @BeforeClass
test.admin.WebtierTests.setup()
[testng] [Invoker 20469344] Invoking
test.admin.WebtierTests.createListener
[testng] [Invoker 20469344] Invoking @BeforeClass
test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
java.lang.String, java.lang.String)
[testng] [Invoker 20469344] Invoking @BeforeClass
test.admin.JdbcConnectionPoolTests.setupEnvironment(java.lang.String)
[testng] [Invoker 20469344] Invoking
test.admin.JdbcConnectionPoolTests.createPool
[testng] [Invoker 20469344] Invoking @BeforeClass
test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
java.lang.String, java.lang.String)
[testng] [Invoker 20469344] Invoking test.admin.JvmOptionTests.createJoe
[testng] [Invoker 20469344] Invoking
test.admin.JvmOptionTests.ensureCreatedJoeExists
[testng] [Invoker 20469344] Invoking test.admin.JvmOptionTests.deleteJoe
[testng] [Invoker 20469344] Invoking
test.admin.WebtierTests.ensureCreatedListenerExists
[testng] [Invoker 20469344] Invoking
test.admin.WebtierTests.deleteListener
[testng] [Invoker 20469344] Invoking
test.admin.JdbcConnectionPoolTests.pingPool
[testng] [Invoker 20469344] Invoking
test.admin.JvmOptionTests.deletedJoeDoesNotExist
[testng] [Invoker 20469344] Invoking
test.admin.JdbcConnectionPoolTests.ensureCreatedPoolExists
[testng] [Invoker 20469344] Invoking
test.admin.JdbcConnectionPoolTests.deletePool
[testng] [Invoker 20469344] Invoking
test.admin.JdbcConnectionPoolTests.deletedPoolDoesNotExist
[testng] [Invoker 20469344] Invoking
test.admin.WebtierTests.createListenerWithOldParam
[testng] *********** INVOKED METHODS
[testng]
test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
java.lang.String, java.lang.String)
http://localhost:4848/__asadmin
admin 7408068
[testng] test.admin.WebtierTests.setup() 7408068
[testng] test.admin.WebtierTests.createListener() 7408068
[testng]
test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
java.lang.String, java.lang.String)
http://localhost:4848/__asadmin
admin 24744797
[testng]
test.admin.JdbcConnectionPoolTests.setupEnvironment(java.lang.String)resources/resources.xml
24744797
[testng] test.admin.JdbcConnectionPoolTests.createPool()
24744797
[testng]
test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
java.lang.String, java.lang.String)
http://localhost:4848/__asadmin
admin 9949215
[testng] test.admin.JvmOptionTests.createJoe() 9949215
[testng]
test.admin.JvmOptionTests.ensureCreatedJoeExists() 9949215
[testng] test.admin.JvmOptionTests.deleteJoe() 9949215
[testng]
test.admin.WebtierTests.ensureCreatedListenerExists() 7408068
[testng] test.admin.WebtierTests.deleteListener() 7408068
[testng] test.admin.JdbcConnectionPoolTests.pingPool()
24744797
[testng]
test.admin.JvmOptionTests.deletedJoeDoesNotExist() 9949215
[testng]
test.admin.JdbcConnectionPoolTests.ensureCreatedPoolExists() 24744797
[testng] test.admin.JdbcConnectionPoolTests.deletePool()
24744797
[testng]
test.admin.JdbcConnectionPoolTests.deletedPoolDoesNotExist() 24744797
[testng]
test.admin.WebtierTests.createListenerWithOldParam() 7408068
[testng] ***********
[testng] Creating
/export/home/ming/v3/quicklook/test-output/QuickLookTests/asadmin_tests.html
[testng] PASSED: createListener
[testng] PASSED: createPool
[testng] PASSED: createJoe
[testng] PASSED: ensureCreatedJoeExists
[testng] PASSED: deleteJoe
[testng] PASSED: ensureCreatedListenerExists
[testng] PASSED: pingPool
[testng] PASSED: deletedJoeDoesNotExist
[testng] PASSED: ensureCreatedPoolExists
[testng] PASSED: deletePool
[testng] PASSED: deletedPoolDoesNotExist
[testng] PASSED: createListenerWithOldParam
[testng] FAILED: deleteListener
[testng] java.lang.RuntimeException: null
[testng] at
test.admin.util.GeneralUtils.handleManifestFailure(GeneralUtils.java:124)
[testng] at
test.admin.WebtierTests.deleteListener(WebtierTests.java:101)
[testng] ... Removed 22 stack frames
[testng] SKIPPED: ensureDeletedListenerDoesNotExist
[testng] ===============================================
[testng] asadmin_tests
[testng] Tests run: 14, Failures: 1, Skips: 1
[testng] ===============================================
[testng] ===============================================
[testng] QuickLookTests
[testng] Total tests run: 14, Failures: 1, Skips: 1
[testng] ===============================================
Ken Cavanaugh wrote:
> Darani and I have been running the QLs on multiple machines in order to
> verify the functioning of ORB b030 (GFv3 is currently using ORB b029),
> and every time we run the tests we get different results. I've had
> the same
> problems with trying to integrate Gmbal b016.
>
> The main tests that do this in the QL appear to be
> iterateAllSanityCheck (an
> AMX test), testAMXComplianceMonitorFailureCount, and deleteListener.
> Over 4 QL runs, I've observed the following:
>
> * Run 1: passed with Gmbal b015
> * Run 2: all 3 tests failed with Gmbal b016
> * Run 3: with CORBA b030, testAMXComplianceMonitorFailureCount failed
> o The AMX compliance monitor complained about the following:
> + Attribute 'Enabled' failed for
> amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
> + Attribute 'JvmOptions' failed for
> amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
> + Attribute 'NativeLibraryPath' failed for
> amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
> + Attribute 'Classpath' failed for
> amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
> + Attribute 'Property' failed for
> amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
> + amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
> General test failure in validateAMXConfig:
> java.lang.NullPointerException: "null"
> org.glassfish.admin.amx.core.AMXValidator.validateAMXConfig(AMXValidator.java:650)
> org.glassfish.admin.amx.core.AMXValidator._validate(AMXValidator.java:618)
> org.glassfish.admin.amx.core.AMXValidator.validate(AMXValidator.java:1213)
> org.glassfish.admin.amx.impl.mbean.ComplianceMonitor$ValidatorThread.doRun(ComplianceMonitor.java:235)
> org.glassfish.admin.amx.impl.mbean.ComplianceMonitor$ValidatorThread.run(ComplianceMonitor.java:206)
> * Run 4: I re-ran the same software as in run 3. This time all 3
> tests failed.
> + There were several Gmbal failures in the log
> (unregister failures). These were all caused by
> InstanceNotFoundExceptions on a number of MBeans:
> # amx:pp=/mon/server-mon[server],type=bean-method-mon,name=remoteview/HelloBean/bean-methods/hello
> # amx:pp=/mon/server-mon[server],type=bean-method-mon,name=remoteview/HelloBean/bean-methods/asyncCancel-int
> # amx:pp=/mon/server-mon[server],type=bean-method-mon,name=remoteview/HelloBean/bean-methods/asyncThrowException-java.lang.String
> # amx:pp=/mon/server-mon[server],type=bean-method-mon,name=remoteview/HelloBean/bean-methods/throwException-java.lang.String
> # amx:pp=/mon/server-mon[server],type=bean-method-mon,name=remoteview/HelloBean/bean-methods/asyncBlock-int
> # etc. for a total of 153 MBeans: each
> unregister failure happened twice in a row.
>
> Now, I initially assumed that the QL failures I observed in Run 2 were
> do to interactions between
> a new Gmbal feature (root parent monitoring) and the rest of GFv3
> monitoring. But the same tests
> seem to fail almost randomly in other situations. Particularly
> disturbing is the fact the Run 3 and Run 4
> shared completely IDENTICAL app server code, and exhibited different
> failures. I also highly doubt that
> the ORB has anything to do with the observed (and unpredictable)
> failures. Note that very similar failures
> are seen both with Gmbal b016 and with CORBA b030. That coincidence
> and the lack of repeatability lead
> me to wonder about the stability of the QLs, and whether the failures
> on Gmbal b016 have anything at all to do
> with the recent Gmbal changes.
>
> This is a big problem. Darani and I have been unable to complete any
> CORBA or Gmbal integrations recently,
> because we keep seeing different failures. Darani had b30 passing ALL
> tests at one point, then when she attempted
> to verify one final time to complete the integration, similar failures
> occurred.
>
> We need help to understand why we see this level of instability in QL
> runs. Without fixing this, we
> are unable to integrate either CORBA or Gmbal, and we are nearly out
> of time for a number
> of important fixes (many of which are ready to go) before hard code
> freeze.
>
> Thanks,
>
> Ken.
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net For
> additional commands, e-mail: dev-help_at_glassfish.dev.java.net