Ming,
Recreating the domain will resolve the AMX test failures. Looks like
QL may not be cleaning-up domain.xml and resetting to the original
configuration.
I see that admin-gui configuration is added to domain.xml after running
QL. This is because admin-gui is getting invoked the first time.
There is a new http-listener, "ls123452". Should that be removed?
I'm not sure what other configurations are added and need to be removed
after running QL.
Can you look into this?
Thanks,
Jane
Ming Zhang wrote:
> I tried to reproduce the issue with the promoted b66
> (http://download.java.net/glassfish/v3/promoted/glassfish-v3-b66.zip)
> and here are what I found:
>
> * Run QL first time in web distribution profile and all passed.
> * Run QL again and the amx or admin tests start having errors
> * To isolate the problem, I deleted the b66 and unzip again. Run
> amx test individually multiple times and tests passed:
> o quicklook/amx % ant
> -Dglassfish.home=/export/home/ming/v3/b66/glassfishv3/glassfish
> all
> * Start domain and derby. Run admin tests individually. The tests
> passed first time and started failed when ran again.
> o quicklook/admin % ant
> -Dglassfish.home=/export/home/ming/v3/b66/glassfishv3/glassfish
> all
>
> The stack trace on server side is clean:
> http://javaweb.sfbay/~mzh777/v3/ql/b66/server.log
> But there are exceptions on client side. Can someone on admin team
> take a look for the error?
>
> Thanks,
> Ming
>
> To turn on the the verbose mode for testng
> quicklook/admin % svn diff
> Index: testng.xml
> ===================================================================
> --- testng.xml (revision 32234)
> +++ testng.xml (working copy)
> @@ -35,7 +35,7 @@
> -->
> <!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
>
> -<suite name="QuickLookTests" verbose="2" >
> +<suite name="QuickLookTests" verbose="4" >
>
> and here are the output on console:
>
> runtest-impl-xml:
> [echo] =============Starting TestNG test at ..//classes/test from
> testng.xml ============
> [testng] [Parser] Running:
> [testng] /export/home/ming/v3/quicklook/admin/testng.xml
>
> [testng] [RunInfo] Adding method selector:
> org.testng.internal.XmlMethodSelector_at_ef2c60 priority: 10
> [testng] [TestClass] Creating TestClass for [ClassImpl
> test.admin.JdbcConnectionPoolTests]
> [testng] [TestClass] Adding method
> test.admin.JdbcConnectionPoolTests.createPool() on TestClass class
> test.admin.JdbcConnectionPoolTests
> [testng] [TestClass] Adding method
> test.admin.JdbcConnectionPoolTests.ensureCreatedPoolExists() on
> TestClass class test.admin.JdbcConnectionPoolTests
> [testng] [TestClass] Adding method
> test.admin.JdbcConnectionPoolTests.deletedPoolDoesNotExist() on
> TestClass class test.admin.JdbcConnectionPoolTests
> [testng] [TestClass] Adding method
> test.admin.JdbcConnectionPoolTests.deletePool() on TestClass class
> test.admin.JdbcConnectionPoolTests
> [testng] [TestClass] Adding method
> test.admin.JdbcConnectionPoolTests.pingPool() on TestClass class
> test.admin.JdbcConnectionPoolTests
> [testng] [XmlMethodSelector] Including method
> test.admin.setUpEnvironment()
> [testng] [XmlMethodSelector] Including method
> test.admin.setupEnvironment()
> [testng] [MethodInheritance]
> test.admin.JdbcConnectionPoolTests.setupEnvironment(java.lang.String)
> DEPENDS ON
> test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
> java.lang.String, java.lang.String)
> [testng] [TestClass] Creating TestClass for [ClassImpl
> test.admin.JvmOptionTests]
> [testng] [TestClass] Adding method
> test.admin.JvmOptionTests.deletedJoeDoesNotExist() on TestClass class
> test.admin.JvmOptionTests
> [testng] [TestClass] Adding method
> test.admin.JvmOptionTests.deleteJoe() on TestClass class
> test.admin.JvmOptionTests
> [testng] [TestClass] Adding method
> test.admin.JvmOptionTests.ensureCreatedJoeExists() on TestClass class
> test.admin.JvmOptionTests
> [testng] [TestClass] Adding method
> test.admin.JvmOptionTests.createJoe() on TestClass class
> test.admin.JvmOptionTests
> [testng] [TestClass] Creating TestClass for [ClassImpl
> test.admin.WebtierTests]
> [testng] [TestClass] Adding method
> test.admin.WebtierTests.deleteListener() on TestClass class
> test.admin.WebtierTests
> [testng] [TestClass] Adding method
> test.admin.WebtierTests.ensureDeletedListenerDoesNotExist() on
> TestClass class test.admin.WebtierTests
> [testng] [TestClass] Adding method
> test.admin.WebtierTests.createListenerWithOldParam() on TestClass
> class test.admin.WebtierTests
> [testng] [TestClass] Adding method
> test.admin.WebtierTests.ensureCreatedListenerExists() on TestClass
> class test.admin.WebtierTests
> [testng] [TestClass] Adding method
> test.admin.WebtierTests.createListener() on TestClass class
> test.admin.WebtierTests
> [testng] [XmlMethodSelector] Including method test.admin.setup()
> [testng] [MethodInheritance] test.admin.WebtierTests.setup()
> DEPENDS ON
> test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
> java.lang.String, java.lang.String)
> [testng] [MethodInheritance]
> test.admin.JdbcConnectionPoolTests.setupEnvironment(java.lang.String)
> DEPENDS ON
> test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
> java.lang.String, java.lang.String)
> [testng] [MethodInheritance] test.admin.WebtierTests.setup()
> DEPENDS ON
> test.admin.JdbcConnectionPoolTests.setupEnvironment(java.lang.String)
> [testng] [XmlMethodSelector] Including method test.admin.createPool()
> [testng] [XmlMethodSelector] Including method
> test.admin.ensureCreatedPoolExists()
> [testng] [XmlMethodSelector] Including method
> test.admin.deletedPoolDoesNotExist()
> [testng] [XmlMethodSelector] Including method test.admin.deletePool()
> [testng] [XmlMethodSelector] Including method test.admin.pingPool()
> [testng] [XmlMethodSelector] Including method
> test.admin.deletedJoeDoesNotExist()
> [testng] [XmlMethodSelector] Including method test.admin.deleteJoe()
> [testng] [XmlMethodSelector] Including method
> test.admin.ensureCreatedJoeExists()
> [testng] [XmlMethodSelector] Including method test.admin.createJoe()
> [testng] [XmlMethodSelector] Including method
> test.admin.deleteListener()
> [testng] [XmlMethodSelector] Including method
> test.admin.ensureDeletedListenerDoesNotExist()
> [testng] [XmlMethodSelector] Including method
> test.admin.createListenerWithOldParam()
> [testng] [XmlMethodSelector] Including method
> test.admin.ensureCreatedListenerExists()
> [testng] [XmlMethodSelector] Including method
> test.admin.createListener()
> [testng] [SuiteRunner] Created 1 TestRunners
> [testng] [TestRunner] Running test asadmin_tests on 3 classes,
> included groups:[] excluded groups:[]
> [testng] [TestClass]
> [testng] ======
> [testng] TESTCLASS: test.admin.JdbcConnectionPoolTests
> [testng] [TestClass] BeforeClass :
> test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
> java.lang.String, java.lang.String)
> [testng] [TestClass] BeforeClass :
> test.admin.JdbcConnectionPoolTests.setupEnvironment(java.lang.String)
> [testng] [TestClass] Test :
> test.admin.JdbcConnectionPoolTests.createPool()
> [testng] [TestClass] Test :
> test.admin.JdbcConnectionPoolTests.ensureCreatedPoolExists()
> [testng] [TestClass] Test :
> test.admin.JdbcConnectionPoolTests.deletedPoolDoesNotExist()
> [testng] [TestClass] Test :
> test.admin.JdbcConnectionPoolTests.deletePool()
> [testng] [TestClass] Test :
> test.admin.JdbcConnectionPoolTests.pingPool()
> [testng] [TestClass]
> [testng] ======
>
> [testng] [TestClass]
> [testng] ======
> [testng] TESTCLASS: test.admin.JvmOptionTests
> [testng] [TestClass] BeforeClass :
> test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
> java.lang.String, java.lang.String)
> [testng] [TestClass] Test :
> test.admin.JvmOptionTests.deletedJoeDoesNotExist()
> [testng] [TestClass] Test :
> test.admin.JvmOptionTests.deleteJoe()
> [testng] [TestClass] Test :
> test.admin.JvmOptionTests.ensureCreatedJoeExists()
> [testng] [TestClass] Test :
> test.admin.JvmOptionTests.createJoe()
> [testng] [TestClass]
> [testng] ======
>
> [testng] [TestClass]
> [testng] ======
> [testng] TESTCLASS: test.admin.WebtierTests
> [testng] [TestClass] BeforeClass :
> test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
> java.lang.String, java.lang.String)
> [testng] [TestClass] BeforeClass : test.admin.WebtierTests.setup()
> [testng] [TestClass] Test :
> test.admin.WebtierTests.deleteListener()
> [testng] [TestClass] Test :
> test.admin.WebtierTests.ensureDeletedListenerDoesNotExist()
> [testng] [TestClass] Test :
> test.admin.WebtierTests.createListenerWithOldParam()
> [testng] [TestClass] Test :
> test.admin.WebtierTests.ensureCreatedListenerExists()
> [testng] [TestClass] Test :
> test.admin.WebtierTests.createListener()
> [testng] [TestClass]
> [testng] ======
>
> [testng] [TestRunner] WILL BE RUN IN RANDOM ORDER:
> [testng] [TestRunner]
> test.admin.WebtierTests.createListenerWithOldParam()
> [testng] [TestRunner] WILL BE RUN SEQUENTIALLY:
> [testng] [TestRunner] test.admin.WebtierTests.createListener()
> [testng] [TestRunner] test.admin.JdbcConnectionPoolTests.createPool()
> [testng] [TestRunner] test.admin.JvmOptionTests.createJoe()
> [testng] [TestRunner]
> test.admin.JvmOptionTests.ensureCreatedJoeExists()
> [testng] [TestRunner] test.admin.JvmOptionTests.deleteJoe()
> [testng] [TestRunner]
> test.admin.WebtierTests.ensureCreatedListenerExists()
> [testng] [TestRunner] test.admin.WebtierTests.deleteListener()
> [testng] [TestRunner]
> test.admin.WebtierTests.ensureDeletedListenerDoesNotExist()
> [testng] [TestRunner] test.admin.JdbcConnectionPoolTests.pingPool()
> [testng] [TestRunner]
> test.admin.JvmOptionTests.deletedJoeDoesNotExist()
> [testng] [TestRunner]
> test.admin.JdbcConnectionPoolTests.ensureCreatedPoolExists()
> [testng] [TestRunner] test.admin.JdbcConnectionPoolTests.deletePool()
> [testng] [TestRunner]
> test.admin.JdbcConnectionPoolTests.deletedPoolDoesNotExist()
> [testng] [TestRunner] ====
> [testng] [TestRunner] ===
> [testng] [TestRunner] Found 2 applicable methods
> [testng] [Invoker 20469344] Invoking @BeforeClass
> test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
> java.lang.String, java.lang.String)
> [testng] [Invoker 20469344] Invoking @BeforeClass
> test.admin.WebtierTests.setup()
> [testng] [Invoker 20469344] Invoking
> test.admin.WebtierTests.createListener
> [testng] [Invoker 20469344] Invoking @BeforeClass
> test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
> java.lang.String, java.lang.String)
> [testng] [Invoker 20469344] Invoking @BeforeClass
> test.admin.JdbcConnectionPoolTests.setupEnvironment(java.lang.String)
> [testng] [Invoker 20469344] Invoking
> test.admin.JdbcConnectionPoolTests.createPool
> [testng] [Invoker 20469344] Invoking @BeforeClass
> test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
> java.lang.String, java.lang.String)
> [testng] [Invoker 20469344] Invoking
> test.admin.JvmOptionTests.createJoe
> [testng] [Invoker 20469344] Invoking
> test.admin.JvmOptionTests.ensureCreatedJoeExists
> [testng] [Invoker 20469344] Invoking
> test.admin.JvmOptionTests.deleteJoe
> [testng] [Invoker 20469344] Invoking
> test.admin.WebtierTests.ensureCreatedListenerExists
> [testng] [Invoker 20469344] Invoking
> test.admin.WebtierTests.deleteListener
> [testng] [Invoker 20469344] Invoking
> test.admin.JdbcConnectionPoolTests.pingPool
> [testng] [Invoker 20469344] Invoking
> test.admin.JvmOptionTests.deletedJoeDoesNotExist
> [testng] [Invoker 20469344] Invoking
> test.admin.JdbcConnectionPoolTests.ensureCreatedPoolExists
> [testng] [Invoker 20469344] Invoking
> test.admin.JdbcConnectionPoolTests.deletePool
> [testng] [Invoker 20469344] Invoking
> test.admin.JdbcConnectionPoolTests.deletedPoolDoesNotExist
> [testng] [Invoker 20469344] Invoking
> test.admin.WebtierTests.createListenerWithOldParam
>
> [testng] *********** INVOKED METHODS
>
> [testng]
> test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
> java.lang.String, java.lang.String)http://localhost:4848/__asadmin
> admin 7408068
> [testng] test.admin.WebtierTests.setup() 7408068
> [testng] test.admin.WebtierTests.createListener() 7408068
> [testng]
> test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
> java.lang.String, java.lang.String)http://localhost:4848/__asadmin
> admin 24744797
> [testng]
> test.admin.JdbcConnectionPoolTests.setupEnvironment(java.lang.String)resources/resources.xml
> 24744797
> [testng]
> test.admin.JdbcConnectionPoolTests.createPool() 24744797
> [testng]
> test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
> java.lang.String, java.lang.String)http://localhost:4848/__asadmin
> admin 9949215
> [testng] test.admin.JvmOptionTests.createJoe() 9949215
> [testng]
> test.admin.JvmOptionTests.ensureCreatedJoeExists() 9949215
> [testng] test.admin.JvmOptionTests.deleteJoe() 9949215
> [testng]
> test.admin.WebtierTests.ensureCreatedListenerExists() 7408068
> [testng] test.admin.WebtierTests.deleteListener() 7408068
> [testng] test.admin.JdbcConnectionPoolTests.pingPool()
> 24744797
> [testng]
> test.admin.JvmOptionTests.deletedJoeDoesNotExist() 9949215
> [testng]
> test.admin.JdbcConnectionPoolTests.ensureCreatedPoolExists() 24744797
> [testng]
> test.admin.JdbcConnectionPoolTests.deletePool() 24744797
> [testng]
> test.admin.JdbcConnectionPoolTests.deletedPoolDoesNotExist() 24744797
> [testng]
> test.admin.WebtierTests.createListenerWithOldParam() 7408068
>
> [testng] ***********
>
> [testng] Creating
> /export/home/ming/v3/quicklook/test-output/QuickLookTests/asadmin_tests.html
> [testng] PASSED: createListener
> [testng] PASSED: createPool
> [testng] PASSED: createJoe
> [testng] PASSED: ensureCreatedJoeExists
> [testng] PASSED: deleteJoe
> [testng] PASSED: ensureCreatedListenerExists
> [testng] PASSED: pingPool
> [testng] PASSED: deletedJoeDoesNotExist
> [testng] PASSED: ensureCreatedPoolExists
> [testng] PASSED: deletePool
> [testng] PASSED: deletedPoolDoesNotExist
> [testng] PASSED: createListenerWithOldParam
> [testng] FAILED: deleteListener
> [testng] java.lang.RuntimeException: null
> [testng] at
> test.admin.util.GeneralUtils.handleManifestFailure(GeneralUtils.java:124)
> [testng] at
> test.admin.WebtierTests.deleteListener(WebtierTests.java:101)
> [testng] ... Removed 22 stack frames
> [testng] SKIPPED: ensureDeletedListenerDoesNotExist
>
> [testng] ===============================================
> [testng] asadmin_tests
> [testng] Tests run: 14, Failures: 1, Skips: 1
> [testng] ===============================================
>
>
> [testng] ===============================================
> [testng] QuickLookTests
> [testng] Total tests run: 14, Failures: 1, Skips: 1
> [testng] ===============================================
>
> Ken Cavanaugh wrote:
>> Darani and I have been running the QLs on multiple machines in order to
>> verify the functioning of ORB b030 (GFv3 is currently using ORB b029),
>> and every time we run the tests we get different results. I've had
>> the same
>> problems with trying to integrate Gmbal b016.
>>
>> The main tests that do this in the QL appear to be
>> iterateAllSanityCheck (an
>> AMX test), testAMXComplianceMonitorFailureCount, and deleteListener.
>> Over 4 QL runs, I've observed the following:
>>
>> * Run 1: passed with Gmbal b015
>> * Run 2: all 3 tests failed with Gmbal b016
>> * Run 3: with CORBA b030, testAMXComplianceMonitorFailureCount failed
>> o The AMX compliance monitor complained about the following:
>> + Attribute 'Enabled' failed for
>> amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
>> + Attribute 'JvmOptions' failed for
>> amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
>> + Attribute 'NativeLibraryPath' failed for
>> amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
>> + Attribute 'Classpath' failed for
>> amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
>> + Attribute 'Property' failed for
>> amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
>> + amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
>> General test failure in validateAMXConfig:
>> java.lang.NullPointerException: "null"
>> org.glassfish.admin.amx.core.AMXValidator.validateAMXConfig(AMXValidator.java:650)
>> org.glassfish.admin.amx.core.AMXValidator._validate(AMXValidator.java:618)
>> org.glassfish.admin.amx.core.AMXValidator.validate(AMXValidator.java:1213)
>> org.glassfish.admin.amx.impl.mbean.ComplianceMonitor$ValidatorThread.doRun(ComplianceMonitor.java:235)
>> org.glassfish.admin.amx.impl.mbean.ComplianceMonitor$ValidatorThread.run(ComplianceMonitor.java:206)
>> * Run 4: I re-ran the same software as in run 3. This time all 3
>> tests failed.
>> + There were several Gmbal failures in the log
>> (unregister failures). These were all caused by
>> InstanceNotFoundExceptions on a number of MBeans:
>> # amx:pp=/mon/server-mon[server],type=bean-method-mon,name=remoteview/HelloBean/bean-methods/hello
>> # amx:pp=/mon/server-mon[server],type=bean-method-mon,name=remoteview/HelloBean/bean-methods/asyncCancel-int
>> # amx:pp=/mon/server-mon[server],type=bean-method-mon,name=remoteview/HelloBean/bean-methods/asyncThrowException-java.lang.String
>> # amx:pp=/mon/server-mon[server],type=bean-method-mon,name=remoteview/HelloBean/bean-methods/throwException-java.lang.String
>> # amx:pp=/mon/server-mon[server],type=bean-method-mon,name=remoteview/HelloBean/bean-methods/asyncBlock-int
>> # etc. for a total of 153 MBeans: each
>> unregister failure happened twice in a row.
>>
>> Now, I initially assumed that the QL failures I observed in Run 2
>> were do to interactions between
>> a new Gmbal feature (root parent monitoring) and the rest of GFv3
>> monitoring. But the same tests
>> seem to fail almost randomly in other situations. Particularly
>> disturbing is the fact the Run 3 and Run 4
>> shared completely IDENTICAL app server code, and exhibited different
>> failures. I also highly doubt that
>> the ORB has anything to do with the observed (and unpredictable)
>> failures. Note that very similar failures
>> are seen both with Gmbal b016 and with CORBA b030. That coincidence
>> and the lack of repeatability lead
>> me to wonder about the stability of the QLs, and whether the failures
>> on Gmbal b016 have anything at all to do
>> with the recent Gmbal changes.
>>
>> This is a big problem. Darani and I have been unable to complete any
>> CORBA or Gmbal integrations recently,
>> because we keep seeing different failures. Darani had b30 passing
>> ALL tests at one point, then when she attempted
>> to verify one final time to complete the integration, similar
>> failures occurred.
>>
>> We need help to understand why we see this level of instability in QL
>> runs. Without fixing this, we
>> are unable to integrate either CORBA or Gmbal, and we are nearly out
>> of time for a number
>> of important fixes (many of which are ready to go) before hard code
>> freeze.
>>
>> Thanks,
>>
>> Ken.
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net For
>> additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>