Jane,
I have checked in one more test to delete http-listener "ls123452". Now
admin tests can be run repeatedly without errors and the domain.xml gets
cleaned up after each admin test run.
Lloyd,
the AMX tests still failed in the 2nd QL run (iterateAllSanityCheck,
testAMXComplianceMonitorFailureCount) after my change for admin tests.
Is this due to the configuration changes in domain.xml after admin GUI
loaded? Is it reasonable to expect AMX tests will accommodate the
configurations of admin GUI in domain.xml?
Thanks,
Ming
Jane Young wrote:
> Ming,
>
> Recreating the domain will resolve the AMX test failures. Looks like
> QL may not be cleaning-up domain.xml and resetting to the original
> configuration.
> I see that admin-gui configuration is added to domain.xml after
> running QL. This is because admin-gui is getting invoked the first
> time. There is a new http-listener, "ls123452". Should that be
> removed? I'm not sure what other configurations are added and need
> to be removed after running QL.
>
> Can you look into this?
>
>
> Thanks,
> Jane
>
>
> Ming Zhang wrote:
>> I tried to reproduce the issue with the promoted b66
>> (http://download.java.net/glassfish/v3/promoted/glassfish-v3-b66.zip)
>> and here are what I found:
>>
>> * Run QL first time in web distribution profile and all passed.
>> * Run QL again and the amx or admin tests start having errors
>> * To isolate the problem, I deleted the b66 and unzip again. Run
>> amx test individually multiple times and tests passed:
>> o quicklook/amx % ant
>> -Dglassfish.home=/export/home/ming/v3/b66/glassfishv3/glassfish
>> all
>> * Start domain and derby. Run admin tests individually. The tests
>> passed first time and started failed when ran again.
>> o quicklook/admin % ant
>> -Dglassfish.home=/export/home/ming/v3/b66/glassfishv3/glassfish
>> all
>>
>> The stack trace on server side is clean:
>> http://javaweb.sfbay/~mzh777/v3/ql/b66/server.log
>> But there are exceptions on client side. Can someone on admin team
>> take a look for the error?
>>
>> Thanks,
>> Ming
>>
>> To turn on the the verbose mode for testng
>> quicklook/admin % svn diff
>> Index: testng.xml
>> ===================================================================
>> --- testng.xml (revision 32234)
>> +++ testng.xml (working copy)
>> @@ -35,7 +35,7 @@
>> -->
>> <!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
>>
>> -<suite name="QuickLookTests" verbose="2" >
>> +<suite name="QuickLookTests" verbose="4" >
>>
>> and here are the output on console:
>>
>> runtest-impl-xml:
>> [echo] =============Starting TestNG test at ..//classes/test
>> from testng.xml ============
>> [testng] [Parser] Running:
>> [testng] /export/home/ming/v3/quicklook/admin/testng.xml
>>
>> [testng] [RunInfo] Adding method selector:
>> org.testng.internal.XmlMethodSelector_at_ef2c60 priority: 10
>> [testng] [TestClass] Creating TestClass for [ClassImpl
>> test.admin.JdbcConnectionPoolTests]
>> [testng] [TestClass] Adding method
>> test.admin.JdbcConnectionPoolTests.createPool() on TestClass class
>> test.admin.JdbcConnectionPoolTests
>> [testng] [TestClass] Adding method
>> test.admin.JdbcConnectionPoolTests.ensureCreatedPoolExists() on
>> TestClass class test.admin.JdbcConnectionPoolTests
>> [testng] [TestClass] Adding method
>> test.admin.JdbcConnectionPoolTests.deletedPoolDoesNotExist() on
>> TestClass class test.admin.JdbcConnectionPoolTests
>> [testng] [TestClass] Adding method
>> test.admin.JdbcConnectionPoolTests.deletePool() on TestClass class
>> test.admin.JdbcConnectionPoolTests
>> [testng] [TestClass] Adding method
>> test.admin.JdbcConnectionPoolTests.pingPool() on TestClass class
>> test.admin.JdbcConnectionPoolTests
>> [testng] [XmlMethodSelector] Including method
>> test.admin.setUpEnvironment()
>> [testng] [XmlMethodSelector] Including method
>> test.admin.setupEnvironment()
>> [testng] [MethodInheritance]
>> test.admin.JdbcConnectionPoolTests.setupEnvironment(java.lang.String)
>> DEPENDS ON
>> test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
>> java.lang.String, java.lang.String)
>> [testng] [TestClass] Creating TestClass for [ClassImpl
>> test.admin.JvmOptionTests]
>> [testng] [TestClass] Adding method
>> test.admin.JvmOptionTests.deletedJoeDoesNotExist() on TestClass class
>> test.admin.JvmOptionTests
>> [testng] [TestClass] Adding method
>> test.admin.JvmOptionTests.deleteJoe() on TestClass class
>> test.admin.JvmOptionTests
>> [testng] [TestClass] Adding method
>> test.admin.JvmOptionTests.ensureCreatedJoeExists() on TestClass class
>> test.admin.JvmOptionTests
>> [testng] [TestClass] Adding method
>> test.admin.JvmOptionTests.createJoe() on TestClass class
>> test.admin.JvmOptionTests
>> [testng] [TestClass] Creating TestClass for [ClassImpl
>> test.admin.WebtierTests]
>> [testng] [TestClass] Adding method
>> test.admin.WebtierTests.deleteListener() on TestClass class
>> test.admin.WebtierTests
>> [testng] [TestClass] Adding method
>> test.admin.WebtierTests.ensureDeletedListenerDoesNotExist() on
>> TestClass class test.admin.WebtierTests
>> [testng] [TestClass] Adding method
>> test.admin.WebtierTests.createListenerWithOldParam() on TestClass
>> class test.admin.WebtierTests
>> [testng] [TestClass] Adding method
>> test.admin.WebtierTests.ensureCreatedListenerExists() on TestClass
>> class test.admin.WebtierTests
>> [testng] [TestClass] Adding method
>> test.admin.WebtierTests.createListener() on TestClass class
>> test.admin.WebtierTests
>> [testng] [XmlMethodSelector] Including method test.admin.setup()
>> [testng] [MethodInheritance] test.admin.WebtierTests.setup()
>> DEPENDS ON
>> test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
>> java.lang.String, java.lang.String)
>> [testng] [MethodInheritance]
>> test.admin.JdbcConnectionPoolTests.setupEnvironment(java.lang.String)
>> DEPENDS ON
>> test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
>> java.lang.String, java.lang.String)
>> [testng] [MethodInheritance] test.admin.WebtierTests.setup()
>> DEPENDS ON
>> test.admin.JdbcConnectionPoolTests.setupEnvironment(java.lang.String)
>> [testng] [XmlMethodSelector] Including method test.admin.createPool()
>> [testng] [XmlMethodSelector] Including method
>> test.admin.ensureCreatedPoolExists()
>> [testng] [XmlMethodSelector] Including method
>> test.admin.deletedPoolDoesNotExist()
>> [testng] [XmlMethodSelector] Including method test.admin.deletePool()
>> [testng] [XmlMethodSelector] Including method test.admin.pingPool()
>> [testng] [XmlMethodSelector] Including method
>> test.admin.deletedJoeDoesNotExist()
>> [testng] [XmlMethodSelector] Including method test.admin.deleteJoe()
>> [testng] [XmlMethodSelector] Including method
>> test.admin.ensureCreatedJoeExists()
>> [testng] [XmlMethodSelector] Including method test.admin.createJoe()
>> [testng] [XmlMethodSelector] Including method
>> test.admin.deleteListener()
>> [testng] [XmlMethodSelector] Including method
>> test.admin.ensureDeletedListenerDoesNotExist()
>> [testng] [XmlMethodSelector] Including method
>> test.admin.createListenerWithOldParam()
>> [testng] [XmlMethodSelector] Including method
>> test.admin.ensureCreatedListenerExists()
>> [testng] [XmlMethodSelector] Including method
>> test.admin.createListener()
>> [testng] [SuiteRunner] Created 1 TestRunners
>> [testng] [TestRunner] Running test asadmin_tests on 3 classes,
>> included groups:[] excluded groups:[]
>> [testng] [TestClass]
>> [testng] ======
>> [testng] TESTCLASS: test.admin.JdbcConnectionPoolTests
>> [testng] [TestClass] BeforeClass :
>> test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
>> java.lang.String, java.lang.String)
>> [testng] [TestClass] BeforeClass :
>> test.admin.JdbcConnectionPoolTests.setupEnvironment(java.lang.String)
>> [testng] [TestClass] Test :
>> test.admin.JdbcConnectionPoolTests.createPool()
>> [testng] [TestClass] Test :
>> test.admin.JdbcConnectionPoolTests.ensureCreatedPoolExists()
>> [testng] [TestClass] Test :
>> test.admin.JdbcConnectionPoolTests.deletedPoolDoesNotExist()
>> [testng] [TestClass] Test :
>> test.admin.JdbcConnectionPoolTests.deletePool()
>> [testng] [TestClass] Test :
>> test.admin.JdbcConnectionPoolTests.pingPool()
>> [testng] [TestClass]
>> [testng] ======
>>
>> [testng] [TestClass]
>> [testng] ======
>> [testng] TESTCLASS: test.admin.JvmOptionTests
>> [testng] [TestClass] BeforeClass :
>> test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
>> java.lang.String, java.lang.String)
>> [testng] [TestClass] Test :
>> test.admin.JvmOptionTests.deletedJoeDoesNotExist()
>> [testng] [TestClass] Test :
>> test.admin.JvmOptionTests.deleteJoe()
>> [testng] [TestClass] Test :
>> test.admin.JvmOptionTests.ensureCreatedJoeExists()
>> [testng] [TestClass] Test :
>> test.admin.JvmOptionTests.createJoe()
>> [testng] [TestClass]
>> [testng] ======
>>
>> [testng] [TestClass]
>> [testng] ======
>> [testng] TESTCLASS: test.admin.WebtierTests
>> [testng] [TestClass] BeforeClass :
>> test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
>> java.lang.String, java.lang.String)
>> [testng] [TestClass] BeforeClass : test.admin.WebtierTests.setup()
>> [testng] [TestClass] Test :
>> test.admin.WebtierTests.deleteListener()
>> [testng] [TestClass] Test :
>> test.admin.WebtierTests.ensureDeletedListenerDoesNotExist()
>> [testng] [TestClass] Test :
>> test.admin.WebtierTests.createListenerWithOldParam()
>> [testng] [TestClass] Test :
>> test.admin.WebtierTests.ensureCreatedListenerExists()
>> [testng] [TestClass] Test :
>> test.admin.WebtierTests.createListener()
>> [testng] [TestClass]
>> [testng] ======
>>
>> [testng] [TestRunner] WILL BE RUN IN RANDOM ORDER:
>> [testng] [TestRunner]
>> test.admin.WebtierTests.createListenerWithOldParam()
>> [testng] [TestRunner] WILL BE RUN SEQUENTIALLY:
>> [testng] [TestRunner] test.admin.WebtierTests.createListener()
>> [testng] [TestRunner]
>> test.admin.JdbcConnectionPoolTests.createPool()
>> [testng] [TestRunner] test.admin.JvmOptionTests.createJoe()
>> [testng] [TestRunner]
>> test.admin.JvmOptionTests.ensureCreatedJoeExists()
>> [testng] [TestRunner] test.admin.JvmOptionTests.deleteJoe()
>> [testng] [TestRunner]
>> test.admin.WebtierTests.ensureCreatedListenerExists()
>> [testng] [TestRunner] test.admin.WebtierTests.deleteListener()
>> [testng] [TestRunner]
>> test.admin.WebtierTests.ensureDeletedListenerDoesNotExist()
>> [testng] [TestRunner] test.admin.JdbcConnectionPoolTests.pingPool()
>> [testng] [TestRunner]
>> test.admin.JvmOptionTests.deletedJoeDoesNotExist()
>> [testng] [TestRunner]
>> test.admin.JdbcConnectionPoolTests.ensureCreatedPoolExists()
>> [testng] [TestRunner]
>> test.admin.JdbcConnectionPoolTests.deletePool()
>> [testng] [TestRunner]
>> test.admin.JdbcConnectionPoolTests.deletedPoolDoesNotExist()
>> [testng] [TestRunner] ====
>> [testng] [TestRunner] ===
>> [testng] [TestRunner] Found 2 applicable methods
>> [testng] [Invoker 20469344] Invoking @BeforeClass
>> test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
>> java.lang.String, java.lang.String)
>> [testng] [Invoker 20469344] Invoking @BeforeClass
>> test.admin.WebtierTests.setup()
>> [testng] [Invoker 20469344] Invoking
>> test.admin.WebtierTests.createListener
>> [testng] [Invoker 20469344] Invoking @BeforeClass
>> test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
>> java.lang.String, java.lang.String)
>> [testng] [Invoker 20469344] Invoking @BeforeClass
>> test.admin.JdbcConnectionPoolTests.setupEnvironment(java.lang.String)
>> [testng] [Invoker 20469344] Invoking
>> test.admin.JdbcConnectionPoolTests.createPool
>> [testng] [Invoker 20469344] Invoking @BeforeClass
>> test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
>> java.lang.String, java.lang.String)
>> [testng] [Invoker 20469344] Invoking
>> test.admin.JvmOptionTests.createJoe
>> [testng] [Invoker 20469344] Invoking
>> test.admin.JvmOptionTests.ensureCreatedJoeExists
>> [testng] [Invoker 20469344] Invoking
>> test.admin.JvmOptionTests.deleteJoe
>> [testng] [Invoker 20469344] Invoking
>> test.admin.WebtierTests.ensureCreatedListenerExists
>> [testng] [Invoker 20469344] Invoking
>> test.admin.WebtierTests.deleteListener
>> [testng] [Invoker 20469344] Invoking
>> test.admin.JdbcConnectionPoolTests.pingPool
>> [testng] [Invoker 20469344] Invoking
>> test.admin.JvmOptionTests.deletedJoeDoesNotExist
>> [testng] [Invoker 20469344] Invoking
>> test.admin.JdbcConnectionPoolTests.ensureCreatedPoolExists
>> [testng] [Invoker 20469344] Invoking
>> test.admin.JdbcConnectionPoolTests.deletePool
>> [testng] [Invoker 20469344] Invoking
>> test.admin.JdbcConnectionPoolTests.deletedPoolDoesNotExist
>> [testng] [Invoker 20469344] Invoking
>> test.admin.WebtierTests.createListenerWithOldParam
>>
>> [testng] *********** INVOKED METHODS
>>
>> [testng]
>> test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
>> java.lang.String, java.lang.String)http://localhost:4848/__asadmin
>> admin 7408068
>> [testng] test.admin.WebtierTests.setup() 7408068
>> [testng] test.admin.WebtierTests.createListener() 7408068
>> [testng]
>> test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
>> java.lang.String, java.lang.String)http://localhost:4848/__asadmin
>> admin 24744797
>> [testng]
>> test.admin.JdbcConnectionPoolTests.setupEnvironment(java.lang.String)resources/resources.xml
>> 24744797
>> [testng]
>> test.admin.JdbcConnectionPoolTests.createPool() 24744797
>> [testng]
>> test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
>> java.lang.String, java.lang.String)http://localhost:4848/__asadmin
>> admin 9949215
>> [testng] test.admin.JvmOptionTests.createJoe() 9949215
>> [testng]
>> test.admin.JvmOptionTests.ensureCreatedJoeExists() 9949215
>> [testng] test.admin.JvmOptionTests.deleteJoe() 9949215
>> [testng]
>> test.admin.WebtierTests.ensureCreatedListenerExists() 7408068
>> [testng] test.admin.WebtierTests.deleteListener() 7408068
>> [testng] test.admin.JdbcConnectionPoolTests.pingPool()
>> 24744797
>> [testng]
>> test.admin.JvmOptionTests.deletedJoeDoesNotExist() 9949215
>> [testng]
>> test.admin.JdbcConnectionPoolTests.ensureCreatedPoolExists() 24744797
>> [testng]
>> test.admin.JdbcConnectionPoolTests.deletePool() 24744797
>> [testng]
>> test.admin.JdbcConnectionPoolTests.deletedPoolDoesNotExist() 24744797
>> [testng]
>> test.admin.WebtierTests.createListenerWithOldParam() 7408068
>>
>> [testng] ***********
>>
>> [testng] Creating
>> /export/home/ming/v3/quicklook/test-output/QuickLookTests/asadmin_tests.html
>> [testng] PASSED: createListener
>> [testng] PASSED: createPool
>> [testng] PASSED: createJoe
>> [testng] PASSED: ensureCreatedJoeExists
>> [testng] PASSED: deleteJoe
>> [testng] PASSED: ensureCreatedListenerExists
>> [testng] PASSED: pingPool
>> [testng] PASSED: deletedJoeDoesNotExist
>> [testng] PASSED: ensureCreatedPoolExists
>> [testng] PASSED: deletePool
>> [testng] PASSED: deletedPoolDoesNotExist
>> [testng] PASSED: createListenerWithOldParam
>> [testng] FAILED: deleteListener
>> [testng] java.lang.RuntimeException: null
>> [testng] at
>> test.admin.util.GeneralUtils.handleManifestFailure(GeneralUtils.java:124)
>> [testng] at
>> test.admin.WebtierTests.deleteListener(WebtierTests.java:101)
>> [testng] ... Removed 22 stack frames
>> [testng] SKIPPED: ensureDeletedListenerDoesNotExist
>>
>> [testng] ===============================================
>> [testng] asadmin_tests
>> [testng] Tests run: 14, Failures: 1, Skips: 1
>> [testng] ===============================================
>>
>>
>> [testng] ===============================================
>> [testng] QuickLookTests
>> [testng] Total tests run: 14, Failures: 1, Skips: 1
>> [testng] ===============================================
>>
>> Ken Cavanaugh wrote:
>>> Darani and I have been running the QLs on multiple machines in order to
>>> verify the functioning of ORB b030 (GFv3 is currently using ORB b029),
>>> and every time we run the tests we get different results. I've had
>>> the same
>>> problems with trying to integrate Gmbal b016.
>>>
>>> The main tests that do this in the QL appear to be
>>> iterateAllSanityCheck (an
>>> AMX test), testAMXComplianceMonitorFailureCount, and deleteListener.
>>> Over 4 QL runs, I've observed the following:
>>>
>>> * Run 1: passed with Gmbal b015
>>> * Run 2: all 3 tests failed with Gmbal b016
>>> * Run 3: with CORBA b030, testAMXComplianceMonitorFailureCount
>>> failed
>>> o The AMX compliance monitor complained about the following:
>>> + Attribute 'Enabled' failed for
>>> amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
>>> + Attribute 'JvmOptions' failed for
>>> amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
>>> + Attribute 'NativeLibraryPath' failed for
>>> amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
>>> + Attribute 'Classpath' failed for
>>> amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
>>> + Attribute 'Property' failed for
>>> amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
>>> + amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
>>> General test failure in validateAMXConfig:
>>> java.lang.NullPointerException: "null"
>>> org.glassfish.admin.amx.core.AMXValidator.validateAMXConfig(AMXValidator.java:650)
>>> org.glassfish.admin.amx.core.AMXValidator._validate(AMXValidator.java:618)
>>> org.glassfish.admin.amx.core.AMXValidator.validate(AMXValidator.java:1213)
>>> org.glassfish.admin.amx.impl.mbean.ComplianceMonitor$ValidatorThread.doRun(ComplianceMonitor.java:235)
>>> org.glassfish.admin.amx.impl.mbean.ComplianceMonitor$ValidatorThread.run(ComplianceMonitor.java:206)
>>> * Run 4: I re-ran the same software as in run 3. This time all
>>> 3 tests failed.
>>> + There were several Gmbal failures in the log
>>> (unregister failures). These were all caused by
>>> InstanceNotFoundExceptions on a number of MBeans:
>>> # amx:pp=/mon/server-mon[server],type=bean-method-mon,name=remoteview/HelloBean/bean-methods/hello
>>> # amx:pp=/mon/server-mon[server],type=bean-method-mon,name=remoteview/HelloBean/bean-methods/asyncCancel-int
>>> # amx:pp=/mon/server-mon[server],type=bean-method-mon,name=remoteview/HelloBean/bean-methods/asyncThrowException-java.lang.String
>>> # amx:pp=/mon/server-mon[server],type=bean-method-mon,name=remoteview/HelloBean/bean-methods/throwException-java.lang.String
>>> # amx:pp=/mon/server-mon[server],type=bean-method-mon,name=remoteview/HelloBean/bean-methods/asyncBlock-int
>>> # etc. for a total of 153 MBeans: each
>>> unregister failure happened twice in a row.
>>>
>>> Now, I initially assumed that the QL failures I observed in Run 2
>>> were do to interactions between
>>> a new Gmbal feature (root parent monitoring) and the rest of GFv3
>>> monitoring. But the same tests
>>> seem to fail almost randomly in other situations. Particularly
>>> disturbing is the fact the Run 3 and Run 4
>>> shared completely IDENTICAL app server code, and exhibited different
>>> failures. I also highly doubt that
>>> the ORB has anything to do with the observed (and unpredictable)
>>> failures. Note that very similar failures
>>> are seen both with Gmbal b016 and with CORBA b030. That coincidence
>>> and the lack of repeatability lead
>>> me to wonder about the stability of the QLs, and whether the
>>> failures on Gmbal b016 have anything at all to do
>>> with the recent Gmbal changes.
>>>
>>> This is a big problem. Darani and I have been unable to complete
>>> any CORBA or Gmbal integrations recently,
>>> because we keep seeing different failures. Darani had b30 passing
>>> ALL tests at one point, then when she attempted
>>> to verify one final time to complete the integration, similar
>>> failures occurred.
>>>
>>> We need help to understand why we see this level of instability in
>>> QL runs. Without fixing this, we
>>> are unable to integrate either CORBA or Gmbal, and we are nearly out
>>> of time for a number
>>> of important fixes (many of which are ready to go) before hard code
>>> freeze.
>>>
>>> Thanks,
>>>
>>> Ken.
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net For
>>> additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>
>