Speaking of QL
QL fails 100% of the time on Windows for me -- for the past 4 or 5 days.
Sreenivas Munnangi wrote:
Ming Zhang wrote:
Jane,
I have checked in one more test to delete http-listener "ls123452". Now
admin tests can be run repeatedly without errors and the domain.xml
gets cleaned up after each admin test run.
Lloyd,
the AMX tests still failed in the 2nd QL run
How many tests failed, if possible could you copy and paste the
failures from ql output.
(iterateAllSanityCheck,
testAMXComplianceMonitorFailureCount) after my change for admin tests.
Is this due to the configuration changes in domain.xml after admin GUI
loaded? Is it reasonable to expect AMX tests will accommodate the
configurations of admin GUI in domain.xml?
Thanks,
Ming
Jane Young wrote:
Ming,
Recreating the domain will resolve the AMX test failures. Looks like
QL may not be cleaning-up domain.xml and resetting to the original
configuration.
I see that admin-gui configuration is added to domain.xml after running
QL. This is because admin-gui is getting invoked the first time.
There is a new http-listener, "ls123452". Should that be removed?
I'm not sure what other configurations are added and need to be removed
after running QL.
Can you look into this?
Thanks,
Jane
Ming Zhang wrote:
I tried to reproduce the issue with the promoted b66
(http://download.java.net/glassfish/v3/promoted/glassfish-v3-b66.zip)
and here are what I found:
- Run QL first time in web distribution profile and all
passed.
- Run QL again and the amx or admin tests start having
errors
- To isolate the problem, I deleted the b66 and unzip
again.
Run
amx test individually multiple times and tests passed:
- quicklook/amx % ant
-Dglassfish.home=/export/home/ming/v3/b66/glassfishv3/glassfish all
- Start domain and derby. Run admin tests individually. The
tests
passed first time and started failed when ran again.
- quicklook/admin % ant
-Dglassfish.home=/export/home/ming/v3/b66/glassfishv3/glassfish all
The stack trace on server side is clean:
http://javaweb.sfbay/~mzh777/v3/ql/b66/server.log
But there are exceptions on client side. Can someone on admin team take
a look for the error?
Thanks,
Ming
To turn on the the verbose mode for testng
quicklook/admin
% svn diff
Index: testng.xml
===================================================================
--- testng.xml (revision 32234)
+++ testng.xml (working copy)
@@ -35,7 +35,7 @@
-->
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd"
>
-<suite name="QuickLookTests" verbose="2" >
+<suite name="QuickLookTests" verbose="4" >
and here are the output on console:
runtest-impl-xml:
[echo] =============Starting TestNG test at ..//classes/test from
testng.xml ============
[testng] [Parser] Running:
[testng] /export/home/ming/v3/quicklook/admin/testng.xml
[testng] [RunInfo] Adding method selector:
org.testng.internal.XmlMethodSelector@ef2c60 priority: 10
[testng] [TestClass] Creating TestClass for [ClassImpl
test.admin.JdbcConnectionPoolTests]
[testng] [TestClass] Adding method
test.admin.JdbcConnectionPoolTests.createPool() on TestClass class
test.admin.JdbcConnectionPoolTests
[testng] [TestClass] Adding method
test.admin.JdbcConnectionPoolTests.ensureCreatedPoolExists() on
TestClass class test.admin.JdbcConnectionPoolTests
[testng] [TestClass] Adding method
test.admin.JdbcConnectionPoolTests.deletedPoolDoesNotExist() on
TestClass class test.admin.JdbcConnectionPoolTests
[testng] [TestClass] Adding method
test.admin.JdbcConnectionPoolTests.deletePool() on TestClass class
test.admin.JdbcConnectionPoolTests
[testng] [TestClass] Adding method
test.admin.JdbcConnectionPoolTests.pingPool() on TestClass class
test.admin.JdbcConnectionPoolTests
[testng] [XmlMethodSelector] Including method
test.admin.setUpEnvironment()
[testng] [XmlMethodSelector] Including method
test.admin.setupEnvironment()
[testng] [MethodInheritance]
test.admin.JdbcConnectionPoolTests.setupEnvironment(java.lang.String)
DEPENDS ON
test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
java.lang.String, java.lang.String)
[testng] [TestClass] Creating TestClass for [ClassImpl
test.admin.JvmOptionTests]
[testng] [TestClass] Adding method
test.admin.JvmOptionTests.deletedJoeDoesNotExist() on TestClass class
test.admin.JvmOptionTests
[testng] [TestClass] Adding method
test.admin.JvmOptionTests.deleteJoe() on TestClass class
test.admin.JvmOptionTests
[testng] [TestClass] Adding method
test.admin.JvmOptionTests.ensureCreatedJoeExists() on TestClass class
test.admin.JvmOptionTests
[testng] [TestClass] Adding method
test.admin.JvmOptionTests.createJoe() on TestClass class
test.admin.JvmOptionTests
[testng] [TestClass] Creating TestClass for [ClassImpl
test.admin.WebtierTests]
[testng] [TestClass] Adding method
test.admin.WebtierTests.deleteListener() on TestClass class
test.admin.WebtierTests
[testng] [TestClass] Adding method
test.admin.WebtierTests.ensureDeletedListenerDoesNotExist() on
TestClass class test.admin.WebtierTests
[testng] [TestClass] Adding method
test.admin.WebtierTests.createListenerWithOldParam() on TestClass class
test.admin.WebtierTests
[testng] [TestClass] Adding method
test.admin.WebtierTests.ensureCreatedListenerExists() on TestClass
class test.admin.WebtierTests
[testng] [TestClass] Adding method
test.admin.WebtierTests.createListener() on TestClass class
test.admin.WebtierTests
[testng] [XmlMethodSelector] Including method test.admin.setup()
[testng] [MethodInheritance] test.admin.WebtierTests.setup() DEPENDS
ON test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
java.lang.String, java.lang.String)
[testng] [MethodInheritance]
test.admin.JdbcConnectionPoolTests.setupEnvironment(java.lang.String)
DEPENDS ON
test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
java.lang.String, java.lang.String)
[testng] [MethodInheritance] test.admin.WebtierTests.setup() DEPENDS
ON test.admin.JdbcConnectionPoolTests.setupEnvironment(java.lang.String)
[testng] [XmlMethodSelector] Including method test.admin.createPool()
[testng] [XmlMethodSelector] Including method
test.admin.ensureCreatedPoolExists()
[testng] [XmlMethodSelector] Including method
test.admin.deletedPoolDoesNotExist()
[testng] [XmlMethodSelector] Including method test.admin.deletePool()
[testng] [XmlMethodSelector] Including method test.admin.pingPool()
[testng] [XmlMethodSelector] Including method
test.admin.deletedJoeDoesNotExist()
[testng] [XmlMethodSelector] Including method test.admin.deleteJoe()
[testng] [XmlMethodSelector] Including method
test.admin.ensureCreatedJoeExists()
[testng] [XmlMethodSelector] Including method test.admin.createJoe()
[testng] [XmlMethodSelector] Including method
test.admin.deleteListener()
[testng] [XmlMethodSelector] Including method
test.admin.ensureDeletedListenerDoesNotExist()
[testng] [XmlMethodSelector] Including method
test.admin.createListenerWithOldParam()
[testng] [XmlMethodSelector] Including method
test.admin.ensureCreatedListenerExists()
[testng] [XmlMethodSelector] Including method
test.admin.createListener()
[testng] [SuiteRunner] Created 1 TestRunners
[testng] [TestRunner] Running test asadmin_tests on 3 classes,
included groups:[] excluded groups:[]
[testng] [TestClass]
[testng] ======
[testng] TESTCLASS: test.admin.JdbcConnectionPoolTests
[testng] [TestClass] BeforeClass :
test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
java.lang.String, java.lang.String)
[testng] [TestClass] BeforeClass :
test.admin.JdbcConnectionPoolTests.setupEnvironment(java.lang.String)
[testng] [TestClass] Test :
test.admin.JdbcConnectionPoolTests.createPool()
[testng] [TestClass] Test :
test.admin.JdbcConnectionPoolTests.ensureCreatedPoolExists()
[testng] [TestClass] Test :
test.admin.JdbcConnectionPoolTests.deletedPoolDoesNotExist()
[testng] [TestClass] Test :
test.admin.JdbcConnectionPoolTests.deletePool()
[testng] [TestClass] Test :
test.admin.JdbcConnectionPoolTests.pingPool()
[testng] [TestClass]
[testng] ======
[testng] [TestClass]
[testng] ======
[testng] TESTCLASS: test.admin.JvmOptionTests
[testng] [TestClass] BeforeClass :
test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
java.lang.String, java.lang.String)
[testng] [TestClass] Test :
test.admin.JvmOptionTests.deletedJoeDoesNotExist()
[testng] [TestClass] Test :
test.admin.JvmOptionTests.deleteJoe()
[testng] [TestClass] Test :
test.admin.JvmOptionTests.ensureCreatedJoeExists()
[testng] [TestClass] Test :
test.admin.JvmOptionTests.createJoe()
[testng] [TestClass]
[testng] ======
[testng] [TestClass]
[testng] ======
[testng] TESTCLASS: test.admin.WebtierTests
[testng] [TestClass] BeforeClass :
test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
java.lang.String, java.lang.String)
[testng] [TestClass] BeforeClass : test.admin.WebtierTests.setup()
[testng] [TestClass] Test :
test.admin.WebtierTests.deleteListener()
[testng] [TestClass] Test :
test.admin.WebtierTests.ensureDeletedListenerDoesNotExist()
[testng] [TestClass] Test :
test.admin.WebtierTests.createListenerWithOldParam()
[testng] [TestClass] Test :
test.admin.WebtierTests.ensureCreatedListenerExists()
[testng] [TestClass] Test :
test.admin.WebtierTests.createListener()
[testng] [TestClass]
[testng] ======
[testng] [TestRunner] WILL BE RUN IN RANDOM ORDER:
[testng] [TestRunner]
test.admin.WebtierTests.createListenerWithOldParam()
[testng] [TestRunner] WILL BE RUN SEQUENTIALLY:
[testng] [TestRunner] test.admin.WebtierTests.createListener()
[testng] [TestRunner]
test.admin.JdbcConnectionPoolTests.createPool()
[testng] [TestRunner] test.admin.JvmOptionTests.createJoe()
[testng] [TestRunner]
test.admin.JvmOptionTests.ensureCreatedJoeExists()
[testng] [TestRunner] test.admin.JvmOptionTests.deleteJoe()
[testng] [TestRunner]
test.admin.WebtierTests.ensureCreatedListenerExists()
[testng] [TestRunner] test.admin.WebtierTests.deleteListener()
[testng] [TestRunner]
test.admin.WebtierTests.ensureDeletedListenerDoesNotExist()
[testng] [TestRunner] test.admin.JdbcConnectionPoolTests.pingPool()
[testng] [TestRunner]
test.admin.JvmOptionTests.deletedJoeDoesNotExist()
[testng] [TestRunner]
test.admin.JdbcConnectionPoolTests.ensureCreatedPoolExists()
[testng] [TestRunner]
test.admin.JdbcConnectionPoolTests.deletePool()
[testng] [TestRunner]
test.admin.JdbcConnectionPoolTests.deletedPoolDoesNotExist()
[testng] [TestRunner] ====
[testng] [TestRunner] ===
[testng] [TestRunner] Found 2 applicable methods
[testng] [Invoker 20469344] Invoking @BeforeClass
test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
java.lang.String, java.lang.String)
[testng] [Invoker 20469344] Invoking @BeforeClass
test.admin.WebtierTests.setup()
[testng] [Invoker 20469344] Invoking
test.admin.WebtierTests.createListener
[testng] [Invoker 20469344] Invoking @BeforeClass
test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
java.lang.String, java.lang.String)
[testng] [Invoker 20469344] Invoking @BeforeClass
test.admin.JdbcConnectionPoolTests.setupEnvironment(java.lang.String)
[testng] [Invoker 20469344] Invoking
test.admin.JdbcConnectionPoolTests.createPool
[testng] [Invoker 20469344] Invoking @BeforeClass
test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
java.lang.String, java.lang.String)
[testng] [Invoker 20469344] Invoking
test.admin.JvmOptionTests.createJoe
[testng] [Invoker 20469344] Invoking
test.admin.JvmOptionTests.ensureCreatedJoeExists
[testng] [Invoker 20469344] Invoking
test.admin.JvmOptionTests.deleteJoe
[testng] [Invoker 20469344] Invoking
test.admin.WebtierTests.ensureCreatedListenerExists
[testng] [Invoker 20469344] Invoking
test.admin.WebtierTests.deleteListener
[testng] [Invoker 20469344] Invoking
test.admin.JdbcConnectionPoolTests.pingPool
[testng] [Invoker 20469344] Invoking
test.admin.JvmOptionTests.deletedJoeDoesNotExist
[testng] [Invoker 20469344] Invoking
test.admin.JdbcConnectionPoolTests.ensureCreatedPoolExists
[testng] [Invoker 20469344] Invoking
test.admin.JdbcConnectionPoolTests.deletePool
[testng] [Invoker 20469344] Invoking
test.admin.JdbcConnectionPoolTests.deletedPoolDoesNotExist
[testng] [Invoker 20469344] Invoking
test.admin.WebtierTests.createListenerWithOldParam
[testng] *********** INVOKED METHODS
[testng]
test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
java.lang.String, java.lang.String)http://localhost:4848/__asadmin
admin 7408068
[testng] test.admin.WebtierTests.setup() 7408068
[testng] test.admin.WebtierTests.createListener() 7408068
[testng]
test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
java.lang.String, java.lang.String)http://localhost:4848/__asadmin
admin 24744797
[testng]
test.admin.JdbcConnectionPoolTests.setupEnvironment(java.lang.String)resources/resources.xml
24744797
[testng] test.admin.JdbcConnectionPoolTests.createPool()
24744797
[testng]
test.admin.BaseAsadminTest.setUpEnvironment(java.lang.String,
java.lang.String, java.lang.String)http://localhost:4848/__asadmin
admin 9949215
[testng] test.admin.JvmOptionTests.createJoe() 9949215
[testng]
test.admin.JvmOptionTests.ensureCreatedJoeExists() 9949215
[testng] test.admin.JvmOptionTests.deleteJoe() 9949215
[testng]
test.admin.WebtierTests.ensureCreatedListenerExists() 7408068
[testng] test.admin.WebtierTests.deleteListener() 7408068
[testng] test.admin.JdbcConnectionPoolTests.pingPool()
24744797
[testng]
test.admin.JvmOptionTests.deletedJoeDoesNotExist() 9949215
[testng]
test.admin.JdbcConnectionPoolTests.ensureCreatedPoolExists() 24744797
[testng] test.admin.JdbcConnectionPoolTests.deletePool()
24744797
[testng]
test.admin.JdbcConnectionPoolTests.deletedPoolDoesNotExist() 24744797
[testng]
test.admin.WebtierTests.createListenerWithOldParam() 7408068
[testng] ***********
[testng] Creating
/export/home/ming/v3/quicklook/test-output/QuickLookTests/asadmin_tests.html
[testng] PASSED: createListener
[testng] PASSED: createPool
[testng] PASSED: createJoe
[testng] PASSED: ensureCreatedJoeExists
[testng] PASSED: deleteJoe
[testng] PASSED: ensureCreatedListenerExists
[testng] PASSED: pingPool
[testng] PASSED: deletedJoeDoesNotExist
[testng] PASSED: ensureCreatedPoolExists
[testng] PASSED: deletePool
[testng] PASSED: deletedPoolDoesNotExist
[testng] PASSED: createListenerWithOldParam
[testng] FAILED: deleteListener
[testng] java.lang.RuntimeException: null
[testng] at
test.admin.util.GeneralUtils.handleManifestFailure(GeneralUtils.java:124)
[testng] at
test.admin.WebtierTests.deleteListener(WebtierTests.java:101)
[testng] ... Removed 22 stack frames
[testng] SKIPPED: ensureDeletedListenerDoesNotExist
[testng] ===============================================
[testng] asadmin_tests
[testng] Tests run: 14, Failures: 1, Skips: 1
[testng] ===============================================
[testng] ===============================================
[testng] QuickLookTests
[testng] Total tests run: 14, Failures: 1, Skips: 1
[testng] ===============================================
Ken Cavanaugh wrote:
Darani
and
I have been running the QLs on multiple machines in order to
verify the functioning of ORB b030 (GFv3 is currently using ORB b029),
and every time we run the tests we get different results. I've had the
same
problems with trying to integrate Gmbal b016.
The main tests that do this in the QL appear to be
iterateAllSanityCheck (an
AMX test), testAMXComplianceMonitorFailureCount, and deleteListener.
Over 4 QL runs, I've observed the following:
- Run 1: passed with Gmbal b015
- Run 2: all 3 tests failed with Gmbal b016
- Run 3: with CORBA b030,
testAMXComplianceMonitorFailureCount
failed
- The AMX compliance monitor complained about the
following:
- Attribute 'Enabled' failed for
amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
- Attribute 'JvmOptions' failed for
amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
- Attribute 'NativeLibraryPath' failed for
amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
- Attribute 'Classpath' failed for
amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
- Attribute 'Property' failed for
amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
- amx:pp=/domain/configs/config[server-config]/java-config,type=profiler
General test failure in validateAMXConfig:
java.lang.NullPointerException: "null"
org.glassfish.admin.amx.core.AMXValidator.validateAMXConfig(AMXValidator.java:650)
org.glassfish.admin.amx.core.AMXValidator._validate(AMXValidator.java:618)
org.glassfish.admin.amx.core.AMXValidator.validate(AMXValidator.java:1213)
org.glassfish.admin.amx.impl.mbean.ComplianceMonitor$ValidatorThread.doRun(ComplianceMonitor.java:235)
org.glassfish.admin.amx.impl.mbean.ComplianceMonitor$ValidatorThread.run(ComplianceMonitor.java:206)
- Run 4: I re-ran the same software as in run 3. This
time
all
3
tests failed.
- There were several Gmbal failures in the log
(unregister
failures). These were all caused by InstanceNotFoundExceptions on a
number of MBeans:
- amx:pp=/mon/server-mon[server],type=bean-method-mon,name=remoteview/HelloBean/bean-methods/hello
- amx:pp=/mon/server-mon[server],type=bean-method-mon,name=remoteview/HelloBean/bean-methods/asyncCancel-int
- amx:pp=/mon/server-mon[server],type=bean-method-mon,name=remoteview/HelloBean/bean-methods/asyncThrowException-java.lang.String
- amx:pp=/mon/server-mon[server],type=bean-method-mon,name=remoteview/HelloBean/bean-methods/throwException-java.lang.String
- amx:pp=/mon/server-mon[server],type=bean-method-mon,name=remoteview/HelloBean/bean-methods/asyncBlock-int
- etc. for a total of 153 MBeans: each unregister
failure
happened twice in a row.
Now, I initially assumed that the QL failures I observed in Run 2 were
do to interactions between
a new Gmbal feature (root parent monitoring) and the rest of GFv3
monitoring. But the same tests
seem to fail almost randomly in other situations. Particularly
disturbing is the fact the Run 3 and Run 4
shared completely IDENTICAL app server code, and exhibited different
failures. I also highly doubt that
the ORB has anything to do with the observed (and unpredictable)
failures. Note that very similar failures
are seen both with Gmbal b016 and with CORBA b030. That coincidence
and the lack of repeatability lead
me to wonder about the stability of the QLs, and whether the failures
on Gmbal b016 have anything at all to do
with the recent Gmbal changes.
This is a big problem. Darani and I have been unable to complete any
CORBA or Gmbal integrations recently,
because we keep seeing different failures. Darani had b30 passing ALL
tests at one point, then when she attempted
to verify one final time to complete the integration, similar failures
occurred.
We need help to understand why we see this level of instability in QL
runs. Without fixing this, we
are unable to integrate either CORBA or Gmbal, and we are nearly out of
time for a number
of important fixes (many of which are ready to go) before hard code
freeze.
Thanks,
Ken.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: dev-help@glassfish.dev.java.net