dev@glassfish.java.net

Re: Instability of QuickLook tests

From: Jane Young <Jane.Young_at_Sun.COM>
Date: Thu, 08 Oct 2009 10:30:37 -0700

Thanks for the update. Is it possible to commit the fix in today?

Thanks,
Jane


Jennifer Chou wrote:
> Hi Jane,
>
> I've been working with Lloyd, Ken, and Marina on a fix for [Issue
> 10042] [monitoring] ejb-application-mon has inconsistent MBeanInfo.
> - an mbean problem.
> Fixing this will fix a major part of the problem.
>
> Hoping to check-in soon.
>
> Thanks,
> Jennifer
>
> Jane Young wrote:
>> Lloyd,
>>
>> We really need to do something about the amx_tests failure in QL.
>> It's showing up randomly on Hudson.
>>
>> See (ignore the adminconsole_tests failures; Ken has committed the fix):
>> http://hudson.glassfish.org/job/gf-trunk-build-continuous/2685/artifact/bundles/QL-WP-report.html
>> http://hudson.glassfish.org/job/gf-trunk-build-continuous/2684/artifact/bundles/QL-WP-report.html
>> http://hudson.glassfish.org/job/gf-trunk-build-continuous/2680/artifact/bundles/QL-WP-report.html
>> http://hudson.glassfish.org/job/gf-trunk-build-continuous/2678/artifact/bundles/QL-WP-report.html
>> http://hudson.glassfish.org/job/gf-trunk-build-continuous/2677/artifact/bundles/QL-WP-report.html
>> http://gf-hudson.sfbay/hudson/job/gf-trunk-build-continuous/2687/artifact/bundles/QL-WP-report.html
>>
>> These are jobs from 2:30am-9:30am today.
>>
>> Can you please take a look? Is this because of new AMX and
>> Monitoring changes or QL tests? As of last Thursday (Oct 1st) QL
>> was stable.
>>
>> Thanks,
>> Jane
>>
>>
>> Lloyd Chambers wrote:
>>> I’ve found at least one problem--
>>>
>>> The QL AMX tests create and delete various config MBeans for test
>>> purposes.
>>>
>>> The AMXValidator is in the middle of validating them when they are
>>> unregistered.
>>>
>>> I am fixing the AMXValidator to recognize that the test failed due
>>> to an MBean disappearing.
>>>
>>> That might not be the only issue, but I will fix it.
>>>
>>> Lloyd
>>>
>>>
>>> On Oct 6, 2009, at 11:45 AM, Lloyd Chambers wrote:
>>>
>>>> Jane,
>>>>
>>>> At least one set of failures is by MBeans disappearing while being
>>>> tested — a race condition.
>>>>
>>>> Who/what is registering and then unregistering in rapid succession
>>>> is unclear, I suspect the monitoring code, which used to have such
>>>> a problem (register/unregister/re-register in a few milliseconds).
>>>>
>>>> I am making that test to be more defensive.
>>>>
>>>> Lloyd
>>>>
>>>> On Oct 6, 2009, at 11:33 AM, Jane Young wrote:
>>>>
>>>>> Hi Lloyd,
>>>>>
>>>>> I agree that the amx-tests failure is detecting valid bugs
>>>>> introduced to v3 workspace. However, one thing I don't
>>>>> understand. Why is it that when I run QL locally it passes in the
>>>>> first run and on Hudon it shows failures and occasionally it
>>>>> passes. If it is a failure, it should always fail.
>>>>>
>>>>> I still don't understand why 2nd QL run always fail in amx-tests.
>>>>>
>>>>> Jane
>>>>>
>>>>>
>>>>> Lloyd Chambers wrote:
>>>>>> My comments on these failures below, inline.
>>>>>>
>>>>>> —Lloyd
>>>>>>
>>>>>> On Oct 4, 2009, at 6:19 PM, Ming Zhang wrote:
>>>>>>
>>>>>>> Sreenivas Munnangi wrote:
>>>>>>>> Ming Zhang wrote:
>>>>>>>>> Jane,
>>>>>>>>>
>>>>>>>>> I have checked in one more test to delete http-listener
>>>>>>>>> "ls123452". Now admin tests can be run repeatedly without
>>>>>>>>> errors and the domain.xml gets cleaned up after each admin
>>>>>>>>> test run.
>>>>>>>>>
>>>>>>>>> Lloyd,
>>>>>>>>> the AMX tests still failed in the 2nd QL run
>>>>>>>> How many tests failed, if possible could you copy and paste the
>>>>>>>> failures from ql output.
>>>>>>>>> (iterateAllSanityCheck, testAMXComplianceMonitorFailureCount)
>>>>>>>>> after my change for admin tests. Is this due to the
>>>>>>>>> configuration changes in domain.xml after admin GUI loaded? Is
>>>>>>>>> it reasonable to expect AMX tests will accommodate the
>>>>>>>>> configurations of admin GUI in domain.xml?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Ming
>>>>>>> The runtestng.output is attached:
>>>>>>>
>>>>>>> [echo] =============Starting TestNG functional tests from
>>>>>>> testng.xml ============
>>>>>>> [testng] [Parser] Running:
>>>>>>> [testng] /export/home/ming/v3/quicklook/testng.xml
>>>>>>> [testng]
>>>>>>> [testng] PASSED: createJDBCPool
>>>>>>> [testng] PASSED: pingJDBCPool
>>>>>>> [testng] PASSED: deleteJDBCPool
>>>>>>> [testng]
>>>>>>> [testng] ===============================================
>>>>>>> [testng] admincli_tests
>>>>>>> [testng] Tests run: 3, Failures: 0, Skips: 0
>>>>>>> [testng] ===============================================
>>>>>>> [testng]
>>>>>>> [testng] Monitoring at HIGH for ThreadPool
>>>>>>> [testng] Monitoring at HIGH for JdbcConnectionPool
>>>>>>> [testng] Monitoring at HIGH for Orb
>>>>>>> [testng] Monitoring at HIGH for ConnectorService
>>>>>>> [testng] Monitoring at HIGH for Security
>>>>>>> [testng] Monitoring at HIGH for JmsService
>>>>>>> [testng] Monitoring at HIGH for WebServicesContainer
>>>>>>> [testng] Monitoring at HIGH for Jvm
>>>>>>> [testng] Monitoring at HIGH for Jpa
>>>>>>> [testng] Monitoring at HIGH for Jersey
>>>>>>> [testng] Monitoring at HIGH for ConnectorConnectionPool
>>>>>>> [testng] Monitoring at HIGH for EjbContainer
>>>>>>> [testng] Monitoring at HIGH for WebContainer
>>>>>>> [testng] Monitoring at HIGH for TransactionService
>>>>>>> [testng] Monitoring at HIGH for HttpService
>>>>>>> [testng]
>>>>>>> [testng] PROBLEMS:
>>>>>>> [testng] Error invoking getParent() on
>>>>>>> amx:pp=/mon/server-mon[server],type=jdbc-connection-pool-mon,name=resources/__TimerPool
>>>>>>> = java.lang.reflect.InvocationTargetException
>>>>>>
>>>>>> This root cause is needed here, but if getParent() is failing
>>>>>> because there is no parent, then this is a critical bug in this
>>>>>> MBean, supplied by the monitoring infrastructure.
>>>>>>
>>>>>>> [testng] CHANGE COUNT: 10
>>>>>>> [testng] PASSED: bootAMX
>>>>>>> [testng] Core AMX tests
>>>>>>> [testng] PASSED: testAmxPref
>>>>>>> [testng] PASSED: connectorConnectionPoolTest
>>>>>>> [testng] PASSED: createChildTest
>>>>>>> [testng] PASSED: testConfigTools
>>>>>>> [testng] PASSED: createProfilerTest
>>>>>>> [testng] PASSED: testAMXConfigAttributeResolver
>>>>>>> [testng] PASSED: testAMXConfigDefaultValues
>>>>>>> [testng] PASSED: testForBogusConfigAnnotations
>>>>>>> [testng] PASSED: testCreateProperties
>>>>>>> [testng] PASSED: testCreateResource
>>>>>>> [testng] PASSED: testCreateProperty
>>>>>>> [testng] PASSED: testConnectorRuntimeAPIProvider
>>>>>>> [testng] PASSED: testMonitoringRoot
>>>>>>> [testng] PASSED: testDomainConfig
>>>>>>> [testng] PASSED: testBulkAccess
>>>>>>> [testng] PASSED: testExt
>>>>>>> [testng] PASSED: testPropertyParent
>>>>>>> [testng] PASSED: testTools
>>>>>>> [testng] PASSED: testPathnames
>>>>>>> [testng] PASSED: testSystemStatus
>>>>>>> [testng] PASSED: testApplications
>>>>>>> [testng] PASSED: testResources
>>>>>>> [testng] PASSED: testChildGetterVariants
>>>>>>> [testng] PASSED: testAllGenerically
>>>>>>> [testng] PASSED: testRuntimeRoot
>>>>>>> [testng] PASSED: testQuery
>>>>>>> [testng] PASSED: testJ2EEDomain
>>>>>>> [testng] PASSED: testSingletonOrNot
>>>>>>> [testng] PASSED: testDomainRoot
>>>>>>> [testng] PASSED: testSystemApplications
>>>>>>> [testng] PASSED: testSystemInfo
>>>>>>> [testng] PASSED: testSystemPropertyParent
>>>>>>> [testng] PASSED: testConfigs
>>>>>>> [testng] PASSED: testServerRuntime
>>>>>>> [testng] PASSED: testLogging
>>>>>>> [testng] PASSED: testServers
>>>>>>> [testng] PASSED: testAutoConvert
>>>>>>> [testng] PASSED: testForBogusAnnotations
>>>>>>> [testng] FAILED: iterateAllSanityCheck
>>>>>>
>>>>>> This should be suppressed as of my Friday checkin. When was this
>>>>>> run?
>>>>>>
>>>>>>> [testng] java.lang.RuntimeException:
>>>>>>> javax.management.InstanceNotFoundException:
>>>>>>> amx:pp=/mon/server-mon[server],type=bean-method-mon,name=slsbnicmt/AnnotatedEJB/bean-methods/persistEntity
>>>>>>> [testng] at
>>>>>>> org.glassfish.admin.amx.core.proxy.ProxyFactory.getMBeanInfo(ProxyFactory.java:447)
>>>>>>> [testng] at
>>>>>>> org.glassfish.admin.amx.core.proxy.ProxyFactory.getProxy(ProxyFactory.java:482)
>>>>>>> [testng] at
>>>>>>> org.glassfish.admin.amx.core.proxy.ProxyFactory.toProxySet(ProxyFactory.java:627)
>>>>>>> [testng] at
>>>>>>> org.glassfish.admin.amx.core.proxy.AMXProxyHandler.autoConvert(AMXProxyHandler.java:653)
>>>>>>> [testng] at
>>>>>>> org.glassfish.admin.amx.core.proxy.AMXProxyHandler._invoke(AMXProxyHandler.java:740)
>>>>>>> [testng] at
>>>>>>> org.glassfish.admin.amx.core.proxy.AMXProxyHandler.invoke(AMXProxyHandler.java:518)
>>>>>>> [testng] at $Proxy19.queryAll(Unknown Source)
>>>>>>> [testng] at amxtest.AMXTestBase.getAllAMX(AMXTestBase.java:295)
>>>>>>> [testng] at
>>>>>>> amxtest.AMXCoreTests.iterateAllSanityCheck(AMXCoreTests.java:86)
>>>>>>> [testng] Caused by:
>>>>>>> javax.management.InstanceNotFoundException:
>>>>>>> amx:pp=/mon/server-mon[server],type=bean-method-mon,name=slsbnicmt/AnnotatedEJB/bean-methods/persistEntity
>>>>>>> [testng] at
>>>>>>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1094)
>>>>>>> [testng] at
>>>>>>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBeanInfo(DefaultMBeanServerInterceptor.java:1372)
>>>>>>> [testng] at
>>>>>>> com.sun.jmx.mbeanserver.JmxMBeanServer.getMBeanInfo(JmxMBeanServer.java:880)
>>>>>>> [testng] at
>>>>>>> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1420)
>>>>>>> [testng] at
>>>>>>> javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
>>>>>>> [testng] at
>>>>>>> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1264)
>>>>>>> [testng] at
>>>>>>> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1359)
>>>>>>> [testng] at
>>>>>>> javax.management.remote.rmi.RMIConnectionImpl.getMBeanInfo(RMIConnectionImpl.java:864)
>>>>>>> [testng] at
>>>>>>> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
>>>>>>> [testng] at sun.rmi.transport.Transport$1.run(Transport.java:159)
>>>>>>> [testng] at java.security.AccessController.doPrivileged(Native
>>>>>>> Method)
>>>>>>> [testng] at
>>>>>>> sun.rmi.transport.Transport.serviceCall(Transport.java:155)
>>>>>>> [testng] at
>>>>>>> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
>>>>>>> [testng] at
>>>>>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
>>>>>>> [testng] at
>>>>>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>>>>>>> [testng] at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>> [testng] at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>> [testng] at java.lang.Thread.run(Thread.java:619)
>>>>>>> [testng] at
>>>>>>> sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:255)
>>>>>>> [testng] at
>>>>>>> sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:233)
>>>>>>> [testng] at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:142)
>>>>>>> [testng] at com.sun.jmx.remote.internal.PRef.invoke(Unknown
>>>>>>> Source)
>>>>>>> [testng] at
>>>>>>> javax.management.remote.rmi.RMIConnectionImpl_Stub.getMBeanInfo(Unknown
>>>>>>> Source)
>>>>>>> [testng] at
>>>>>>> javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.getMBeanInfo(RMIConnector.java:1037)
>>>>>>> [testng] at
>>>>>>> org.glassfish.admin.amx.core.proxy.ProxyFactory.getMBeanInfo(ProxyFactory.java:437)
>>>>>>> [testng] ... 30 more
>>>>>>> [testng] ... Removed 25 stack frames
>>>>>>> [testng] FAILED: testAMXComplianceMonitorFailureCount
>>>>>>> [testng] java.lang.AssertionError: Server indicates that there
>>>>>>> are non-compliant AMX MBean validator failures, failure count =
>>>>>>> 1, examine the server log for failures
>>>>>>> [testng] at
>>>>>>> amxtest.AMXCoreTests.testAMXComplianceMonitorFailureCount(AMXCoreTests.java:120)
>>>>>>> [testng] ... Removed 22 stack frames
>>>>>>
>>>>>> This indicates non-compliant MBeans. I will see if I can make
>>>>>> the message spell out which MBeans and what failed.
>>>>>>
>>>>>>> [testng]
>>>>>>> [testng] ===============================================
>>>>>>> [testng] amx_tests
>>>>>>> [testng] Tests run: 41, Failures: 2, Skips: 0
>>>>>>> [testng] ===============================================
>>>>>>> [testng]
>>>>>>> [testng] ***** Login page not found. Sleeping to allow app to
>>>>>>> deploy (1 of 60)...
>>>>>>> [testng] ***** Login page not found. Sleeping to allow app to
>>>>>>> deploy (2 of 60)...
>>>>>>> [testng] ***** Login page not found. Sleeping to allow app to
>>>>>>> deploy (3 of 60)...
>>>>>>> [testng] ***** Login page not found. Sleeping to allow app to
>>>>>>> deploy (4 of 60)...
>>>>>>> [testng] ***** Login page not found. Sleeping to allow app to
>>>>>>> deploy (5 of 60)...
>>>>>>> [testng] Oct 4, 2009 5:51:08 PM
>>>>>>> org.apache.commons.httpclient.HttpMethodDirector isRedirectNeeded
>>>>>>> [testng] INFO: Redirect requested but followRedirects is disabled
>>>>>>> [testng] PASSED: testDeployedAppPage
>>>>>>> [testng] PASSED: testCommonTasks
>>>>>>> [testng] PASSED: testHelpPage
>>>>>>> [testng] PASSED: testRealmsList
>>>>>>> [testng]
>>>>>>> [testng] ===============================================
>>>>>>> [testng] adminconsole_tests
>>>>>>> [testng] Tests run: 4, Failures: 0, Skips: 0
>>>>>>> [testng] ===============================================
>>>>>>> [testng]
>>>>>>> [testng] PASSED: staticHTMLPageTest
>>>>>>> [testng] PASSED: testServerRunning
>>>>>>> [testng] PASSED: simpleServletTest
>>>>>>> [testng] PASSED: simpleJSPTestPage
>>>>>>> [testng]
>>>>>>> [testng] ===============================================
>>>>>>> [testng] web_jspservlet_war
>>>>>>> [testng] Tests run: 4, Failures: 0, Skips: 0
>>>>>>> [testng] ===============================================
>>>>>>> [testng]
>>>>>>> [testng] PASSED: jsfIndexPageBasicTest
>>>>>>> [testng] PASSED: jsfAppDeployedFirstPagetest
>>>>>>> [testng]
>>>>>>> [testng] ===============================================
>>>>>>> [testng] jsf_hello_world
>>>>>>> [testng] Tests run: 2, Failures: 0, Skips: 0
>>>>>>> [testng] ===============================================
>>>>>>> [testng]
>>>>>>> [testng] Injection2 matched.
>>>>>>> [testng] PASSED: injectedValuesTestPage
>>>>>>> [testng]
>>>>>>> [testng] ===============================================
>>>>>>> [testng] jsf_injection
>>>>>>> [testng] Tests run: 1, Failures: 0, Skips: 0
>>>>>>> [testng] ===============================================
>>>>>>> [testng]
>>>>>>> [testng] PASSED: testNoLeak
>>>>>>> [testng] PASSED: testUserTx
>>>>>>> [testng]
>>>>>>> [testng] ===============================================
>>>>>>> [testng] jdbc_user_tx
>>>>>>> [testng] Tests run: 2, Failures: 0, Skips: 0
>>>>>>> [testng] ===============================================
>>>>>>> [testng]
>>>>>>> [testng] PASSED: testJpaPersistWithInjectEMF
>>>>>>> [testng] PASSED: testJpaLazyLoadingByQuery
>>>>>>> [testng]
>>>>>>> [testng] ===============================================
>>>>>>> [testng] jpa_inject_emf
>>>>>>> [testng] Tests run: 2, Failures: 0, Skips: 0
>>>>>>> [testng] ===============================================
>>>>>>> [testng]
>>>>>>> [testng] PASSED: staticHTMLPageTest
>>>>>>> [testng] PASSED: simpleServletTest
>>>>>>> [testng] PASSED: simpleJSPTestPage
>>>>>>> [testng]
>>>>>>> [testng] ===============================================
>>>>>>> [testng] security_hello_jsp
>>>>>>> [testng] Tests run: 3, Failures: 0, Skips: 0
>>>>>>> [testng] ===============================================
>>>>>>> [testng]
>>>>>>> [testng] PASSED: testAuthGroupMappedUser
>>>>>>> [testng] PASSED: testAuthNotValidPassword
>>>>>>> [testng] PASSED: testAuthRoleMappedUser
>>>>>>> [testng] PASSED: testAuthNotAuthorizedUser
>>>>>>> [testng]
>>>>>>> [testng] ===============================================
>>>>>>> [testng] security_hello_basicauth
>>>>>>> [testng] Tests run: 4, Failures: 0, Skips: 0
>>>>>>> [testng] ===============================================
>>>>>>> [testng]
>>>>>>> [testng] PASSED: SetName
>>>>>>> [testng] PASSED: EJBInject
>>>>>>> [testng] PASSED: GetName
>>>>>>> [testng] PASSED: JpaPersist
>>>>>>> [testng] PASSED: JpaRemove
>>>>>>> [testng] PASSED: JpaVerify
>>>>>>> [testng]
>>>>>>> [testng] ===============================================
>>>>>>> [testng] ejb_stateful_cmt_ni_war
>>>>>>> [testng] Tests run: 6, Failures: 0, Skips: 0
>>>>>>> [testng] ===============================================
>>>>>>> [testng]
>>>>>>> [testng] PASSED: executeServlet
>>>>>>> [testng]
>>>>>>> [testng] ===============================================
>>>>>>> [testng] bv_servlet_simple
>>>>>>> [testng] Tests run: 1, Failures: 0, Skips: 0
>>>>>>> [testng] ===============================================
>>>>>>> [testng]
>>>>>>> [testng] PASSED: appDeployedFirstPagetest
>>>>>>> [testng]
>>>>>>> [testng] ===============================================
>>>>>>> [testng] webbeans_numberguess
>>>>>>> [testng] Tests run: 1, Failures: 0, Skips: 0
>>>>>>> [testng] ===============================================
>>>>>>> [testng]
>>>>>>> [testng] PASSED: createListener
>>>>>>> [testng] PASSED: createPool
>>>>>>> [testng] PASSED: createJoe
>>>>>>> [testng] PASSED: ensureCreatedJoeExists
>>>>>>> [testng] PASSED: deleteJoe
>>>>>>> [testng] PASSED: ensureCreatedListenerExists
>>>>>>> [testng] PASSED: deleteListener
>>>>>>> [testng] PASSED: ensureDeletedListenerDoesNotExist
>>>>>>> [testng] PASSED: createListenerWithOldParam
>>>>>>> [testng] PASSED: deleteListener2
>>>>>>> [testng] PASSED: pingPool
>>>>>>> [testng] PASSED: deletedJoeDoesNotExist
>>>>>>> [testng] PASSED: ensureCreatedPoolExists
>>>>>>> [testng] PASSED: deletePool
>>>>>>> [testng] PASSED: deletedPoolDoesNotExist
>>>>>>> [testng]
>>>>>>> [testng] ===============================================
>>>>>>> [testng] asadmin_tests
>>>>>>> [testng] Tests run: 15, Failures: 0, Skips: 0
>>>>>>> [testng] ===============================================
>>>>>>> [testng]
>>>>>>> [testng]
>>>>>>> [testng] ===============================================
>>>>>>> [testng] QuickLookTests
>>>>>>> [testng] Total tests run: 89, Failures: 2, Skips: 0
>>>>>>> [testng] ===============================================
>>>>>>> [testng]
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>>>>> <mailto:dev-unsubscribe_at_glassfish.dev.java.net>
>>>>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>>>>> <mailto:dev-help_at_glassfish.dev.java.net>
>>>>>>
>>>>>> Lloyd Chambers
>>>>>> lloyd.chambers_at_sun.com <mailto:lloyd.chambers_at_sun.com>
>>>>>> GlassFish Team
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> Lloyd Chambers
>>>> lloyd.chambers_at_sun.com <mailto:lloyd.chambers_at_sun.com>
>>>> GlassFish Team
>>>>
>>>>
>>>>
>>>
>>> Lloyd Chambers
>>> lloyd.chambers_at_sun.com <mailto:lloyd.chambers_at_sun.com>
>>> GlassFish Team
>>>
>>>
>>>
>>
>