Jane,
It's not unstable really, it just reflects reality of a threaded system.
AMX is a framework; some problems are due to MBeans written in various
modules; these are being addressed by Jennifer and Ken.
I will be committing shortly to address certain issues.
Bug fixes from Jennifer and Ken will address some other issues that
show up.
Lloyd
On Oct 8, 2009, at 9:58 AM, Jane Young wrote:
> Lloyd,
>
> We really need to do something about the amx_tests failure in QL.
> It's showing up randomly on Hudson.
>
> See (ignore the adminconsole_tests failures; Ken has committed the
> fix):
> http://hudson.glassfish.org/job/gf-trunk-build-continuous/2685/artifact/bundles/QL-WP-report.html
> http://hudson.glassfish.org/job/gf-trunk-build-continuous/2684/artifact/bundles/QL-WP-report.html
> http://hudson.glassfish.org/job/gf-trunk-build-continuous/2680/artifact/bundles/QL-WP-report.html
> http://hudson.glassfish.org/job/gf-trunk-build-continuous/2678/artifact/bundles/QL-WP-report.html
> http://hudson.glassfish.org/job/gf-trunk-build-continuous/2677/artifact/bundles/QL-WP-report.html
> http://gf-hudson.sfbay/hudson/job/gf-trunk-build-continuous/2687/artifact/bundles/QL-WP-report.html
>
> These are jobs from 2:30am-9:30am today.
>
> Can you please take a look? Is this because of new AMX and
> Monitoring changes or QL tests? As of last Thursday (Oct 1st) QL
> was stable.
>
> Thanks,
> Jane
>
>
> Lloyd Chambers wrote:
>>
>> I’ve found at least one problem--
>>
>> The QL AMX tests create and delete various config MBeans for test
>> purposes.
>>
>> The AMXValidator is in the middle of validating them when they are
>> unregistered.
>>
>> I am fixing the AMXValidator to recognize that the test failed due
>> to an MBean disappearing.
>>
>> That might not be the only issue, but I will fix it.
>>
>> Lloyd
>>
>>
>> On Oct 6, 2009, at 11:45 AM, Lloyd Chambers wrote:
>>
>>> Jane,
>>>
>>> At least one set of failures is by MBeans disappearing while being
>>> tested — a race condition.
>>>
>>> Who/what is registering and then unregistering in rapid succession
>>> is unclear, I suspect the monitoring code, which used to have such
>>> a problem (register/unregister/re-register in a few milliseconds).
>>>
>>> I am making that test to be more defensive.
>>>
>>> Lloyd
>>>
>>> On Oct 6, 2009, at 11:33 AM, Jane Young wrote:
>>>
>>>> Hi Lloyd,
>>>>
>>>> I agree that the amx-tests failure is detecting valid bugs
>>>> introduced to v3 workspace. However, one thing I don't
>>>> understand. Why is it that when I run QL locally it passes in
>>>> the first run and on Hudon it shows failures and occasionally it
>>>> passes. If it is a failure, it should always fail.
>>>>
>>>> I still don't understand why 2nd QL run always fail in amx-tests.
>>>>
>>>> Jane
>>>>
>>>>
>>>> Lloyd Chambers wrote:
>>>>>
>>>>> My comments on these failures below, inline.
>>>>>
>>>>> —Lloyd
>>>>>
>>>>> On Oct 4, 2009, at 6:19 PM, Ming Zhang wrote:
>>>>>
>>>>>> Sreenivas Munnangi wrote:
>>>>>>>
>>>>>>> Ming Zhang wrote:
>>>>>>>> Jane,
>>>>>>>>
>>>>>>>> I have checked in one more test to delete http-listener
>>>>>>>> "ls123452". Now admin tests can be run repeatedly without
>>>>>>>> errors and the domain.xml gets cleaned up after each admin
>>>>>>>> test run.
>>>>>>>>
>>>>>>>> Lloyd,
>>>>>>>> the AMX tests still failed in the 2nd QL run
>>>>>>> How many tests failed, if possible could you copy and paste
>>>>>>> the failures from ql output.
>>>>>>>> (iterateAllSanityCheck, testAMXComplianceMonitorFailureCount)
>>>>>>>> after my change for admin tests. Is this due to the
>>>>>>>> configuration changes in domain.xml after admin GUI loaded?
>>>>>>>> Is it reasonable to expect AMX tests will accommodate the
>>>>>>>> configurations of admin GUI in domain.xml?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Ming
>>>>>> The runtestng.output is attached:
>>>>>>
>>>>>> [echo] =============Starting TestNG functional tests from
>>>>>> testng.xml ============
>>>>>> [testng] [Parser] Running:
>>>>>> [testng] /export/home/ming/v3/quicklook/testng.xml
>>>>>> [testng]
>>>>>> [testng] PASSED: createJDBCPool
>>>>>> [testng] PASSED: pingJDBCPool
>>>>>> [testng] PASSED: deleteJDBCPool
>>>>>> [testng]
>>>>>> [testng] ===============================================
>>>>>> [testng] admincli_tests
>>>>>> [testng] Tests run: 3, Failures: 0, Skips: 0
>>>>>> [testng] ===============================================
>>>>>> [testng]
>>>>>> [testng] Monitoring at HIGH for ThreadPool
>>>>>> [testng] Monitoring at HIGH for JdbcConnectionPool
>>>>>> [testng] Monitoring at HIGH for Orb
>>>>>> [testng] Monitoring at HIGH for ConnectorService
>>>>>> [testng] Monitoring at HIGH for Security
>>>>>> [testng] Monitoring at HIGH for JmsService
>>>>>> [testng] Monitoring at HIGH for WebServicesContainer
>>>>>> [testng] Monitoring at HIGH for Jvm
>>>>>> [testng] Monitoring at HIGH for Jpa
>>>>>> [testng] Monitoring at HIGH for Jersey
>>>>>> [testng] Monitoring at HIGH for ConnectorConnectionPool
>>>>>> [testng] Monitoring at HIGH for EjbContainer
>>>>>> [testng] Monitoring at HIGH for WebContainer
>>>>>> [testng] Monitoring at HIGH for TransactionService
>>>>>> [testng] Monitoring at HIGH for HttpService
>>>>>> [testng]
>>>>>> [testng] PROBLEMS:
>>>>>> [testng] Error invoking getParent() on amx:pp=/mon/server-mon
>>>>>> [server],type=jdbc-connection-pool-mon,name=resources/
>>>>>> __TimerPool = java.lang.reflect.InvocationTargetException
>>>>>
>>>>> This root cause is needed here, but if getParent() is failing
>>>>> because there is no parent, then this is a critical bug in this
>>>>> MBean, supplied by the monitoring infrastructure.
>>>>>
>>>>>> [testng] CHANGE COUNT: 10
>>>>>> [testng] PASSED: bootAMX
>>>>>> [testng] Core AMX tests
>>>>>> [testng] PASSED: testAmxPref
>>>>>> [testng] PASSED: connectorConnectionPoolTest
>>>>>> [testng] PASSED: createChildTest
>>>>>> [testng] PASSED: testConfigTools
>>>>>> [testng] PASSED: createProfilerTest
>>>>>> [testng] PASSED: testAMXConfigAttributeResolver
>>>>>> [testng] PASSED: testAMXConfigDefaultValues
>>>>>> [testng] PASSED: testForBogusConfigAnnotations
>>>>>> [testng] PASSED: testCreateProperties
>>>>>> [testng] PASSED: testCreateResource
>>>>>> [testng] PASSED: testCreateProperty
>>>>>> [testng] PASSED: testConnectorRuntimeAPIProvider
>>>>>> [testng] PASSED: testMonitoringRoot
>>>>>> [testng] PASSED: testDomainConfig
>>>>>> [testng] PASSED: testBulkAccess
>>>>>> [testng] PASSED: testExt
>>>>>> [testng] PASSED: testPropertyParent
>>>>>> [testng] PASSED: testTools
>>>>>> [testng] PASSED: testPathnames
>>>>>> [testng] PASSED: testSystemStatus
>>>>>> [testng] PASSED: testApplications
>>>>>> [testng] PASSED: testResources
>>>>>> [testng] PASSED: testChildGetterVariants
>>>>>> [testng] PASSED: testAllGenerically
>>>>>> [testng] PASSED: testRuntimeRoot
>>>>>> [testng] PASSED: testQuery
>>>>>> [testng] PASSED: testJ2EEDomain
>>>>>> [testng] PASSED: testSingletonOrNot
>>>>>> [testng] PASSED: testDomainRoot
>>>>>> [testng] PASSED: testSystemApplications
>>>>>> [testng] PASSED: testSystemInfo
>>>>>> [testng] PASSED: testSystemPropertyParent
>>>>>> [testng] PASSED: testConfigs
>>>>>> [testng] PASSED: testServerRuntime
>>>>>> [testng] PASSED: testLogging
>>>>>> [testng] PASSED: testServers
>>>>>> [testng] PASSED: testAutoConvert
>>>>>> [testng] PASSED: testForBogusAnnotations
>>>>>> [testng] FAILED: iterateAllSanityCheck
>>>>>
>>>>> This should be suppressed as of my Friday checkin. When was
>>>>> this run?
>>>>>
>>>>>> [testng] java.lang.RuntimeException:
>>>>>> javax.management.InstanceNotFoundException: amx:pp=/mon/server-
>>>>>> mon[server],type=bean-method-mon,name=slsbnicmt/AnnotatedEJB/
>>>>>> bean-methods/persistEntity
>>>>>> [testng] at
>>>>>> org.glassfish.admin.amx.core.proxy.ProxyFactory.getMBeanInfo
>>>>>> (ProxyFactory.java:447)
>>>>>> [testng] at
>>>>>> org.glassfish.admin.amx.core.proxy.ProxyFactory.getProxy
>>>>>> (ProxyFactory.java:482)
>>>>>> [testng] at
>>>>>> org.glassfish.admin.amx.core.proxy.ProxyFactory.toProxySet
>>>>>> (ProxyFactory.java:627)
>>>>>> [testng] at
>>>>>> org.glassfish.admin.amx.core.proxy.AMXProxyHandler.autoConvert
>>>>>> (AMXProxyHandler.java:653)
>>>>>> [testng] at
>>>>>> org.glassfish.admin.amx.core.proxy.AMXProxyHandler._invoke
>>>>>> (AMXProxyHandler.java:740)
>>>>>> [testng] at
>>>>>> org.glassfish.admin.amx.core.proxy.AMXProxyHandler.invoke
>>>>>> (AMXProxyHandler.java:518)
>>>>>> [testng] at $Proxy19.queryAll(Unknown Source)
>>>>>> [testng] at amxtest.AMXTestBase.getAllAMX(AMXTestBase.java:
>>>>>> 295)
>>>>>> [testng] at amxtest.AMXCoreTests.iterateAllSanityCheck
>>>>>> (AMXCoreTests.java:86)
>>>>>> [testng] Caused by:
>>>>>> javax.management.InstanceNotFoundException: amx:pp=/mon/server-
>>>>>> mon[server],type=bean-method-mon,name=slsbnicmt/AnnotatedEJB/
>>>>>> bean-methods/persistEntity
>>>>>> [testng] at
>>>>>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean
>>>>>> (DefaultMBeanServerInterceptor.java:1094)
>>>>>> [testng] at
>>>>>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBeanInfo
>>>>>> (DefaultMBeanServerInterceptor.java:1372)
>>>>>> [testng] at
>>>>>> com.sun.jmx.mbeanserver.JmxMBeanServer.getMBeanInfo
>>>>>> (JmxMBeanServer.java:880)
>>>>>> [testng] at
>>>>>> javax.management.remote.rmi.RMIConnectionImpl.doOperation
>>>>>> (RMIConnectionImpl.java:1420)
>>>>>> [testng] at
>>>>>> javax.management.remote.rmi.RMIConnectionImpl.access$200
>>>>>> (RMIConnectionImpl.java:72)
>>>>>> [testng] at javax.management.remote.rmi.RMIConnectionImpl
>>>>>> $PrivilegedOperation.run(RMIConnectionImpl.java:1264)
>>>>>> [testng] at
>>>>>> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation
>>>>>> (RMIConnectionImpl.java:1359)
>>>>>> [testng] at
>>>>>> javax.management.remote.rmi.RMIConnectionImpl.getMBeanInfo
>>>>>> (RMIConnectionImpl.java:864)
>>>>>> [testng] at sun.rmi.server.UnicastServerRef.dispatch
>>>>>> (UnicastServerRef.java:305)
>>>>>> [testng] at sun.rmi.transport.Transport$1.run(Transport.java:
>>>>>> 159)
>>>>>> [testng] at java.security.AccessController.doPrivileged
>>>>>> (Native Method)
>>>>>> [testng] at sun.rmi.transport.Transport.serviceCall
>>>>>> (Transport.java:155)
>>>>>> [testng] at sun.rmi.transport.tcp.TCPTransport.handleMessages
>>>>>> (TCPTransport.java:535)
>>>>>> [testng] at sun.rmi.transport.tcp.TCPTransport
>>>>>> $ConnectionHandler.run0(TCPTransport.java:790)
>>>>>> [testng] at sun.rmi.transport.tcp.TCPTransport
>>>>>> $ConnectionHandler.run(TCPTransport.java:649)
>>>>>> [testng] at java.util.concurrent.ThreadPoolExecutor
>>>>>> $Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>> [testng] at java.util.concurrent.ThreadPoolExecutor
>>>>>> $Worker.run(ThreadPoolExecutor.java:908)
>>>>>> [testng] at java.lang.Thread.run(Thread.java:619)
>>>>>> [testng] at
>>>>>> sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer
>>>>>> (StreamRemoteCall.java:255)
>>>>>> [testng] at sun.rmi.transport.StreamRemoteCall.executeCall
>>>>>> (StreamRemoteCall.java:233)
>>>>>> [testng] at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:
>>>>>> 142)
>>>>>> [testng] at com.sun.jmx.remote.internal.PRef.invoke(Unknown
>>>>>> Source)
>>>>>> [testng] at
>>>>>> javax.management.remote.rmi.RMIConnectionImpl_Stub.getMBeanInfo
>>>>>> (Unknown Source)
>>>>>> [testng] at javax.management.remote.rmi.RMIConnector
>>>>>> $RemoteMBeanServerConnection.getMBeanInfo(RMIConnector.java:1037)
>>>>>> [testng] at
>>>>>> org.glassfish.admin.amx.core.proxy.ProxyFactory.getMBeanInfo
>>>>>> (ProxyFactory.java:437)
>>>>>> [testng] ... 30 more
>>>>>> [testng] ... Removed 25 stack frames
>>>>>> [testng] FAILED: testAMXComplianceMonitorFailureCount
>>>>>> [testng] java.lang.AssertionError: Server indicates that
>>>>>> there are non-compliant AMX MBean validator failures, failure
>>>>>> count = 1, examine the server log for failures
>>>>>> [testng] at
>>>>>> amxtest.AMXCoreTests.testAMXComplianceMonitorFailureCount
>>>>>> (AMXCoreTests.java:120)
>>>>>> [testng] ... Removed 22 stack frames
>>>>>
>>>>> This indicates non-compliant MBeans. I will see if I can make
>>>>> the message spell out which MBeans and what failed.
>>>>>
>>>>>> [testng]
>>>>>> [testng] ===============================================
>>>>>> [testng] amx_tests
>>>>>> [testng] Tests run: 41, Failures: 2, Skips: 0
>>>>>> [testng] ===============================================
>>>>>> [testng]
>>>>>> [testng] ***** Login page not found. Sleeping to allow app
>>>>>> to deploy (1 of 60)...
>>>>>> [testng] ***** Login page not found. Sleeping to allow app
>>>>>> to deploy (2 of 60)...
>>>>>> [testng] ***** Login page not found. Sleeping to allow app
>>>>>> to deploy (3 of 60)...
>>>>>> [testng] ***** Login page not found. Sleeping to allow app
>>>>>> to deploy (4 of 60)...
>>>>>> [testng] ***** Login page not found. Sleeping to allow app
>>>>>> to deploy (5 of 60)...
>>>>>> [testng] Oct 4, 2009 5:51:08 PM
>>>>>> org.apache.commons.httpclient.HttpMethodDirector isRedirectNeeded
>>>>>> [testng] INFO: Redirect requested but followRedirects is
>>>>>> disabled
>>>>>> [testng] PASSED: testDeployedAppPage
>>>>>> [testng] PASSED: testCommonTasks
>>>>>> [testng] PASSED: testHelpPage
>>>>>> [testng] PASSED: testRealmsList
>>>>>> [testng]
>>>>>> [testng] ===============================================
>>>>>> [testng] adminconsole_tests
>>>>>> [testng] Tests run: 4, Failures: 0, Skips: 0
>>>>>> [testng] ===============================================
>>>>>> [testng]
>>>>>> [testng] PASSED: staticHTMLPageTest
>>>>>> [testng] PASSED: testServerRunning
>>>>>> [testng] PASSED: simpleServletTest
>>>>>> [testng] PASSED: simpleJSPTestPage
>>>>>> [testng]
>>>>>> [testng] ===============================================
>>>>>> [testng] web_jspservlet_war
>>>>>> [testng] Tests run: 4, Failures: 0, Skips: 0
>>>>>> [testng] ===============================================
>>>>>> [testng]
>>>>>> [testng] PASSED: jsfIndexPageBasicTest
>>>>>> [testng] PASSED: jsfAppDeployedFirstPagetest
>>>>>> [testng]
>>>>>> [testng] ===============================================
>>>>>> [testng] jsf_hello_world
>>>>>> [testng] Tests run: 2, Failures: 0, Skips: 0
>>>>>> [testng] ===============================================
>>>>>> [testng]
>>>>>> [testng] Injection2 matched.
>>>>>> [testng] PASSED: injectedValuesTestPage
>>>>>> [testng]
>>>>>> [testng] ===============================================
>>>>>> [testng] jsf_injection
>>>>>> [testng] Tests run: 1, Failures: 0, Skips: 0
>>>>>> [testng] ===============================================
>>>>>> [testng]
>>>>>> [testng] PASSED: testNoLeak
>>>>>> [testng] PASSED: testUserTx
>>>>>> [testng]
>>>>>> [testng] ===============================================
>>>>>> [testng] jdbc_user_tx
>>>>>> [testng] Tests run: 2, Failures: 0, Skips: 0
>>>>>> [testng] ===============================================
>>>>>> [testng]
>>>>>> [testng] PASSED: testJpaPersistWithInjectEMF
>>>>>> [testng] PASSED: testJpaLazyLoadingByQuery
>>>>>> [testng]
>>>>>> [testng] ===============================================
>>>>>> [testng] jpa_inject_emf
>>>>>> [testng] Tests run: 2, Failures: 0, Skips: 0
>>>>>> [testng] ===============================================
>>>>>> [testng]
>>>>>> [testng] PASSED: staticHTMLPageTest
>>>>>> [testng] PASSED: simpleServletTest
>>>>>> [testng] PASSED: simpleJSPTestPage
>>>>>> [testng]
>>>>>> [testng] ===============================================
>>>>>> [testng] security_hello_jsp
>>>>>> [testng] Tests run: 3, Failures: 0, Skips: 0
>>>>>> [testng] ===============================================
>>>>>> [testng]
>>>>>> [testng] PASSED: testAuthGroupMappedUser
>>>>>> [testng] PASSED: testAuthNotValidPassword
>>>>>> [testng] PASSED: testAuthRoleMappedUser
>>>>>> [testng] PASSED: testAuthNotAuthorizedUser
>>>>>> [testng]
>>>>>> [testng] ===============================================
>>>>>> [testng] security_hello_basicauth
>>>>>> [testng] Tests run: 4, Failures: 0, Skips: 0
>>>>>> [testng] ===============================================
>>>>>> [testng]
>>>>>> [testng] PASSED: SetName
>>>>>> [testng] PASSED: EJBInject
>>>>>> [testng] PASSED: GetName
>>>>>> [testng] PASSED: JpaPersist
>>>>>> [testng] PASSED: JpaRemove
>>>>>> [testng] PASSED: JpaVerify
>>>>>> [testng]
>>>>>> [testng] ===============================================
>>>>>> [testng] ejb_stateful_cmt_ni_war
>>>>>> [testng] Tests run: 6, Failures: 0, Skips: 0
>>>>>> [testng] ===============================================
>>>>>> [testng]
>>>>>> [testng] PASSED: executeServlet
>>>>>> [testng]
>>>>>> [testng] ===============================================
>>>>>> [testng] bv_servlet_simple
>>>>>> [testng] Tests run: 1, Failures: 0, Skips: 0
>>>>>> [testng] ===============================================
>>>>>> [testng]
>>>>>> [testng] PASSED: appDeployedFirstPagetest
>>>>>> [testng]
>>>>>> [testng] ===============================================
>>>>>> [testng] webbeans_numberguess
>>>>>> [testng] Tests run: 1, Failures: 0, Skips: 0
>>>>>> [testng] ===============================================
>>>>>> [testng]
>>>>>> [testng] PASSED: createListener
>>>>>> [testng] PASSED: createPool
>>>>>> [testng] PASSED: createJoe
>>>>>> [testng] PASSED: ensureCreatedJoeExists
>>>>>> [testng] PASSED: deleteJoe
>>>>>> [testng] PASSED: ensureCreatedListenerExists
>>>>>> [testng] PASSED: deleteListener
>>>>>> [testng] PASSED: ensureDeletedListenerDoesNotExist
>>>>>> [testng] PASSED: createListenerWithOldParam
>>>>>> [testng] PASSED: deleteListener2
>>>>>> [testng] PASSED: pingPool
>>>>>> [testng] PASSED: deletedJoeDoesNotExist
>>>>>> [testng] PASSED: ensureCreatedPoolExists
>>>>>> [testng] PASSED: deletePool
>>>>>> [testng] PASSED: deletedPoolDoesNotExist
>>>>>> [testng]
>>>>>> [testng] ===============================================
>>>>>> [testng] asadmin_tests
>>>>>> [testng] Tests run: 15, Failures: 0, Skips: 0
>>>>>> [testng] ===============================================
>>>>>> [testng]
>>>>>> [testng]
>>>>>> [testng] ===============================================
>>>>>> [testng] QuickLookTests
>>>>>> [testng] Total tests run: 89, Failures: 2, Skips: 0
>>>>>> [testng] ===============================================
>>>>>> [testng]
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>>>
>>>>> Lloyd Chambers
>>>>> lloyd.chambers_at_sun.com
>>>>> GlassFish Team
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>> Lloyd Chambers
>>> lloyd.chambers_at_sun.com
>>> GlassFish Team
>>>
>>>
>>>
>>
>> Lloyd Chambers
>> lloyd.chambers_at_sun.com
>> GlassFish Team
>>
>>
>>
>
Lloyd Chambers
lloyd.chambers_at_sun.com
GlassFish Team