Hi All,
We are facing a issue with GenericJMSRA.
Background:
We have a J2EE component (EAR) deployed on Sun appserver 8.1
The component is basically a MDB that listens to a queue hosted by
ORACLE AQ .
The component consumes the messages off the queue(MIC_OUTBOUND_QUEUE)
and then does a http to another component.
The connector module to consume messages from ORACLE AQ is GenericJMSRA
deployed on Sun appserver 8.1.
The end to end flow works perfectly fine.
However , there seems to be a problem when we are running a non
functional capacity test to find out the throughput of this
component(The name of the component is MIC Outbound).
Test Description:
The flow is : MIC_OUTBOUND_QUEUE (OracleAQ on DB) ------>[J2EE listener]
MIC_OUBOUND_COMPONENT [http to Stub]----------> Stub returning 202
To run this test approx 60,000 messages are piled up on the
MIC_OUTBOUND_QUEUE .
Then the MIC_OUBOUND_COMPONENT (listener) is enabled on the appserver so
that the messages get dequeued from MIC_OUTBOUND_QUEUE.
The throughput is then calculated based on time taken to consume 60,000
messages off the queue.
It is observed that while the test is running, the database CPU reaches
100% and all 60,000 messages get consumed successfully.
So far so good.
The problem actually begins when the test has finished execution.
Problem Summary:
The problem starts after the execution of the test is over.
Though there is not test running on the appserver and database, the
database CPU continuously fluctuates between 60% to 100%.
There are only three ways of getting the database CPU back to normal ie:
0%.
1)The CPU on database becomes 0% after the SUN appserver 8.1 instance
is stopped.
However it again goes up and fluctuates betweeb 60% and 100% after the
appserver instance is started.
2)The CPU on database goes down after MIC_OUTBOUND_COMPONENT on Sun
appserver 8.1 is disabled .
Sometimes it is observed that though the component is disabled manually
from the console, the appserver though shows it in disabled state it
does not disable it internally and hence the CPU on the database still
fluctuates between 60% and 100%.
Then we have to go for the 3rd option.
3)Undeploy the MIC_OUTBOUND component from the SUN appserver 8.1.
Sometimes it is observed that though the component gets undeployed from
the console it does not get undeployed internally and component fliles
exist under nodeagent/serverinstance/application/j2ee-apps
and the database CPU still fluctuates between 60% and 100%.
Then the files under nodeagent/serverinstance/application/j2ee-apps
need to be manually deleted and then the database CPU goes to 0%.
The obvious question is what are the processes that run on the Database
server and are responsible for 60 to 100% of CPU utilization on the
database when actually it is supposed to remain idle as there are no
messages on the queue?
The below process continuously runs and multiplies in n umbers on the
database and hogs the database CPU :
begin dbms_aqin.aq$_dequeue_in( :1, :2, :3, :4, :5, :6, :7, :8, :9, :10,
:11, :12, :13, :14, :15, :16, :17, :18, :19, :20, :21, :22, :23, :24,
:25, :26, :27, :28, :29); end;
There are 65 connections from Appserver to database which are
continuously in Established state.
As we have a MDB (MIC_OUTBOUND_COMPONENT ) running continuously
listening to the queue , the Established state of the connections is
justified.
However, the problem is that genericJMSRA is triggerring dequeue process
on the DB though there are no messages on the queue.
These processes multiply in numbers and CPU on the database goes up
though there is no test running and there are no messages on the queue.
The probable cause for this problem is the way in which genericJMSRA
works.
GenericJMSRA is suppose to trigger a dequeue process on the Oracle AQ
only when there are messages on the queue.
After we are through with running the MIC outbound test and dequeueing
approx. 60,000 messages off the Oracle AQ, genericJMSRA is not
stabalizing .
It tends to still trigger the dequeue process on the database when it is
not required.
Note:
The same Sun Appserver 8.1 after the execution of the test when pointed
to a different database server by changing the IP address in the OID,
it seems to start the begin dbms_aqin.aq$_dequeue_in( :1, :2, :3, :4, :
process on the different database and gets the CPU of the database to
fluctuate between 60 to 100%.
This suggests that the issue is with GenericJMSRA and not OracleAQ
/Database server.
Please let me know if you need any furtehr information.
Regards,
Ashish