Hi Nandaja,
I am also new to Jersey. However, interestingly, I am facing another problem
<
https://java.net/jira/browse/JERSEY-2952> related to the same code (Jersey
1.18).
I don't know if there is any already reported issue or not. However, I am
sharing my thoughts with the hope that it may help you.
You have mentioned that several threads are blocked at
"com.sun.jersey.core.provider.jaxb.AbstractJAXBProvider.getStoredJAXBContext
: 192". The source code (
https://github.com/jersey/jersey-1.x/blob/1.19/jersey-core/src/main/java/com/sun/jersey/core/provider/jaxb/AbstractJAXBProvider.java)
indicates all of them are waiting to enter the synchronized block. It will
be interesting to know what the other thread is doing which is actually
holding the lock.
In this method, Jersey loads JAXBContexts for each type (during marshalling
and unmarshallings). Each call to JAXBContext.newInstance(type) freshly
loads the class. In our case (we didn't observe any slowness, but observed
that the number of class loaded count is increasing), we observed that the
same class is loaded for each request. So there is a possibility that the
thread with the lock is busy loading the classes. In your application, are
you loading huge number of classes (large request/wadl etc)?
Few suggestions/questions:
1. Could you please take 5 thread dumps in a gap of 10 seconds and check?
Ideally, if there is a "deadlock", dump should clearly state that. If
possible share them with the community.
2. Enable the logging for classes loaded and unloaded or simply monitor the
class loaded count through JConsole. Is it giving any clue?
3. How is the GC? Another possibility is GC is eating CPUs. It's expected
that all the weak references are being frequently collected by the
collectors :
private static final Map<Class<?>, WeakReference<JAXBContext>> jaxbContexts
=
new WeakHashMap<Class<?>, WeakReference<JAXBContext>>();
5. If you reduce/stop the load on your server is it able to recover?
Thanks,
Arnab
On Tue, Sep 15, 2015 at 12:34 AM, Nandaja A <nandaja_at_gmail.com> wrote:
>
> Hi,
>
> We use Jetty web server with Jersey for hosting REST based web services.
> We heavily rely on JAXB / Jackson to process the incoming JSON / XML
> payloads and also to write out JSON / XML response for requests received on
> our web service endpoints. We use Jersey-1.19 version on Jetty container
> 9.2, with Guice 4.0 for dependency injection.
>
> Our application, on heavy loads, cannot seem to sustain well and hangs -
> warranting a restart of the Jetty server to recover from this state. We
> used thread dumps / Yourkit profiling to analyze the state of the server a
> little further to see that we had thread contention on monitor at
> com.sun.jersey.core.provider.jaxb.AbstractJAXBProvider.getStoredJAXBContext.
>
> A thread dump in the state where Jetty was stuck without being able ot
> recover yielded several threads blocked on this object monitor :
>
> "qtp1747352992-2233" #2233 prio=5 os_prio=0 tid=0x0000000008b39000
> nid=0x3d07 waiting for monitor entry [0x00007f6647c32000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> com.sun.jersey.core.provider.jaxb.AbstractJAXBProvider.getStoredJAXBContext(AbstractJAXBProvider.java:192)
> - waiting to lock <0x00000005c0f683d0> (a java.util.WeakHashMap)
> at
> com.sun.jersey.core.provider.jaxb.AbstractJAXBProvider.getJAXBContext(AbstractJAXBProvider.java:188)
> at
> com.sun.jersey.core.provider.jaxb.AbstractJAXBProvider.getUnmarshaller(AbstractJAXBProvider.java:140)
> at
> com.sun.jersey.core.provider.jaxb.AbstractJAXBProvider.getUnmarshaller(AbstractJAXBProvider.java:123)
> at
> com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.readFrom(AbstractRootElementProvider.java:111)
>
>
> Analyzing through Yourkit profiling on the application also revealed
> several threads in state below :
>
> Frozen threads found (potential deadlock)
>
> It seems that the following threads have not changed their stack for more
> than 10 seconds.
> These threads are possibly (but not necessarily!) in a deadlock or hung.
>
> qtp1072601481-2254 <--- Frozen for at least 26 sec
> com.sun.jersey.core.impl.provider.xml.ThreadLocalSingletonContextProvider$1.initialValue()
> ThreadLocalSingletonContextProvider.java:64
> java.lang.ThreadLocal.setInitialValue() ThreadLocal.java:180
> java.lang.ThreadLocal.get() ThreadLocal.java:170
> com.sun.jersey.core.impl.provider.xml.ThreadLocalSingletonContextProvider$2.getValue()
> ThreadLocalSingletonContextProvider.java:77
> com.sun.jersey.core.impl.provider.entity.XMLRootElementProvider.readFrom(Class,
> MediaType, Unmarshaller, InputStream) XMLRootElementProvider.java:113
> com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.readFrom(Class,
> Type, Annotation[], MediaType, MultivaluedMap, InputStream)
> AbstractRootElementProvider.java:111
> com.sun.jersey.spi.container.ContainerRequest.getEntity(Class, Type,
> Annotation[]) ContainerRequest.java:490
> com.sun.jersey.server.impl.model.method.dispatch.EntityParamDispatchProvider$EntityInjectable.getValue(HttpContext)
> EntityParamDispatchProvider.java:123
> com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(HttpContext)
> InjectableValuesProvider.java:86
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(HttpContext)
> AbstractResourceMethodDispatchProvider.java:153
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(Object,
> HttpContext) AbstractResourceMethodDispatchProvider.java:203
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(Object,
> HttpContext) ResourceJavaMethodDispatcher.java:75
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(CharSequence,
> Object, UriRuleContext) HttpMethodRule.java:302
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(CharSequence,
> Object, UriRuleContext) RightHandPathRule.java:147
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(CharSequence,
> Object, UriRuleContext) ResourceClassRule.java:108
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(CharSequence,
> Object, UriRuleContext) RightHandPathRule.java:147
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(CharSequence,
> Object, UriRuleContext) RootResourceClassesRule.java:84
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationContext,
> ContainerRequest) WebApplicationImpl.java:1542
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationContext,
> ContainerRequest, ContainerResponse) WebApplicationImpl.java:1473
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(ContainerRequest,
> ContainerResponse) WebApplicationImpl.java:1419
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(ContainerRequest,
> ContainerResponseWriter) WebApplicationImpl.java:1409
> com.sun.jersey.spi.container.servlet.WebComponent.service(URI, URI,
> HttpServletRequest, HttpServletResponse) WebComponent.java:409
> com.sun.jersey.spi.container.servlet.ServletContainer.service(URI, URI,
> HttpServletRequest, HttpServletResponse) ServletContainer.java:558
> com.sun.jersey.spi.container.servlet.ServletContainer.service(HttpServletRequest,
> HttpServletResponse) ServletContainer.java:733
> javax.servlet.http.HttpServlet.service(ServletRequest, ServletResponse)
> HttpServlet.java:790
> com.google.inject.servlet.ServletDefinition.doServiceImpl(HttpServletRequest,
> HttpServletResponse) ServletDefinition.java:287
> com.google.inject.servlet.ServletDefinition.doService(ServletRequest,
> ServletResponse) ServletDefinition.java:277
> com.google.inject.servlet.ServletDefinition.service(ServletRequest,
> ServletResponse) ServletDefinition.java:182
> com.google.inject.servlet.ManagedServletPipeline.service(ServletRequest,
> ServletResponse) ManagedServletPipeline.java:91
>
>
> Is this a known issue ? Any pointers on how to alleviate this are
> appreciated.
>
> Thanks,
> Nandaja
>