Hi,
We use Jetty web server with Jersey for hosting REST based web services. We
heavily rely on JAXB / Jackson to process the incoming JSON / XML payloads
and also to write out JSON / XML response for requests received on our web
service endpoints. We use Jersey-1.19 version on Jetty container 9.2, with
Guice 4.0 for dependency injection.
Our application, on heavy loads, cannot seem to sustain well and hangs -
warranting a restart of the Jetty server to recover from this state. We
used thread dumps / Yourkit profiling to analyze the state of the server a
little further to see that we had thread contention on monitor at
com.sun.jersey.core.provider.jaxb.AbstractJAXBProvider.getStoredJAXBContext.
A thread dump in the state where Jetty was stuck without being able ot
recover yielded several threads blocked on this object monitor :
"qtp1747352992-2233" #2233 prio=5 os_prio=0 tid=0x0000000008b39000
nid=0x3d07 waiting for monitor entry [0x00007f6647c32000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
com.sun.jersey.core.provider.jaxb.AbstractJAXBProvider.getStoredJAXBContext(AbstractJAXBProvider.java:192)
- waiting to lock <0x00000005c0f683d0> (a java.util.WeakHashMap)
at
com.sun.jersey.core.provider.jaxb.AbstractJAXBProvider.getJAXBContext(AbstractJAXBProvider.java:188)
at
com.sun.jersey.core.provider.jaxb.AbstractJAXBProvider.getUnmarshaller(AbstractJAXBProvider.java:140)
at
com.sun.jersey.core.provider.jaxb.AbstractJAXBProvider.getUnmarshaller(AbstractJAXBProvider.java:123)
at
com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.readFrom(AbstractRootElementProvider.java:111)
Analyzing through Yourkit profiling on the application also revealed
several threads in state below :
Frozen threads found (potential deadlock)
It seems that the following threads have not changed their stack for more
than 10 seconds.
These threads are possibly (but not necessarily!) in a deadlock or hung.
qtp1072601481-2254 <--- Frozen for at least 26 sec
com.sun.jersey.core.impl.provider.xml.ThreadLocalSingletonContextProvider$1.initialValue()
ThreadLocalSingletonContextProvider.java:64
java.lang.ThreadLocal.setInitialValue() ThreadLocal.java:180
java.lang.ThreadLocal.get() ThreadLocal.java:170
com.sun.jersey.core.impl.provider.xml.ThreadLocalSingletonContextProvider$2.getValue()
ThreadLocalSingletonContextProvider.java:77
com.sun.jersey.core.impl.provider.entity.XMLRootElementProvider.readFrom(Class,
MediaType, Unmarshaller, InputStream) XMLRootElementProvider.java:113
com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.readFrom(Class,
Type, Annotation[], MediaType, MultivaluedMap, InputStream)
AbstractRootElementProvider.java:111
com.sun.jersey.spi.container.ContainerRequest.getEntity(Class, Type,
Annotation[]) ContainerRequest.java:490
com.sun.jersey.server.impl.model.method.dispatch.EntityParamDispatchProvider$EntityInjectable.getValue(HttpContext)
EntityParamDispatchProvider.java:123
com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(HttpContext)
InjectableValuesProvider.java:86
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(HttpContext)
AbstractResourceMethodDispatchProvider.java:153
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(Object,
HttpContext) AbstractResourceMethodDispatchProvider.java:203
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(Object,
HttpContext) ResourceJavaMethodDispatcher.java:75
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(CharSequence,
Object, UriRuleContext) HttpMethodRule.java:302
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(CharSequence,
Object, UriRuleContext) RightHandPathRule.java:147
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(CharSequence,
Object, UriRuleContext) ResourceClassRule.java:108
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(CharSequence,
Object, UriRuleContext) RightHandPathRule.java:147
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(CharSequence,
Object, UriRuleContext) RootResourceClassesRule.java:84
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationContext,
ContainerRequest) WebApplicationImpl.java:1542
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationContext,
ContainerRequest, ContainerResponse) WebApplicationImpl.java:1473
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(ContainerRequest,
ContainerResponse) WebApplicationImpl.java:1419
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(ContainerRequest,
ContainerResponseWriter) WebApplicationImpl.java:1409
com.sun.jersey.spi.container.servlet.WebComponent.service(URI, URI,
HttpServletRequest, HttpServletResponse) WebComponent.java:409
com.sun.jersey.spi.container.servlet.ServletContainer.service(URI, URI,
HttpServletRequest, HttpServletResponse) ServletContainer.java:558
com.sun.jersey.spi.container.servlet.ServletContainer.service(HttpServletRequest,
HttpServletResponse) ServletContainer.java:733
javax.servlet.http.HttpServlet.service(ServletRequest, ServletResponse)
HttpServlet.java:790
com.google.inject.servlet.ServletDefinition.doServiceImpl(HttpServletRequest,
HttpServletResponse) ServletDefinition.java:287
com.google.inject.servlet.ServletDefinition.doService(ServletRequest,
ServletResponse) ServletDefinition.java:277
com.google.inject.servlet.ServletDefinition.service(ServletRequest,
ServletResponse) ServletDefinition.java:182
com.google.inject.servlet.ManagedServletPipeline.service(ServletRequest,
ServletResponse) ManagedServletPipeline.java:91
Is this a known issue ? Any pointers on how to alleviate this are
appreciated.
Thanks,
Nandaja