users@jersey.java.net

[Jersey] Re: Jersey 2.11 throws IllegalStateException: ServiceLocatorImpl has been shut down

From: Sobieralski, Damian Michael <dsobiera_at_indiana.edu>
Date: Mon, 20 Apr 2015 16:38:00 +0000

The Jersey client code that was hiccupping was sitting in the business logic tier of a webapp and in turn called a REST api elsewhere in the world.

It turns out that we had a load balancer that was timing out before the jersey client request to the backend service was complete. It resulted in putting the application in an odd state. Increasing the timeout value for our load balancer rectified this problem.



From: Sobieralski, Damian Michael [mailto:dsobiera_at_indiana.edu]
Sent: Monday, March 30, 2015 6:02 PM
To: users_at_jersey.java.net
Subject: [Jersey] Re: Jersey 2.11 throws IllegalStateException: ServiceLocatorImpl has been shut down

I am experiencing this error as well. It is happening in Jersey client (not a unit test it is happening in production). I am making a Jersey REST call to a third party endpoint. It's an odd thing because it only happens when there is a long delay in the results coming back to my Jersey client (large dataset is also being returned). For smaller datasets/quicker response it works fine. This third party endpoint also is notoriously slow. At first I thought it was a timeout issue.

Does anyone have any ideas? I have zero ideas how to start to debug this.

I am running 2.11 of Jersey client (although I tried 2.17 and 2.9 and 2.9.1 w/ the same issue).

30 Mar 2015 16:46:45,504 -- DEBUG [http-bio-192.168.220.188-8443-exec-113] (CourseServiceImpl.java:135) - Exception: A MultiException has 1 exceptions. They are:
1. java.lang.IllegalStateException: ServiceLocatorImpl(__HK2_Generated_4,5,782323610) has been shut down

- Damian


From: cowwoc [mailto:cowwoc_at_bbs.darktech.org]
Sent: Saturday, July 26, 2014 11:15 PM
To: users_at_jersey.java.net<mailto:users_at_jersey.java.net>
Subject: [Jersey] Re: Jersey 2.11 throws IllegalStateException: ServiceLocatorImpl has been shut down

I'm having a hard time reproducing this issue now. It still occurs but it's rare. I don't know whether I changed something or it was always this rare.

In any case, I've noticed that HK2 recently fixed this bug which might be related: https://java.net/jira/browse/HK2-130

So forcing an HK2 update might be a solution:

<dependency>
    <groupId>org.glassfish.hk2</groupId>
    <artifactId>hk2-api</artifactId>
    <version>2.3.0-b10</version>
</dependency>
<dependency>
    <groupId>org.glassfish.hk2</groupId>
    <artifactId>hk2-locator</artifactId>
    <version>2.3.0-b10</version>
</dependency>
<dependency>
    <groupId>org.glassfish.hk2</groupId>
    <artifactId>hk2-utils</artifactId>
    <version>2.3.0-b10</version>
</dependency>

Gili

On 24/07/2014 7:18 PM, Adam Lindenthal wrote:
Hi Gili,

could you please share one of your failing unit tests (some skeleton is enough, of course)? You know, to make reproducing faster/easier...

Thanks,
Adam

On 25.7.2014 00:19, cowwoc wrote:
Hi,

I migrated from Jersey 2.9.1 to 2.11 and now my unit tests fail with the following exception:

17:47:01.058 [qtp990182778-927] WARN org.eclipse.jetty.servlet.ServletHandler.doHandle() -
javax.servlet.ServletException: A MultiException has 1 exceptions. They are:
1. java.lang.IllegalStateException: ServiceLocatorImpl(__HK2_Generated_84,85,1111434133) has been shut down

    at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:391)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:381)
    at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:534)
    at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:482)
    at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:419)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1636)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:564)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:498)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:98)
    at org.eclipse.jetty.server.Server.handle(Server.java:461)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:284)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
    at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.glassfish.hk2.api.MultiException: A MultiException has 1 exceptions. They are:
1. java.lang.IllegalStateException: ServiceLocatorImpl(__HK2_Generated_84,85,1111434133) has been shut down

    at org.jvnet.hk2.internal.FactoryCreator.getFactoryHandle(FactoryCreator.java:80)
    at org.jvnet.hk2.internal.FactoryCreator.dispose(FactoryCreator.java:110)
    at org.jvnet.hk2.internal.SystemDescriptor.dispose(SystemDescriptor.java:481)
    at org.glassfish.jersey.process.internal.RequestScope$Instance.remove(RequestScope.java:512)
    at org.glassfish.jersey.process.internal.RequestScope$Instance.release(RequestScope.java:529)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:299)
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1028)
    at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:372)
    ... 22 common frames omitted
Caused by: java.lang.IllegalStateException: ServiceLocatorImpl(__HK2_Generated_84,85,1111434133) has been shut down
    at org.jvnet.hk2.internal.ServiceLocatorImpl.checkState(ServiceLocatorImpl.java:2182)
    at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetServiceHandle(ServiceLocatorImpl.java:580)
    at org.jvnet.hk2.internal.ServiceLocatorImpl.getServiceHandle(ServiceLocatorImpl.java:573)
    at org.jvnet.hk2.internal.FactoryCreator.getFactoryHandle(FactoryCreator.java:77)
    ... 30 common frames omitted

The stack-trace is so vague I have no idea what action is triggering the problem. Any ideas?

PS: Version 2.10.1 works fine as well so this is a new regression as of 2.11.

Gili