users@jersey.java.net

[Jersey] Re: Jersey2 Client OutOfMemoryError when uploading large file?

From: Piers Powlesland <piers_at_aptusinteractive.com>
Date: Fri, 9 Aug 2013 15:51:47 +0100

Hi Mira

Thanks for getting back to me on this one, I was going a bit crazy trying
to get it to work. In my search for solutions I came across the following
stack overflow entry a part of which is quoted below -
http://stackoverflow.com/questions/11176824/preventing-the-jersey-client-from-causing-an-outofmemory-error-when-posting-larg

"Unfortunately, this does not seem to work reliably - that's why we have
chunked encoding commented out by default in the Jersey code. There seems
to be a nasty bug in HttpURLConnection which makes some requests fail in a
weird way and semi-randomly (subset of our tests fail, if we re-run just
the failing tests a smaller subset of them fails, if we re-run just that
subset, even a smaller subset fails and eventually all pass). So, I'd
recommend using a more robust client connector for any serious work. Martin
Matula <http://stackoverflow.com/users/950199/martin-matula> Jun 24 '12 at
18:00<http://stackoverflow.com/questions/11176824/preventing-the-jersey-client-from-causing-an-outofmemory-error-when-posting-larg#comment14669192_11176825>
"


I was wondering if the problems highlighted by Martin have been resolved
and if there were any way to use a different client implementation for
Jersey 2 such as the jersey-apache-http-client which was available for
Jersey 1?

Thanks

Piers


On 8 August 2013 17:23, Piers Powlesland <piers_at_aptusinteractive.com> wrote:

> Hi I am trying to create a simple file server and in the interest of
> making it scalable I decided that data should be streamed through it. As
> such I decided to use "application/octet-stream" for the upload and
> download media type.
>
> The issue I am facing is that I run out of heap space when uploading large
> files (downloading seems to work fine), I've tested with 700 MB and it
> failed. I used jconsole to monitor server and client and saw that for a
> 358.4 MB file the client peak memory usage is 772.2 MB so a bit over double
> the file size. I used wireshark to see what was being sent and saw that the
> client was sending the content length correctly and therefore must be
> buffering the stream at least once. I would like to disable this behaviour
> as I am able to set the "content-length" header myself, I was hoping that
> setting the header my prevent this behaviour but I tried it and it does
> not. Does anyone have an idea about how to prevent the client from
> buffering any large amounts of data?
>
> Thanks
>
> Piers
>
> Below is the client code
>
> Client client = ClientBuilder.newClient();
> WebTarget storageServiceContextRootWebTarget = client.target("
> http://localhost:8080/storage-service-test-war");
> File bigFile = new File("bigfile");
> InputStream bigFileContentInputStream = new FileInputStream(bigFile);
> Response response =
> storageServiceContextRootWebTarget.path(FILE_ONLY_PATH).request().put(Entity.entity(bigFileContentInputStream,
> MediaType.APPLICATION_OCTET_STREAM), Response.class);
>
> And server side code
>
> @PUT
> @Path("{" + FILE_PATH_PATH_PARAMETER + ":.+}")
> @Consumes(MediaType.APPLICATION_OCTET_STREAM)
> public Response putFileToPath(@PathParam(FILE_PATH_PATH_PARAMETER) String
> filePath,_at_HeaderParam("Content-Length") long contentLength, InputStream
> inputStream) throws IOException {
> repository.writeFileToPath(inputStream, contentLength,
> Paths.get(filePath));
> return Response.noContent().build();
> }
>
>
> writeFileToPath() uses Apache's IOUtils.copy() method to copy the data out
> of the stream into a file.
>
>
> javax.ws.rs.ProcessingException: Java heap space
> at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:224)
> at
> org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:650)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
> at
> org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:421)
> at
> org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:646)
> at
> org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:402)
> at
> org.glassfish.jersey.client.JerseyInvocation$Builder.put(JerseyInvocation.java:290)
> at
> com.aptusinteractive.acceptancetests.StorageServiceAcceptanceTest.givenNoFileAtGivenNonHierarchicalPath_whenPuttingBigFileToGivenNonHierarchicalPath_thenFilePutToGivenNonHierarchicalPathAndResponseWithStatusCodeNoContentReturned(StorageServiceAcceptanceTest.java:99)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:69)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:48)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:292)
> at
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:2271)
> at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
> at
> java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
> at sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:78)
> at
> org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:227)
> at
> org.glassfish.jersey.message.internal.ReaderWriter.writeTo(ReaderWriter.java:111)
> at
> org.glassfish.jersey.message.internal.AbstractMessageReaderWriterProvider.writeTo(AbstractMessageReaderWriterProvider.java:77)
> at
> org.glassfish.jersey.message.internal.InputStreamProvider.writeTo(InputStreamProvider.java:103)
> at
> org.glassfish.jersey.message.internal.InputStreamProvider.writeTo(InputStreamProvider.java:58)
> at
> org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:194)
> at
> org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
> at
> org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1005)
> at
> org.glassfish.jersey.client.ClientRequest.writeEntity(ClientRequest.java:430)
> at
> org.glassfish.jersey.client.HttpUrlConnector._apply(HttpUrlConnector.java:287)
> at
> org.glassfish.jersey.client.HttpUrlConnector.apply(HttpUrlConnector.java:200)
> at
> org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:215)
> at
> org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:650)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
> at
> org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:421)
> at
> org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:646)
> at
> org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:402)
> at
> org.glassfish.jersey.client.JerseyInvocation$Builder.put(JerseyInvocation.java:290)
> at
> com.aptusinteractive.acceptancetests.StorageServiceAcceptanceTest.givenNoFileAtGivenNonHierarchicalPath_whenPuttingBigFileToGivenNonHierarchicalPath_thenFilePutToGivenNonHierarchicalPathAndResponseWithStatusCodeNoContentReturned(StorageServiceAcceptanceTest.java:99)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>
>