Thanks Mira, I will use this workaround for now, do you have any idea when
https://java.net/jira/browse/JERSEY-2024 will be fixed?
Piers
On 3 September 2013 17:34, Piers Powlesland <piers_at_aptusinteractive.com>wrote:
> Hi Mira
>
> Thanks for the advice, the example you showed me works although the client
> it still takes 200~400 MB to stream a 1GB file. The apache client seems to
> take up to 200Mb but seems slower at making many small requests, prohibits
> explicit setting of the content length and also requires closing responses
> which would mean modifying a fair bit of existing code. What works best is
> if I set the content length explicitly via the connection factory using
>
> conn.setFixedLengthStreamingMode(LENGTH);
>
> Then the client uses very little memory, but It seems like the client is
> not configurable enough for me to do this.
>
> Although using the chunked encoding has solved the out of memory error, it
> is really not a viable solution as the server side to this application will
> be integrated with Amazon's S3 which requires the content length set, if I
> do not set this at the client I will need to buffer streams on the server
> before sending them to S3.
>
> It seems like the most intuitive solution would be to set
> "conn.setFixedLengthStreamingMode(LENGTH);" if the content length has been
> explicitly set.
>
> In the meantime are you aware of any way that I can ensure the client uses
> conn.setFixedLengthStreamingMode(XXXX) instead of buffering or chunked
> encoding?
>
> Thanks
>
> Piers
>
>
> On 8 August 2013 17:23, Piers Powlesland <piers_at_aptusinteractive.com>wrote:
>
>> Hi I am trying to create a simple file server and in the interest of
>> making it scalable I decided that data should be streamed through it. As
>> such I decided to use "application/octet-stream" for the upload and
>> download media type.
>>
>> The issue I am facing is that I run out of heap space when uploading
>> large files (downloading seems to work fine), I've tested with 700 MB and
>> it failed. I used jconsole to monitor server and client and saw that for a
>> 358.4 MB file the client peak memory usage is 772.2 MB so a bit over double
>> the file size. I used wireshark to see what was being sent and saw that the
>> client was sending the content length correctly and therefore must be
>> buffering the stream at least once. I would like to disable this behaviour
>> as I am able to set the "content-length" header myself, I was hoping that
>> setting the header my prevent this behaviour but I tried it and it does
>> not. Does anyone have an idea about how to prevent the client from
>> buffering any large amounts of data?
>>
>> Thanks
>>
>> Piers
>>
>> Below is the client code
>>
>> Client client = ClientBuilder.newClient();
>> WebTarget storageServiceContextRootWebTarget = client.target("
>> http://localhost:8080/storage-service-test-war");
>> File bigFile = new File("bigfile");
>> InputStream bigFileContentInputStream = new FileInputStream(bigFile);
>> Response response =
>> storageServiceContextRootWebTarget.path(FILE_ONLY_PATH).request().put(Entity.entity(bigFileContentInputStream,
>> MediaType.APPLICATION_OCTET_STREAM), Response.class);
>>
>> And server side code
>>
>> @PUT
>> @Path("{" + FILE_PATH_PATH_PARAMETER + ":.+}")
>> @Consumes(MediaType.APPLICATION_OCTET_STREAM)
>> public Response putFileToPath(@PathParam(FILE_PATH_PATH_PARAMETER)
>> String filePath,_at_HeaderParam("Content-Length") long contentLength,
>> InputStream inputStream) throws IOException {
>> repository.writeFileToPath(inputStream, contentLength,
>> Paths.get(filePath));
>> return Response.noContent().build();
>> }
>>
>>
>> writeFileToPath() uses Apache's IOUtils.copy() method to copy the data
>> out of the stream into a file.
>>
>>
>> javax.ws.rs.ProcessingException: Java heap space
>> at
>> org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:224)
>> at
>> org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:650)
>> at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
>> at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
>> at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
>> at
>> org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:421)
>> at
>> org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:646)
>> at
>> org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:402)
>> at
>> org.glassfish.jersey.client.JerseyInvocation$Builder.put(JerseyInvocation.java:290)
>> at
>> com.aptusinteractive.acceptancetests.StorageServiceAcceptanceTest.givenNoFileAtGivenNonHierarchicalPath_whenPuttingBigFileToGivenNonHierarchicalPath_thenFilePutToGivenNonHierarchicalPathAndResponseWithStatusCodeNoContentReturned(StorageServiceAcceptanceTest.java:99)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at
>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
>> at
>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>> at
>> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
>> at
>> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
>> at
>> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
>> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
>> at
>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:69)
>> at
>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:48)
>> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
>> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
>> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
>> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
>> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
>> at
>> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
>> at
>> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
>> at org.junit.runners.ParentRunner.run(ParentRunner.java:292)
>> at
>> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
>> at
>> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
>> Caused by: java.lang.OutOfMemoryError: Java heap space
>> at java.util.Arrays.copyOf(Arrays.java:2271)
>> at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
>> at
>> java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
>> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
>> at sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:78)
>> at
>> org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:227)
>> at
>> org.glassfish.jersey.message.internal.ReaderWriter.writeTo(ReaderWriter.java:111)
>> at
>> org.glassfish.jersey.message.internal.AbstractMessageReaderWriterProvider.writeTo(AbstractMessageReaderWriterProvider.java:77)
>> at
>> org.glassfish.jersey.message.internal.InputStreamProvider.writeTo(InputStreamProvider.java:103)
>> at
>> org.glassfish.jersey.message.internal.InputStreamProvider.writeTo(InputStreamProvider.java:58)
>> at
>> org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:194)
>> at
>> org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
>> at
>> org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1005)
>> at
>> org.glassfish.jersey.client.ClientRequest.writeEntity(ClientRequest.java:430)
>> at
>> org.glassfish.jersey.client.HttpUrlConnector._apply(HttpUrlConnector.java:287)
>> at
>> org.glassfish.jersey.client.HttpUrlConnector.apply(HttpUrlConnector.java:200)
>> at
>> org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:215)
>> at
>> org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:650)
>> at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
>> at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
>> at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
>> at
>> org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:421)
>> at
>> org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:646)
>> at
>> org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:402)
>> at
>> org.glassfish.jersey.client.JerseyInvocation$Builder.put(JerseyInvocation.java:290)
>> at
>> com.aptusinteractive.acceptancetests.StorageServiceAcceptanceTest.givenNoFileAtGivenNonHierarchicalPath_whenPuttingBigFileToGivenNonHierarchicalPath_thenFilePutToGivenNonHierarchicalPathAndResponseWithStatusCodeNoContentReturned(StorageServiceAcceptanceTest.java:99)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at
>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
>> at
>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>>
>>
>