Hi Mira
Thanks for the advice, the example you showed me works although the client
it still takes 200~400 MB to stream a 1GB file. The apache client seems to
take up to 200Mb but seems slower at making many small requests, prohibits
explicit setting of the content length and also requires closing responses
which would mean modifying a fair bit of existing code. What works best is
if I set the content length explicitly via the connection factory using
conn.setFixedLengthStreamingMode(LENGTH);
Then the client uses very little memory, but It seems like the client is
not configurable enough for me to do this.
Although using the chunked encoding has solved the out of memory error, it
is really not a viable solution as the server side to this application will
be integrated with Amazon's S3 which requires the content length set, if I
do not set this at the client I will need to buffer streams on the server
before sending them to S3.
It seems like the most intuitive solution would be to set
"conn.setFixedLengthStreamingMode(LENGTH);" if the content length has been
explicitly set.
In the meantime are you aware of any way that I can ensure the client uses
conn.setFixedLengthStreamingMode(XXXX) instead of buffering or chunked
encoding?
Thanks
Piers
On 8 August 2013 17:23, Piers Powlesland <piers_at_aptusinteractive.com> wrote:
> Hi I am trying to create a simple file server and in the interest of
> making it scalable I decided that data should be streamed through it. As
> such I decided to use "application/octet-stream" for the upload and
> download media type.
>
> The issue I am facing is that I run out of heap space when uploading large
> files (downloading seems to work fine), I've tested with 700 MB and it
> failed. I used jconsole to monitor server and client and saw that for a
> 358.4 MB file the client peak memory usage is 772.2 MB so a bit over double
> the file size. I used wireshark to see what was being sent and saw that the
> client was sending the content length correctly and therefore must be
> buffering the stream at least once. I would like to disable this behaviour
> as I am able to set the "content-length" header myself, I was hoping that
> setting the header my prevent this behaviour but I tried it and it does
> not. Does anyone have an idea about how to prevent the client from
> buffering any large amounts of data?
>
> Thanks
>
> Piers
>
> Below is the client code
>
> Client client = ClientBuilder.newClient();
> WebTarget storageServiceContextRootWebTarget = client.target("
> http://localhost:8080/storage-service-test-war");
> File bigFile = new File("bigfile");
> InputStream bigFileContentInputStream = new FileInputStream(bigFile);
> Response response =
> storageServiceContextRootWebTarget.path(FILE_ONLY_PATH).request().put(Entity.entity(bigFileContentInputStream,
> MediaType.APPLICATION_OCTET_STREAM), Response.class);
>
> And server side code
>
> @PUT
> @Path("{" + FILE_PATH_PATH_PARAMETER + ":.+}")
> @Consumes(MediaType.APPLICATION_OCTET_STREAM)
> public Response putFileToPath(@PathParam(FILE_PATH_PATH_PARAMETER) String
> filePath,_at_HeaderParam("Content-Length") long contentLength, InputStream
> inputStream) throws IOException {
> repository.writeFileToPath(inputStream, contentLength,
> Paths.get(filePath));
> return Response.noContent().build();
> }
>
>
> writeFileToPath() uses Apache's IOUtils.copy() method to copy the data out
> of the stream into a file.
>
>
> javax.ws.rs.ProcessingException: Java heap space
> at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:224)
> at
> org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:650)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
> at
> org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:421)
> at
> org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:646)
> at
> org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:402)
> at
> org.glassfish.jersey.client.JerseyInvocation$Builder.put(JerseyInvocation.java:290)
> at
> com.aptusinteractive.acceptancetests.StorageServiceAcceptanceTest.givenNoFileAtGivenNonHierarchicalPath_whenPuttingBigFileToGivenNonHierarchicalPath_thenFilePutToGivenNonHierarchicalPathAndResponseWithStatusCodeNoContentReturned(StorageServiceAcceptanceTest.java:99)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:69)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:48)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:292)
> at
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:2271)
> at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
> at
> java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
> at sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:78)
> at
> org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:227)
> at
> org.glassfish.jersey.message.internal.ReaderWriter.writeTo(ReaderWriter.java:111)
> at
> org.glassfish.jersey.message.internal.AbstractMessageReaderWriterProvider.writeTo(AbstractMessageReaderWriterProvider.java:77)
> at
> org.glassfish.jersey.message.internal.InputStreamProvider.writeTo(InputStreamProvider.java:103)
> at
> org.glassfish.jersey.message.internal.InputStreamProvider.writeTo(InputStreamProvider.java:58)
> at
> org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:194)
> at
> org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
> at
> org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1005)
> at
> org.glassfish.jersey.client.ClientRequest.writeEntity(ClientRequest.java:430)
> at
> org.glassfish.jersey.client.HttpUrlConnector._apply(HttpUrlConnector.java:287)
> at
> org.glassfish.jersey.client.HttpUrlConnector.apply(HttpUrlConnector.java:200)
> at
> org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:215)
> at
> org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:650)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
> at
> org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:421)
> at
> org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:646)
> at
> org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:402)
> at
> org.glassfish.jersey.client.JerseyInvocation$Builder.put(JerseyInvocation.java:290)
> at
> com.aptusinteractive.acceptancetests.StorageServiceAcceptanceTest.givenNoFileAtGivenNonHierarchicalPath_whenPuttingBigFileToGivenNonHierarchicalPath_thenFilePutToGivenNonHierarchicalPathAndResponseWithStatusCodeNoContentReturned(StorageServiceAcceptanceTest.java:99)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>
>