Hi Piers,
This is currently opened in Unplanned, so it might be fixed until next
release (in few weeks) but I cannot promise it.
Mira
On 09/04/2013 10:28 AM, Piers Powlesland wrote:
> Thanks Mira, I will use this workaround for now, do you have any idea
> when https://java.net/jira/browse/JERSEY-2024 will be fixed?
>
> Piers
>
>
> On 3 September 2013 17:34, Piers Powlesland
> <piers_at_aptusinteractive.com <mailto:piers_at_aptusinteractive.com>> wrote:
>
> Hi Mira
>
> Thanks for the advice, the example you showed me works although
> the client it still takes 200~400 MB to stream a 1GB file. The
> apache client seems to take up to 200Mb but seems slower at making
> many small requests, prohibits explicit setting of the content
> length and also requires closing responses which would mean
> modifying a fair bit of existing code. What works best is if I set
> the content length explicitly via the connection factory using
>
> conn.setFixedLengthStreamingMode(LENGTH);
>
> Then the client uses very little memory, but It seems like the
> client is not configurable enough for me to do this.
>
> Although using the chunked encoding has solved the out of memory
> error, it is really not a viable solution as the server side to
> this application will be integrated with Amazon's S3 which
> requires the content length set, if I do not set this at the
> client I will need to buffer streams on the server before sending
> them to S3.
>
> It seems like the most intuitive solution would be to set
> "conn.setFixedLengthStreamingMode(LENGTH);" if the content length
> has been explicitly set.
>
> In the meantime are you aware of any way that I can ensure the
> client uses conn.setFixedLengthStreamingMode(XXXX) instead of
> buffering or chunked encoding?
>
> Thanks
>
> Piers
>
>
> On 8 August 2013 17:23, Piers Powlesland
> <piers_at_aptusinteractive.com <mailto:piers_at_aptusinteractive.com>>
> wrote:
>
> Hi I am trying to create a simple file server and in the
> interest of making it scalable I decided that data should be
> streamed through it. As such I decided to
> use "application/octet-stream" for the upload and download
> media type.
>
> The issue I am facing is that I run out of heap space when
> uploading large files (downloading seems to work fine), I've
> tested with 700 MB and it failed. I used jconsole to monitor
> server and client and saw that for a 358.4 MB file the client
> peak memory usage is 772.2 MB so a bit over double the file
> size. I used wireshark to see what was being sent and saw that
> the client was sending the content length correctly and
> therefore must be buffering the stream at least once. I would
> like to disable this behaviour as I am able to set the
> "content-length" header myself, I was hoping that setting the
> header my prevent this behaviour but I tried it and it does
> not. Does anyone have an idea about how to prevent the client
> from buffering any large amounts of data?
>
> Thanks
>
> Piers
>
> Below is the client code
>
> Client client = ClientBuilder.newClient();
> WebTarget storageServiceContextRootWebTarget =
> client.target("http://localhost:8080/storage-service-test-war");
> File bigFile = new File("bigfile");
> InputStream bigFileContentInputStream = new
> FileInputStream(bigFile);
> Response response =
> storageServiceContextRootWebTarget.path(FILE_ONLY_PATH).request().put(Entity.entity(bigFileContentInputStream,
> MediaType.APPLICATION_OCTET_STREAM), Response.class);
>
> And server side code
>
> @PUT
> @Path("{" + FILE_PATH_PATH_PARAMETER + ":.+}")
> @Consumes(MediaType.APPLICATION_OCTET_STREAM)
> public Response
> putFileToPath(@PathParam(FILE_PATH_PATH_PARAMETER) String
> filePath,_at_HeaderParam("Content-Length") long contentLength,
> InputStream inputStream) throws IOException {
> repository.writeFileToPath(inputStream, contentLength,
> Paths.get(filePath));
> return Response.noContent().build();
> }
>
>
> writeFileToPath() uses Apache's IOUtils.copy() method to copy
> the data out of the stream into a file.
>
>
> javax.ws.rs.ProcessingException: Java heap space
> at
> org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:224)
> at
> org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:650)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
> at
> org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:421)
> at
> org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:646)
> at
> org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:402)
> at
> org.glassfish.jersey.client.JerseyInvocation$Builder.put(JerseyInvocation.java:290)
> at
> com.aptusinteractive.acceptancetests.StorageServiceAcceptanceTest.givenNoFileAtGivenNonHierarchicalPath_whenPuttingBigFileToGivenNonHierarchicalPath_thenFilePutToGivenNonHierarchicalPathAndResponseWithStatusCodeNoContentReturned(StorageServiceAcceptanceTest.java:99)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:69)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:48)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:292)
> at
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:2271)
> at
> java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
> at
> java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
> at
> java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
> at
> sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:78)
> at
> org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:227)
> at
> org.glassfish.jersey.message.internal.ReaderWriter.writeTo(ReaderWriter.java:111)
> at
> org.glassfish.jersey.message.internal.AbstractMessageReaderWriterProvider.writeTo(AbstractMessageReaderWriterProvider.java:77)
> at
> org.glassfish.jersey.message.internal.InputStreamProvider.writeTo(InputStreamProvider.java:103)
> at
> org.glassfish.jersey.message.internal.InputStreamProvider.writeTo(InputStreamProvider.java:58)
> at
> org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:194)
> at
> org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
> at
> org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1005)
> at
> org.glassfish.jersey.client.ClientRequest.writeEntity(ClientRequest.java:430)
> at
> org.glassfish.jersey.client.HttpUrlConnector._apply(HttpUrlConnector.java:287)
> at
> org.glassfish.jersey.client.HttpUrlConnector.apply(HttpUrlConnector.java:200)
> at
> org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:215)
> at
> org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:650)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
> at
> org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:421)
> at
> org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:646)
> at
> org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:402)
> at
> org.glassfish.jersey.client.JerseyInvocation$Builder.put(JerseyInvocation.java:290)
> at
> com.aptusinteractive.acceptancetests.StorageServiceAcceptanceTest.givenNoFileAtGivenNonHierarchicalPath_whenPuttingBigFileToGivenNonHierarchicalPath_thenFilePutToGivenNonHierarchicalPathAndResponseWithStatusCodeNoContentReturned(StorageServiceAcceptanceTest.java:99)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>
>
>