Hi I am trying to create a simple file server and in the interest of making
it scalable I decided that data should be streamed through it. As such I
decided to use "application/octet-stream" for the upload and download media
type.
The issue I am facing is that I run out of heap space when uploading large
files (downloading seems to work fine), I've tested with 700 MB and it
failed. I used jconsole to monitor server and client and saw that for a
358.4 MB file the client peak memory usage is 772.2 MB so a bit over double
the file size. I used wireshark to see what was being sent and saw that the
client was sending the content length correctly and therefore must be
buffering the stream at least once. I would like to disable this behaviour
as I am able to set the "content-length" header myself, I was hoping that
setting the header my prevent this behaviour but I tried it and it does
not. Does anyone have an idea about how to prevent the client from
buffering any large amounts of data?
Thanks
Piers
Below is the client code
Client client = ClientBuilder.newClient();
WebTarget storageServiceContextRootWebTarget = client.target("
http://localhost:8080/storage-service-test-war");
File bigFile = new File("bigfile");
InputStream bigFileContentInputStream = new FileInputStream(bigFile);
Response response =
storageServiceContextRootWebTarget.path(FILE_ONLY_PATH).request().put(Entity.entity(bigFileContentInputStream,
MediaType.APPLICATION_OCTET_STREAM), Response.class);
And server side code
@PUT
@Path("{" + FILE_PATH_PATH_PARAMETER + ":.+}")
@Consumes(MediaType.APPLICATION_OCTET_STREAM)
public Response putFileToPath(@PathParam(FILE_PATH_PATH_PARAMETER) String
filePath,_at_HeaderParam("Content-Length") long contentLength, InputStream
inputStream) throws IOException {
repository.writeFileToPath(inputStream, contentLength,
Paths.get(filePath));
return Response.noContent().build();
}
writeFileToPath() uses Apache's IOUtils.copy() method to copy the data out
of the stream into a file.
javax.ws.rs.ProcessingException: Java heap space
at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:224)
at
org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:650)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
at
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:421)
at
org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:646)
at
org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:402)
at
org.glassfish.jersey.client.JerseyInvocation$Builder.put(JerseyInvocation.java:290)
at
com.aptusinteractive.acceptancetests.StorageServiceAcceptanceTest.givenNoFileAtGivenNonHierarchicalPath_whenPuttingBigFileToGivenNonHierarchicalPath_thenFilePutToGivenNonHierarchicalPathAndResponseWithStatusCodeNoContentReturned(StorageServiceAcceptanceTest.java:99)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:69)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:48)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.ParentRunner.run(ParentRunner.java:292)
at
org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2271)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
at
java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
at sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:78)
at
org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:227)
at
org.glassfish.jersey.message.internal.ReaderWriter.writeTo(ReaderWriter.java:111)
at
org.glassfish.jersey.message.internal.AbstractMessageReaderWriterProvider.writeTo(AbstractMessageReaderWriterProvider.java:77)
at
org.glassfish.jersey.message.internal.InputStreamProvider.writeTo(InputStreamProvider.java:103)
at
org.glassfish.jersey.message.internal.InputStreamProvider.writeTo(InputStreamProvider.java:58)
at
org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:194)
at
org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
at
org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1005)
at
org.glassfish.jersey.client.ClientRequest.writeEntity(ClientRequest.java:430)
at
org.glassfish.jersey.client.HttpUrlConnector._apply(HttpUrlConnector.java:287)
at
org.glassfish.jersey.client.HttpUrlConnector.apply(HttpUrlConnector.java:200)
at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:215)
at
org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:650)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
at
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:421)
at
org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:646)
at
org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:402)
at
org.glassfish.jersey.client.JerseyInvocation$Builder.put(JerseyInvocation.java:290)
at
com.aptusinteractive.acceptancetests.StorageServiceAcceptanceTest.givenNoFileAtGivenNonHierarchicalPath_whenPuttingBigFileToGivenNonHierarchicalPath_thenFilePutToGivenNonHierarchicalPathAndResponseWithStatusCodeNoContentReturned(StorageServiceAcceptanceTest.java:99)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)