Hi Piers,
I think there is no nice workaround to set content length on the
connection. The only solution comes to my mind is not so nice. You can
implement HttpUrlConnector.ConnectionFactory and use it to setup the
client as you already do. The connection factory will look something
like this:
public class ConnFactory implements HttpUrlConnector.ConnectionFactory {
private volatile int currentContentLenght;
public void setCurrentContentLenght(int currentContentLenght) {
this.currentContentLenght = currentContentLenght;
}
@Override
public HttpURLConnection getConnection(URL endpointUrl) throws
IOException {
HttpURLConnection conn = (HttpURLConnection)
endpointUrl.openConnection();
conn.setFixedLengthStreamingMode(currentContentLenght);
return conn;
}
}
and you will have such a client code:
ConnFactory factory = new ConnFactory();
ClientConfig clientConfig = new ClientConfig();
Connector connector = new HttpUrlConnector(factory);
clientConfig.connector(connector);
Client client = ClientBuilder.newClient(clientConfig);
factory.setCurrentContentLenght(...CONTENT LENGTH of entity..);
// here you will setup content length for the next entity that you want
to post
Response response
= client.target(getBaseUri()).path("resource")
.request().post(Entity.entity(is,
MediaType.APPLICATION_OCTET_STREAM));
// and you can send another entity (file)
factory.setCurrentContentLenght(...CONTENT LENGTH of next
entity..);
final Response response
= client.target(getBaseUri()).path("resource")
.request().post(Entity.entity(is,
MediaType.APPLICATION_OCTET_STREAM));
...
So, your factory will always setup content length of the entity that
will be send. Such a client can be used only by one thread. I have not
tested it and I am not sure this will definitely work but it looks like
a possible solution to me.
I hope this helps.
After the issue
https://java.net/jira/browse/JERSEY-2024 is fixed, such
a workaround will not be needed.
Mira
On 09/03/2013 06:34 PM, Piers Powlesland wrote:
> Hi Mira
>
> Thanks for the advice, the example you showed me works although the
> client it still takes 200~400 MB to stream a 1GB file. The apache
> client seems to take up to 200Mb but seems slower at making many small
> requests, prohibits explicit setting of the content length and also
> requires closing responses which would mean modifying a fair bit of
> existing code. What works best is if I set the content length
> explicitly via the connection factory using
>
> conn.setFixedLengthStreamingMode(LENGTH);
>
> Then the client uses very little memory, but It seems like the client
> is not configurable enough for me to do this.
>
> Although using the chunked encoding has solved the out of memory
> error, it is really not a viable solution as the server side to this
> application will be integrated with Amazon's S3 which requires the
> content length set, if I do not set this at the client I will need to
> buffer streams on the server before sending them to S3.
>
> It seems like the most intuitive solution would be to set
> "conn.setFixedLengthStreamingMode(LENGTH);" if the content length has
> been explicitly set.
>
> In the meantime are you aware of any way that I can ensure the client
> uses conn.setFixedLengthStreamingMode(XXXX) instead of buffering or
> chunked encoding?
>
> Thanks
>
> Piers
>
>
> On 8 August 2013 17:23, Piers Powlesland <piers_at_aptusinteractive.com
> <mailto:piers_at_aptusinteractive.com>> wrote:
>
> Hi I am trying to create a simple file server and in the interest
> of making it scalable I decided that data should be streamed
> through it. As such I decided to use "application/octet-stream"
> for the upload and download media type.
>
> The issue I am facing is that I run out of heap space when
> uploading large files (downloading seems to work fine), I've
> tested with 700 MB and it failed. I used jconsole to monitor
> server and client and saw that for a 358.4 MB file the client peak
> memory usage is 772.2 MB so a bit over double the file size. I
> used wireshark to see what was being sent and saw that the client
> was sending the content length correctly and therefore must be
> buffering the stream at least once. I would like to disable this
> behaviour as I am able to set the "content-length" header myself,
> I was hoping that setting the header my prevent this behaviour but
> I tried it and it does not. Does anyone have an idea about how to
> prevent the client from buffering any large amounts of data?
>
> Thanks
>
> Piers
>
> Below is the client code
>
> Client client = ClientBuilder.newClient();
> WebTarget storageServiceContextRootWebTarget =
> client.target("http://localhost:8080/storage-service-test-war");
> File bigFile = new File("bigfile");
> InputStream bigFileContentInputStream = new FileInputStream(bigFile);
> Response response =
> storageServiceContextRootWebTarget.path(FILE_ONLY_PATH).request().put(Entity.entity(bigFileContentInputStream,
> MediaType.APPLICATION_OCTET_STREAM), Response.class);
>
> And server side code
>
> @PUT
> @Path("{" + FILE_PATH_PATH_PARAMETER + ":.+}")
> @Consumes(MediaType.APPLICATION_OCTET_STREAM)
> public Response
> putFileToPath(@PathParam(FILE_PATH_PATH_PARAMETER) String
> filePath,_at_HeaderParam("Content-Length") long contentLength,
> InputStream inputStream) throws IOException {
> repository.writeFileToPath(inputStream, contentLength,
> Paths.get(filePath));
> return Response.noContent().build();
> }
>
>
> writeFileToPath() uses Apache's IOUtils.copy() method to copy the
> data out of the stream into a file.
>
>
> javax.ws.rs.ProcessingException: Java heap space
> at
> org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:224)
> at
> org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:650)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
> at
> org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:421)
> at
> org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:646)
> at
> org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:402)
> at
> org.glassfish.jersey.client.JerseyInvocation$Builder.put(JerseyInvocation.java:290)
> at
> com.aptusinteractive.acceptancetests.StorageServiceAcceptanceTest.givenNoFileAtGivenNonHierarchicalPath_whenPuttingBigFileToGivenNonHierarchicalPath_thenFilePutToGivenNonHierarchicalPathAndResponseWithStatusCodeNoContentReturned(StorageServiceAcceptanceTest.java:99)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:69)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:48)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:292)
> at
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:2271)
> at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
> at
> java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
> at
> sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:78)
> at
> org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:227)
> at
> org.glassfish.jersey.message.internal.ReaderWriter.writeTo(ReaderWriter.java:111)
> at
> org.glassfish.jersey.message.internal.AbstractMessageReaderWriterProvider.writeTo(AbstractMessageReaderWriterProvider.java:77)
> at
> org.glassfish.jersey.message.internal.InputStreamProvider.writeTo(InputStreamProvider.java:103)
> at
> org.glassfish.jersey.message.internal.InputStreamProvider.writeTo(InputStreamProvider.java:58)
> at
> org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:194)
> at
> org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
> at
> org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1005)
> at
> org.glassfish.jersey.client.ClientRequest.writeEntity(ClientRequest.java:430)
> at
> org.glassfish.jersey.client.HttpUrlConnector._apply(HttpUrlConnector.java:287)
> at
> org.glassfish.jersey.client.HttpUrlConnector.apply(HttpUrlConnector.java:200)
> at
> org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:215)
> at
> org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:650)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
> at
> org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:421)
> at
> org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:646)
> at
> org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:402)
> at
> org.glassfish.jersey.client.JerseyInvocation$Builder.put(JerseyInvocation.java:290)
> at
> com.aptusinteractive.acceptancetests.StorageServiceAcceptanceTest.givenNoFileAtGivenNonHierarchicalPath_whenPuttingBigFileToGivenNonHierarchicalPath_thenFilePutToGivenNonHierarchicalPathAndResponseWithStatusCodeNoContentReturned(StorageServiceAcceptanceTest.java:99)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>
>