Hi,
Currently, it looks like we cannot disable buffering of entity in Jersey
client using HttpUrlConnector. I have filed an issue:
https://java.net/jira/browse/JERSEY-2024.
As a workaround you can use the following approach:
...
import org.glassfish.jersey.client.ClientConfig;
import org.glassfish.jersey.client.HttpUrlConnector;
import org.glassfish.jersey.client.spi.Connector;
import java.net.HttpURLConnection;
....
public class ClientBufferingTest extends JerseyTest {
private static final long LENGTH = 1000000000l;
@Override
protected Application configure() {
return new ResourceConfig(MyResource.class);
}
@Path("resource")
public static class MyResource {
@POST
public long post(InputStream is) throws IOException {
int b;
long count = 0;
while ((b = is.read()) != -1) {
count++;
}
return count;
}
}
@Test
public void test() {
HttpUrlConnector.ConnectionFactory factory = new
HttpUrlConnector.ConnectionFactory() {
@Override
public HttpURLConnection getConnection(URL endpointUrl)
throws IOException {
HttpURLConnection conn = (HttpURLConnection)
endpointUrl.openConnection();
conn.setChunkedStreamingMode(2048);
return conn;
}
};
InputStream is = new InputStream() {
private int cnt = 0;
@Override
public int read() throws IOException {
if (cnt++ < LENGTH) {
return 'a';
} else {
return -1;
}
}
};
ClientConfig clientConfig = new ClientConfig();
Connector connector = new HttpUrlConnector(factory);
clientConfig.connector(connector);
Client client = ClientBuilder.newClient(clientConfig);
final Response response
= client.target(getBaseUri()).path("resource")
.request().post(Entity.entity(is,
MediaType.APPLICATION_OCTET_STREAM));
Assert.assertEquals(200, response.getStatus());
final long count = response.readEntity(long.class);
System.out.println(count);
Assert.assertEquals(LENGTH, count);
}
}
This test will not throw OutOfMemoryError. The workaround is in using
HttpUrlConnector.ConnectionFactory which will set
conn.setChunkedStreamingMode(<chunk size>); on underlying
HttpUrlConnection. You pass this factory to the constructor of
HttpUrlConnector that you will register in ClientConfig. This workaround
does not use content-length at all.
You can of course setup different chunk size.
Mira
On 08/08/2013 06:23 PM, Piers Powlesland wrote:
> Hi I am trying to create a simple file server and in the interest of
> making it scalable I decided that data should be streamed through it.
> As such I decided to use "application/octet-stream" for the upload and
> download media type.
>
> The issue I am facing is that I run out of heap space when uploading
> large files (downloading seems to work fine), I've tested with 700 MB
> and it failed. I used jconsole to monitor server and client and saw
> that for a 358.4 MB file the client peak memory usage is 772.2 MB so a
> bit over double the file size. I used wireshark to see what was being
> sent and saw that the client was sending the content length correctly
> and therefore must be buffering the stream at least once. I would like
> to disable this behaviour as I am able to set the "content-length"
> header myself, I was hoping that setting the header my prevent this
> behaviour but I tried it and it does not. Does anyone have an idea
> about how to prevent the client from buffering any large amounts of data?
>
> Thanks
>
> Piers
>
> Below is the client code
>
> Client client = ClientBuilder.newClient();
> WebTarget storageServiceContextRootWebTarget =
> client.target("http://localhost:8080/storage-service-test-war");
> File bigFile = new File("bigfile");
> InputStream bigFileContentInputStream = new FileInputStream(bigFile);
> Response response =
> storageServiceContextRootWebTarget.path(FILE_ONLY_PATH).request().put(Entity.entity(bigFileContentInputStream,
> MediaType.APPLICATION_OCTET_STREAM), Response.class);
>
> And server side code
>
> @PUT
> @Path("{" + FILE_PATH_PATH_PARAMETER + ":.+}")
> @Consumes(MediaType.APPLICATION_OCTET_STREAM)
> public Response putFileToPath(@PathParam(FILE_PATH_PATH_PARAMETER)
> String filePath,_at_HeaderParam("Content-Length") long contentLength,
> InputStream inputStream) throws IOException {
> repository.writeFileToPath(inputStream, contentLength,
> Paths.get(filePath));
> return Response.noContent().build();
> }
>
>
> writeFileToPath() uses Apache's IOUtils.copy() method to copy the data
> out of the stream into a file.
>
>
> javax.ws.rs.ProcessingException: Java heap space
> at
> org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:224)
> at
> org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:650)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
> at
> org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:421)
> at
> org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:646)
> at
> org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:402)
> at
> org.glassfish.jersey.client.JerseyInvocation$Builder.put(JerseyInvocation.java:290)
> at
> com.aptusinteractive.acceptancetests.StorageServiceAcceptanceTest.givenNoFileAtGivenNonHierarchicalPath_whenPuttingBigFileToGivenNonHierarchicalPath_thenFilePutToGivenNonHierarchicalPathAndResponseWithStatusCodeNoContentReturned(StorageServiceAcceptanceTest.java:99)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:69)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:48)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:292)
> at
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:2271)
> at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
> at
> java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
> at sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:78)
> at
> org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:227)
> at
> org.glassfish.jersey.message.internal.ReaderWriter.writeTo(ReaderWriter.java:111)
> at
> org.glassfish.jersey.message.internal.AbstractMessageReaderWriterProvider.writeTo(AbstractMessageReaderWriterProvider.java:77)
> at
> org.glassfish.jersey.message.internal.InputStreamProvider.writeTo(InputStreamProvider.java:103)
> at
> org.glassfish.jersey.message.internal.InputStreamProvider.writeTo(InputStreamProvider.java:58)
> at
> org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:194)
> at
> org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
> at
> org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1005)
> at
> org.glassfish.jersey.client.ClientRequest.writeEntity(ClientRequest.java:430)
> at
> org.glassfish.jersey.client.HttpUrlConnector._apply(HttpUrlConnector.java:287)
> at
> org.glassfish.jersey.client.HttpUrlConnector.apply(HttpUrlConnector.java:200)
> at
> org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:215)
> at
> org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:650)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
> at
> org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:421)
> at
> org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:646)
> at
> org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:402)
> at
> org.glassfish.jersey.client.JerseyInvocation$Builder.put(JerseyInvocation.java:290)
> at
> com.aptusinteractive.acceptancetests.StorageServiceAcceptanceTest.givenNoFileAtGivenNonHierarchicalPath_whenPuttingBigFileToGivenNonHierarchicalPath_thenFilePutToGivenNonHierarchicalPathAndResponseWithStatusCodeNoContentReturned(StorageServiceAcceptanceTest.java:99)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>