Using rest-client to download a file to disk without loading it all in memory first
Another way is to use raw_response
. This saves directly to a file, usually in /tmp
and handles redirects without a problem.
See Streaming Responses. Here's their example:
>> raw = RestClient::Request.execute(
method: :get,
url: 'http://releases.ubuntu.com/16.04.2/ubuntu-16.04.2-desktop-amd64.iso',
raw_response: true)
=> <RestClient::RawResponse @code=200, @file=#<Tempfile:/tmp/rest-client.20170522-5346-1pptjm1>, @request=<RestClient::Request @method="get", @url="http://releases.ubuntu.com/16.04.2/ubuntu-16.04.2-desktop-amd64.iso">>
>> raw.file.size
=> 1554186240
>> raw.file.path
=> "/tmp/rest-client.20170522-5346-1pptjm1"
How to use restsharp to download file
With RestSharp, it's right there in the readme:
var client = new RestClient("http://example.com");
client.DownloadData(request).SaveAs(path);
With HttpClient
, it's a bit more involved. Have a look at this blog post.
Another option is Flurl.Http (disclaimer: I'm the author). It uses HttpClient
under the hood and provides a fluent interface and lots of convenient helper methods, including:
await "http://example.com".DownloadFileAsync(folderPath, "foo.xml");
Get it on NuGet.
How to download a file using a Java REST service and a data stream
"How can I directly (without saving the file on 2nd server) download the file from 1st server to client's machine?"
Just use the Client
API and get the InputStream
from the response
Client client = ClientBuilder.newClient();
String url = "...";
final InputStream responseStream = client.target(url).request().get(InputStream.class);
There are two flavors to get the InputStream
. You can also use
Response response = client.target(url).request().get();
InputStream is = (InputStream)response.getEntity();
Which one is the more efficient? I'm not sure, but the returned InputStream
s are different classes, so you may want to look into that if you care to.
From 2nd server I can get a ByteArrayOutputStream to get the file from 1st server, can I pass this stream further to the client using the REST service?
So most of the answers you'll see in the link provided by @GradyGCooper seem to favor the use of StreamingOutput
. An example implementation might be something like
final InputStream responseStream = client.target(url).request().get(InputStream.class);
System.out.println(responseStream.getClass());
StreamingOutput output = new StreamingOutput() {
@Override
public void write(OutputStream out) throws IOException, WebApplicationException {
int length;
byte[] buffer = new byte[1024];
while((length = responseStream.read(buffer)) != -1) {
out.write(buffer, 0, length);
}
out.flush();
responseStream.close();
}
};
return Response.ok(output).header(
"Content-Disposition", "attachment, filename=\"...\"").build();
But if we look at the source code for StreamingOutputProvider, you'll see in the writeTo
, that it simply writes the data from one stream to another. So with our implementation above, we have to write twice.
How can we get only one write? Simple return the InputStream
as the Response
final InputStream responseStream = client.target(url).request().get(InputStream.class);
return Response.ok(responseStream).header(
"Content-Disposition", "attachment, filename=\"...\"").build();
If we look at the source code for InputStreamProvider, it simply delegates to ReadWriter.writeTo(in, out)
, which simply does what we did above in the StreamingOutput
implementation
public static void writeTo(InputStream in, OutputStream out) throws IOException {
int read;
final byte[] data = new byte[BUFFER_SIZE];
while ((read = in.read(data)) != -1) {
out.write(data, 0, read);
}
}
Asides:
Client
objects are expensive resources. You may want to reuse the sameClient
for request. You can extract aWebTarget
from the client for each request.WebTarget target = client.target(url);
InputStream is = target.request().get(InputStream.class);I think the
WebTarget
can even be shared. I can't find anything in the Jersey 2.x documentation (only because it is a larger document, and I'm too lazy to scan through it right now :-), but in the Jersey 1.x documentation, it says theClient
andWebResource
(which is equivalent toWebTarget
in 2.x) can be shared between threads. So I'm guessing Jersey 2.x would be the same. but you may want to confirm for yourself.You don't have to make use of the
Client
API. A download can be easily achieved with thejava.net
package APIs. But since you're already using Jersey, it doesn't hurt to use its APIsThe above is assuming Jersey 2.x. For Jersey 1.x, a simple Google search should get you a bunch of hits for working with the API (or the documentation I linked to above)
UPDATE
I'm such a dufus. While the OP and I are contemplating ways to turn a ByteArrayOutputStream
to an InputStream
, I missed the simplest solution, which is simply to write a MessageBodyWriter
for the ByteArrayOutputStream
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.lang.annotation.Annotation;
import java.lang.reflect.Type;
import javax.ws.rs.WebApplicationException;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.MultivaluedMap;
import javax.ws.rs.ext.MessageBodyWriter;
import javax.ws.rs.ext.Provider;
@Provider
public class OutputStreamWriter implements MessageBodyWriter<ByteArrayOutputStream> {
@Override
public boolean isWriteable(Class<?> type, Type genericType,
Annotation[] annotations, MediaType mediaType) {
return ByteArrayOutputStream.class == type;
}
@Override
public long getSize(ByteArrayOutputStream t, Class<?> type, Type genericType,
Annotation[] annotations, MediaType mediaType) {
return -1;
}
@Override
public void writeTo(ByteArrayOutputStream t, Class<?> type, Type genericType,
Annotation[] annotations, MediaType mediaType,
MultivaluedMap<String, Object> httpHeaders, OutputStream entityStream)
throws IOException, WebApplicationException {
t.writeTo(entityStream);
}
}
Then we can simply return the ByteArrayOutputStream
in the response
return Response.ok(baos).build();
D'OH!
UPDATE 2
Here are the tests I used (
Resource class
@Path("test")
public class TestResource {
final String path = "some_150_mb_file";
@GET
@Produces(MediaType.APPLICATION_OCTET_STREAM)
public Response doTest() throws Exception {
InputStream is = new FileInputStream(path);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
int len;
byte[] buffer = new byte[4096];
while ((len = is.read(buffer, 0, buffer.length)) != -1) {
baos.write(buffer, 0, len);
}
System.out.println("Server size: " + baos.size());
return Response.ok(baos).build();
}
}
Client test
public class Main {
public static void main(String[] args) throws Exception {
Client client = ClientBuilder.newClient();
String url = "http://localhost:8080/api/test";
Response response = client.target(url).request().get();
String location = "some_location";
FileOutputStream out = new FileOutputStream(location);
InputStream is = (InputStream)response.getEntity();
int len = 0;
byte[] buffer = new byte[4096];
while((len = is.read(buffer)) != -1) {
out.write(buffer, 0, len);
}
out.flush();
out.close();
is.close();
}
}
UPDATE 3
So the final solution for this particular use case was for the OP to simply pass the OutputStream
from the StreamingOutput
's write
method. Seems the third-party API, required a OutputStream
as an argument.
StreamingOutput output = new StreamingOutput() {
@Override
public void write(OutputStream out) {
thirdPartyApi.downloadFile(.., .., .., out);
}
}
return Response.ok(output).build();
Not quite sure, but seems the reading/writing within the resource method, using ByteArrayOutputStream`, realized something into memory.
The point of the downloadFile
method accepting an OutputStream
is so that it can write the result directly to the OutputStream
provided. For instance a FileOutputStream
, if you wrote it to file, while the download is coming in, it would get directly streamed to the file.
It's not meant for us to keep a reference to the OutputStream
, as you were trying to do with the baos
, which is where the memory realization comes in.
So with the way that works, we are writing directly to the response stream provided for us. The method write
doesn't actually get called until the writeTo
method (in the MessageBodyWriter
), where the OutputStream
is passed to it.
You can get a better picture looking at the MessageBodyWriter
I wrote. Basically in the writeTo
method, replace the ByteArrayOutputStream
with StreamingOutput
, then inside the method, call streamingOutput.write(entityStream)
. You can see the link I provided in the earlier part of the answer, where I link to the StreamingOutputProvider
. This is exactly what happens
Scala & Finatra: send file server response directly from disk to network w/o loading into memory
It's unfortunately not possible with Finatra 1.6. Streaming seems to be on the roadmap for 2.0, but there is no official word on its release. Right now, Request => Response
relies on memorized inputs and outputs. The most efficient way to deal with bodies in Finatra is to keep them in ChannelBuffers
so that there is at least only one instance of the bytes materialized.
Download Large file from server using REST template Java Spring MVC
Here is how I do it. Based on hints from this Spring Jira issue.
RestTemplate restTemplate // = ...;
// Optional Accept header
RequestCallback requestCallback = request -> request.getHeaders()
.setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL));
// Streams the response instead of loading it all in memory
ResponseExtractor<Void> responseExtractor = response -> {
// Here I write the response to a file but do what you like
Path path = Paths.get("some/path");
Files.copy(response.getBody(), path);
return null;
};
restTemplate.execute(URI.create("www.something.com"), HttpMethod.GET, requestCallback, responseExtractor);
From the aforementioned Jira issue:
Note that you cannot simply return the InputStream from the extractor, because by the time the execute method returns, the underlying connection and stream are already closed.
Update for Spring 5
Spring 5 introduced the WebClient
class which allows asynchronous (e.g. non-blocking) http requests. From the doc:
By comparison to the RestTemplate, the WebClient is:
- non-blocking, reactive, and supports higher concurrency with less hardware resources.
- provides a functional API that takes advantage of Java 8 lambdas.
- supports both synchronous and asynchronous scenarios.
- supports streaming up or down from a server.
To get WebClient
in Spring Boot, you need this dependency:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-webflux</artifactId>
</dependency>
For the moment, I'm sticking with RestTemplate
because I don't want to pull in another dependency only to get access to WebClient
.
c# Dropbox RestSharp download file to stream
I found the best answer in link:
string url = String.Format("https://content.dropboxapi.com/1/files/auto{0}?oauth_consumer_key={1}&oauth_token={2}&oauth_signature={3}%26{4}", path, app-key, access-token, app-secret, access-token-secret);
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(url);
webRequest.Method = "Get";
WebResponse webResponse = null;
try
{
webResponse = webRequest.GetResponse();
return webResponse.GetResponseStream();
}
catch (Exception ex)
{
throw ex;
}
Related Topics
How to Export a Rails Model to JSON Schema
How to Merge Two Hashes That Have Same Keys in Ruby
What the Purpose of Bind/Unbind Methods in Ruby
Ruby on Rails:How to Implement Cancel Button in Form_Tag
How to Join a Table and Count Records in Rails 3
Failure to Install Old MySQL2 Gem
How to Completely Wipe Rubygems Along with Rails etc
Ruby 'split': Invalid Byte Sequence in Utf-8 (Argumenterror)
How to Change Case of Letters in String Using Regex in Ruby
How to Download a File Over Http Using Ruby
How to Trigger Mouse Event in Capybara Test
How to Remove All Characters in a String Until a Substring Is Matched, in Ruby
Can't Install Rmagick and Imagemagick on Windows 7
How to Upload a Local File to a Carrierwave Model
Killing Process Group from Ruby Kills My Whole Computer