Uncompressing Gzip Response from Webclient

Automatically decompress gzip response via WebClient.DownloadData

WebClient uses HttpWebRequest under the covers. And HttpWebRequest supports gzip/deflate decompression. See HttpWebRequest AutomaticDecompression property

However, WebClient class does not expose this property directly. So you will have to derive from it to set the property on the underlying HttpWebRequest.

class MyWebClient : WebClient
{
protected override WebRequest GetWebRequest(Uri address)
{
HttpWebRequest request = base.GetWebRequest(address) as HttpWebRequest;
request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
return request;
}
}

Uncompressing gzip response from WebClient

The easiest way to do this is to use the built in automatic decompression with the HttpWebRequest class.

var request = (HttpWebRequest)HttpWebRequest.Create("http://stackoverflow.com");
request.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate");
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;

To do this with a WebClient you have to make your own class derived from WebClient and override the GetWebRequest() method.

public class GZipWebClient : WebClient
{
protected override WebRequest GetWebRequest(Uri address)
{
HttpWebRequest request = (HttpWebRequest)base.GetWebRequest(address);
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
return request;
}
}

Also see this SO thread: Does .NET's HttpWebResponse uncompress automatically GZiped and Deflated responses?

Use Webclient to get page resource in C#

By setting client.Headers["Accept-Encoding"] = "gzip"; you are asking the server to send a compressed response. However, you are not decompressing it. This is causing the incorrect response.

As per https://stackoverflow.com/a/4914874/23633, you can get WebClient to automatically decompress responses by modifying the HttpWebRequest it creates:

class MyWebClient : WebClient
{
protected override WebRequest GetWebRequest(Uri address)
{
var request = (HttpWebRequest) base.GetWebRequest(address);
request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
return request;
}
}

string url = "http://www.tsetmc.com/Loader.aspx?ParTree=15131F";
WebClient client = new MyWebClient();
// don't set the Accept-Encoding header here; it will be done automatically
string pageSource = client.DownloadString(url);

Prevent server from encoding in gzip when downloading with WebClient

See this Jeff Atwood post for the solution:

http://www.­codinghorror.­com/­blog/­2004/­08/­netwebclient-­and-­gzip.­html

How to decompress a FluxDataBuffer (and how to write one)?

public class HttpResponseHeadersHandler extends ChannelInboundHandlerAdapter {
private final HttpHeaders httpHeaders;

@Override
public void channelRead(ChannelHandlerContext ctx, Object msg) {
if (msg instanceof HttpResponse &&
!HttpStatus.resolve(((HttpResponse) msg).status().code()).is1xxInformational()) {
HttpHeaders headers = ((HttpResponse) msg).headers();

httpHeaders.forEach(e -> {
log.warn("Modifying {} from: {} to: {}.", e.getKey(), headers.get(e.getKey()), e.getValue());
headers.set(e.getKey(), e.getValue());
});
}
ctx.fireChannelRead(msg);
}
}

Then I create a ClientHttpConnector to use with WebClient and in afterNettyContextInit add the handler:

ctx.addHandlerLast(new ReadTimeoutHandler(readTimeoutMillis, TimeUnit.MILLISECONDS));
ctx.addHandlerLast(new Slf4JLoggingHandler());
if (forceDecompression) {
io.netty.handler.codec.http.HttpHeaders httpHeaders = new ReadOnlyHttpHeaders(
true,
CONTENT_ENCODING, GZIP,
CONTENT_TYPE, APPLICATION_JSON
);
HttpResponseHeadersHandler headersModifier = new HttpResponseHeadersHandler(httpHeaders);
ctx.addHandlerFirst(headersModifier);
}
ctx.addHandlerLast(new HttpContentDecompressor());

This, of course, would fail for responses that are not GZIP compressed, so I use this instance of WebClient for a particular use case only, where I know for sure that the response is compressed.

Writing is easy: Spring has a ResourceEncoder, so InputStream can simply be converted to InputStreamResource, and voila!

Does .NET's HttpWebResponse uncompress automatically GZiped and Deflated responses?

I found the answer.

You can change the code to:

var request = (HttpWebRequest)HttpWebRequest.Create(requestUri);
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;

And you will have automatic decompression. No need to change the rest of the code.



Related Topics



Leave a reply



Submit