Automatically decompress gzip response via WebClient.DownloadData
WebClient uses HttpWebRequest under the covers. And HttpWebRequest supports gzip/deflate decompression. See HttpWebRequest AutomaticDecompression property
However, WebClient class does not expose this property directly. So you will have to derive from it to set the property on the underlying HttpWebRequest.
class MyWebClient : WebClient
{
protected override WebRequest GetWebRequest(Uri address)
{
HttpWebRequest request = base.GetWebRequest(address) as HttpWebRequest;
request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
return request;
}
}
Uncompressing gzip response from WebClient
The easiest way to do this is to use the built in automatic decompression with the HttpWebRequest
class.
var request = (HttpWebRequest)HttpWebRequest.Create("http://stackoverflow.com");
request.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate");
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
To do this with a WebClient
you have to make your own class derived from WebClient
and override the GetWebRequest()
method.
public class GZipWebClient : WebClient
{
protected override WebRequest GetWebRequest(Uri address)
{
HttpWebRequest request = (HttpWebRequest)base.GetWebRequest(address);
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
return request;
}
}
Also see this SO thread: Does .NET's HttpWebResponse uncompress automatically GZiped and Deflated responses?
Use Webclient to get page resource in C#
By setting client.Headers["Accept-Encoding"] = "gzip";
you are asking the server to send a compressed response. However, you are not decompressing it. This is causing the incorrect response.
As per https://stackoverflow.com/a/4914874/23633, you can get WebClient
to automatically decompress responses by modifying the HttpWebRequest
it creates:
class MyWebClient : WebClient
{
protected override WebRequest GetWebRequest(Uri address)
{
var request = (HttpWebRequest) base.GetWebRequest(address);
request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
return request;
}
}
string url = "http://www.tsetmc.com/Loader.aspx?ParTree=15131F";
WebClient client = new MyWebClient();
// don't set the Accept-Encoding header here; it will be done automatically
string pageSource = client.DownloadString(url);
Prevent server from encoding in gzip when downloading with WebClient
See this Jeff Atwood post for the solution:
http://www.codinghorror.com/blog/2004/08/netwebclient-and-gzip.html
How to decompress a FluxDataBuffer (and how to write one)?
public class HttpResponseHeadersHandler extends ChannelInboundHandlerAdapter {
private final HttpHeaders httpHeaders;
@Override
public void channelRead(ChannelHandlerContext ctx, Object msg) {
if (msg instanceof HttpResponse &&
!HttpStatus.resolve(((HttpResponse) msg).status().code()).is1xxInformational()) {
HttpHeaders headers = ((HttpResponse) msg).headers();
httpHeaders.forEach(e -> {
log.warn("Modifying {} from: {} to: {}.", e.getKey(), headers.get(e.getKey()), e.getValue());
headers.set(e.getKey(), e.getValue());
});
}
ctx.fireChannelRead(msg);
}
}
Then I create a ClientHttpConnector
to use with WebClient
and in afterNettyContextInit
add the handler:
ctx.addHandlerLast(new ReadTimeoutHandler(readTimeoutMillis, TimeUnit.MILLISECONDS));
ctx.addHandlerLast(new Slf4JLoggingHandler());
if (forceDecompression) {
io.netty.handler.codec.http.HttpHeaders httpHeaders = new ReadOnlyHttpHeaders(
true,
CONTENT_ENCODING, GZIP,
CONTENT_TYPE, APPLICATION_JSON
);
HttpResponseHeadersHandler headersModifier = new HttpResponseHeadersHandler(httpHeaders);
ctx.addHandlerFirst(headersModifier);
}
ctx.addHandlerLast(new HttpContentDecompressor());
This, of course, would fail for responses that are not GZIP compressed, so I use this instance of WebClient
for a particular use case only, where I know for sure that the response is compressed.
Writing is easy: Spring has a ResourceEncoder
, so InputStream
can simply be converted to InputStreamResource
, and voila!
Does .NET's HttpWebResponse uncompress automatically GZiped and Deflated responses?
I found the answer.
You can change the code to:
var request = (HttpWebRequest)HttpWebRequest.Create(requestUri);
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
And you will have automatic decompression. No need to change the rest of the code.
Related Topics
Why Does Timespan.Parseexact Not Work
Determining If File Exists Using C# and Resolving Unc Path
How to Loop Through a List<T> and Grab Each Item
What's a Good Threadsafe Singleton Generic Template Pattern in C#
My C# Application Is Returning 0Xe0434352 to Windows Task Scheduler But It Is Not Crashing
Invoking Methods with Optional Parameters Through Reflection
How to Get the Taskbar's Position and Size
How to Keep Console Window Open
Delete Specific Line from a Text File
What Is the Best Practice for "Copy Local" and with Project References
Creating PDF Files at Runtime in C#
Interfaces VS. Abstract Classes