Decompressing Gzip Stream from Httpclient Response

Decompressing GZip Stream from HTTPClient Response

Just instantiate HttpClient like this:

HttpClientHandler handler = new HttpClientHandler()
{
    AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};

using (var client = new HttpClient(handler)) //see update below
{
    // your code
}

Update June 19, 2020:
It's not recommended to use httpclient in a 'using' block as it might cause port exhaustion.

private static HttpClient client = null;
    
ContructorMethod()
{
   if(client == null)
   {
        HttpClientHandler handler = new HttpClientHandler()
        {
            AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
        };        
        client = new HttpClient(handler);
   }
// your code            
 }

If using .Net Core 2.1+, consider using IHttpClientFactory and injecting like this in your startup code.

 var timeout = Policy.TimeoutAsync<HttpResponseMessage>(
            TimeSpan.FromSeconds(60));

 services.AddHttpClient<XApiClient>().ConfigurePrimaryHttpMessageHandler(() => new HttpClientHandler
        {
            AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
        }).AddPolicyHandler(request => timeout);

Decompressing (Gzip) chunks of response from http.client call

The most probable reason for that is, the response your received is indeed not a gzipped file.

I notice that in your code, you pass a variable called auth. Typically, a server won't send you a compressed response if you don't specify in the request headers that you can accept it. If there is only auth-related keys in your headers like your variable name suggests, you won't receive a gzipped response. First, make sure you have 'Accept-Encoding': 'gzip' in your headers.

Going forward, you will face another problem:

Basically, I am chunking so that I don't have to load the entire response in memory.

gzip.decompress will expect a complete file, so you would need to reconstruct it and load it entirely in memory before doing that, which would undermine the whole point of chunking the response. Trying to decompress a part of a gzip with gzip.decompress will most likely give you an EOFError saying something like Compressed file ended before the end-of-stream marker was reached.

I don't know if you can manage that directly with the gzip library, but I know how to do it with zlib. You will also need to convert your chunk to a file-like object, you can do that with io.BytesIO. I see you have very strong constraints on libraries, but zlib and io are part of the python default, so hopefully you have them available. Here is a rework of your code that should help you going on:

import http
import ssl
import gzip
import zlib
from io import BytesIO

# your variables here
api = 'your_api_host'
api_url = 'your_api_endpoint'
auth = {'AuhtKeys': 'auth_values'}

# add the gzip header
auth['Accept-Encoding'] = 'gzip'

# prepare decompressing object
decompressor = zlib.decompressobj(16 + zlib.MAX_WBITS)

connection = http.client.HTTPSConnection(api, context = ssl._create_unverified_context())
connection.request('GET', api_url, headers = auth)
response = connection.getresponse()

while chunk := response.read(20):
    data = decompressor.decompress(BytesIO(chunk).read())
    print(data)

How Decompress Gzipped Http Get Response in c#

Just Change My Function as Follows,Which is Perfectly Working For me:

private JObject PostingToPKFAndDecompress(string sData, string sUrl)
        {
            var jOBj = new JObject();
            try
            {

                try
                {
                    string urlStr = @"" + sUrl + "?param=" + sData;


                    HttpWebRequest request = (HttpWebRequest)WebRequest.Create(urlStr);
                    HttpWebResponse response = (HttpWebResponse)request.GetResponse();
                    Stream resStream = response.GetResponseStream();

                    var t = ReadFully(resStream);
                    var y = Decompress(t);

                    using (var ms = new MemoryStream(y))
                    using (var streamReader = new StreamReader(ms))
                    using (var jsonReader = new JsonTextReader(streamReader))
                    {
                        jOBj = (JObject)JToken.ReadFrom(jsonReader);
                    }


                }
                catch (System.Net.Sockets.SocketException)
                {
                    // The remote site is currently down. Try again next time. 
                }

            }
            catch (Exception ex)
            {
                throw new Exception(ex.ToString());
            }
            return jOBj;
        }

        public static byte[] ReadFully(Stream input)
        {
            byte[] buffer = new byte[16 * 1024];
            using (MemoryStream ms = new MemoryStream())
            {
                int read;
                while ((read = input.Read(buffer, 0, buffer.Length)) > 0)
                {
                    ms.Write(buffer, 0, read);
                }
                return ms.ToArray();
            }
        }

        public static byte[] Decompress(byte[] data)
        {
            using (var compressedStream = new MemoryStream(data))
            using (var zipStream = new GZipStream(compressedStream, CompressionMode.Decompress))
            using (var resultStream = new MemoryStream())
            {
                zipStream.CopyTo(resultStream);
                return resultStream.ToArray();
            }
        }

HttpResponseMessage - Decompression of Gzip respone from an API Call throws the following exceptiong - The response ended prematurely

Try adding HttpCompletionOption.ResponseContentRead to your GetAsync call:

var result = await httpClient.GetAsync(url, HttpCompletionOption.ResponseContentRead);

This should make your code wait until the entire response has been read.

HttpClient Gzip Compression

The problem was due to the IP of the Azure server not being on the API whitelist therefore it was returning a 403 forbidden error in the response.

decompressing a gzip encoded response

Probably you should copy GZip decompressed stream and not the memory one:

using (var stram = new GZipStream(stream, CompressionMode.Decompress ))
using (var file = File.Create("../../../Downloads/login_result.txt"))
{
    stram.CopyTo(file); //stream.CopyTo(file);
}

Decompressing GZIP stream

Try

public static string UnZip(string value)
{
    // Removing brackets from string
    value = value.TrimStart('[');
    value = value.TrimEnd(']');

    //Transform string into byte[]
    string[] strArray = value.Split(',');
    byte[] byteArray = new byte[strArray.Length];
    for (int i = 0; i < strArray.Length; i++)
    {
        byteArray[i] = unchecked((byte)Convert.ToSByte(strArray[i]));
    }

    //Prepare for decompress
    using (System.IO.MemoryStream output = new System.IO.MemoryStream())
    {
        using (System.IO.MemoryStream ms = new System.IO.MemoryStream(byteArray))
        using (System.IO.Compression.GZipStream sr = new System.IO.Compression.GZipStream(ms, System.IO.Compression.CompressionMode.Decompress))
        {
            sr.CopyTo(output);
        }

        string str = Encoding.UTF8.GetString(output.GetBuffer(), 0, (int)output.Length);
        return str;
    }
}

The MemoryBuffer() doesn't "duplicate" the byteArray but is directly backed by it, so you can't reuse the byteArray.

I'll add that I find funny that they "compressed" a json of 277 characters to a stringized byte array of 620 characters.

As a sidenote, the memory occupation of this method is out-of-the-roof... The 620 character string (that in truth is a 277 byte array) to be decompressed causes the creation of strings/arrays for a total size of 4887 bytes (including the 620 initial character string) (disclaimer: the GC can reclaim part of this memory during the execution of the method). This is ok for byte arrays of 277 bytes... But for bigger ones the memory occupation will become quite big.

---

Related Topics