Compress Http Get Response

Compress HTTP GET Response

The easiest is to enable compression directly at IIS level.

If you want to do it at the application level you could write a custom delegating message handler as shown in the following post:

public class CompressHandler : DelegatingHandler
{
    protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        return base.SendAsync(request, cancellationToken).ContinueWith<HttpResponseMessage>((responseToCompleteTask) =>
        {
            HttpResponseMessage response = responseToCompleteTask.Result;

            if (response.RequestMessage.Headers.AcceptEncoding != null)
            {
                string encodingType = response.RequestMessage.Headers.AcceptEncoding.First().Value;

                response.Content = new CompressedContent(response.Content, encodingType);
            }

            return response;
        },
        TaskContinuationOptions.OnlyOnRanToCompletion);
    }
}

public class CompressedContent : HttpContent
{
    private HttpContent originalContent;
    private string encodingType;

    public CompressedContent(HttpContent content, string encodingType)
    {
        if (content == null)
        {
            throw new ArgumentNullException("content");
        }

        if (encodingType == null)
        {
            throw new ArgumentNullException("encodingType");
        }

        originalContent = content;
        this.encodingType = encodingType.ToLowerInvariant();

        if (this.encodingType != "gzip" && this.encodingType != "deflate")
        {
            throw new InvalidOperationException(string.Format("Encoding '{0}' is not supported. Only supports gzip or deflate encoding.", this.encodingType));
        }

        // copy the headers from the original content
        foreach (KeyValuePair<string, IEnumerable<string>> header in originalContent.Headers)
        {
            this.Headers.AddWithoutValidation(header.Key, header.Value);
        }

        this.Headers.ContentEncoding.Add(encodingType);
    }

    protected override bool TryComputeLength(out long length)
    {
        length = -1;

        return false;
    }

    protected override Task SerializeToStreamAsync(Stream stream, TransportContext context)
    {
        Stream compressedStream = null;

        if (encodingType == "gzip")
        {
            compressedStream = new GZipStream(stream, CompressionMode.Compress, leaveOpen: true);
        }
        else if (encodingType == "deflate")
        {
            compressedStream = new DeflateStream(stream, CompressionMode.Compress, leaveOpen: true);
        }

        return originalContent.CopyToAsync(compressedStream).ContinueWith(tsk =>
        {
            if (compressedStream != null)
            {
                compressedStream.Dispose();
            }
        });
    }
}

All that's left now is to register the handler in Application_Start:

GlobalConfiguration.Configuration.MessageHandlers.Add(new CompressHandler());

HTTP request compression

It appears [Content-Encoding] is not a valid request header.

That is actually not quite true. As per RFC 2616, sec 14.11, Content-Encoding is an entity header which means it can be applied on the entities of both, http responses and requests. Through the powers of multipart MIME messages, even selected parts of a request (or response) can be compressed.

However, webserver support for compressed request bodies is rather slim. Apache supports it to a degree via the mod_deflate module. It's not entirely clear to me if nginx can handle compressed requests.

How to handle compressed (gzip) HTTP requests (NOT response) in JAVA servlet - Simple Example?

A simple solution is by using a filter. (See servlet-filter tutorial)

Create a Servlet Filter:

• Make sure that the filter is called either first/before any filters which use request body.

I. Register filter in web.xml:

<filter>
    <filter-name>GzipRequestFilter</filter-name>
    <filter-class>com...pkg...GzipRequestFilter</filter-class>
</filter>
<filter-mapping>
    <filter-name>GzipRequestFilter</filter-name>
    <url-pattern>/*</url-pattern>
</filter-mapping>

---

II. Code for filter class:

public class GzipRequestFilter implements Filter {
    // Optional but recommended.
    private static final Set<String> METHODS_TO_IGNORE = ImmutableSet.of("GET", "OPTIONS", "HEAD");

    @Override
    public void doFilter(
            final ServletRequest request,
            final ServletResponse response,
            final FilterChain chain) throws IOException, ServletException {
        HttpServletRequest httpServletRequest = (HttpServletRequest) request;
        
        String method = httpServletRequest.getMethod().toUpperCase();
        String encoding = Strings.nullToEmpty(
            httpServletRequest.getHeader(HttpHeaders.CONTENT_ENCODING));

        if (METHODS_TO_IGNORE.contains(method) || !encoding.contains("application/gzip")) {
            chain.doFilter(request, response); // pass through
            return;
        }
        
        HttpServletRequestWrapper requestInflated = new GzippedInputStreamWrapper(httpServletRequest);
        chain.doFilter(requestInflated, response);
    }

    @Override
    public void init(final FilterConfig filterConfig) throws ServletException {}
    @Override
    public void destroy() {}
}

---

III. Followed by code for GzipInputStream wrapper:

// Simple Wrapper class to inflate body of a gzipped HttpServletRequest.
final class GzippedInputStreamWrapper extends HttpServletRequestWrapper {
    private GZIPInputStream inputStream;

    GzippedInputStreamWrapper(final HttpServletRequest request) throws IOException {
        super(request);
        inputStream = new GZIPInputStream(request.getInputStream());
    }

    @Override
    public ServletInputStream getInputStream() throws IOException {
        return new ServletInputStream() {
            // NOTE: Later versions of javax.servlet library may require more overrides.
            public int read() throws IOException {
                return inputStream.read();
            }
            public void close() throws IOException {
                super.close();
                inputStream.close();
            }
        };
    }

    @Override
    public BufferedReader getReader() throws IOException {
        return new BufferedReader(new InputStreamReader(inputStream));
    }
}

---

Now what remains is how to send a compressed request?

Postman does not yet support sending compressed HttpRequest bodies. You can still make it work by using the binary option and use a gzipped file containing the properly encoded request body.

One way is using a nodejs script with pako compression library. For a multipart/form-data request see form-data library

const pako = require('pako')
const axios = require('axios')

var params = qs.stringify({
  'num': 42,
  'str': 'A string param',
});

data = pako.gzip(Buffer.from(params));

var config = {
  headers: {
    'Content-Type': 'application/x-www-form-urlencoded',
    'Content-Encoding': 'application/gzip';
  },
}

await axios.post(
  'http://url-for-post-api-accepting-urlencoded',
  data,
  config,
).then((res) => {
  console.log(`status: ${res.status} | data: ${res.data}`)
}).catch((error) => {
  console.error(error)
})

---

NOTES:

• We are using Content-Encoding: application/gzip header to specify that a request is compressed. Yes this is standard.
• Do not use Content-Type as it will not work with multipart/form-data.
• The HTTP protocol has been running under the assumption that size of HttpRequests are dwarfed by HttpResponses.
• Further, due to assumed limited computing power in browser/client side, the norm has been to compress response and not requests. Browsers cannot natively compress but can do decompression natively.
• But, unfortunately after years of many developers pushing code; some HTTP APIs evolve to consume large strings/data!!
• It's a piece of cake to allow java servlets to have the option of working with compressed requests.

How Decompress Gzipped Http Get Response in c#

Just Change My Function as Follows,Which is Perfectly Working For me:

private JObject PostingToPKFAndDecompress(string sData, string sUrl)
        {
            var jOBj = new JObject();
            try
            {

                try
                {
                    string urlStr = @"" + sUrl + "?param=" + sData;

                    HttpWebRequest request = (HttpWebRequest)WebRequest.Create(urlStr);
                    HttpWebResponse response = (HttpWebResponse)request.GetResponse();
                    Stream resStream = response.GetResponseStream();

                    var t = ReadFully(resStream);
                    var y = Decompress(t);

                    using (var ms = new MemoryStream(y))
                    using (var streamReader = new StreamReader(ms))
                    using (var jsonReader = new JsonTextReader(streamReader))
                    {
                        jOBj = (JObject)JToken.ReadFrom(jsonReader);
                    }

                }
                catch (System.Net.Sockets.SocketException)
                {
                    // The remote site is currently down. Try again next time. 
                }

            }
            catch (Exception ex)
            {
                throw new Exception(ex.ToString());
            }
            return jOBj;
        }

        public static byte[] ReadFully(Stream input)
        {
            byte[] buffer = new byte[16 * 1024];
            using (MemoryStream ms = new MemoryStream())
            {
                int read;
                while ((read = input.Read(buffer, 0, buffer.Length)) > 0)
                {
                    ms.Write(buffer, 0, read);
                }
                return ms.ToArray();
            }
        }

        public static byte[] Decompress(byte[] data)
        {
            using (var compressedStream = new MemoryStream(data))
            using (var zipStream = new GZipStream(compressedStream, CompressionMode.Decompress))
            using (var resultStream = new MemoryStream())
            {
                zipStream.CopyTo(resultStream);
                return resultStream.ToArray();
            }
        }

What is the canonical method for an HTTP client to instruct an HTTP server to disable gzip responses?

Do you wish encoding to be disabled altogether?
Then skip the Accept-Encoding header itself within http request headers.

Do you wish only gzip compression to be absent in the http response?
Then skip gzip from the values list in the http request header.

Do you wish to prioritize different compression techniques that servers support? then use different values between 0 and 1 along-with q argument for each value in the Accept-Encoding http request header. (Currently you are using conflicting value and indicating by weight=0 that you don't know how you'll manage, but you want response to be encoded anyhow)

How to do GZIP compression while writing directly to response

Just realised I was not closing the writer.
After adding fwriter.close(); in the end, the problem was solved.

Compressing HTTP Post Data sent from browser

Will the browser automatically gzip-encode your data for you? The short answer is no.

The long answer is that some user-agents can do things like this, but you definitely can't rely on it. The apache mod_deflate docs state:

some special applications actually do support request compression, for instance some WebDAV clients.

So, no, that's not going to work. You'll need to generate the appropriate HTTP request message yourself. The appropriate header in this case is Content-Encoding: gzip and NOT Content-Type: because the content itself is application/json, you're just looking to encode the entity body of your HTTP request message for transport.

Note that you need to also add the appropriate Content-Length: header specifying the size in bytes of the message entity body after compression -OR- send your HTTP message using Transfer-Encoding: chunked and forego the content-length specification.

On the receiving end, you can instruct mod_deflate to use an input filter to decompress the information:

<Location /dav-area>
SetInputFilter DEFLATE
</Location>

This is a bit heavy handed if you're only receiving compressed message bodies for a couple of resources. Instead, you should probably just use the client-side script to check for the Content-Encoding: gzip header and decompress the request body manually. How to do this in say, PHP, is another question entirely. If you need details for that you should post another question.

How to keep the gzip representation of HTTP body

First of all, I continued using the gzip and gin packages.

Now, I compress my responses with a function similar to this:

func compress(response interface{}) ([]byte, error) {
    body, err := json.Marshal(response)
    if err != nil {
        return nil, err
    }
    var buffer bytes.Buffer
    zw := gzip.NewWriter(&buffer)
    _, err = zw.Write(body)
    closeErr := zw.Close()
    if err != nil {
        return nil, err
    }
    if closeErr != nil {
        return nil, err
    }
    return buffer.Bytes(), nil
}

When I need to respond to a request, I execute something like this:

    gzipOutput, err := compressResponse(output)
    if err != nil {
        c.JSON(http.StatusInternalServerError, err.Error())
        return
    }

    c.Writer.Header().Set("Accept-Encoding", "gzip")
    c.Writer.Header().Set("Content-Encoding", "gzip")
    c.Writer.Header().Set("Content-Type", "application/json")
    c.Data(http.StatusOK, "gzip", gzipOutput)

As we can see, the idea is to tell gin that the response is compressed by setting the HTML header.

It has been tested on five different APIs since eight months.

I hope it will be useful to another just like it was for me.

WebAPI Gzip when returning HttpResponseMessage

If you have access to IIS configuration

You cant just apply the header and hope it will be gzipped - the response will not be zipped.

You need remove the header you added and ensure you have the dynamic compression and static content compression are enabled on your IIS server.

One of the commenter's mentioned a good resource link here at stakoverflow that show how to do that:

Enable IIS7 gzip

Note it will only work setting the value in web.config if dynamic compression is already installed (which is not in a default install of IIS)

You can find the information about this on MSDN documentation: http://www.iis.net/configreference/system.webserver/httpcompression

Simple compression

Below is using a simple example of doing your own compression this example is using the Web Api MVC 4 project from visual studio project templates. To get compression working for HttpResponseMessages you have to implement a custom MessageHandler. See below a working example.

See the code implementation below.

Please note that I tried to keep the method doing the same as your example.

using System;
using System.Collections.Generic;
using System.IO;
using System.IO.Compression;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using System.Web;
using System.Web.Http;

namespace MvcApplication1.Controllers
{
    public class ValuesController : ApiController
    {
        public class Person
        {
            public string name { get; set; }
        }
        // GET api/values
        public IEnumerable<string> Get()
        {
            HttpContext.Current.Response.Cache.VaryByHeaders["accept-encoding"] = true;

            return new [] { "value1", "value2" };
        }

        // GET api/values/5
        public HttpResponseMessage Get(int id)
        {
            HttpContext.Current.Response.Cache.VaryByHeaders["accept-encoding"] = true;

            var TheHTTPResponse = new HttpResponseMessage(System.Net.HttpStatusCode.OK); 
            TheHTTPResponse.Content = new StringContent("{\"asdasdasdsadsad\": 123123123 }", Encoding.UTF8, "text/json"); 

            return TheHTTPResponse;
        }

        public class EncodingDelegateHandler : DelegatingHandler
        {
            protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
            {
                return base.SendAsync(request, cancellationToken).ContinueWith<HttpResponseMessage>((responseToCompleteTask) =>
                {
                    HttpResponseMessage response = responseToCompleteTask.Result;

                    if (response.RequestMessage.Headers.AcceptEncoding != null &&
                        response.RequestMessage.Headers.AcceptEncoding.Count > 0)
                    {
                        string encodingType = response.RequestMessage.Headers.AcceptEncoding.First().Value;

                        response.Content = new CompressedContent(response.Content, encodingType);
                    }

                    return response;
                },
                TaskContinuationOptions.OnlyOnRanToCompletion);
            }
        }

        public class CompressedContent : HttpContent
        {
            private HttpContent originalContent;
            private string encodingType;

            public CompressedContent(HttpContent content, string encodingType)
            {
                if (content == null)
                {
                    throw new ArgumentNullException("content");
                }

                if (encodingType == null)
                {
                    throw new ArgumentNullException("encodingType");
                }

                originalContent = content;
                this.encodingType = encodingType.ToLowerInvariant();

                if (this.encodingType != "gzip" && this.encodingType != "deflate")
                {
                    throw new InvalidOperationException(string.Format("Encoding '{0}' is not supported. Only supports gzip or deflate encoding.", this.encodingType));
                }

                // copy the headers from the original content
                foreach (KeyValuePair<string, IEnumerable<string>> header in originalContent.Headers)
                {
                    this.Headers.TryAddWithoutValidation(header.Key, header.Value);
                }

                this.Headers.ContentEncoding.Add(encodingType);
            }

            protected override bool TryComputeLength(out long length)
            {
                length = -1;

                return false;
            }

            protected override Task SerializeToStreamAsync(Stream stream, TransportContext context)
            {
                Stream compressedStream = null;

                if (encodingType == "gzip")
                {
                    compressedStream = new GZipStream(stream, CompressionMode.Compress, leaveOpen: true);
                }
                else if (encodingType == "deflate")
                {
                    compressedStream = new DeflateStream(stream, CompressionMode.Compress, leaveOpen: true);
                }

                return originalContent.CopyToAsync(compressedStream).ContinueWith(tsk =>
                {
                    if (compressedStream != null)
                    {
                        compressedStream.Dispose();
                    }
                });
            }
        }
    }
}

Also add the new message handler to the config of your app.

using System.Web.Http;
using MvcApplication1.Controllers;

namespace MvcApplication1
{
    public static class WebApiConfig
    {
        public static void Register(HttpConfiguration config)
        {
            config.Routes.MapHttpRoute(
                name: "DefaultApi",
                routeTemplate: "api/{controller}/{id}",
                defaults: new { id = RouteParameter.Optional }
            );

            config.MessageHandlers.Add(new ValuesController.EncodingDelegateHandler());

            config.EnableSystemDiagnosticsTracing();
        }
    }
}

The Custom handler was put together by - Kiran Challa (http://blogs.msdn.com/b/kiranchalla/archive/2012/09/04/handling-compression-accept-encoding-sample.aspx)

There are better examples that implement deflating of inbound streams too you can see examples of that below:

• http://www.codeproject.com/Articles/557232/Implementing-a-Custom-DelegatingHandler-in-ASP-NET
• http://ronaldrosiernet.azurewebsites.net/blog/2013/07/16/implement_compression_in_aspnet_web_api

Additionally I found a really nice project that supports all of this on github.

• https://github.com/azzlack/Microsoft.AspNet.WebApi.MessageHandlers.Compression

Note while I arrived to this answer by myself Simon in your comments suggested this approach 2 days ago from the date of this answer.

---

Related Topics