Preserving Case in Http Headers with Ruby's Net:Http

How to preserve custom headers case in ruby

If at all possible, I would try to change the server, which is violating HTTP standards by treating request header keys as case-sensitive - "Field names are case-insensitive". That error will mess with browsers, caches and so on.

If you can't fix it, I would probably try another HTTP client library that preserves case, not Net::HTTP. Just make sure that library doesn't use Net::HTTP behind the scenes. You could try Excon for example (I'm not sure if it preserves case but it has a lot of low-level control).

How to preserve custom headers case in ruby 2.6.5

Sorry, I needed to patch net/http as we have large existing project and its working with below code for ruby 2.5 and above

module Net::HTTPHeader 
    def capitalize(name)
     name 
    end 
    private :capitalize 
end

How do I preserve case with http.get?

Based on the Tin Man's answer that the Net::HTTP library is calling #downcase on your custom header key (and all header keys), here are some additional options that don't monkey-patch the whole of Net::HTTP.

You could try this:

custom_header_key = "X-miXEd-cASe"
def custom_header_key.downcase
  self
end

To avoid clearing the method cache, either store the result of the above in a class-level constant:

custom_header_key = "X-miXEd-cASe"
def custom_header_key.downcase
  self
end
CUSTOM_HEADER_KEY = custom_header_key

or subclass String to override that particular behavior:

class StringWithIdentityDowncase < String
  def downcase
    self
  end
end

custom_header_key = StringWithIdentityDowncase.new("X-miXEd-cASe")

How do I preserve case with http.get?

Based on the Tin Man's answer that the Net::HTTP library is calling #downcase on your custom header key (and all header keys), here are some additional options that don't monkey-patch the whole of Net::HTTP.

You could try this:

custom_header_key = "X-miXEd-cASe"
def custom_header_key.downcase
  self
end

To avoid clearing the method cache, either store the result of the above in a class-level constant:

custom_header_key = "X-miXEd-cASe"
def custom_header_key.downcase
  self
end
CUSTOM_HEADER_KEY = custom_header_key

or subclass String to override that particular behavior:

class StringWithIdentityDowncase < String
  def downcase
    self
  end
end

custom_header_key = StringWithIdentityDowncase.new("X-miXEd-cASe")

stop ruby http request modifying header name

According to the HTTP spec (RFC 2616), header field names are case-insensitive. So the third-party server has a broken implementation.

If you really needed to, you could monkey-patch Net::HTTP to preserve case, because it downcases the field names when it stores them and then writes them with initial caps.

Here's the storage method you used (Net::HTTPHeader#[]=):

# File net/http.rb, line 1160
    def []=(key, val)
      unless val
        @header.delete key.downcase
        return val
      end
      @header[key.downcase] = [val]
    end

And here is where it writes the header (Net::HTTPGenericRequest#write_header):

# File lib/net/http.rb, line 2071
    def write_header(sock, ver, path)
      buf = "#{@method} #{path} HTTP/#{ver}\r\n"
      each_capitalized do |k,v|
        buf << "#{k}: #{v}\r\n"
      end
      buf << "\r\n"
      sock.write buf
    end

Those are likely the only methods you'd need to override, but I'm not 100% certain.

How to force Ruby to respect underscore in Net::HTTP header

Net::HTTP forces headers to meet the spec with regard to capitalization and punctuation. You can monkey patch it in a variety of ways (depending on the version of Net::HTTP) but those solutions are all fairly old at this point. Regardless, monkey patching third party libraries is a recipe for disaster.

Any client that relies on Net::HTTP, like HTTParty, has the same problem. You can read about some of these workarounds at https://github.com/jnunemaker/httparty/issues/406, but again I don't recommend them.

You can read some more about issues with underscores in HTTP headers at Why is my custom header not present sometimes? and Why do HTTP servers forbid underscores in HTTP header names.

The easier solution is to use typhoeus which wraps libcurl rather than relying on Net::HTTP. Here's the quickest demonstration of how this works in typhoeus:

require 'typhoeus'
request = Typhoeus.get('www.example.com', headers: {'foo_bar' => 'baz'})
=> #<Typhoeus::Response:0x00007fdf3aa717e8 @options={:httpauth_avail=>0, :total_time=>0.336714, :starttransfer_time=>0.336496, :appconnect_time=>0.0, :pretransfer_time=>0.26573, :connect_time=>0.265662, :namelookup_time=>0.00133, :redirect_time=>0.0, :effective_url=>"www.example.com", :primary_ip=>"93.184.216.34", :response_code=>200, :request_size=>129, :redirect_count=>0, :return_code=>:ok, :response_headers=>"HTTP/1.1 200 OK\r\nAge: 459799\r\nCache-Control: max-age=604800\r\nContent-Type: text/html; charset=UTF-8\r\nDate: Tue, 24 Mar 2020 21:29:22 GMT\r\nEtag: \"3147526947+ident\"\r\nExpires: Tue, 31 Mar 2020 21:29:22 GMT\r\nLast-Modified: Thu, 17 Oct 2019 07:18:26 GMT\r\nServer: ECS (ord/4CD5)\r\nVary: Accept-Encoding\r\nX-Cache: HIT\r\nContent-Length: 1256\r\n\r\n", :response_body=>"<!doctype html>\n<html>\n<head>\n    <title>Example Domain</title>\n\n    <meta charset=\"utf-8\" />\n    <meta http-equiv=\"Content-type\" content=\"text/html; charset=utf-8\" />\n    <meta name=\"viewport\" content=\"width=device-width, initial-scale=1\" />\n    <style type=\"text/css\">\n    body {\n        background-color: #f0f0f2;\n        margin: 0;\n        padding: 0;\n        font-family: -apple-system, system-ui, BlinkMacSystemFont, \"Segoe UI\", \"Open Sans\", \"Helvetica Neue\", Helvetica, Arial, sans-serif;\n        \n    }\n    div {\n        width: 600px;\n        margin: 5em auto;\n        padding: 2em;\n        background-color: #fdfdff;\n        border-radius: 0.5em;\n        box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);\n    }\n    a:link, a:visited {\n        color: #38488f;\n        text-decoration: none;\n    }\n    @media (max-width: 700px) {\n        div {\n            margin: 0 auto;\n            width: auto;\n        }\n    }\n    </style>    \n</head>\n\n<body>\n<div>\n    <h1>Example Domain</h1>\n    <p>This domain is for use in illustrative examples in documents. You may use this\n    domain in literature without prior coordination or asking for permission.</p>\n    <p><a href=\"https://www.iana.org/domains/example\">More information...</a></p>\n</div>\n</body>\n</html>\n", :debug_info=>#<Ethon::Easy::DebugInfo:0x00007fdf3b3dbf40 @messages=[]>}, @request=#<Typhoeus::Request:0x00007fdf3aa72c60 @base_url="www.example.com", @original_options={:headers=>{"foo_bar"=>"baz"}, :method=>:get}, @options={:headers=>{"User-Agent"=>"Typhoeus - https://github.com/typhoeus/typhoeus", "foo_bar"=>"baz", "Expect"=>""}, :method=>:get, :maxredirs=>50}, @on_progress=[], @on_headers=[], @response=#<Typhoeus::Response:0x00007fdf3aa717e8 ...>, @on_complete=[], @on_success=[]>>

Then validate that the headers in your request were set properly:

request.request.options[:headers]
=> {
    "User-Agent" => "Typhoeus - https://github.com/typhoeus/typhoeus",
       "foo_bar" => "baz",
        "Expect" => ""
}

But even so, pay attention to the full stack that may be processing these headers as underscores are still at times problematic for various components in the stack.

I answered a similar question once before at https://stackoverflow.com/a/58459132/3784008.

Sending custom HTTP headers with Ruby

Finally figured it out. When setting up an HTTP request, using the 'https' scheme does not automatically enable TLS/SSL. You must do this explicitly before the request starts. Here's my updated version:

#!/usr/bin/env ruby -w
# frozen_string_literal: true

require 'fileutils'
require 'net/http'
require 'time'

cached_response = 'index.html'                                     # Added
FileUtils.touch cached_response unless File.exist? cached_response # Added
uri = URI("https://www.apple.com/#{cached_response}")              # Changed
file = File.stat cached_response

req = Net::HTTP::Get.new(uri)
req['If-Modified-Since'] = file.mtime.rfc2822

http = Net::HTTP.new(uri.hostname, uri.port)                       # Added
http.use_ssl = uri.scheme == 'https'                               # Added
res = http.start { |h| h.request(req) }                            # Changed

if res.is_a?(Net::HTTPSuccess)
  File.open cached_response, 'w' do |io|
    io.write res.body
  end
end

Ruby NET::HTTP Read the header BEFORE the body (without HEAD request)?

net/http supports streaming, you can use this to read the header before the body.

Code example,

url = URI('http://stackoverflow.com/questions/41306082/ruby-nethttp-read-the-header-before-the-body-without-head-request')

Net::HTTP.start(url.host, url.port) do |http|
  request = Net::HTTP::Get.new(url)
  http.request(request) do |response|

    # check headers here, body has not yet been read
    # then call read_body or just body to read the body

    if true  
      response.read_body do |chunk|
        # process body chunks here
      end
    end
  end
end

---

Related Topics