Cannot Set Some Http Headers When Using System.Net.Webrequest

Cannot set some HTTP headers when using System.Net.WebRequest

If you need the short and technical answer go right to the last section of the answer.

If you want to know better, read it all, and i hope you'll enjoy...

I countered this problem too today, and what i discovered today is that:

the above answers are true, as:
1.1 it's telling you that the header you are trying to add already exist and you should then modify its value using the appropriate property (the indexer, for instance), instead of trying to add it again.
1.2 Anytime you're changing the headers of an HttpWebRequest, you need to use the appropriate properties on the object itself, if they exist.

Thanks FOR and Jvenema for the leading guidelines...

But, What i found out, and that was the missing piece in the puzzle is that:
2.1 The WebHeaderCollection class is generally accessed through WebRequest.Headers or WebResponse.Headers. Some common headers are considered restricted and are either exposed directly by the API (such as Content-Type) or protected by the system and cannot be changed.

The restricted headers are:

Accept
Connection
Content-Length
Content-Type
Date
Expect
Host
If-Modified-Since
Range
Referer
Transfer-Encoding
User-Agent
Proxy-Connection

So, next time you are facing this exception and don't know how to solve this, remember that there are some restricted headers, and the solution is to modify their values using the appropriate property explicitly from the WebRequest/HttpWebRequest class.

Edit: (useful, from comments, comment by user Kaido)

Solution is to check if the header exists already or is restricted (WebHeaderCollection.IsRestricted(key)) before calling add

How to correctly add WebRequest header?

You need to set UserAgent directly, use CreateHttp instead which returns HttpWebRequest, then you can do:

webRequest.UserAgent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36";

UserAgent is a restricted header and must be set via a property, there is a list of all restricted headers here.

Strict ordering of HTTP headers in HttpWebrequest

.Net Core

If you set the headers yourself, you can specify the order. When the common headers are added it will find the existing headers instead of appending them:

using System.Net;

namespace ConsoleApp2
{
    class Program
    {
        static void Main(string[] args)
        {
            var request = WebRequest.Create("http://www.google.com");
            request.Headers.Add("Host", "www.google.com");
            // this will be set within GetResponse.
            request.Headers.Add("Connection", "");
            request.Headers.Add("Accept", "*/*");
            request.Headers.Add("User-Agent", "Mozilla/5.0 etc");
            request.GetResponse();
        }
    }
}

Sample Image

Here is an example with HttpClient:

using System.Net.Http;
using System.Threading.Tasks;

namespace ConsoleApp3
{
    class Program
    {
        static async Task Main(string[] args)
        {
            var client = new HttpClient();
            client.DefaultRequestHeaders.Add("Host", "www.google.com");
            client.DefaultRequestHeaders.Add("Connection", "keep-alive");
            client.DefaultRequestHeaders.Add("Accept", "*/*");
            client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 etc");
            await client.GetAsync("http://www.google.com");
            await client.PostAsync("http://www.google.com", new StringContent(""));
        }
    }
}

GET with ordered headers
POST with ordered headers

Edit
The above code did not work on .Net Framework only .Net Core

.Net Framework

On .Net Framework the headers are reserved so they cannot be set like this, see Cannot set some HTTP headers when using System.Net.WebRequest.

One work around is to use reflection to modify the behavior of the framework class, but be warned this could break if the libraries are updated so it's not recommended!.

Essentially, HttpWebRequest calls ToString on WebHeaderCollection to serialize.
See https://referencesource.microsoft.com/#System/net/System/Net/HttpWebRequest.cs,5079

So a custom class can be made to override ToString. Unfortunately reflection is needed to set the headers as WebRequest copies the collection on assignment to Headers, instead of taking the new reference.

WARNING, THE FOLLOWING CODE CAN BREAK IF FRAMEWORK CHANGES

If you use this, write some unit tests that verify the behavior still stays consistent after updates to .NET Framework

using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Reflection;

namespace ConsoleApp2
{
    class Program
    {
        static void Main(string[] args)
        {
            // WARNING, CODE CAN BREAK IF FRAMEWORK CHANGES
            // If you use this, write some unit tests that verify the behavior still stays consistent after updates to .NET Framework
            var request = (HttpWebRequest)WebRequest.Create("http://www.google.com");
            var field = typeof(HttpWebRequest).GetField("_HttpRequestHeaders", BindingFlags.Instance | BindingFlags.NonPublic);
            var headers = new CustomWebHeaderCollection(new Dictionary<string, string>
            {
                ["Host"] = "www.google.com",
                ["Connection"] = "keep-alive",
                ["Accept"] = "*/*",
                ["User-Agent"] = "Mozilla/5.0 etc"
            });
            field.SetValue(request, headers);
            request.GetResponse();
        }
    }

    internal class CustomWebHeaderCollection : WebHeaderCollection
    {
        private readonly Dictionary<string, string> _customHeaders;

        public CustomWebHeaderCollection(Dictionary<string, string> customHeaders)
        {
            _customHeaders = customHeaders;
        }

        public override string ToString()
        {
            // Could call base.ToString() split on Newline and sort as needed

            var lines = _customHeaders
                .Select(kvp => $"{kvp.Key}: {kvp.Value}")
                // These two new lines are needed after the HTTP header
                .Concat(new [] { string.Empty, string.Empty });

            var headers = string.Join("\r\n", lines);

            return headers;
        }
    }
}

Sample Image

Is it possible to change headers order using HttpWebRequest?

There was an outstanding complaint that .NET doesn't let you modify the Host header a while back. It might not have been resolved. If it is really that important, you could always write socket-level code to send a prepared request (since it's just text).

Add custom header in HttpWebRequest

You use the Headers property with a string index:

request.Headers["X-My-Custom-Header"] = "the-value";

According to MSDN, this has been available since:

Universal Windows Platform 4.5
.NET Framework 1.1
Portable Class Library
Silverlight 2.0
Windows Phone Silverlight 7.0
Windows Phone 8.1

https://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.headers(v=vs.110).aspx

How can I set Headers with Silverlight GET HttpWebRequest?

Silverlight only supports setting headers using the POST method not the GET method. This is due to a limitation in how the TCP/IP stack is implemented in Silverlight. It uses the browser extension APIs instead of going directly against the host OS's APIs.

Custom headers in WebRequest NET Core 3.1

Let's take a little tour at the related source code. In System.txt there is a row:

net_WebHeaderInvalidHeaderChars=Specified value has invalid HTTP Header characters.

That means we should look for this net_WebHeaderInvalidHeaderChars key in the source code of WebHeaderCollection:

//
// CheckBadChars - throws on invalid chars to be not found in header name/value
//
internal static string CheckBadChars(string name, bool isHeaderValue) {

    ...

    if (isHeaderValue) {
        ...
    }
    else {
        // NAME check
        //First, check for absence of separators and spaces
        if (name.IndexOfAny(ValidationHelper.InvalidParamChars) != -1) {
            throw new ArgumentException(SR.GetString(SR.net_WebHeaderInvalidHeaderChars), "name");
        }
        
        ...
    }
    return name;
}

This means that the error is thrown if the provided name contains some invalid characters.

The InvalidParamChars are defined inside the Internal class like this:

internal static readonly char[]  InvalidParamChars =
    new char[]{
    '(',
    ')',
    '<',
    '>',
    '@',
    ',',
    ';',
    ':',
    '\\',
    '"',
    '\'',
    '/',
    '[',
    ']',
    '?',
    '=',
    '{',
    '}',
    ' ',
    '\t',
    '\r',
    '\n'};

So, all you have to do is to make sure that the request header name does not contain any of the not allowed characters.

How to set User Agent with System.Net.WebRequest in c#

Use the UserAgent property on HttpWebRequest by casting it to a HttpWebRequest.

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.UserAgent = "my user agent";

Alternatively, instead of casting you can use WebRequest.CreateHttp instead.

How to set custom Host header in HttpWebRequest?

There is a roundabout way to do this, as described here:

http://blogs.msdn.com/feroze_daud/archive/2005/03/31/404328.aspx

However, the next version of the framework (.NET Framework 4.0) will make it easier.

http://blogs.msdn.com/ncl/archive/2009/07/20/new-ncl-features-in-net-4-0-beta-2.aspx

Hope this helps.

Cannot Set Some Http Headers When Using System.Net.Webrequest