Parse raw HTTP Headers
There are excellent tools in the Standard Library both for parsing RFC 821 headers, and also for parsing entire HTTP requests. Here is an example request string (note that Python treats it as one big string, even though we are breaking it across several lines for readability) that we can feed to my examples:Update: It’s 2019, so I have rewritten this answer for Python 3, following a confused comment from a programmer trying to use the code. The original Python 2 code is now down at the bottom of the answer.
request_text = (
b'GET /who/ken/trust.html HTTP/1.1\r\n'
b'Host: cm.bell-labs.com\r\n'
b'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3\r\n'
b'Accept: text/html;q=0.9,text/plain\r\n'
b'\r\n'
)
As @TryPyPy points out, you can use Python’s email message library to parse the headers — though we should add that the resulting Message
object acts like a dictionary of headers once you are done creating it:from email.parser import BytesParser
request_line, headers_alone = request_text.split(b'\r\n', 1)
headers = BytesParser().parsebytes(headers_alone)
print(len(headers)) # -> "3"
print(headers.keys()) # -> ['Host', 'Accept-Charset', 'Accept']
print(headers['Host']) # -> "cm.bell-labs.com"
But this, of course, ignores the request line, or makes you parse it yourself. It turns out that there is a much better solution.The Standard Library will parse HTTP for you if you use its BaseHTTPRequestHandler
. Though its documentation is a bit obscure — a problem with the whole suite of HTTP and URL tools in the Standard Library — all you have to do to make it parse a string is (a) wrap your string in a BytesIO()
, (b) read the raw_requestline
so that it stands ready to be parsed, and (c) capture any error codes that occur during parsing instead of letting it try to write them back to the client (since we do not have one!).
So here is our specialization of the Standard Library class:
from http.server import BaseHTTPRequestHandler
from io import BytesIO
class HTTPRequest(BaseHTTPRequestHandler):
def __init__(self, request_text):
self.rfile = BytesIO(request_text)
self.raw_requestline = self.rfile.readline()
self.error_code = self.error_message = None
self.parse_request()
def send_error(self, code, message):
self.error_code = code
self.error_message = message
Again, I wish the Standard Library folks had realized that HTTP parsing should be broken out in a way that did not require us to write nine lines of code to properly call it, but what can you do? Here is how you would use this simple class:# Using this new class is really easy!
request = HTTPRequest(request_text)
print(request.error_code) # None (check this first)
print(request.command) # "GET"
print(request.path) # "/who/ken/trust.html"
print(request.request_version) # "HTTP/1.1"
print(len(request.headers)) # 3
print(request.headers.keys()) # ['Host', 'Accept-Charset', 'Accept']
print(request.headers['host']) # "cm.bell-labs.com"
If there is an error during parsing, the error_code
will not be None
:# Parsing can result in an error code and message
request = HTTPRequest(b'GET\r\nHeader: Value\r\n\r\n')
print(request.error_code) # 400
print(request.error_message) # "Bad request syntax ('GET')"
I prefer using the Standard Library like this because I suspect that they have already encountered and resolved any edge cases that might bite me if I try re-implementing an Internet specification myself with regular expressions.Old Python 2 code
Here’s the original code for this answer, back when I first wrote it:
request_text = (
'GET /who/ken/trust.html HTTP/1.1\r\n'
'Host: cm.bell-labs.com\r\n'
'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3\r\n'
'Accept: text/html;q=0.9,text/plain\r\n'
'\r\n'
)
And:# Ignore the request line and parse only the headers
from mimetools import Message
from StringIO import StringIO
request_line, headers_alone = request_text.split('\r\n', 1)
headers = Message(StringIO(headers_alone))
print len(headers) # -> "3"
print headers.keys() # -> ['accept-charset', 'host', 'accept']
print headers['Host'] # -> "cm.bell-labs.com"
And:from BaseHTTPServer import BaseHTTPRequestHandler
from StringIO import StringIO
class HTTPRequest(BaseHTTPRequestHandler):
def __init__(self, request_text):
self.rfile = StringIO(request_text)
self.raw_requestline = self.rfile.readline()
self.error_code = self.error_message = None
self.parse_request()
def send_error(self, code, message):
self.error_code = code
self.error_message = message
And:# Using this new class is really easy!
request = HTTPRequest(request_text)
print request.error_code # None (check this first)
print request.command # "GET"
print request.path # "/who/ken/trust.html"
print request.request_version # "HTTP/1.1"
print len(request.headers) # 3
print request.headers.keys() # ['accept-charset', 'host', 'accept']
print request.headers['host'] # "cm.bell-labs.com"
And:# Parsing can result in an error code and message
request = HTTPRequest('GET\r\nHeader: Value\r\n\r\n')
print request.error_code # 400
print request.error_message # "Bad request syntax ('GET')"
How to parse raw HTTP request in Python 3?
You could use the email.message.Message
class from the email
module in the standard library.
By modifying the answer from the question you linked, below is a Python3 example of parsing HTTP headers.
Suppose you wanted to create a dictionary containing all of your header fields:
import email
import pprint
from io import StringIO
request_string = 'GET / HTTP/1.1\r\nHost: localhost\r\nConnection: keep-alive\r\nCache-Control: max-age=0\r\nUpgrade-Insecure-Requests: 1\r\nUser-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\r\nAccept-Encoding: gzip, deflate, sdch\r\nAccept-Language: en-US,en;q=0.8'
# pop the first line so we only process headers
_, headers = request_string.split('\r\n', 1)
# construct a message from the request string
message = email.message_from_file(StringIO(headers))
# construct a dictionary containing the headers
headers = dict(message.items())
# pretty-print the dictionary of headers
pprint.pprint(headers, width=160)
if you ran this at a python prompt, the result would look like:{'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'Accept-Encoding': 'gzip, deflate, sdch',
'Accept-Language': 'en-US,en;q=0.8',
'Cache-Control': 'max-age=0',
'Connection': 'keep-alive',
'Host': 'localhost',
'Upgrade-Insecure-Requests': '1',
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36'}
Parsing raw HTTP Request
I [am] working on [an] HTTP Traffic Data set which is composed of complete POST and GET request[s]So you want to parse a file or list that contains multiple HTTP requests. What data do you want to extract? Anyway here is a Java HTTP parsing class, which can read the method, version and URI used in the request-line, and that reads all headers into a Hashtable.
You can use that one or write one yourself if you feel like reinventing the wheel. Take a look at the RFC to see what a request looks like in order to parse it correctly:
Request = Request-Line ; Section 5.1
*(( general-header ; Section 4.5
| request-header ; Section 5.3
| entity-header ) CRLF) ; Section 7.1
CRLF
[ message-body ] ; Section 4.3
How can I parse a raw HTTP response in .net?
I ended up using this Nuget package:
https://www.nuget.org/packages/HttpMachine.PCL/4.0.3
using (var handler = new HttpParserDelegate())
using (var parser = new HttpCombinedParser(handler))
{
bArray = TestReponse();
Console.WriteLine(parser.Execute(bArray) == bArray.Length
? $"Reponse test succeed. Type identified is; {handler.HttpRequestResponse.MessageType} \r\n" +
$"Headers: \r\n" +
$"{string.Join("\r\n", handler.HttpRequestResponse.Headers.Select(h => $"{h.Key}: {string.Join(", ", h.Value)} "))}"
: $"Response test failed");
handler.HttpRequestResponse.Body.Position = 0;
var reader = new StreamReader(handler.HttpRequestResponse.Body);
var body = reader.ReadToEnd();
}
python parse http response (string)
You might find this useful, keep in mind that HTTPResponse wasn't designed to be "instantiated directly by user."
Also note that the content-length header in your response string may not be valid any more (it depends on how you've aquired these responses) this just means that the call to HTTPResponse.read() needs to have value larger than the content in order to get it all.
In python 2 it can be run this way.
from httplib import HTTPResponse
from StringIO import StringIO
http_response_str = """HTTP/1.1 200 OK
Date: Thu, Jul 3 15:27:54 2014
Content-Type: text/xml; charset="utf-8"
Connection: close
Content-Length: 626"""
class FakeSocket():
def __init__(self, response_str):
self._file = StringIO(response_str)
def makefile(self, *args, **kwargs):
return self._file
source = FakeSocket(http_response_str)
response = HTTPResponse(source)
response.begin()
print "status:", response.status
print "single header:", response.getheader('Content-Type')
print "content:", response.read(len(http_response_str)) # the len here will give a 'big enough' value to read the whole content
In python 3, the HTTPResponse
is imported from http.client
, and the response to be parsed needs to be byte encoded. Depending on where the data is gotten from this may be done already or need to be called explicitlyfrom http.client import HTTPResponse
from io import BytesIO
http_response_str = """HTTP/1.1 200 OK
Date: Thu, Jul 3 15:27:54 2014
Content-Type: text/xml; charset="utf-8"
Connection: close
Content-Length: 626
teststring"""
http_response_bytes = http_response_str.encode()
class FakeSocket():
def __init__(self, response_bytes):
self._file = BytesIO(response_bytes)
def makefile(self, *args, **kwargs):
return self._file
source = FakeSocket(http_response_bytes)
response = HTTPResponse(source)
response.begin()
print( "status:", response.status)
# status: 200
print( "single header:", response.getheader('Content-Type'))
# single header: text/xml; charset="utf-8"
print( "content:", response.read(len(http_response_str)))
# content: b'teststring'
How to parse HTTP raw bytes and get the HTTP content in python?
There's no built-in way to parse a raw HTTP response like this and handle compression properly. I would use urllib3
:
import urllib3
from io import BytesIO
from http.client import HTTPResponse
class BytesIOSocket:
def __init__(self, content):
self.handle = BytesIO(content)
def makefile(self, mode):
return self.handle
def response_from_bytes(data):
sock = BytesIOSocket(data)
response = HTTPResponse(sock)
response.begin()
return urllib3.HTTPResponse.from_httplib(response)
if __name__ == '__main__':
import socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect(('httpbin.org', 80))
sock.send(b'GET /gzip HTTP/1.1\r\nHost: httpbin.org\r\n\r\n')
raw_response = sock.recv(8192)
response = response_from_bytes(raw_response)
print(response.headers)
print(response.data)
Parsing RAW HTTP headers and converting it to JSON
Split by carriage returns ⏎, then split each line by colon, two parts at most, then convert the result to the hash:
input.split($/).map { |line| line.split(':', 2).map(&:strip) }.to_h
The strip
operation in between is optional. Parse Raw HTTP in Python
http://docs.python.org/library/httplib.html I believe this is the library you are looking for. A little change in name for python 3 but otherwise good to go.
Related Topics
Python 'Requests' Library - Define Specific Dns
Python: Why Does ("Hello" Is "Hello") Evaluate as True
How to Change a Widget's Font Style Without Knowing the Widget's Font Family/Size
Find All Upper, Lower and Mixed Case Combinations of a String
What Is the Max Length of a Python String
Suppressing Scientific Notation in Pandas
Differencebetween 'Py' and 'Python' in the Windows Terminal
Shell Script: Execute a Python Program from Within a Shell Script
Why Isn't Assigning to an Empty List (E.G. [] = "") an Error
Is There a Library Function for Root Mean Square Error (Rmse) in Python
How to Ignore Hidden Files Using Os.Listdir()
How to Change Tcp Keepalive Timer Using Python Script
Vectorized Numpy Linspace for Multiple Start and Stop Values
How to Set Selenium Python Webdriver Default Timeout
Python & Selenium: Difference Between Driver.Implicitly_Wait() and Time.Sleep()