What's the Best Way to Parse a JSON Response from the Requests Library

What's the best way to parse a JSON response from the requests library?

You can use json.loads:

import json
import requests

response = requests.get(...)
json_data = json.loads(response.text)

This converts a given string into a dictionary which allows you to access your JSON data easily within your code.

Or you can use @Martijn's helpful suggestion, and the higher voted answer, response.json().

HTTP requests and JSON parsing in Python

I recommend using the awesome requests library:

import requests

url = 'http://maps.googleapis.com/maps/api/directions/json'

params = dict(
    origin='Chicago,IL',
    destination='Los+Angeles,CA',
    waypoints='Joplin,MO|Oklahoma+City,OK',
    sensor='false'
)

resp = requests.get(url=url, params=params)
data = resp.json() # Check the JSON Response Content documentation below

JSON Response Content: https://requests.readthedocs.io/en/master/user/quickstart/#json-response-content

How do I parse a JSON response from Python Requests?

The manual suggests: if self.response.status_code == requests.codes.ok:

If that doesn't work:

if json.loads(self.response.text)['result'] == 'success':
   whatever()

How to parse JSON data from API response in Python?

Like @danil-kondratiev said, you can use response.json() and you don't need to encode/decode. Will this work for you?

import requests

response = requests.get('url with keys')

json_data = response.json() if response and response.status_code == 200 else None

if json_data and 'hoststatuslist' in json_data:
    if 'hoststatus' in json_data['hoststatuslist']:
        for hoststatus in json_data['hoststatuslist']['hoststatus']:
            host_name = hoststatus.get('name')
            status_text = hoststatus.get('status_text')

Python Requests Library - Scraping separate JSON and HTML responses from POST request

This is probably because the javascript the page sends to your browser is making a request to an API to get the json info about the movies.

You could either try sending the request directly to their API (see edit 2), parse the html with a library like Beautiful Soup or you can use a dedicated scraping library in python. I've had great experiences with scrapy. It is much faster than requests

Edit:

If the page uses dynamically loaded content, which I think is the case, you'd have to use selenium with the PhantomJS browser instead of requests. here is an example:

from bs4 import BeautifulSoup
from selenium import webdriver

url = "your url"
browser = webdriver.PhantomJS()
browser.get(url)
html = browser.page_source
soup = BeautifulSoup(html, 'lxml')

# Then parse the html code here

Or you could load the dynamic content with scrapy

I recommend the latter if you want to get into scraping. It would take a bit more time to learn but it is a better solution.

Edit 2:

To make a request directly to their api you can just reproduce the request you see. Using google chrome, you can see the request if you click on it and go to 'Headers':

After that, you simply reproduce the request using the requests library:

import requests
import json

url = 'http://paste.the.url/?here='

response = requests.get(url)

content = response.content

# in my case content was byte string 
# (it looks like b'data' instead of 'data' when you print it)
# if this is you case, convert it to string, like so

content_string = content.decode()

content_json = json.loads(content_string)

# do whatever you like with the data

You can modify the url as you see fit, for example if it is something like http://api.movies.com/?page=1&movietype=3 you could modify movietype=3 to movietype=2 to see a different type of movie, etc

---

Related Topics