Python: Read Several Json Files from a Folder

Python: Read several json files from a folder

One option is listing all files in a directory with os.listdir and then finding only those that end in '.json':

import os, json
import pandas as pd

path_to_json = 'somedir/'
json_files = [pos_json for pos_json in os.listdir(path_to_json) if pos_json.endswith('.json')]
print(json_files)  # for me this prints ['foo.json']

Now you can use pandas DataFrame.from_dict to read in the json (a python dictionary at this point) to a pandas dataframe:

montreal_json = pd.DataFrame.from_dict(many_jsons[0])
print montreal_json['features'][0]['geometry']

Prints:

{u'type': u'Point', u'coordinates': [-73.6051013, 45.5115944]}

In this case I had appended some jsons to a list many_jsons. The first json in my list is actually a geojson with some geo data on Montreal. I'm familiar with the content already so I print out the 'geometry' which gives me the lon/lat of Montreal.

The following code sums up everything above:

import os, json
import pandas as pd

# this finds our json files
path_to_json = 'json/'
json_files = [pos_json for pos_json in os.listdir(path_to_json) if pos_json.endswith('.json')]

# here I define my pandas Dataframe with the columns I want to get from the json
jsons_data = pd.DataFrame(columns=['country', 'city', 'long/lat'])

# we need both the json and an index number so use enumerate()
for index, js in enumerate(json_files):
    with open(os.path.join(path_to_json, js)) as json_file:
        json_text = json.load(json_file)

        # here you need to know the layout of your json and each json has to have
        # the same structure (obviously not the structure I have here)
        country = json_text['features'][0]['properties']['country']
        city = json_text['features'][0]['properties']['name']
        lonlat = json_text['features'][0]['geometry']['coordinates']
        # here I push a list of data into a pandas DataFrame at row given by 'index'
        jsons_data.loc[index] = [country, city, lonlat]

# now that we have the pertinent json data in our DataFrame let's look at it
print(jsons_data)

for me this prints:

  country           city                   long/lat
0  Canada  Montreal city  [-73.6051013, 45.5115944]
1  Canada        Toronto  [-79.3849008, 43.6529206]

It may be helpful to know that for this code I had two geojsons in a directory name 'json'. Each json had the following structure:

{"features":
[{"properties":
{"osm_key":"boundary","extent":
[-73.9729016,45.7047897,-73.4734865,45.4100756],
"name":"Montreal city","state":"Quebec","osm_id":1634158,
"osm_type":"R","osm_value":"administrative","country":"Canada"},
"type":"Feature","geometry":
{"type":"Point","coordinates":
[-73.6051013,45.5115944]}}],
"type":"FeatureCollection"}

How to read multiple json files stored in a folder into different dictionaries in Python?

You may use a list to hold the filenames, then apply the dict reading for each

import json
from pathlib import Path

files = ['a.txt', 'b.txt', 'c.txt']  # ...
dicts = [json.loads(Path(file).read_text()) for file in files]

How can i open json files in folder and read them or print them in python

Try this:

for jsonfile in json_files:
    print(jsonfile)
    with open(os.path.join(path_to_json, jsonfile)) as file:
            data = json.load(file)
            print(data)

Reading all json files in a directory

Your code has two errors:

1.file is only the file name. You have to write full filepath (including its folder).

2.You have to use append inside the loop.

To sum up, this should work:

alldicts = []
for file in os.listdir(path_to_jsonfiles):
    full_filename = "%s/%s" % (path_to_jsonfiles, file)
    with open(full_filename,'r') as fi:
        dict = json.load(fi)
        alldicts.append(dict)

how to import and read multiple json files in pandas?

You can use the method os.listdir("relative path to where the folders are") (take a look at the domumentation) to get all subdirectories in the cwd. And you should not use loads for getting the contents of a file, instead pass the file object to the json.load() method. Implementing it in your code would be something like this:

import os

for i in os.listdir(path) :
   new_file = path + i + file
   my_file = open(new_file,"r")
   obj = json.load(my_file)
   my_file.close()

---

Related Topics