Use Python Requests to Download CSV

Use python requests to download CSV

This should help:

import csv
import requests

CSV_URL = 'http://samplecsvs.s3.amazonaws.com/Sacramentorealestatetransactions.csv'

with requests.Session() as s:
    download = s.get(CSV_URL)

    decoded_content = download.content.decode('utf-8')

    cr = csv.reader(decoded_content.splitlines(), delimiter=',')
    my_list = list(cr)
    for row in my_list:
        print(row)

Ouput sample:

['street', 'city', 'zip', 'state', 'beds', 'baths', 'sq__ft', 'type', 'sale_date', 'price', 'latitude', 'longitude']
['3526 HIGH ST', 'SACRAMENTO', '95838', 'CA', '2', '1', '836', 'Residential', 'Wed May 21 00:00:00 EDT 2008', '59222', '38.631913', '-121.434879']
['51 OMAHA CT', 'SACRAMENTO', '95823', 'CA', '3', '1', '1167', 'Residential', 'Wed May 21 00:00:00 EDT 2008', '68212', '38.478902', '-121.431028']
['2796 BRANCH ST', 'SACRAMENTO', '95815', 'CA', '2', '1', '796', 'Residential', 'Wed May 21 00:00:00 EDT 2008', '68880', '38.618305', '-121.443839']
['2805 JANETTE WAY', 'SACRAMENTO', '95815', 'CA', '2', '1', '852', 'Residential', 'Wed May 21 00:00:00 EDT 2008', '69307', '38.616835', '-121.439146']
[...]

Related question with answer: https://stackoverflow.com/a/33079644/295246

---

Edit: Other answers are useful if you need to download large files (i.e. stream=True).

How to only download updated CSV files via Python requests?

Have you tried to pull the json feed instead of the download csv? Requires a slight change to your csv_file_names list. (Which if you need me to, we can work off your original list and just use regex to grab the relevant parts to place in the url).

import requests
import pandas as pd
import re

csv_file_names = [
    ['epl-2021','chelsea'] ,
    ['champions-league-2021','chelsea'],
    ['la-liga-2021','fc-barcelona'],
    ['champions-league-2021','barcelona'],
    ['ligue-1-2021','paris-saint-germain'],
    ['champions-league-2021','paris'],
    ['epl-2021', ''],
    ['champions-league-2021',''],
    ['mlb-2021','baltimore-orioles'],
    ['nfl-2020','pittsburgh-steelers']
]

for count, each in enumerate(csv_file_names, start=1):
    url = 'https://fixturedownload.com/feed/json/%s/%s' %(each[0], each[-1])
    jsonData = requests.get(url).json()
    df = pd.DataFrame(jsonData)
    
    csv = '%s-%s-.csv' %(each[0], each[-1])
    print("Downloading...", count, "of", len(csv_file_names), "-", csv  )
    
    for col in ['HomeTeamScore', 'AwayTeamScore']:
        df[col] = df[col].fillna(99).astype(int).astype(str)
    
    df['Result'] = df['HomeTeamScore'] + ' - ' + df['AwayTeamScore']
    df['Result'] = df['Result'].replace('99 - 99', '')
    
    for col in df.columns:
        if 'Date' in col:
            newColName = 'Date'
        else:
            newColName = ' '.join(re.sub('([A-Z][a-z]+)', r' \1', re.sub('([A-Z]+)', r' \1', col)).split())
        df = df.rename(columns={col:newColName})
        
    df = df.drop(['Group', 'Home Team Score', 'Away Team Score'], axis=1)

    
    df.to_csv('/home/pi/Score-Checker/CSV-Files/'+csv, index=False)

Output: of 'ligue-1-2021','paris-saint-germain'

Use python request to download and save CSV

def main():

    import requests

    url = "https://coronavirus.data.gov.uk/downloads/csv/coronavirus-deaths_latest.csv"

    with requests.get(url, stream=True) as response:
        response.raise_for_status()

        with open("deaths_latest.csv", "wb") as file:
            for chunk in response.iter_content(chunk_size=8192):
                file.write(chunk)
            file.flush()

    print("Done")

    return 0

if __name__ == "__main__":
    import sys
    sys.exit(main())

Saving a downloaded CSV file using Python

If you're trying to write this data to a CSV file, you can first download it using requests.get, then save each line to a CSV file.

import csv
import requests

url = 'https://www.alphavantage.co/query?function=TIME_SERIES_DAILY_ADJUSTED&symbol=MSFT&apikey=demo&datatype=csv'
response = requests.get(url)        

with open('out.csv', 'w') as f:
    writer = csv.writer(f)
    for line in response.iter_lines():
        writer.writerow(line.decode('utf-8').split(','))

---

Alternatively, if you have pandas installed (pip install --user pandas), you can load data by passing a URL directly.

import pandas as pd

df = pd.read_csv(url)   
df.head()

    timestamp    open    high     low   close  adjusted_close    volume  dividend_amount  split_coefficient
0  2019-06-19  135.00  135.93  133.81  135.69          135.69  17946556              0.0                1.0
1  2019-06-18  134.19  135.24  133.57  135.16          135.16  25908534              0.0                1.0
2  2019-06-17  132.63  133.73  132.53  132.85          132.85  14517785              0.0                1.0
3  2019-06-14  132.26  133.79  131.64  132.45          132.45  17821703              0.0                1.0
4  2019-06-13  131.98  132.67  131.56  132.32          132.32  17200848              0.0                1.0

df.to_csv('out.csv')

Downloading a csv using http get request?

You can get the text content by .text then store the value in a csv file locally.

import requests

data = requests.get('https://eodhistoricaldata.com/api/eod/AAPL.US?from=2020-01-05&to=2020-02-10&period=d&fmt=csv&api_token=OeAFFmMliFG5orCUuwAKQ8l4WWFQ67YX').text

with open('file.csv', 'w') as f:
    f.write(data)

---

Related Topics