How to read a few lines in a large CSV file with pandas?
Try this
train = pd.read_csv('file.csv', iterator=True, chunksize=150000)
If you only want to read the first n rows:
train = pd.read_csv(..., nrows=n)
If you only want to read rows from n to n+100
train = pd.read_csv(..., skiprows=n, nrows=n+100)
How to read a large csv with pandas?
Try to use the chunksize
parameter, filter in chunks and then concat
t_min, t_max, n_min, n_max, c_min, c_max = map(float, raw_input('t_min, t_max, n_min, n_max, c_min, c_max: ').split())
num_of_rows = 1024
TextFileReader = pd.read_csv(path, header=None, chunksize=num_of_rows)
dfs = []
for chunk_df in TextFileReader:
dfs.append(chunk_df.loc[(chunk_df[0] >= t_min) & (chunk_df[0] <= t_max) & (chunk_df[1] >= n_min) & (chunk_df[1] <= n_max) & (chunk_df[2] >= c_min) & (chunk_df[2] <= c_max)])
df = pd.concat(dfs,sort=False)
Related Topics
Convert a String Representation of a Dictionary to a Dictionary
How to Read/Process Command Line Arguments
Correct Way to Write Line to File
How to Do a Case-Insensitive String Comparison
Use Different Python Version With Virtualenv
Sorting List Based on Values from Another List
What's the Canonical Way to Check For Type in Python
Difference Between Class and Instance Attributes
Link to Flask Static Files With Url_For
Why Does Python Use 'Else' After For and While Loops
Local Variables in Nested Functions
How to Type Hint a Method With the Type of the Enclosing Class
Why Do Backslashes Appear Twice
How to Change the Size of Figures Drawn With Matplotlib
How to Set the 'Backend' in Matplotlib in Python
Download Large File in Python With Requests
How to Provide a Reproducible Copy of Your Dataframe With To_Clipboard()