How do I expand the output display to see more columns of a Pandas DataFrame?
Update: Pandas 0.23.4 onwards
This is not necessary. Pandas autodetects the size of your terminal window if you set pd.options.display.width = 0. (For older versions see at bottom.)
pandas.set_printoptions(...) is deprecated. Instead, use pandas.set_option(optname, val), or equivalently pd.options.<opt.hierarchical.name> = val. Like:
import pandas as pd
pd.set_option('display.max_rows', 500)
pd.set_option('display.max_columns', 500)
pd.set_option('display.width', 1000)
Here is the help for set_option:
set_option(pat,value) - Sets the value of the specified option
Available options:
display.[chop_threshold, colheader_justify, column_space, date_dayfirst,
date_yearfirst, encoding, expand_frame_repr, float_format, height,
line_width, max_columns, max_colwidth, max_info_columns, max_info_rows,
max_rows, max_seq_items, mpl_style, multi_sparse, notebook_repr_html,
pprint_nest_depth, precision, width]
mode.[sim_interactive, use_inf_as_null]
Parameters
----------
pat - str/regexp which should match a single option.
Note: partial matches are supported for convenience, but unless you use the
full option name (e.g., *x.y.z.option_name*), your code may break in future
versions if new options with similar names are introduced.
value - new value of option.
Returns
-------
None
Raises
------
KeyError if no such option exists
display.chop_threshold: [default: None] [currently: None]
: float or None
if set to a float value, all float values smaller then the given threshold
will be displayed as exactly 0 by repr and friends.
display.colheader_justify: [default: right] [currently: right]
: 'left'/'right'
Controls the justification of column headers. used by DataFrameFormatter.
display.column_space: [default: 12] [currently: 12]No description available.
display.date_dayfirst: [default: False] [currently: False]
: boolean
When True, prints and parses dates with the day first, eg 20/01/2005
display.date_yearfirst: [default: False] [currently: False]
: boolean
When True, prints and parses dates with the year first, e.g., 2005/01/20
display.encoding: [default: UTF-8] [currently: UTF-8]
: str/unicode
Defaults to the detected encoding of the console.
Specifies the encoding to be used for strings returned by to_string,
these are generally strings meant to be displayed on the console.
display.expand_frame_repr: [default: True] [currently: True]
: boolean
Whether to print out the full DataFrame repr for wide DataFrames
across multiple lines, `max_columns` is still respected, but the output will
wrap-around across multiple "pages" if it's width exceeds `display.width`.
display.float_format: [default: None] [currently: None]
: callable
The callable should accept a floating point number and return
a string with the desired format of the number. This is used
in some places like SeriesFormatter.
See core.format.EngFormatter for an example.
display.height: [default: 60] [currently: 1000]
: int
Deprecated.
(Deprecated, use `display.height` instead.)
display.line_width: [default: 80] [currently: 1000]
: int
Deprecated.
(Deprecated, use `display.width` instead.)
display.max_columns: [default: 20] [currently: 500]
: int
max_rows and max_columns are used in __repr__() methods to decide if
to_string() or info() is used to render an object to a string. In case
python/IPython is running in a terminal this can be set to 0 and Pandas
will correctly auto-detect the width the terminal and swap to a smaller
format in case all columns would not fit vertically. The IPython notebook,
IPython qtconsole, or IDLE do not run in a terminal and hence it is not
possible to do correct auto-detection.
'None' value means unlimited.
display.max_colwidth: [default: 50] [currently: 50]
: int
The maximum width in characters of a column in the repr of
a Pandas data structure. When the column overflows, a "..."
placeholder is embedded in the output.
display.max_info_columns: [default: 100] [currently: 100]
: int
max_info_columns is used in DataFrame.info method to decide if
per column information will be printed.
display.max_info_rows: [default: 1690785] [currently: 1690785]
: int or None
max_info_rows is the maximum number of rows for which a frame will
perform a null check on its columns when repr'ing To a console.
The default is 1,000,000 rows. So, if a DataFrame has more
1,000,000 rows there will be no null check performed on the
columns and thus the representation will take much less time to
display in an interactive session. A value of None means always
perform a null check when repr'ing.
display.max_rows: [default: 60] [currently: 500]
: int
This sets the maximum number of rows Pandas should output when printing
out various output. For example, this value determines whether the repr()
for a dataframe prints out fully or just a summary repr.
'None' value means unlimited.
display.max_seq_items: [default: None] [currently: None]
: int or None
when pretty-printing a long sequence, no more then `max_seq_items`
will be printed. If items are ommitted, they will be denoted by the addition
of "..." to the resulting string.
If set to None, the number of items to be printed is unlimited.
display.mpl_style: [default: None] [currently: None]
: bool
Setting this to 'default' will modify the rcParams used by matplotlib
to give plots a more pleasing visual style by default.
Setting this to None/False restores the values to their initial value.
display.multi_sparse: [default: True] [currently: True]
: boolean
"sparsify" MultiIndex display (don't display repeated
elements in outer levels within groups)
display.notebook_repr_html: [default: True] [currently: True]
: boolean
When True, IPython notebook will use html representation for
Pandas objects (if it is available).
display.pprint_nest_depth: [default: 3] [currently: 3]
: int
Controls the number of nested levels to process when pretty-printing
display.precision: [default: 7] [currently: 7]
: int
Floating point output precision (number of significant digits). This is
only a suggestion
display.width: [default: 80] [currently: 1000]
: int
Width of the display in characters. In case python/IPython is running in
a terminal this can be set to None and Pandas will correctly auto-detect the
width.
Note that the IPython notebook, IPython qtconsole, or IDLE do not run in a
terminal and hence it is not possible to correctly detect the width.
mode.sim_interactive: [default: False] [currently: False]
: boolean
Whether to simulate interactive mode for purposes of testing
mode.use_inf_as_null: [default: False] [currently: False]
: boolean
True means treat None, NaN, INF, -INF as null (old way),
False means None and NaN are null, but INF, -INF are not null
(new way).
Call def: pd.set_option(self, *args, **kwds)
Older version information. Much of this has been deprecated.
As @bmu mentioned, Pandas auto detects (by default) the size of the display area, a summary view will be used when an object repr does not fit on the display. You mentioned resizing the IDLE window, to no effect. If you do print df.describe().to_string() does it fit on the IDLE window?
The terminal size is determined by pandas.util.terminal.get_terminal_size() (deprecated and removed), this returns a tuple containing the (width, height) of the display. Does the output match the size of your IDLE window? There might be an issue (there was one before when running a terminal in Emacs).
Note that it is possible to bypass the autodetect, pandas.set_printoptions(max_rows=200, max_columns=10) will never switch to summary view if number of rows, columns does not exceed the given limits.
The 'max_colwidth' option helps in seeing untruncated form of each column.
Show the entire output when pandas dataframe is big (when many nan values in multiple columns need to be shown)
You can try:
DataFrame.to_string()
Or:
DataFrame.to_markdown()
Or:
pandas.set_option('display.max_rows', None)
Now you could decide to just display NaN values like this:
-by column:
df[df['column name'].isna()]
-or for entire dataframe:
df[df.isna().any(axis=1)]
How to display different columns and remove them using pandas
You can first drop the columns or parameters which are empty and select the rows with only 1 or 0 values.
To get column names which are having all null values
df.columns[df.isna().all()]
Next step you can drop null columns.
df.dropna(how='all', axis=1, inplace=True)
df.loc[:, ((df==0) | (df==1)).all()]
KeyError after renaming the first n column names of python pandas dataframe
You can try updating the column labels like this:
df2 = df2.rename(columns=dict(zip(list(df2.columns)[0:3], ["symbol","field","abc"])))
... or like this:
df2.columns = ["symbol","field","abc"] + list(df2.columns)[3:]
Output:
COLA COL_B testC
0 1 2 3
1 10 11 12
<class 'pandas.core.frame.DataFrame'>
Index(['COLA', 'COL_B', 'testC'], dtype='object')
<class 'pandas.core.indexes.base.Index'>
['COLA' 'COL_B' 'testC']
<class 'numpy.ndarray'>
COLA COL_B testC
0 1 2 3
1 10 11 12
<class 'pandas.core.frame.DataFrame'>
Index(['COLA', 'COL_B', 'testC'], dtype='object')
<class 'pandas.core.indexes.base.Index'>
['COLA' 'COL_B' 'testC']
<class 'numpy.ndarray'>
after renaming the columns:
symbol field abc
0 1 2 3
1 10 11 12
df2["symbol"]
0 1
1 10
Name: symbol, dtype: int64
Note that the docs for Index.values have a warning which reads:
We recommend using Index.array or Index.to_numpy(), depending on whether you need a reference to the underlying data or a NumPy array.
Pandas: How to expand a dataframe between dates and add NaNs to new rows
Create DatetimeIndex by column date first, so possible use custom lambda function with DataFrame.asfreq, remove first level of MultiIndex and convert index to column date, last use Series.dt.strftime for original format DD/MM/YYYY:
First is possible test duplicated rows by columns ID, date:
print (df[df.duplicated(['ID','date'], keep=False)])
ID date StartDate EndDate Count Cov1 Cov2 Cov3
0 A 05/05/2005 04/04/05 06/06/2006 3 1 F 1
1 A 05/05/2005 04/04/05 06/06/2006 3 1 F 1
If possible, remove duplicates:
df = df.drop_duplicates(['ID','date'])
df['date'] = pd.to_datetime(df['date'], dayfirst=True)
df1 = (df.set_index('date').groupby('ID')
.apply(lambda x: x.asfreq('D', method='ffill'))
.droplevel(0)
.reset_index())
print (df1)
date ID StartDate EndDate Count Cov1 Cov2 Cov3
0 2005-05-05 A 04/04/05 06/06/2006 3 1 F 1
1 2005-05-06 A 04/04/05 06/06/2006 5 1 F 1
2 2005-05-07 A 04/04/05 06/06/2006 2 1 F 1
3 2005-05-08 A 04/04/05 06/06/2006 2 1 F 1
4 2005-05-09 A 04/04/05 06/06/2006 2 1 F 1
5 2005-05-10 A 04/04/05 06/06/2006 7 1 F 1
6 2005-05-05 B 04/04/05 06/06/2006 6 0 M 2
7 2005-05-06 B 04/04/05 06/06/2006 6 0 M 2
8 2005-05-07 B 04/04/05 06/06/2006 1 0 M 2
9 2005-05-01 C 04/04/05 06/06/2006 3 1 F 1
10 2005-05-02 C 04/04/05 06/06/2006 3 1 F 1
11 2005-05-03 C 04/04/05 06/06/2006 7 1 F 1
print (df1.index.name)
None
If possible in real data it is ID use:
df1 = df1.rename_axis(None)
m = df1.merge(df, indicator=True, how='left')['_merge'].eq('left_only')
df1.loc[m, 'Count'] = 0
df1['date'] = df1['date'].dt.strftime('%d/%m/%Y')
print (df1)
date ID StartDate EndDate Count Cov1 Cov2 Cov3
0 05/05/2005 A 04/04/05 06/06/2006 3 1 F 1
1 06/05/2005 A 04/04/05 06/06/2006 5 1 F 1
2 07/05/2005 A 04/04/05 06/06/2006 2 1 F 1
3 08/05/2005 A 04/04/05 06/06/2006 0 1 F 1
4 09/05/2005 A 04/04/05 06/06/2006 0 1 F 1
5 10/05/2005 A 04/04/05 06/06/2006 7 1 F 1
6 05/05/2005 B 04/04/05 06/06/2006 6 0 M 2
7 06/05/2005 B 04/04/05 06/06/2006 0 0 M 2
8 07/05/2005 B 04/04/05 06/06/2006 1 0 M 2
9 01/05/2005 C 04/04/05 06/06/2006 3 1 F 1
10 02/05/2005 C 04/04/05 06/06/2006 0 1 F 1
11 03/05/2005 C 04/04/05 06/06/2006 7 1 F 1
How to create new column with calculation of days between date in 2 other columns in DataFrame in Pandas Python?
Try:
# ensure datetime:
df["col1"] = pd.to_datetime(df["col1"])
df["col2"] = pd.to_datetime(df["col2"])
df["col3"] = (df["col1"] - df["col2"]).dt.days
print(df)
Prints:
col1 col2 col3
0 2020-01-10 19:45:49 2020-01-11 -1.0
1 2020-01-24 20:14:33 2020-01-24 0.0
2 2020-01-24 11:43:15 2020-01-20 4.0
3 NaT 2020-08-14 NaN
Related Topics
How to Improve Performance of This Code
Why Do Python Classes Inherit Object
"Inconsistent Use of Tabs and Spaces in Indentation"
How to Print a Single Backslash
Pandas Groupby With Delimiter Join
How to Pass a String into Subprocess.Popen (Using the Stdin Argument)
How to Fix: "Unicodedecodeerror: 'Ascii' Codec Can't Decode Byte"
What Is an Alternative to Execfile in Python 3
Append Existing Excel Sheet With New Dataframe Using Python Pandas
How to Format a Floating Number to Fixed Width in Python
Best Way to Return Multiple Values from a Function
How to Clear the Interpreter Console
Use a List of Values to Select Rows from a Pandas Dataframe
Is Explicitly Closing Files Important
Deleting Dataframe Row in Pandas Based on Column Value
How to Save and Load Cookies Using Python + Selenium Webdriver