Python: Pandas Series - Why Use Loc

Python: Pandas Series - Why use loc?

• Explicit is better than implicit.

  df[boolean_mask] selects rows where boolean_mask is True, but there is a corner case when you might not want it to: when df has boolean-valued column labels:

  In [229]: df = pd.DataFrame({True:[1,2,3],False:[3,4,5]}); df
  Out[229]: 
     False  True 
  0      3      1
  1      4      2
  2      5      3
  

  You might want to use df[[True]] to select the True column. Instead it raises a ValueError:

  In [230]: df[[True]]
  ValueError: Item wrong length 1 instead of 3.
  

  Versus using loc:

  In [231]: df.loc[[True]]
  Out[231]: 
     False  True 
  0      3      1
  

  In contrast, the following does not raise ValueError even though the structure of df2 is almost the same as df1 above:

  In [258]: df2 = pd.DataFrame({'A':[1,2,3],'B':[3,4,5]}); df2
  Out[258]: 
     A  B
  0  1  3
  1  2  4
  2  3  5
  
  In [259]: df2[['B']]
  Out[259]: 
     B
  0  3
  1  4
  2  5
  

  Thus, df[boolean_mask] does not always behave the same as df.loc[boolean_mask]. Even though this is arguably an unlikely use case, I would recommend always using df.loc[boolean_mask] instead of df[boolean_mask] because the meaning of df.loc's syntax is explicit. With df.loc[indexer] you know automatically that df.loc is selecting rows. In contrast, it is not clear if df[indexer] will select rows or columns (or raise ValueError) without knowing details about indexer and df.

• df.loc[row_indexer, column_index] can select rows and columns. df[indexer] can only select rows or columns depending on the type of values in indexer and the type of column values df has (again, are they boolean?).

  In [237]: df2.loc[[True,False,True], 'B']
  Out[237]: 
  0    3
  2    5
  Name: B, dtype: int64
  

• When a slice is passed to df.loc the end-points are included in the range. When a slice is passed to df[...], the slice is interpreted as a half-open interval:

  In [239]: df2.loc[1:2]
  Out[239]: 
     A  B
  1  2  4
  2  3  5
  
  In [271]: df2[1:2]
  Out[271]: 
     A  B
  1  2  4
  

What is the difference between using loc and using just square brackets to filter for columns in Pandas/Python?

In the following situations, they behave the same:

1. Selecting a single column (df['A'] is the same as df.loc[:, 'A'] -> selects column A)
2. Selecting a list of columns (df[['A', 'B', 'C']] is the same as df.loc[:, ['A', 'B', 'C']] -> selects columns A, B and C)
3. Slicing by rows (df[1:3] is the same as df.iloc[1:3] -> selects rows 1 and 2. Note, however, if you slice rows with loc, instead of iloc, you'll get rows 1, 2 and 3 assuming you have a RangeIndex. See details here.)

However, [] does not work in the following situations:

1. You can select a single row with df.loc[row_label]
2. You can select a list of rows with df.loc[[row_label1, row_label2]]
3. You can slice columns with df.loc[:, 'A':'C']

These three cannot be done with [].
More importantly, if your selection involves both rows and columns, then assignment becomes problematic.

df[1:3]['A'] = 5

This selects rows 1 and 2 then selects column 'A' of the returning object and assigns value 5 to it. The problem is, the returning object might be a copy so this may not change the actual DataFrame. This raises SettingWithCopyWarning. The correct way of making this assignment is:

df.loc[1:3, 'A'] = 5

With .loc, you are guaranteed to modify the original DataFrame. It also allows you to slice columns (df.loc[:, 'C':'F']), select a single row (df.loc[5]), and select a list of rows (df.loc[[1, 2, 5]]).

Also note that these two were not included in the API at the same time. .loc was added much later as a more powerful and explicit indexer. See unutbu's answer for more detail.

Note: Getting columns with [] vs . is a completely different topic. . is only there for convenience. It only allows accessing columns whose names are valid Python identifiers (i.e. they cannot contain spaces, they cannot be composed of numbers...). It cannot be used when the names conflict with Series/DataFrame methods. It also cannot be used for non-existing columns (i.e. the assignment df.a = 1 won't work if there is no column a). Other than that, . and [] are the same.

When to use .loc and when not to use (Pandas Dataframe)?

Why is .loc not used in 2nd criteria?

df = pd.DataFrame({

         'col_1':[0,3,0,7,1,0],
         'col_2':[0,3,6,9,2,4],
         'col3':list('aaabbb')
})

No, you are wrong, it working in both.

print (df.loc[df['col_1']==0])
   col_1  col_2 col3
0      0      0    a
2      0      6    a
5      0      4    b

print (df.loc[(df['col_1']==0) & (df['col_2']>0)])
   col_1  col_2 col3
2      0      6    a
5      0      4    b

print (df[df['col_1']==0])
   col_1  col_2 col3
0      0      0    a
2      0      6    a
5      0      4    b

print (df[(df['col_1']==0) & (df['col_2']>0)])
   col_1  col_2 col3
2      0      6    a
5      0      4    b

Reason for using is if need also filter columns names, e.g. col_1:

print (df.loc[df['col_1']==0, 'col_2'])
0    0
2    6
5    4
Name: col_2, dtype: int64

print (df.loc[(df['col_1']==0) & (df['col_2']>0), 'col_2'])
2    6
5    4
Name: col_2, dtype: int64

If need filter 2 or more columns use list, e.g for col_1,col3 use:

print (df.loc[df['col_1']==0, ['col_1','col3']])
   col_1 col3
0      0    a
2      0    a
5      0    b

print (df.loc[(df['col_1']==0) & (df['col_2']>0), ['col_1','col3']])
   col_1 col3
2      0    a
5      0    b

If omit loc it failed:

df[df['col_1']==0, 'col_1']
df[(df['col_1']==0) & (df['col_2']>0), 'col_1']

TypeError

Also, why can't we use and in the second code, i.e.

df[(df['col_1']==0) and (df['col_2']>0)]

becasue and is for processing by scalars, in pandas are used & for bitwise AND - &. More info is here.

How are iloc and loc different?

Label vs. Location

The main distinction between the two methods is:

• loc gets rows (and/or columns) with particular labels.

• iloc gets rows (and/or columns) at integer locations.

To demonstrate, consider a series s of characters with a non-monotonic integer index:

>>> s = pd.Series(list("abcdef"), index=[49, 48, 47, 0, 1, 2]) 
49    a
48    b
47    c
0     d
1     e
2     f

>>> s.loc[0]    # value at index label 0
'd'

>>> s.iloc[0]   # value at index location 0
'a'

>>> s.loc[0:1]  # rows at index labels between 0 and 1 (inclusive)
0    d
1    e

>>> s.iloc[0:1] # rows at index location between 0 and 1 (exclusive)
49    a

Here are some of the differences/similarities between s.loc and s.iloc when passed various objects:

In [25]: %timeit df.loc[('a', 'A'), ('c', 'C')]
10000 loops, best of 3: 187 µs per loop

In [26]: %timeit df.at[('a', 'A'), ('c', 'C')]
100000 loops, best of 3: 8.33 µs per loop

In [35]: %timeit df.get_value(('a', 'A'), ('c', 'C'))
100000 loops, best of 3: 3.62 µs per loop

>>> df['a'] >= 15
>>> 
0    False
1    False
2    False
3    False
4    False
5    False
6     True
7     True
Name: a, dtype: bool

import numpy as np
import pandas as pd
df = pd.DataFrame(np.random.randint(10,100, (5, 4)), columns = list("ABCD"))

df.loc[1:3, "A":"C"]

df.iloc[1:3, 1:3]