And' (Boolean) VS '&' (Bitwise) - Why Difference in Behavior with Lists VS Numpy Arrays

and' (boolean) vs '&' (bitwise) - Why difference in behavior with lists vs numpy arrays?

and tests whether both expressions are logically True while & (when used with True/False values) tests if both are True.

In Python, empty built-in objects are typically treated as logically False while non-empty built-ins are logically True. This facilitates the common use case where you want to do something if a list is empty and something else if the list is not. Note that this means that the list [False] is logically True:

>>> if [False]:
...    print 'True'
...
True

So in Example 1, the first list is non-empty and therefore logically True, so the truth value of the and is the same as that of the second list. (In our case, the second list is non-empty and therefore logically True, but identifying that would require an unnecessary step of calculation.)

For example 2, lists cannot meaningfully be combined in a bitwise fashion because they can contain arbitrary unlike elements. Things that can be combined bitwise include: Trues and Falses, integers.

NumPy objects, by contrast, support vectorized calculations. That is, they let you perform the same operations on multiple pieces of data.

Example 3 fails because NumPy arrays (of length > 1) have no truth value as this prevents vector-based logic confusion.

Example 4 is simply a vectorized bit and operation.

Bottom Line

• If you are not dealing with arrays and are not performing math manipulations of integers, you probably want and.

• If you have vectors of truth values that you wish to combine, use numpy with &.

Boolean operators vs Bitwise operators

Here are a couple of guidelines:

• Boolean operators are usually used on boolean values but bitwise operators are usually used on integer values.
• Boolean operators are short-circuiting but bitwise operators are not short-circuiting.

The short-circuiting behaviour is useful in expressions like this:

if x is not None and x.foo == 42:
    # ...

This would not work correctly with the bitwise & operator because both sides would always be evaluated, giving AttributeError: 'NoneType' object has no attribute 'foo'. When you use the boolean andoperator the second expression is not evaluated when the first is False. Similarly or does not evaluate the second argument if the first is True.

What is the difference between | and or ?

and is python's logical-AND operator.

| is python's bitwise-OR operator (not bitwise-AND, as your question might suggest), which numpy overrides in order to make it the numpy element-wise-OR.

Numpy couldn't have overridden and nor or to work element-wise, because and and or are not overridable in python. Therefor, the bitwise operators are overridden in numpy.

difference between & and and in pandas

The len(df_temp > 0) and len(df_temp4 > 0) probably don't do what you expect. The comparison operators with pandas DataFrames return element-wise results, that means they create a boolean DataFrame where each value indicates if the corresponding value in the DataFrame is greater than zero:

>>> import pandas as pd
>>> df = pd.DataFrame({'a': [-1,0,1], 'b': [-1,0,1]})
>>> df
   a  b
0 -1 -1
1  0  0
2  1  1
>>> df > 0
       a      b
0  False  False
1  False  False
2   True   True

So the len of df is the same as the len of df > 0:

>>> len(df)
3
>>> len(df > 0)
3

difference between "&" and "and"

They mean different things:

• & is bitwise and
• and is logical and (and short-circuiting)

Since you asked specifically about pandas (assuming at least one operand is a NumPy array, pandas Series, or pandas DataFrame):

• & also refers to the element-wise "bitwise and".
• The element-wise "logical and" for pandas isn't and but one has to use a function, i.e. numpy.logical_and.

For more explanation you can refer to "Difference between 'and' (boolean) vs. '&' (bitwise) in python. Why difference in behavior with lists vs numpy arrays?"

not sure what would cause this statement to fail all of a sudden.

You did not provide the "fail" nor the expected behavior so unfortunately I cannot help you there.

Where are the ampersand and vertical bar characters used in Python?

The wikipedia page is wrong, I've corrected it. | and & are not boolean operators, even though they are eager operators, which just means that they are not short circuit operators. As you probably know, here's how the python and and or operators work:

>>> def talk(x):
...     print "Evaluating: ", bool(x)
...     return x
... 
>>> talk(1 == 1) or talk(2 == 1)   # 2 == 1 is not evaluated
Evaluating:  True
True
>>> talk(1 == 1) and talk(2 == 1)
Evaluating:  True
Evaluating:  False
False
>>> talk(1 == 2) and talk(1 == 3)  # 1 == 3 is not evaluated
Evaluating:  False
False

As far as I know, python has no eager boolean operators, they would have to be explicitly coded, for instance like this:

>>> def eager_or(a, b):
...    return a or b
...
>>> eager_or(talk(1 == 1), talk(2 == 1))
Evaluating:  True
Evaluating:  False
True

Now a and b are automatically evaluated when the function is called, even though or still short circuits.

As for the usage of | and &, when used with numbers, they are binary operators:

>>> bin(0b11110000 & 0b10101010)
'0b10100000'
>>> bin(0b11110000 | 0b10101010)
'0b11111010'

You're most likely to use | this way with python bindings to libraries that uses flags, like wxWidgets:

>>> frame = wx.Frame(title="My Frame", style=wx.MAXIMIZE | wx.STAY_ON_TOP)
>>> bin(wx.MAXIMIZE)
'0b10000000000000'
>>> bin(wx.STAY_ON_TOP)
'0b1000000000000000'
>>> bin(wx.MAXIMIZE | wx.STAY_ON_TOP)
'0b1010000000000000'

When used with sets, they do the intersection and union operations, respectively:

>>> set("abcd") & set("cdef")
set(['c', 'd'])
>>> set("abcd") | set("cdef")
set(['a', 'c', 'b', 'e', 'd', 'f'])

In python 3 logical operator giving me an opposite result

In Python '&' has higher precedence than '==' so We are getting the wrong result
Try this:

if (a==b) & (a==20):
    print "a is equal to b"
else:
    print "a is not equal to b"

Logical operators for Boolean indexing in Pandas

When you say

(a['x']==1) and (a['y']==10)

You are implicitly asking Python to convert (a['x']==1) and (a['y']==10) to Boolean values.

NumPy arrays (of length greater than 1) and Pandas objects such as Series do not have a Boolean value -- in other words, they raise

ValueError: The truth value of an array is ambiguous. Use a.empty, a.any() or a.all().

when used as a Boolean value. That's because it's unclear when it should be True or False. Some users might assume they are True if they have non-zero length, like a Python list. Others might desire for it to be True only if all its elements are True. Others might want it to be True if any of its elements are True.

Because there are so many conflicting expectations, the designers of NumPy and Pandas refuse to guess, and instead raise a ValueError.

Instead, you must be explicit, by calling the empty(), all() or any() method to indicate which behavior you desire.

In this case, however, it looks like you do not want Boolean evaluation, you want element-wise logical-and. That is what the & binary operator performs:

(a['x']==1) & (a['y']==10)

returns a boolean array.

---

By the way, as alexpmil notes,
the parentheses are mandatory since & has a higher operator precedence than ==.

Without the parentheses, a['x']==1 & a['y']==10 would be evaluated as a['x'] == (1 & a['y']) == 10 which would in turn be equivalent to the chained comparison (a['x'] == (1 & a['y'])) and ((1 & a['y']) == 10). That is an expression of the form Series and Series.
The use of and with two Series would again trigger the same ValueError as above. That's why the parentheses are mandatory.

---

Related Topics