How to Catch a Numpy Warning Like It's an Exception (Not Just for Testing)

How do I catch a numpy warning like it's an exception (not just for testing)?

It seems that your configuration is using the print option for numpy.seterr:

>>> import numpy as np
>>> np.array([1])/0   #'warn' mode
__main__:1: RuntimeWarning: divide by zero encountered in divide
array([0])
>>> np.seterr(all='print')
{'over': 'warn', 'divide': 'warn', 'invalid': 'warn', 'under': 'ignore'}
>>> np.array([1])/0   #'print' mode
Warning: divide by zero encountered in divide
array([0])

This means that the warning you see is not a real warning, but it's just some characters printed to stdout(see the documentation for seterr). If you want to catch it you can:

1. Use numpy.seterr(all='raise') which will directly raise the exception. This however changes the behaviour of all the operations, so it's a pretty big change in behaviour.
2. Use numpy.seterr(all='warn'), which will transform the printed warning in a real warning and you'll be able to use the above solution to localize this change in behaviour.

Once you actually have a warning, you can use the warnings module to control how the warnings should be treated:

>>> import warnings
>>> 
>>> warnings.filterwarnings('error')
>>> 
>>> try:
...     warnings.warn(Warning())
... except Warning:
...     print 'Warning was raised as an exception!'
... 
Warning was raised as an exception!

Read carefully the documentation for filterwarnings since it allows you to filter only the warning you want and has other options. I'd also consider looking at catch_warnings which is a context manager which automatically resets the original filterwarnings function:

>>> import warnings
>>> with warnings.catch_warnings():
...     warnings.filterwarnings('error')
...     try:
...         warnings.warn(Warning())
...     except Warning: print 'Raised!'
... 
Raised!
>>> try:
...     warnings.warn(Warning())
... except Warning: print 'Not raised!'
... 
__main__:2: Warning: 

Catching numpy runtimewarning as exception and suppressing them

Thanks to Jürg Merlin Spaak for his comment, I found a better and simpler solution. It's obviously better to catch the exception outside the function which I reverted back to the original version:

def ExtractValues(d):
    for v in d.values():
        if isinstance(v, dict):
            yield from ExtractValues(v)
        else:
            if isinstance(v,list):
                v = np.mean(v)  
            yield v

I've set everything on warn in the main piece of the of the code:

np.seterr(all='warn')

Then catch them:

with warnings.catch_warnings():
                    warnings.filterwarnings('error')
                    try:
                        raw_features = list(ExtractValues(data)) 
                    except Warning as e:
                        print('Houston, we have a warning:', e)
                        print('The bad guy is: ' + current_file)
                        print('This sample will not be considered.')
                        pass
                    else:
                        #Whatever

Worth noting for whoever comes here for the same exception. I succeeded to catch both warnings, but print(e) will tell you only "mean of empty slice". I can guess why, but I'm too tired to further investigate.

In Python, how does one catch warnings as if they were exceptions?

To handle warnings as errors simply use this:

import warnings
warnings.filterwarnings("error")

After this you will be able to catch warnings same as errors, e.g. this will work:

try:
    some_heavy_calculations()
except RuntimeWarning:
    breakpoint()

You can also reset the behaviour of warnings by running:

warnings.resetwarnings()

P.S. Added this answer because the best answer in comments contains misspelling: filterwarnigns instead of filterwarnings.

Catch numpy ComplexWarning as Exception

Using stdlib warnings filter causes these to raise instead of print:

>>> warnings.filterwarnings(action="error", category=np.ComplexWarning)
>>> b = np.array(a, dtype=np.float64)
ComplexWarning: Casting complex values to real discards the imaginary part

You can reset it to default filters with warnings.resetwarnings.

Python/numpy locating Runtime Warning

If you put

np.seterr(all='raise')

near the beginning of your script, exceptions will be raised instead of warnings. That will halt your script with a nice traceback which will give you information about where the error is occurring.

You could then put a try...except around the line in your code that raises the exception, and use the except clause to log the value of relevant variables.

---

Also, the RuntimeWarning you posted says the warning is originating in stats.py, line 2417. This seems to be in the pearsonr function. Googling "invalid value encountered in double_scalars" yielded this SO question which suggests

from scipy.stats.stats import pearsonr

X = [4, 4, 4, 4, 4, 4]
Y = [4, 5, 5, 4, 4, 4]

pearsonr(X, Y)

raise the RuntimeWarning. This suggests that you are occasionally calling pearsonr with inputs that result in division by zero (as user3453425 stated)-- maybe due to one of the inputs being constant, and hence having a standard deviation of zero.

In this case pearsonr(X, Y) returns (nan, 1.0). So make sure you handle the case when the pearson correlation coefficient is undefined (nan).

Debugging Numpy VisibleDeprecationWarning (ndarray from ragged nested sequences)

With a function that creates a ragged array:

In [60]: def foo(): 
    ...:     print('one') 
    ...:     x = np.array([[1],[1,2]]) 
    ...:     return x 
    ...:                                                                                             
In [61]: foo()                                                                                       
one
/usr/local/bin/ipython3:3: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  # -*- coding: utf-8 -*-
Out[61]: array([list([1]), list([1, 2])], dtype=object)

I get the warning, but also the expected result.

I can control the warnings.

For example to turn if off:

In [68]: np.warnings.filterwarnings('ignore', category=np.VisibleDeprecationWarning)                 
In [69]: foo()                                                                                       
one
Out[69]: array([list([1]), list([1, 2])], dtype=object)

Or to raise an error:

In [70]: np.warnings.filterwarnings('error', category=np.VisibleDeprecationWarning)                  
In [71]: foo()                                                                                       
one
---------------------------------------------------------------------------
VisibleDeprecationWarning                 Traceback (most recent call last)
<ipython-input-71-c19b6d9633cf> in <module>
----> 1 foo()

<ipython-input-60-6ad21d9e07b4> in foo()
      1 def foo():
      2     print('one')
----> 3     x = np.array([[1],[1,2]])
      4     return x
      5 

VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray

The error gives a traceback telling me where the warning was raised.

There may be ways of refining the warning filter to catch just this one, and not others of the same category. I haven't used this mechanism much.

Read np.warnings.filterwarnings docs for more details.

Catch matplotlib warning

You probably don't want to catch this warning as an exception. That will interrupt the function call.

Use the warnings standard library module to control warnings.

You can suppress a warning from a specific function call using a context manager:

import warnings
with warnings.catch_warnings():
    warnings.simplefilter("ignore")
    fig.tight_layout()

---

To ignore all warnings from matplotlib:

warnings.filterwarnings("ignore", module="matplotlib")

---

To ignore only UserWarnings from matplotlib:

warnings.filterwarnings("ignore", category=UserWarning, module="matplotlib")

---

Related Topics