How to Remove Nan Values from a Numpy Array

How do I remove NaN values from a NumPy array?

To remove NaN values from a NumPy array x:

x = x[~numpy.isnan(x)]

Explanation

The inner function numpy.isnan returns a boolean/logical array which has the value True everywhere that x is not-a-number. Since we want the opposite, we use the logical-not operator ~ to get an array with Trues everywhere that x is a valid number.

Lastly, we use this logical array to index into the original array x, in order to retrieve just the non-NaN values.

Numpy way to temporarily remove NaN values from array, with ability to place them back later

You can save the indices where the nan values existed via as a boolean mask. Then use that mask to fill in the the value you want.

a = np.array([nan,  1.,  2.,  3.,  4.,  5., nan,  6.,  7.,  8.,  9.])  
nan_mask = np.isnan(a)
a_no_nan = a[~nan_mask]

So far it is basically what you have. If you want to push the values of a_no_nan to a new array with same shape as a, but keep the nan locations, you can create an empty array, fill it with nan values, then push values using the mask.

c = np.empty_like(a)
c.fill(np.nan)
c[~nan_mask] = a_no_nan
c
# returns:
array([nan,  1.,  2.,  3.,  4.,  5., nan,  6.,  7.,  8.,  9.])  

How to remove nan values from numpy.ndarray

This is probably because the np.isnan() is failing to deal with string types among your possible element types in collection. You could try to use panda's isnull() to remove NaN values.

import pandas as pa
import numpy as np

a = ['A', np.nan, np.nan, 1.67, 8]
a = [x for x in a if not pa.isnull(x)]
print(a)

how to remove NaN from numpy subarray

First for some reason, the provided array is an array of strings. So before proceeding further we need to convert it to an array of floats:

# assuming your original array is arr
new_arr = arr.astype(float)

Then, we can filter the list elements, in a way to only keep the subarrays which second element is not NaN

filtered_list = np.array(list(filter(lambda x: not np.isnan(x[1]), new_arr)))

Removing nan entries from a 2-D Numpy array without for loops

Use x = x[~np.isnan(x).any(axis=1)]

Ref: https://www.w3resource.com/python-exercises/numpy/python-numpy-exercise-91.php

Remove NaN rows from numpy array that also contain objects

In [314]: x = np.array([[1, [2,3], np.nan], [3, [5,6,7], 8]])                                                
In [315]: x                                                                                                  
Out[315]: 
array([[1, list([2, 3]), nan],
       [3, list([5, 6, 7]), 8]], dtype=object)
In [316]: x.shape                                                                                            
Out[316]: (2, 3)
In [317]: x[0]                                                                                               
Out[317]: array([1, list([2, 3]), nan], dtype=object)
In [318]: x[1]                                                                                               
Out[318]: array([3, list([5, 6, 7]), 8], dtype=object)

isnan works on a float dtype array; object dtype can't be converted to that:

In [320]: np.isnan(x)                                                                                        
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-320-3b2be83a8ed7> in <module>
----> 1 np.isnan(x)

TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

We can, though, test the elements one by one with a is np.nan test:

In [325]: np.frompyfunc(lambda i: i is np.nan,1,1)(x)                                                        
Out[325]: 
array([[False, False, True],
       [False, False, False]], dtype=object)

frompyfunc returns an object dtype; let's convert that to bool:

In [328]: np.frompyfunc(lambda i: i is np.nan,1,1)(x).astype(bool)                                           
Out[328]: 
array([[False, False,  True],
       [False, False, False]])
In [329]: np.any(_, axis=1)           # test whole rows                                                                       
Out[329]: array([ True, False])
In [330]: x[~_, :]                    # use that as mask to keep other rows                                                      
Out[330]: array([[3, list([5, 6, 7]), 8]], dtype=object)

The pandas isnull suggested in the other answer, can do a similar element by element test:

In [335]: pd.isnull(x)                                                                                       
Out[335]: 
array([[False, False,  True],
       [False, False, False]])

Remove all nan from nested list

nan is not equal to itself (this goes for float('nan') as well as np.nan). So, we can use filter(), removing elements which are not equal to itself.

l1 = [['a', 'b', 'c', 'd', 'e', 'f'], 
      [1.0, 2.0, 3.0, 4.0, 5.0, nan], 
      ['red', 'orange', 'blue', nan, nan, nan]]

result = [list(filter(lambda x: x == x, inner_list)) for inner_list in l1]

print(result)

---

Related Topics