Converting numpy dtypes to native python types
Use val.item()
to convert most NumPy values to a native Python type:
import numpy as np
# for example, numpy.float32 -> python float
val = np.float32(0)
pyval = val.item()
print(type(pyval)) # <class 'float'>
# and similar...
type(np.float64(0).item()) # <class 'float'>
type(np.uint32(0).item()) # <class 'int'>
type(np.int16(0).item()) # <class 'int'>
type(np.cfloat(0).item()) # <class 'complex'>
type(np.datetime64(0, 'D').item()) # <class 'datetime.date'>
type(np.datetime64('2001-01-01 00:00:00').item()) # <class 'datetime.datetime'>
type(np.timedelta64(0, 'D').item()) # <class 'datetime.timedelta'>
...
(Another method is np.asscalar(val)
, however it is deprecated since NumPy 1.16).For the curious, to build a table of conversions of NumPy array scalars for your system:
for name in dir(np):
obj = getattr(np, name)
if hasattr(obj, 'dtype'):
try:
if 'time' in name:
npn = obj(0, 'D')
else:
npn = obj(0)
nat = npn.item()
print('{0} ({1!r}) -> {2}'.format(name, npn.dtype.char, type(nat)))
except:
pass
There are a few NumPy types that have no native Python equivalent on some systems, including: clongdouble
, clongfloat
, complex192
, complex256
, float128
, longcomplex
, longdouble
and longfloat
. These need to be converted to their nearest NumPy equivalent before using .item()
. Convert numpy object type to float type
Try adding:
np.set_printoptions(suppress=True)
As the first line under import numpy as np
. Convert numpy type to python
It looks like you're correct:
>>> import numpy
>>> import json
>>> json.dumps(numpy.int32(685))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/json/__init__.py", line 243, in dumps
return _default_encoder.encode(obj)
File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
return _iterencode(o, 0)
File "/usr/lib/python2.7/json/encoder.py", line 184, in default
raise TypeError(repr(o) + " is not JSON serializable")
TypeError: 685 is not JSON serializable
The unfortunate thing here is that numpy numbers' __repr__
doesn't give you any hint about what type they are. They're running around masquerading as int
s when they aren't (gasp). Ultimately, it looks like json
is telling you that an int
isn't serializable, but really, it's telling you that this particular np.int32 (or whatever type you actually have) isn't serializable. (No real surprise there -- No np.int32 is serializable). This is also why the dict that you inevitably printed before passing it to json.dumps
looks like it just has integers in it as well.The easiest workaround here is probably to write your own serializer1:
class MyEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, numpy.integer):
return int(obj)
elif isinstance(obj, numpy.floating):
return float(obj)
elif isinstance(obj, numpy.ndarray):
return obj.tolist()
else:
return super(MyEncoder, self).default(obj)
You use it like this:json.dumps(numpy.float32(1.2), cls=MyEncoder)
json.dumps(numpy.arange(12), cls=MyEncoder)
json.dumps({'a': numpy.int32(42)}, cls=MyEncoder)
etc.1Or you could just write the default function and pass that as the defaut
keyword argument to json.dumps
. In this scenario, you'd replace the last line with raise TypeError
, but ... meh. The class is more extensible :-)
Converting native python types to numpy dtypes
numpy.float
is just the regular Python float
type. It's not a NumPy dtype. It's almost certainly not what you need:
>>> import numpy
>>> numpy.float is float
True
If you want the dtype NumPy would coerce your scalar to, just make an array and get its dtype:>>> numpy.array(7.7).dtype
dtype('float64')
If you want the type NumPy uses for scalars of this dtype, access the dtype's type
attribute:>>> numpy.array(7.7).dtype.type
<class 'numpy.float64'>
Convert list of numpy.float64 to float in Python quickly
The tolist()
method should do what you want. If you have a numpy array, just call tolist()
:
In [17]: a
Out[17]:
array([ 0. , 0.14285714, 0.28571429, 0.42857143, 0.57142857,
0.71428571, 0.85714286, 1. , 1.14285714, 1.28571429,
1.42857143, 1.57142857, 1.71428571, 1.85714286, 2. ])
In [18]: a.dtype
Out[18]: dtype('float64')
In [19]: b = a.tolist()
In [20]: b
Out[20]:
[0.0,
0.14285714285714285,
0.2857142857142857,
0.42857142857142855,
0.5714285714285714,
0.7142857142857142,
0.8571428571428571,
1.0,
1.1428571428571428,
1.2857142857142856,
1.4285714285714284,
1.5714285714285714,
1.7142857142857142,
1.857142857142857,
2.0]
In [21]: type(b)
Out[21]: list
In [22]: type(b[0])
Out[22]: float
If, in fact, you really have python list of numpy.float64 objects, then @Alexander's answer is great, or you could convert the list to an array and then use the tolist()
method. E.g.In [46]: c
Out[46]:
[0.0,
0.33333333333333331,
0.66666666666666663,
1.0,
1.3333333333333333,
1.6666666666666665,
2.0]
In [47]: type(c)
Out[47]: list
In [48]: type(c[0])
Out[48]: numpy.float64
@Alexander's suggestion, a list comprehension:In [49]: [float(v) for v in c]
Out[49]:
[0.0,
0.3333333333333333,
0.6666666666666666,
1.0,
1.3333333333333333,
1.6666666666666665,
2.0]
Or, convert to an array and then use the tolist()
method.In [50]: np.array(c).tolist()
Out[50]:
[0.0,
0.3333333333333333,
0.6666666666666666,
1.0,
1.3333333333333333,
1.6666666666666665,
2.0]
If you are concerned with the speed, here's a comparison. The input,
x
, is a python list of numpy.float64 objects:In [8]: type(x)
Out[8]: list
In [9]: len(x)
Out[9]: 1000
In [10]: type(x[0])
Out[10]: numpy.float64
Timing for the list comprehension:In [11]: %timeit list1 = [float(v) for v in x]
10000 loops, best of 3: 109 µs per loop
Timing for conversion to numpy array and then tolist()
:In [12]: %timeit list2 = np.array(x).tolist()
10000 loops, best of 3: 70.5 µs per loop
So it is faster to convert the list to an array and then call tolist()
. How can I convert a numpy array of float type into a numpy array of int type?
In [385]: npW_x = np.random.rand(4,5)*10
...: npW_round_111 = np.around(npW_x, decimals=0)
...: sum_x_111 = np.sum(npW_round_111, axis=1)
...:
In [386]: sum_x_111
Out[386]: array([38., 25., 24., 30.])
save with default fmt
(Read and reread the np.savetxt
docs!)In [387]: np.savetxt('test',sum_x_111)
In [388]: cat test
3.800000000000000000e+01
2.500000000000000000e+01
2.400000000000000000e+01
3.000000000000000000e+01
save with int format:In [389]: np.savetxt('test',sum_x_111, fmt='%10d')
In [390]: cat test
38
25
24
30
conversion in memory to int:In [391]: In [391]: sum_x_111.astype(int)
Out[391]: array([38, 25, 24, 30])
Convert numpy array type and values from Float64 to Float32
For more info, you can follow this link:Actually i tried hard but not able to do as the
'sklearn.tree._tree.Tree' objects is not writable.It is causing a precision issue while generating a PMML file, so i
raised a bug over there and they gave an updated solution for it by
not converting it in to the Float64 internally.
Precision Issue
Tensorflow - ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type float)
TL;DR Several possible errors, most fixed with x = np.asarray(x).astype('float32')
.
Others may be faulty data preprocessing; ensure everything is properly formatted (categoricals, nans, strings, etc). Below shows what the model expects:
[print(i.shape, i.dtype) for i in model.inputs]
[print(o.shape, o.dtype) for o in model.outputs]
[print(l.name, l.input_shape, l.dtype) for l in model.layers]
The problem's rooted in using lists as inputs, as opposed to Numpy arrays; Keras/TF doesn't support former. A simple conversion is:
x_array = np.asarray(x_list)
.The next step's to ensure data is fed in expected format; for LSTM, that'd be a 3D tensor with dimensions (batch_size, timesteps, features)
- or equivalently, (num_samples, timesteps, channels)
. Lastly, as a debug pro-tip, print ALL the shapes for your data. Code accomplishing all of the above, below:
Sequences = np.asarray(Sequences)
Targets = np.asarray(Targets)
show_shapes()
Sequences = np.expand_dims(Sequences, -1)
Targets = np.expand_dims(Targets, -1)
show_shapes()
# OUTPUTS
Expected: (num_samples, timesteps, channels)
Sequences: (200, 1000)
Targets: (200,)
Expected: (num_samples, timesteps, channels)
Sequences: (200, 1000, 1)
Targets: (200, 1)
As a bonus tip, I notice you're running via
main()
, so your IDE probably lacks a Jupyter-like cell-based execution; I strongly recommend the Spyder IDE. It's as simple as adding # In[]
, and pressing Ctrl + Enter
below:Function used:
def show_shapes(): # can make yours to take inputs; this'll use local variable values
print("Expected: (num_samples, timesteps, channels)")
print("Sequences: {}".format(Sequences.shape))
print("Targets: {}".format(Targets.shape))
Related Topics
Plotting Results of Hierarchical Clustering Ontop of a Matrix of Data in Python
Adding a Y-Axis Label to Secondary Y-Axis in Matplotlib
Python Ignore Certificate Validation Urllib2
Python Pip on Windows - Command 'Cl.Exe' Failed
Runtimeerror: This Event Loop Is Already Running in Python
Hide Chromedriver Console in Python
Counting Cars Opencv + Python Issue
Django/Python Beginner: Error When Executing Python Manage.Py Syncdb - Psycopg2 Not Found
Pandas Dataframe Column to List
Getting Data from Ctypes Array into Numpy
How to Troubleshoot Python "Could Not Find Platform Independent Libraries <Prefix>"
How to Exit from Python Without Traceback
Destructuring-Bind Dictionary Contents