Converting Between Datetime and Pandas Timestamp Objects

Converting between datetime and Pandas Timestamp objects

You can use the to_pydatetime method to be more explicit:

In [11]: ts = pd.Timestamp('2014-01-23 00:00:00', tz=None)

In [12]: ts.to_pydatetime()
Out[12]: datetime.datetime(2014, 1, 23, 0, 0)

It's also available on a DatetimeIndex:

In [13]: rng = pd.date_range('1/10/2011', periods=3, freq='D')

In [14]: rng.to_pydatetime()
Out[14]:
array([datetime.datetime(2011, 1, 10, 0, 0),
       datetime.datetime(2011, 1, 11, 0, 0),
       datetime.datetime(2011, 1, 12, 0, 0)], dtype=object)

Converting between datetime, Timestamp and datetime64

To convert numpy.datetime64 to datetime object that represents time in UTC on numpy-1.8:

>>> from datetime import datetime
>>> import numpy as np
>>> dt = datetime.utcnow()
>>> dt
datetime.datetime(2012, 12, 4, 19, 51, 25, 362455)
>>> dt64 = np.datetime64(dt)
>>> ts = (dt64 - np.datetime64('1970-01-01T00:00:00Z')) / np.timedelta64(1, 's')
>>> ts
1354650685.3624549
>>> datetime.utcfromtimestamp(ts)
datetime.datetime(2012, 12, 4, 19, 51, 25, 362455)
>>> np.__version__
'1.8.0.dev-7b75899'

The above example assumes that a naive datetime object is interpreted by np.datetime64 as time in UTC.

---

To convert datetime to np.datetime64 and back (numpy-1.6):

>>> np.datetime64(datetime.utcnow()).astype(datetime)
datetime.datetime(2012, 12, 4, 13, 34, 52, 827542)

It works both on a single np.datetime64 object and a numpy array of np.datetime64.

Think of np.datetime64 the same way you would about np.int8, np.int16, etc and apply the same methods to convert between Python objects such as int, datetime and corresponding numpy objects.

Your "nasty example" works correctly:

>>> from datetime import datetime
>>> import numpy 
>>> numpy.datetime64('2002-06-28T01:00:00.000000000+0100').astype(datetime)
datetime.datetime(2002, 6, 28, 0, 0)
>>> numpy.__version__
'1.6.2' # current version available via pip install numpy

I can reproduce the long value on numpy-1.8.0 installed as:

pip install git+https://github.com/numpy/numpy.git#egg=numpy-dev

The same example:

>>> from datetime import datetime
>>> import numpy
>>> numpy.datetime64('2002-06-28T01:00:00.000000000+0100').astype(datetime)
1025222400000000000L
>>> numpy.__version__
'1.8.0.dev-7b75899'

It returns long because for numpy.datetime64 type .astype(datetime) is equivalent to .astype(object) that returns Python integer (long) on numpy-1.8.

To get datetime object you could:

>>> dt64.dtype
dtype('<M8[ns]')
>>> ns = 1e-9 # number of seconds in a nanosecond
>>> datetime.utcfromtimestamp(dt64.astype(int) * ns)
datetime.datetime(2002, 6, 28, 0, 0)

To get datetime64 that uses seconds directly:

>>> dt64 = numpy.datetime64('2002-06-28T01:00:00.000000000+0100', 's')
>>> dt64.dtype
dtype('<M8[s]')
>>> datetime.utcfromtimestamp(dt64.astype(int))
datetime.datetime(2002, 6, 28, 0, 0)

The numpy docs say that the datetime API is experimental and may change in future numpy versions.

Pandas: Convert Timestamp to datetime.date

Use the .date method:

In [11]: t = pd.Timestamp('2013-12-25 00:00:00')

In [12]: t.date()
Out[12]: datetime.date(2013, 12, 25)

In [13]: t.date() == datetime.date(2013, 12, 25)
Out[13]: True

---

To compare against a DatetimeIndex (i.e. an array of Timestamps), you'll want to do it the other way around:

In [21]: pd.Timestamp(datetime.date(2013, 12, 25))
Out[21]: Timestamp('2013-12-25 00:00:00')

In [22]: ts = pd.DatetimeIndex([t])

In [23]: ts == pd.Timestamp(datetime.date(2013, 12, 25))
Out[23]: array([ True], dtype=bool)

How to properly convert a UNIX timestamp to pd.Timestamp object via pandas?

Problem is quite simple but not obvious. utcnow() gives you a naive datetime object, meaning that it is not aware of the fact that it represents UTC. Therefor, once you call .timestamp(), Python assumes local time because the datetime object is naive! Thus converts to UTC first before calculating Unix time, adding any UTC offset that your local tz might have.

Solution: construct a datetime object that is aware of UTC. Same goes for fromtimestamp: set UTC as tz !

from datetime import datetime, timezone
import pandas as pd

d = datetime.now(timezone.utc)
timestamp = d.timestamp()

assert datetime.fromtimestamp(timestamp, tz=timezone.utc) == d
assert pd.to_datetime(timestamp, unit="s", utc=True).to_pydatetime() == d

pandas is kind of a different story; naive datetime is treated internally as UTC, so pd.to_datetime(timestamp, unit="s") gives you the UTC timestamp. But the conversion to Python datetime does not take into account that Python will treat it as local time again... Here, keeping it consistent and setting utc=True (i.e. using an aware Timestamp) makes it work nicely.

• Further reading: Stop using utcnow and utcfromtimestamp

pandas convert from datetime to integer timestamp

You can typecast to int using astype(int) and divide it by 10**9 to get the number of seconds to the unix epoch start.

import pandas as pd
df = pd.DataFrame({'time': [pd.to_datetime('2019-01-15 13:25:43')]})
df_unix_sec = pd.to_datetime(df['time']).astype(int)/ 10**9
print(df_unix_sec)

Python pandas convert datetime to timestamp effectively through dt accessor

I think you need convert first to numpy array by values and cast to int64 - output is in ns, so need divide by 10 ** 9:

df['ts'] = df.datetime.values.astype(np.int64) // 10 ** 9
print (df)
              datetime          ts
0  2016-01-01 00:00:01  1451606401
1  2016-01-01 01:00:01  1451610001
2  2016-01-01 02:00:01  1451613601
3  2016-01-01 03:00:01  1451617201
4  2016-01-01 04:00:01  1451620801
5  2016-01-01 05:00:01  1451624401
6  2016-01-01 06:00:01  1451628001
7  2016-01-01 07:00:01  1451631601
8  2016-01-01 08:00:01  1451635201
9  2016-01-01 09:00:01  1451638801
10 2016-01-01 10:00:01  1451642401
11 2016-01-01 11:00:01  1451646001
12 2016-01-01 12:00:01  1451649601
13 2016-01-01 13:00:01  1451653201
14 2016-01-01 14:00:01  1451656801
15 2016-01-01 15:00:01  1451660401
16 2016-01-01 16:00:01  1451664001
17 2016-01-01 17:00:01  1451667601
18 2016-01-01 18:00:01  1451671201
19 2016-01-01 19:00:01  1451674801
20 2016-01-01 20:00:01  1451678401
21 2016-01-01 21:00:01  1451682001
22 2016-01-01 22:00:01  1451685601
23 2016-01-01 23:00:01  1451689201
24 2016-01-02 00:00:01  1451692801

to_timestamp is used for converting from period to datetime index.

Dates to timestamp in pandas

We can do strftime

df=df.apply(pd.to_datetime)
print(df.start_date.dt.strftime('%Y-%m-%d %H:%M:%S'))
0    2013-01-10 00:00:00
1    2013-01-26 00:00:00
2    2013-01-26 00:00:00
3    2013-01-26 00:00:00
4    2013-02-13 00:00:00
Name: start_date, dtype: object

How do I convert timestamp to datetime.date in pandas dataframe?

I got some help from a colleague.

This appears to solve the problem posted above

pd.to_datetime(df['mydates']).apply(lambda x: x.date())

Converting Pandas Timestamp to python native Datetime not working

Pandas uses its own timestamp object for their indexes (makes sense because the native python datetime does not implement allt the interface pandas needs).

You can check it with this example:

>>> df.index.map(lambda x: f'{x} as {type(x)}')
Index(['2018-01-01 00:00:00 as <class 'pandas._libs.tslibs.timestamps.Timestamp'>',
       '2018-01-01 01:00:00 as <class 'pandas._libs.tslibs.timestamps.Timestamp'>',
       '2018-01-01 02:00:00 as <class 'pandas._libs.tslibs.timestamps.Timestamp'>',
       '2018-01-01 03:00:00 as <class 'pandas._libs.tslibs.timestamps.Timestamp'>',
       '2018-01-01 04:00:00 as <class 'pandas._libs.tslibs.timestamps.Timestamp'>'],
      dtype='object')

So, what happens when try to set the index with pydatetime objects like this?

df.index = df.index.to_pydatetime()

• First python calls df.index.to_pydatetime()
• And you have an ndarray of datetime objects on the right hand side of the assignment.
• Then python calls
  df.index.equals ( returned ndarray ) which performs the type conversion.

---

Related Topics