How to use type hints in python 3.6?
Is that possible? Maybe mypy could do it, but I'd prefer to use Python-3.6-style type checking (like
a: List[str]
) instead of the comment-style (like# type: List[str]
) used in mypy. And I'm curious if there's a switch in native python 3.6 to achieve the two points I said above.
There's no way Python will do this for you; you can use mypy
to get type checking (and PyCharms built-in checker should do it too). In addition to that, mypy
also doesn't restrict you to only type comments # type List[str]
, you can use variable annotations as you do in Python 3.6 so a: List[str]
works equally well.
With mypy
as is, because the release is fresh, you'll need to install typed_ast
and execute mypy
with --fast-parser
and --python-version 3.6
as documented in mypy's docs. This will probably change soon but for now you'll need them to get it running smoothly
Update: --fast-parser
and --python-version 3.6
aren't needed now.
After you do that, mypy detects the incompatibility of the second operation on your a: List[str]
just fine. Let's say your file is called tp_check.py
with statements:
from typing import List
a: List[str] = []
a.append('a')
a.append(1)
print(a)
Running mypy
with the aforementioned arguments (you must first pip install -U typed_ast
):
python -m mypy --fast-parser --python-version 3.6 tp_check.py
catches the error:
tp_check.py:5: error: Argument 1 to "append" of "list" has incompatible type "int"; expected "str"
As noted in many other answers on type hinting with Python, mypy
and PyCharm
s' type-checkers are the ones performing the validation, not Python itself. Python doesn't use this information currently, it only stores it as metadata and ignores it during execution.
What are type hints in Python 3.5?
I would suggest reading PEP 483 and PEP 484 and watching this presentation by Guido on type hinting.
In a nutshell: Type hinting is literally what the words mean. You hint the type of the object(s) you're using.
Due to the dynamic nature of Python, inferring or checking the type of an object being used is especially hard. This fact makes it hard for developers to understand what exactly is going on in code they haven't written and, most importantly, for type checking tools found in many IDEs (PyCharm and PyDev come to mind) that are limited due to the fact that they don't have any indicator of what type the objects are. As a result they resort to trying to infer the type with (as mentioned in the presentation) around 50% success rate.
To take two important slides from the type hinting presentation:
Why type hints?
- Helps type checkers: By hinting at what type you want the object to be the type checker can easily detect if, for instance, you're passing an object with a type that isn't expected.
- Helps with documentation: A third person viewing your code will know what is expected where, ergo, how to use it without getting them
TypeErrors
. - Helps IDEs develop more accurate and robust tools: Development Environments will be better suited at suggesting appropriate methods when know what type your object is. You have probably experienced this with some IDE at some point, hitting the
.
and having methods/attributes pop up which aren't defined for an object.
Why use static type checkers?
- Find bugs sooner: This is self-evident, I believe.
- The larger your project the more you need it: Again, makes sense. Static languages offer a robustness and control that
dynamic languages lack. The bigger and more complex your application becomes the more control and predictability (from
a behavioral aspect) you require. - Large teams are already running static analysis: I'm guessing this verifies the first two points.
As a closing note for this small introduction: This is an optional feature and, from what I understand, it has been introduced in order to reap some of the benefits of static typing.
You generally do not need to worry about it and definitely don't need to use it (especially in cases where you use Python as an auxiliary scripting language). It should be helpful when developing large projects as it offers much needed robustness, control and additional debugging capabilities.
Type hinting with mypy:
In order to make this answer more complete, I think a little demonstration would be suitable. I'll be using mypy
, the library which inspired Type Hints as they are presented in the PEP. This is mainly written for anybody bumping into this question and wondering where to begin.
Before I do that let me reiterate the following: PEP 484 doesn't enforce anything; it is simply setting a direction for function
annotations and proposing guidelines for how type checking can/should be performed. You can annotate your functions and
hint as many things as you want; your scripts will still run regardless of the presence of annotations because Python itself doesn't use them.
Anyways, as noted in the PEP, hinting types should generally take three forms:
- Function annotations (PEP 3107).
- Stub files for built-in/user modules.
- Special
# type: type
comments that complement the first two forms. (See: What are variable annotations? for a Python 3.6 update for# type: type
comments)
Additionally, you'll want to use type hints in conjunction with the new typing
module introduced in Py3.5
. In it, many (additional) ABCs (abstract base classes) are defined along with helper functions and decorators for use in static checking. Most ABCs in collections.abc
are included, but in a generic form in order to allow subscription (by defining a __getitem__()
method).
For anyone interested in a more in-depth explanation of these, the mypy documentation
is written very nicely and has a lot of code samples demonstrating/describing the functionality of their checker; it is definitely worth a read.
Function annotations and special comments:
First, it's interesting to observe some of the behavior we can get when using special comments. Special # type: type
comments
can be added during variable assignments to indicate the type of an object if one cannot be directly inferred. Simple assignments are
generally easily inferred but others, like lists (with regard to their contents), cannot.
Note: If we want to use any derivative of containers and need to specify the contents for that container we must use the generic types from the typing
module. These support indexing.
# Generic List, supports indexing.
from typing import List
# In this case, the type is easily inferred as type: int.
i = 0
# Even though the type can be inferred as of type list
# there is no way to know the contents of this list.
# By using type: List[str] we indicate we want to use a list of strings.
a = [] # type: List[str]
# Appending an int to our list
# is statically not correct.
a.append(i)
# Appending a string is fine.
a.append("i")
print(a) # [0, 'i']
If we add these commands to a file and execute them with our interpreter, everything works just fine and print(a)
just prints
the contents of list a
. The # type
comments have been discarded, treated as plain comments which have no additional semantic meaning.
By running this with mypy
, on the other hand, we get the following response:
(Python3)jimmi@jim: mypy typeHintsCode.py
typesInline.py:14: error: Argument 1 to "append" of "list" has incompatible type "int"; expected "str"
Indicating that a list of str
objects cannot contain an int
, which, statically speaking, is sound. This can be fixed by either abiding to the type of a
and only appending str
objects or by changing the type of the contents of a
to indicate that any value is acceptable (Intuitively performed with List[Any]
after Any
has been imported from typing
).
Function annotations are added in the form param_name : type
after each parameter in your function signature and a return type is specified using the -> type
notation before the ending function colon; all annotations are stored in the __annotations__
attribute for that function in a handy dictionary form. Using a trivial example (which doesn't require extra types from the typing
module):
def annotated(x: int, y: str) -> bool:
return x < y
The annotated.__annotations__
attribute now has the following values:
{'y': <class 'str'>, 'return': <class 'bool'>, 'x': <class 'int'>}
If we're a complete newbie, or we are familiar with Python 2.7 concepts and are consequently unaware of the TypeError
lurking in the comparison of annotated
, we can perform another static check, catch the error and save us some trouble:
(Python3)jimmi@jim: mypy typeHintsCode.py
typeFunction.py: note: In function "annotated":
typeFunction.py:2: error: Unsupported operand types for > ("str" and "int")
Among other things, calling the function with invalid arguments will also get caught:
annotated(20, 20)
# mypy complains:
typeHintsCode.py:4: error: Argument 2 to "annotated" has incompatible type "int"; expected "str"
These can be extended to basically any use case and the errors caught extend further than basic calls and operations. The types you
can check for are really flexible and I have merely given a small sneak peak of its potential. A look in the typing
module, the
PEPs or the mypy
documentation will give you a more comprehensive idea of the capabilities offered.
Stub files:
Stub files can be used in two different non mutually exclusive cases:
- You need to type check a module for which you do not want to directly alter the function signatures
- You want to write modules and have type-checking but additionally want to separate annotations from content.
What stub files (with an extension of .pyi
) are is an annotated interface of the module you are making/want to use. They contain
the signatures of the functions you want to type-check with the body of the functions discarded. To get a feel of this, given a set
of three random functions in a module named randfunc.py
:
def message(s):
print(s)
def alterContents(myIterable):
return [i for i in myIterable if i % 2 == 0]
def combine(messageFunc, itFunc):
messageFunc("Printing the Iterable")
a = alterContents(range(1, 20))
return set(a)
We can create a stub file randfunc.pyi
, in which we can place some restrictions if we wish to do so. The downside is that
somebody viewing the source without the stub won't really get that annotation assistance when trying to understand what is supposed
to be passed where.
Anyway, the structure of a stub file is pretty simplistic: Add all function definitions with empty bodies (pass
filled) and
supply the annotations based on your requirements. Here, let's assume we only want to work with int
types for our Containers.
# Stub for randfucn.py
from typing import Iterable, List, Set, Callable
def message(s: str) -> None: pass
def alterContents(myIterable: Iterable[int])-> List[int]: pass
def combine(
messageFunc: Callable[[str], Any],
itFunc: Callable[[Iterable[int]], List[int]]
)-> Set[int]: pass
The combine
function gives an indication of why you might want to use annotations in a different file, they some times clutter up
the code and reduce readability (big no-no for Python). You could of course use type aliases but that sometime confuses more than it
helps (so use them wisely).
This should get you familiarized with the basic concepts of type hints in Python. Even though the type checker used has beenmypy
you should gradually start to see more of them pop-up, some internally in IDEs (PyCharm,) and others as standard Python modules.
I'll try and add additional checkers/related packages in the following list when and if I find them (or if suggested).
Checkers I know of:
- Mypy: as described here.
- PyType: By Google, uses different notation from what I gather, probably worth a look.
Related Packages/Projects:
- typeshed: Official Python repository housing an assortment of stub files for the standard library.
The typeshed
project is actually one of the best places you can look to see how type hinting might be used in a project of your own. Let's take as an example the __init__
dunders of the Counter
class in the corresponding .pyi
file:
class Counter(Dict[_T, int], Generic[_T]):
@overload
def __init__(self) -> None: ...
@overload
def __init__(self, Mapping: Mapping[_T, int]) -> None: ...
@overload
def __init__(self, iterable: Iterable[_T]) -> None: ...
Where _T = TypeVar('_T')
is used to define generic classes. For the Counter
class we can see that it can either take no arguments in its initializer, get a single Mapping
from any type to an int
or take an Iterable
of any type.
Notice: One thing I forgot to mention was that the typing
module has been introduced on a provisional basis. From PEP 411:
A provisional package may have its API modified prior to "graduating" into a "stable" state. On one hand, this state provides the package with the benefits of being formally part of the Python distribution. On the other hand, the core development team explicitly states that no promises are made with regards to the the stability of the package's API, which may change for the next release. While it is considered an unlikely outcome, such packages may even be removed from the standard library without a deprecation period if the concerns regarding their API or maintenance prove well-founded.
So take things here with a pinch of salt; I'm doubtful it will be removed or altered in significant ways, but one can never know.
** Another topic altogether, but valid in the scope of type-hints: PEP 526
: Syntax for Variable Annotations is an effort to replace # type
comments by introducing new syntax which allows users to annotate the type of variables in simple varname: type
statements.
See What are variable annotations?, as previously mentioned, for a small introduction to these.
Python 3.6 type hinting for a function accepting generic class type and instance type of the same generic type
But Type[T] is TypeVar, so it's not the way to go.
No, you are on the right track - TypeVar
is definitely the way to go. The problem here is rather in pykube.objects.APIObject
class being wrapped in a decorator that mypy
cannot deal with yet. Adding type stubs for pykube.objects
will resolve the issue. Create a directory _typeshed/pykube
and add minimal type stubs for pykube
:
_typeshed/pykube/__init__.pyi
:from typing import Any
def __getattr__(name: str) -> Any: ... # incomplete_typeshed/pykube/objects.pyi
:from typing import Any, ClassVar, Optional
from pykube.query import Query
def __getattr__(name: str) -> Any: ... # incomplete
class ObjectManager:
def __getattr__(self, name: str) -> Any: ... # incomplete
def __call__(self, api: Any, namespace: Optional[Any] = None) -> Query: ...
class APIObject:
objects: ClassVar[ObjectManager]
def __getattr__(self, name: str) -> Any: ... # incomplete
class NamespacedAPIObject(APIObject): ...
Now running
$ MYPYPATH=_typeshed mypy pytest_helm_charts/
resolves obj_type.objects
correctly:
T = TypeVar('T', bound=NamespacedAPIObject)
def wait_for_namespaced_objects_condition(obj_type: Type[T]) -> List[T]:
reveal_type(obj_type.objects)
Output:
pytest_helm_charts/utils.py:29: note: Revealed type is 'pykube.objects.ObjectManager'
How do I type hint a method with the type of the enclosing class?
TL;DR: As of today (2019), in Python 3.7+ you can turn this feature on using a "future" statement, from __future__ import annotations
.
(The behaviour enabled by from __future__ import annotations
might become the default in future versions of Python, and was going to be made the default in Python 3.10. However, the change in 3.10 was reverted at the last minute, and now may not happen at all.)
In Python 3.6 or below, you should use a string.
I guess you got this exception:
NameError: name 'Position' is not defined
This is because Position
must be defined before you can use it in an annotation, unless you are using Python with PEP 563 changes enabled.
Python 3.7+: from __future__ import annotations
Python 3.7 introduces PEP 563: postponed evaluation of annotations. A module that uses the future statement from __future__ import annotations
will store annotations as strings automatically:
from __future__ import annotations
class Position:
def __add__(self, other: Position) -> Position:
...
This had been scheduled to become the default in Python 3.10, but this change has now been postponed. Since Python still is a dynamically typed language so no type-checking is done at runtime, typing annotations should have no performance impact, right? Wrong! Before Python 3.7, the typing module used to be one of the slowest python modules in core so for code that involves importing the typing
module, you will see an up to 7 times increase in performance when you upgrade to 3.7.
Python <3.7: use a string
According to PEP 484, you should use a string instead of the class itself:
class Position:
...
def __add__(self, other: 'Position') -> 'Position':
...
If you use the Django framework, this may be familiar, as Django models also use strings for forward references (foreign key definitions where the foreign model is self
or is not declared yet). This should work with Pycharm and other tools.
Sources
The relevant parts of PEP 484 and PEP 563, to spare you the trip:
Forward references
When a type hint contains names that have not been defined yet, that definition may be expressed as a string literal, to be resolved later.
A situation where this occurs commonly is the definition of a container class, where the class being defined occurs in the signature of some of the methods. For example, the following code (the start of a simple binary tree implementation) does not work:
class Tree:
def __init__(self, left: Tree, right: Tree):
self.left = left
self.right = rightTo address this, we write:
class Tree:
def __init__(self, left: 'Tree', right: 'Tree'):
self.left = left
self.right = rightThe string literal should contain a valid Python expression (i.e., compile(lit, '', 'eval') should be a valid code object) and it should evaluate without errors once the module has been fully loaded. The local and global namespace in which it is evaluated should be the same namespaces in which default arguments to the same function would be evaluated.
and PEP 563:
Implementation
In Python 3.10, function and variable annotations will no longer be evaluated at definition time. Instead, a string form will be preserved in the respective
__annotations__
dictionary. Static type checkers will see no difference in behavior, whereas tools using annotations at runtime will have to perform postponed evaluation....
Enabling the future behavior in Python 3.7
The functionality described above can be enabled starting from Python 3.7 using the following special import:
from __future__ import annotations
Things that you may be tempted to do instead
A. Define a dummy Position
Before the class definition, place a dummy definition:
class Position(object):
pass
class Position(object):
...
This will get rid of the NameError
and may even look OK:
>>> Position.__add__.__annotations__
{'other': __main__.Position, 'return': __main__.Position}
But is it?
>>> for k, v in Position.__add__.__annotations__.items():
... print(k, 'is Position:', v is Position)
return is Position: False
other is Position: False
B. Monkey-patch in order to add the annotations:
You may want to try some Python metaprogramming magic and write a decorator
to monkey-patch the class definition in order to add annotations:
class Position:
...
def __add__(self, other):
return self.__class__(self.x + other.x, self.y + other.y)
The decorator should be responsible for the equivalent of this:
Position.__add__.__annotations__['return'] = Position
Position.__add__.__annotations__['other'] = Position
At least it seems right:
>>> for k, v in Position.__add__.__annotations__.items():
... print(k, 'is Position:', v is Position)
return is Position: True
other is Position: True
Probably too much trouble.
What is the correct way in python to annotate a path with type hints?
I assume that typical path objects are either Path
s or str
s, as such you could use a Union
:
import pathlib
import typing
typing.Union[str, pathlib.Path]
Related Topics
What Is the Most Compatible Way to Install Python Modules on a MAC
Flask-Sqlalchemy Import/Context Issue
Why Doesn't Print Work in a Lambda
How to Combine Two Lists into a Dictionary in Python
How to Read File with Space Separated Values in Pandas
Memory-Efficient Built-In SQLalchemy Iterator/Generator
How to Import Data from Mongodb to Pandas
How to Break Up This Long Line in Python
Calculate Time Difference Between Pandas Dataframe Indices
Python - Activate Conda Env Through Shell Script
Python: Changing Methods and Attributes at Runtime
How to Concatenate Two Layers in Keras
Passing Csrftoken with Python Requests
How to Bind the Enter Key to a Function in Tkinter
Pandas Deleting Row with Df.Drop Doesn't Work
Running Jupyter via Command Line on Windows