Can you explain closures (as they relate to Python)?
Closure on closures
Objects are data with methods
attached, closures are functions with
data attached.
def make_counter():
i = 0
def counter(): # counter() is a closure
nonlocal i
i += 1
return i
return counter
c1 = make_counter()
c2 = make_counter()
print (c1(), c1(), c2(), c2())
# -> 1 2 1 2
Why aren't python nested functions called closures?
A closure occurs when a function has access to a local variable from an enclosing scope that has finished its execution.
def make_printer(msg):
def printer():
print(msg)
return printer
printer = make_printer('Foo!')
printer()
When make_printer is called, a new frame is put on the stack with the compiled code for the printer function as a constant and the value of msg as a local. It then creates and returns the function. Because the function printer references the msg variable, it is kept alive after the make_printer function has returned.
So, if your nested functions don't
- access variables that are local to enclosing scopes,
- do so when they are executed outside of that scope,
then they are not closures.
Here's an example of a nested function which is not a closure.
def make_printer(msg):
def printer(msg=msg):
print(msg)
return printer
printer = make_printer("Foo!")
printer() #Output: Foo!
Here, we are binding the value to the default value of a parameter. This occurs when the function printer is created and so no reference to the value of msg external to printer needs to be maintained after make_printer returns. msg is just a normal local variable of the function printer in this context.
Can anybody explain about the closure of function in Python?
Higher-order functions
Functions like makeInc that in turn, return another function are called higher order functions. Usually, functions are known to accept data as input and return data as output. With higher order functions, functions instead of data, either return code as output or accept code as input. This code is wrapped into a function. In Python, functions are first class citizens which means functions, just like data, can be passed around. For instance:
myvariable = print
Notice, how I have assigned print to myvariable and how I have dropped the parentheses after print Functions without parentheses are called function objects. This means myvariable now is just another name for print:
print("Hello World!")
myvariable("Hello World!")
Both of the above statements do the exact same thing. What can be assigned to variables can also be returned from functions:
def myfunction():
return print
myfunction()("Hello World!");
Now let's look at your example:
def makeInc(x):
def inc(y):
return y + x
return inc
makeInc is a function that accepts a parameter called x. It then defines another nested inner function called inc which takes in a parameter called y. The thing about nested functions is that they have access to the variables of the enclosing function as well. Here, inc is the inner function but it has access to x which is a variable of the enclosing outer scope.
The last statement return inc returns the inner function to the caller of makeInc. What makeInc essentially is doing, is creating a custom function based on the parameter it receives.
For instance:
x = makeInc(10)
makeInc will first accept 10 and then return a function that takes in an argument y and it increments y by 10.
Here, x is a function that takes in any argument y and then increments it by 10:
x(42) # Returns 52
nonlocal
However, there is a caveat when using nested functions:
def outer():
x = 10
def inner():
x = 20
inner()
print(x) # prints 10
Here, you would assume that the last print statement will print 20. But no! When you assign x = 20 in the inner function, it creates a new local variable called x which is initialized to 20. The outer x remains untouched. To modify the outer x, use the nonlocal keyword:
def outer():
x = 10
def inner():
nonlocal x = 20
inner()
print(x) # prints 20
If you are directly reading x inside inner() instead of assigning to it, you do not need nonlocal.
Python: How can a closure continue its existence after the outer function is reassigned?
In CPython (i.e. the reference implementation written in C that most people think of as just "Python"), lexical closures are implemented as 'flat closures' (see PEP 227) that use cell object references instead of searching a linked list of frame objects (nested scopes) at runtime. This allows for quick lookup and improves garbage collection when returning a closure function.
The bytecode in outer_function is specialized to access a cell object in the stack frame instead of directly referencing the message object. The interpreter knows about this when setting up the stack frame for the call because the code object has this variable defined as a cell variable:
>>> outer_function.__code__.co_cellvars
('message',)
The bytecode in inner_function also dereferences a cell object for the value of message, but since it's not the source of the object, it's classified instead as a free variable:
>>> type(outer_function.__code__.co_consts[1])
<class 'code'>
>>> outer_function.__code__.co_consts[1].co_name
'inner_function'
>>> outer_function.__code__.co_consts[1].co_freevars
('message',)
Each inner_function function object that gets instantiated has a __closure__ tuple that references the cell for the enclosed free variable. For example:
>>> function = outer_function('hello')
>>> type(function.__closure__[0])
<class 'cell'>
>>> function.__closure__[0].cell_contents
'hello'
The cells in this __closure__ tuple are loaded in the stack frame when function is called.
This tuple of cells is what makes it flattened. No matter how deeply you nest the scopes, the __closure__ will always propagate all of the required cells. For example:
def a():
x = 1
def b():
def c():
def d():
x
print('d.__closure__[0].cell_contents:',
d.__closure__[0].cell_contents)
print('c.__closure__[0].cell_contents:',
c.__closure__[0].cell_contents)
c()
print('b.__closure__[0].cell_contents:',
b.__closure__[0].cell_contents)
b()
>>> a()
b.__closure__[0].cell_contents: 1
c.__closure__[0].cell_contents: 1
d.__closure__[0].cell_contents: 1
Functions b and c don't directly reference x, but they have to propagate the cell for the inner function d to reference it.
The above inspection relies on CPython implementation details. In Python 3.3+ you can instead call inspect.getclosurevars to inspect closure variables. For example:
import inspect
def outer_function(hello):
message = hello
def inner_function():
print(message)
return inner_function
>>> function = outer_function('hello')
>>> inspect.getclosurevars(function)
ClosureVars(nonlocals={'message': 'hello'},
globals={},
builtins={'print': <built-in function print>},
unbound=set())
What limitations have closures in Python compared to language X closures?
The most important limitation, currently, is that you cannot assign to an outer-scope variable. In other words, closures are read-only:
>>> def outer(x):
... def inner_reads():
... # Will return outer's 'x'.
... return x
... def inner_writes(y):
... # Will assign to a local 'x', not the outer 'x'
... x = y
... def inner_error(y):
... # Will produce an error: 'x' is local because of the assignment,
... # but we use it before it is assigned to.
... tmp = x
... x = y
... return tmp
... return inner_reads, inner_writes, inner_error
...
>>> inner_reads, inner_writes, inner_error = outer(5)
>>> inner_reads()
5
>>> inner_writes(10)
>>> inner_reads()
5
>>> inner_error(10)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 11, in inner_error
UnboundLocalError: local variable 'x' referenced before assignment
A name that gets assigned to in a local scope (a function) is always local, unless declared otherwise. While there is the 'global' declaration to declare a variable global even when it is assigned to, there is no such declaration for enclosed variables -- yet. In Python 3.0, there is (will be) the 'nonlocal' declaration that does just that.
You can work around this limitation in the mean time by using a mutable container type:
>>> def outer(x):
... x = [x]
... def inner_reads():
... # Will return outer's x's first (and only) element.
... return x[0]
... def inner_writes(y):
... # Will look up outer's x, then mutate it.
... x[0] = y
... def inner_error(y):
... # Will now work, because 'x' is not assigned to, just referenced.
... tmp = x[0]
... x[0] = y
... return tmp
... return inner_reads, inner_writes, inner_error
...
>>> inner_reads, inner_writes, inner_error = outer(5)
>>> inner_reads()
5
>>> inner_writes(10)
>>> inner_reads()
10
>>> inner_error(15)
10
>>> inner_reads()
15
what are python closures good for?
I think the most used example of closure is for caching functions with a decorator.
def cache_decorator(f):
cache = {}
def wrapper(*args):
if args not in cache:
cache[args] = f(*args)
return cache[args]
return wrapper
@cache_decorator
def some_function(*args):
...
This way the cache cannot be referenced from anywhere, since you do not want your users to tamper with it.
What is a 'Closure'?
Variable scope
When you declare a local variable, that variable has a scope. Generally, local variables exist only within the block or function in which you declare them.
function() {
var a = 1;
console.log(a); // works
}
console.log(a); // fails
If I try to access a local variable, most languages will look for it in the current scope, then up through the parent scopes until they reach the root scope.
var a = 1;
function() {
console.log(a); // works
}
console.log(a); // works
When a block or function is done with, its local variables are no longer needed and are usually blown out of memory.
This is how we normally expect things to work.
A closure is a persistent local variable scope
A closure is a persistent scope which holds on to local variables even after the code execution has moved out of that block. Languages which support closure (such as JavaScript, Swift, and Ruby) will allow you to keep a reference to a scope (including its parent scopes), even after the block in which those variables were declared has finished executing, provided you keep a reference to that block or function somewhere.
The scope object and all its local variables are tied to the function and will persist as long as that function persists.
This gives us function portability. We can expect any variables that were in scope when the function was first defined to still be in scope when we later call the function, even if we call the function in a completely different context.
For example
Here's a really simple example in JavaScript that illustrates the point:
outer = function() {
var a = 1;
var inner = function() {
console.log(a);
}
return inner; // this returns a function
}
var fnc = outer(); // execute outer to get inner
fnc();
Here I have defined a function within a function. The inner function gains access to all the outer function's local variables, including a. The variable a is in scope for the inner function.
Normally when a function exits, all its local variables are blown away. However, if we return the inner function and assign it to a variable fnc so that it persists after outer has exited, all of the variables that were in scope when inner was defined also persist. The variable a has been closed over -- it is within a closure.
Note that the variable a is totally private to fnc. This is a way of creating private variables in a functional programming language such as JavaScript.
As you might be able to guess, when I call fnc() it prints the value of a, which is "1".
In a language without closure, the variable a would have been garbage collected and thrown away when the function outer exited. Calling fnc would have thrown an error because a no longer exists.
In JavaScript, the variable a persists because the variable scope is created when the function is first declared and persists for as long as the function continues to exist.
a belongs to the scope of outer. The scope of inner has a parent pointer to the scope of outer. fnc is a variable which points to inner. a persists as long as fnc persists. a is within the closure.
Further reading (watching)
I made a YouTube video looking at this code with some practical examples of usage.
How are python closures implemented?
Overview
Python doesn't directly use variables the same way one might expect coming from a statically-typed language like C or Java, rather it uses names and tags instances of objects with them
In your example, closure is simply an instance of a function with that name
It's really nonlocal here which causes LOAD_CLOSURE and BUILD_TUPLE to be used as described in When is the existence of nonlocal variables checked? and further in How to define free-variable in python? and refers to x, not the inner function named literally closure
3 4 LOAD_CLOSURE 0 (x)
About nonlocal
For your case, nonlocal is asserting that x exists in the outer scope excluding globals at compile time, but is practically redundant because it's not used elsewhere docs
Originally I'd written that this was redundant due to redeclaration, but that's not true - nonlocal prevents re-using the name, but x simply isn't shown anywhere else so the effect isn't obvious
I've added a 3rd example with a very ugly generator to illustrate the effect
Example of use with a global (note the SyntaxError is at compile time, not at runtime!)
>>> x = 3
>>> def closure_test():
... def closure():
... nonlocal x
... print(x)
... return closure
...
File "<stdin>", line 3
SyntaxError: no binding for nonlocal 'x' found
>>> def closure_test():
... def closure():
... print(x)
... return closure
...
>>> closure_test()()
3
Examples of SyntaxErrors related to invalid locals use
>>> def closure_test():
... def closure():
... nonlocal x
... x = 2
... print(x)
... return closure
...
File "<stdin>", line 3
SyntaxError: no binding for nonlocal 'x' found
>>> def closure_test():
... x = 1
... def closure():
... x = 2
... nonlocal x
... print(x)
... return closure
...
File "<stdin>", line 5
SyntaxError: name 'x' is assigned to before nonlocal declaration
Example which makes use of nonlocal to set the outer value
(Note this is badly-behaved because a more normal approach wrapping yield with try:finally displays before closure is actually called)
>>> def closure_test():
... x = 1
... print(f"x outer A: {x}")
... def closure():
... nonlocal x
... x = 2
... print(f"x inner: {x}")
... yield closure
... print(f"x outer B: {x}")
...
>>> list(x() for x in closure_test())
x outer A: 1
x inner: 2
x outer B: 2
[None]
Original Example without nonlocal (note absence of BUILD_TUPLE and LOAD_CLOSURE!)
>>> def closure_test():
... x = 1
... def closure():
... x = 2
... print(x)
... return closure
...
>>>
>>> import dis
>>> dis.dis(closure_test)
2 0 LOAD_CONST 1 (1)
2 STORE_FAST 0 (x)
3 4 LOAD_CONST 2 (<code object closure at 0x10d8132f0, file "<stdin>", line 3>)
6 LOAD_CONST 3 ('closure_test.<locals>.closure')
8 MAKE_FUNCTION 0
10 STORE_FAST 1 (closure)
6 12 LOAD_FAST 1 (closure)
14 RETURN_VALUE
Disassembly of <code object closure at 0x10d8132f0, file "<stdin>", line 3>:
4 0 LOAD_CONST 1 (2)
2 STORE_FAST 0 (x)
5 4 LOAD_GLOBAL 0 (print)
6 LOAD_FAST 0 (x)
8 CALL_FUNCTION 1
10 POP_TOP
12 LOAD_CONST 0 (None)
14 RETURN_VALUE
About the ByteCode and a Simple Comparison
Reducing your example to remove all the names, it's simply
>>> import dis
>>> dis.dis(lambda: print(2))
1 0 LOAD_GLOBAL 0 (print)
2 LOAD_CONST 1 (2)
4 CALL_FUNCTION 1
6 RETURN_VALUE
The rest of the bytecode just moves the names around
xfor1and2closureandclosure_test.<locals>.closurefor inner function (located at some memory address)printliterally the print functionNoneliterally theNonesingleton
Specific DIS opcodes
STORE_DEREFputs a value in slotiLOAD_DEREFretrieves a value from slotiMAKE_FUNCTIONcreates a new function on the stack and puts it in slotiLOAD_CLOSUREdoes just that, putting it on the stack ati
You can see the constants, names, and free variables with dis.show_code()
>>> dis.show_code(closure_test)
Name: closure_test
Filename: <stdin>
Argument count: 0
Positional-only arguments: 0
Kw-only arguments: 0
Number of locals: 1
Stack size: 3
Flags: OPTIMIZED, NEWLOCALS
Constants:
0: None
1: 1
2: <code object closure at 0x10db282f0, file "<stdin>", line 3>
3: 'closure_test.<locals>.closure'
Variable names:
0: closure
Cell variables:
0: x
Digging at the closure itself
>>> dis.show_code(closure_test()) # call outer
Name: closure
Filename: <stdin>
Argument count: 0
Positional-only arguments: 0
Kw-only arguments: 0
Number of locals: 0
Stack size: 2
Flags: OPTIMIZED, NEWLOCALS, NESTED
Constants:
0: None
1: 2
Names:
0: print
Free variables:
0: x
>>> dis.show_code(lambda: print(2))
Name: <lambda>
Filename: <stdin>
Argument count: 0
Positional-only arguments: 0
Kw-only arguments: 0
Number of locals: 0
Stack size: 2
Flags: OPTIMIZED, NEWLOCALS, NOFREE
Constants:
0: None
1: 2
Names:
0: print
Using Python 3.9.10
Other related questions
- Python nonlocal statement
- How does exec work with locals?
Related Topics
Transpose Column to Row with Spark
Typeerror: a Bytes-Like Object Is Required, Not 'Str'
Partial Coloring of Text in Matplotlib
Alternative to Dict Comprehension Prior to Python 2.7
In Practice, What Are the Main Uses for the "Yield From" Syntax in Python 3.3
What's the How to Install Pip, Virtualenv, and Distribute for Python
What's the Difference Between Select_Related and Prefetch_Related in Django Orm
Multiple Python Versions on the Same MAChine
How to Bypass the Google Captcha with Selenium and Python
Reading File Using Relative Path in Python Project
Get Md5 Hash of Big Files in Python
How to Take Column-Slices of Dataframe in Pandas