Read/Write Python Closures

Read/Write Python Closures

To expand on Ignacio's answer:

def counter():
    count = 0
    def c():
        nonlocal count
        count += 1
        return count
    return c

x = counter()
print([x(),x(),x()])

gives [1,2,3] in Python 3; invocations of counter() give independent counters. Other solutions - especially using itertools/yield are more idiomatic.

Closures in Python - an example

Here are 3 options:

1. use a list for your counter:

   def make_progress_report(n):
       i = [0]
       def progress_report():
           i[0] = i[0] + 1
           if i[0] % n == 0:
               print i[0]
       return progress_report
   

2. use itertools.count to track your counter:

   from itertools import count
   def make_progress_report(n):
       i = count(1)
       def progress_report():
           cur = i.next()
           if cur % n == 0:
               print cur
       return progress_report
   

3. Use nonlocal for your counter (Python 3+ only!):

   def make_progress_report(n):
       i = 0
       def progress_report():
           nonlocal i
           i = i + 1
           if i % n == 0:
               print i
       return progress_report
   

Can you explain closures (as they relate to Python)?

Closure on closures

Objects are data with methods
attached, closures are functions with
data attached.

def make_counter():
    i = 0
    def counter(): # counter() is a closure
        nonlocal i
        i += 1
        return i
    return counter

c1 = make_counter()
c2 = make_counter()

print (c1(), c1(), c2(), c2())
# -> 1 2 1 2

Closure in python?

Python 2.x has a syntax limitation that doesn't allow to capture a variable in read/write.

The reason is that if a variable is assigned in a function there are only two possibilities:

1. the variable is a global and has been declared so with global x
2. the variable is a local of the function

more specifically it's ruled out that the variable is a local of an enclosing function scope

This has been superseded in Python 3.x with the addition of nonlocal declaration. Your code would work as expected in Python 3 by changing it to

def make_adder_and_setter(x):
    def setter(n):
        nonlocal x
        x = n

    return (lambda y: x + y, setter)

The python 2.x runtime is able to handle read-write closed over variable at a bytecode level, however the limitation is in the syntax that the compiler accepts.

You can see a lisp compiler that generates python bytecode directly that creates an adder closure with read-write captured state at the end of this video. The compiler can generate bytecode for Python 2.x, Python 3.x or PyPy.

If you need closed-over mutable state in Python 2.x a trick is to use a list:

def make_adder_and_setter(x):
    x = [x]
    def setter(n):
        x[0] = n

    return (lambda y: x[0] + y, setter)

CLOSURE : retrieve a variable given with the outer function

__closure__ is an (read-only) attribute of a user-defined function object:
it is tuple (or None) of cells that contain bindings for the function’s free variables.

A cell object has the (read & write) attribute cell_contents. This can be used to get/set the value of the cell.

Note cell types can be accessed through types module

---

code objects are an internal type used by the interpreter and accessible by the user.

co_freevars is a tuple containing the names of free variable

---

Here an example on how to get the value of the function's parameter and its identifier:

# html_tag: from the question

print_h1 = html_tag('h1')

# identifier of the parameter - with code obj
print(print_h1.__code__.co_freevars[0])
#tag

# value of the function's parameter - read
print(print_h1.__closure__[0].cell_contents)
#h1

# value of the function's parameter - write
print_h1.__closure__[0].cell_contents = 'h19'
print_h1('a')
#<h19>a<h19>

Why can't Python increment variable in closure?

You can't re-bind closure variables in Python 2. In Python 3, which you appear to be using due to your print(), you can declare them nonlocal:

def foo():

  counter = 1

  def bar():
    nonlocal counter
    counter += 1
    print("bar", counter)

  return bar

bar = foo()
bar()

Otherwise, the assignment within bar() makes the variable local, and since you haven't assigned a value to the variable in the local scope, trying to access it is an error.

In Python 2, my favorite workaround looks like this:

def foo():

  class nonlocal:
    counter = 1

  def bar():
    nonlocal.counter += 1
    print("bar", nonlocal.counter)

  return bar

bar = foo()
bar()

This works because mutating a mutable object doesn't require changing what the variable name points to. In this case, nonlocal is the closure variable and it is never reassigned; only its contents are changed. Other workarounds use lists or dictionaries.

Or you could use a class for the whole thing, as @naomik suggests in a comment. Define __call__() to make the instance callable.

class Foo(object):

    def __init__(self, counter=1):
       self.counter = counter

    def __call__(self):
       self.counter += 1
       print("bar", self.counter)

bar = Foo()
bar()

How to maintain state in Python without classes?

You can also accomplish this with default arguments:

def try_match(m, re_match=re.compile(r'sldkjlsdjf').match):
    return re_match(m)

since default arguments are only evaluated once, at module import time.

Or even simpler:

try_match = lambda m, re_match=re.compile(r'sldkjlsdjf').match: re_match(m)

Or simplest yet:

try_match = re.compile(r'sldkjlsdjf').match

This saves not only the re compile time (which is actually cached internally in the re module anyway), but also the lookup of the '.match' method. In a busy function or a tight loop, those '.' resolutions can add up.

Python - locals() and closure

locals() built-in function prints local symbol table which is bound to a code object, and filled up when interpreter receives a name in a source code.

Second example, when disassembled, will contain LOAD_GLOBAL foo bytecode instruction in b function code. This LOAD_GLOBAL instruction will move up the scope, find outer foo name and bind it to the code object by adding name offset into co_names attribute of the closure's (function b) code object.

locals() function prints local symbol table (as was said before, co_names attribute of function's code object).

Read more about code objects here.

confused by python3 closure

def gen(x):
    def f():
        nonlocal x # add this line
        x += 1
        return x

    return f

print(gen(1)())

Why doesn't this closure modify the variable in the enclosing scope?

There are two "better" / more Pythonic ways to do this on Python 2.x than using a container just to get around the lack of a nonlocal keyword.

One you mentioned in a comment in your code -- bind to a local variable. There is another way to do that:

Using a default argument

def make_incrementer(start):
    def closure(start = start):
        while True:
            yield start
            start += 1
    return closure

x = make_incrementer(100)
iter = x()
print iter.next()

This has all the benefits of a local variable without an additional line of code. It also happens on the x = make_incrememter(100) line rather than the iter = x() line, which may or may not matter depending on the situation.

You can also use the "don't actually assign to the referenced variable" method, in a more elegant way than using a container:

Using a function attribute

def make_incrementer(start):
    def closure():
        # You can still do x = closure.start if you want to rebind to local scope
        while True:
            yield closure.start
            closure.start += 1
    closure.start = start
    return closure

x = make_incrementer(100)
iter = x()
print iter.next()    

This works in all recent versions of Python and utilizes the fact that in this situation, you already have an object you know the name of you can references attributes on -- there is no need to create a new container for just this purpose.

---

Related Topics