What Do Lambda Function Closures Capture

What do lambda function closures capture?

Your second question has been answered, but as for your first:

what does the closure capture exactly?

Scoping in Python is dynamic and lexical. A closure will always remember the name and scope of the variable, not the object it's pointing to. Since all the functions in your example are created in the same scope and use the same variable name, they always refer to the same variable.

Regarding your other question of how to overcome this, there are two ways that come to mind:

1. The most concise, but not strictly equivalent way is the one recommended by Adrien Plisson. Create a lambda with an extra argument, and set the extra argument's default value to the object you want preserved.

2. A little more verbose but less hacky would be to create a new scope each time you create the lambda:

    >>> adders = [0,1,2,3]
    >>> for i in [0,1,2,3]:
    ...     adders[i] = (lambda b: lambda a: b + a)(i)
    ...     
    >>> adders[1](3)
    4
    >>> adders[2](3)
    5
   

The scope here is created using a new function (a lambda, for brevity), which binds its argument, and passing the value you want to bind as the argument. In real code, though, you most likely will have an ordinary function instead of the lambda to create the new scope:

def createAdder(x):
    return lambda y: y + x
adders = [createAdder(i) for i in range(4)]

Python lambda closure scoping

The reason is that closures (lambdas or otherwise) close over names, not values. When you define lambda x: test_fun(n, x), the n is not evaluated, because it is inside the function. It is evaluated when the function is called, at which time the value that is there is the last value from the loop.

You say at the beginning that you want to "use closures to eliminate a variable from a function signature", but it doesn't really work that way. (See below, though, for a way that may satisfy you, depending on what you mean by "eliminate".) Variables inside the function body will not be evaluated when the function is defined. In order to get the function to take a "snapshot" of the variable as it exists at function-definition time, you must pass the variable as an argument. The usual way to do this is to give the function an argument whose default value is the variable from the outer scope. Look at the difference between these two examples:

>>> stuff = [lambda x: n+x for n in [1, 2, 3]]
>>> for f in stuff:
...     print f(1)
4
4
4
>>> stuff = [lambda x, n=n: n+x for n in [1, 2, 3]]
>>> for f in stuff:
...     print f(1)
2
3
4

In the second example, passing n as an argument to the function "locks in" the current value of n to that function. You have to do something like this if you want to lock in the value in this way. (If it didn't work this way, things like global variables wouldn't work at all; it's essential that free variables be looked up at the time of use.)

Note that nothing about this behavior is specific to lambdas. The same scoping rules are in effect if you use def to define a function that references variables from the enclosing scope.

If you really want to, you can avoid adding the extra argument to your returned function, but to do so you must wrap that function in yet another function, like so:

>>> def makeFunc(n):
...     return lambda x: x+n
>>> stuff = [makeFunc(n) for n in [1, 2, 3]]
>>> for f in stuff:
...     print f(1)
2
3
4

Here, the inner lambda still looks up the value of n when it is called. But the n it refers to is no longer a global variable but a local variable inside the enclosing function makeFunc. A new value of this local variable is created every time makeFunc is called, and the returned lambda creates a closure that "saves" the local variable value that was in effect for that invocation of makeFunc. Thus each function created in the loop has its own "private" variable called x. (For this simple case, this can also be done using a lambda for the outer function --- stuff = [(lambda n: lambda x: x+n)(n) for n in [1, 2, 3]] --- but this is less readable.)

Notice that you still have to pass your n as an argument, it's just that, by doing it this way, you don't pass it as an argument to the same function that winds up going into the stuff list; instead you pass it as an argument to a helper function that creates the function you want to put into stuff. The advantage of using this two-function approach is that the returned function is "clean" and doesn't have the extra argument; this could be useful if you were wrapping functions that accepted a lot of arguments, in which case it could become confusing to remember where the n argument was in the list. The disadvantage is that, doing it this way, the process of making the functions is more complicated, since you need another enclosing function.

The upshot is that there is a tradeoff: you can make the function-creation process simpler (i.e., no need for two nested functions), but then you must make the resulting function a bit more complicated (i.e., it has this extra n=n argument). Or you can make the function simpler (i.e., it has no n=n argument), but then you must make the function-creation process more complicated (i.e., you need two nested functions to implement the mechanism).

What is the difference between a 'closure' and a 'lambda'?

A lambda is just an anonymous function - a function defined with no name. In some languages, such as Scheme, they are equivalent to named functions. In fact, the function definition is re-written as binding a lambda to a variable internally. In other languages, like Python, there are some (rather needless) distinctions between them, but they behave the same way otherwise.

A closure is any function which closes over the environment in which it was defined. This means that it can access variables not in its parameter list. Examples:

def func(): return h
def anotherfunc(h):
   return func()

This will cause an error, because func does not close over the environment in anotherfunc - h is undefined. func only closes over the global environment. This will work:

def anotherfunc(h):
    def func(): return h
    return func()

Because here, func is defined in anotherfunc, and in python 2.3 and greater (or some number like this) when they almost got closures correct (mutation still doesn't work), this means that it closes over anotherfunc's environment and can access variables inside of it. In Python 3.1+, mutation works too when using the nonlocal keyword.

Another important point - func will continue to close over anotherfunc's environment even when it's no longer being evaluated in anotherfunc. This code will also work:

def anotherfunc(h):
    def func(): return h
    return func

print anotherfunc(10)()

This will print 10.

This, as you notice, has nothing to do with lambdas - they are two different (although related) concepts.

When python closures are exactly doing their capture?

at compile-time, when python encounters a function, it makes some determinations. It looks at everything in the function, it does not create a scope or a namespace but it determines which variables are going to be local or nonlocal.

When you run foo(), which means at run-time of foo, bar function gets created and python determines "free" and "capture" as non-local, therefore bar already has a reference to free variables.

"free" and "capture" are in two different scopes but always reference the same "value". When python determines the local variables of "bar", it creates a cell object.

So when outer function "foo" finishes running, this "Cell" object still exists so when inner function "bar" is called, it still gets the same value.

How do I capture the CURRENT values of closure variables immutably in Python?

For simple cases, such as when the code is short and we don't have many variables to capture, we can create a temporary lambda and call it:

def f():
    a = 1
    g = (lambda a: lambda: a)(a)
    a = 2
    return g

The issue here is that the code can quickly become harder to read.

Alternatively, we can capture the variable as an optional argument:

def f():
    a = 1
    g = lambda a=a: a
    a = 2
    return g

The issue here is, of course, that we might not want the caller to be able to specify this parameter.

(And the code can be a little less readable, too.)

A fully general solution might be the following, except that it does not capture globals:

def bind(*args, **kwargs):
    # Use '*args' so that callers aren't limited in what names they can specify
    func               = args[0]
    include_by_default = args[1] if len(args) > 1 else None
    # if include_by_default == False, variables are NOT bound by default
    if include_by_default == None: include_by_default = not kwargs
    fc = func.__code__
    fv = fc.co_freevars
    q = func.__closure__
    if q:
        ql = []
        i = 0
        for qi in q:
            fvi = fv[i]
            ql.append((lambda v: (lambda: v).__closure__[0])(
                kwargs.get(fvi, qi.cell_contents))
                if include_by_default or fvi in kwargs
                else qi)
            i += 1
        q = q.__class__(ql)
        del ql
    return func.__class__(fc, func.__globals__, func.__name__, func.__defaults__, q)

The reason I do not attempt to capture globals here is that the semantics can get confusing -- if an inner function says global x; x = 1, it certainly does want the global x to be modified, so suppressing this change would quickly make the code very counterintuitive.

However, barring that, we would be able to simply use it as follows:

def f():
    a = 1
    g = bind(lambda: a)
    a = 2
    return g
print(f()())

And voilà, a is instantly captured. And if we want to only capture some variables, we can do:

def f():
    a = 1
    b = 2
    g = bind(lambda: a + b, b=5)  # capture 'b' as 5; leave 'a' free
    a = 2
    b = 3
    return g
print(f()())

Difference between a lambda function and a closure (in PHP)?

A closure is a lambda function in php that encapsulates variables so they can be used once their original references are out of scope.

A closure is a lambda function, but a lambda function is not a closure unless you specify the use keyword.

This is a much better answer: https://stackoverflow.com/a/220728/1152375

Python lambda's binding to local values

Change x.append(lambda : pv(v)) to x.append(lambda v=v: pv(v)).

You expect "python lambdas to bind to the reference a local variable is pointing to, behind the scene", but that is not how Python works. Python looks up the variable name at the time the function is called, not when it is created. Using a default argument works because default arguments are evaluated when the function is created, not when it is called.

This is not something special about lambdas. Consider:

x = "before foo defined"
def foo():
    print x
x = "after foo was defined"
foo()

prints

after foo was defined

Are Lambda expressions in C# closures?

A lambda may be implemented using a closure, but it is not itself necessarily a closure.

A closure is "a function together with a referencing environment for the non-local variables of that function.".

When you make a lambda expression that uses variables defined outside of the method, then the lambda must be implemented using a closure. For example:

int i = 42;

Action lambda = () => { Console.WriteLine(i); }; 

In this case, the compiler generated method must have access to the variable (i) defined in a completely different scope. In order for this to work, the method it generates is a "function together with the referencing environment" - basically, it's creating a "closure" to retrieve access to the variable.

However, this lambda:

Action lambda2 = () => { Console.WriteLine("Foo"); }

does not rely on any "referencing environment", since it's a fully contained method. In this case, the compiler generates a normal static method, and there is no closure involved at all.

In both cases, the lambda is creating a delegate ("function object"), but it's only creating a closure in the first case, as the lambda doesn't necessarily need to "capture" the referencing environment in all cases.

Python lambda capturing external variables

By capturing the variable in the lambda definition, the original variable seems to be retained.

Becaused you made it the default value for the test argument of the lambda, and Python only evaluates arguments defaults once when the function is created. FWIW, this has nothing to do with the use of the lambda keyword - lambda is just syntactic sugar and you'd get exactly the same result with a "full-blown" function, ie:

test = 123
def f(x, test=test):
    print(x, test)
test = 456
f('hello')

Is the lambda ok to use when you could also use a simple object to store the same data?

Oranges and apples, really. A function is not meant to be use to "store" anything, but to perform some computation. The fact that it can capture some values either via the arguments defaults or via a closure, while quite useful for some use cases, is not meant as a replacement for proper objects or collections. So the answer is : depends on what you want to do with those values and your function.

NB : technically what you're doing here is (almost) what is known as "partial application" (you'd just have to rename Test.f to Test.__call__ and use t("hello") in the second example)). A partial application of a function is the mechanism by which a function of N arguments, when called with N - x (with x < N) arguments, returns a function of N-x arguments that when called with the missing arguments will return the result of the original function, ie:

def foo(a, b=None):
   if b is None:
       return lambda b, a=a: foo(b, a)
   return a + b

# here, `f` is a partial application of function `foo` to `1`
f = foo(1)
print(f)
f(2)

In this case we use a closure to capture a, in the functional programming tradition - "partial application" is also mostly a functionnal programming concept FWIW. Now while it does support some FP features and idioms, Python is first and foremost an OO language, and since closures are the FP equivalent of objects (closures are a way to encapsulate state and behaviour together), it also makes sense to implement partial application as proper class, either with "ad hoc" specialized objects (your Test class but also Method objects), or with a more generic "partial" class - which already exists in the stdlib as functools.partial

---

Related Topics