Python Lambda Closure Scoping

What do lambda function closures capture?

What do the closures capture exactly?

Closures in Python use lexical scoping: they remember the name and scope of the closed-over variable where it is created. However, they are still late binding: the name is looked up when the code in the closure is used, not when the closure is created. Since all the functions in your example are created in the same scope and use the same variable name, they always refer to the same variable.

There are at least two ways to get early binding instead:

The most concise, but not strictly equivalent way is the one recommended by Adrien Plisson. Create a lambda with an extra argument, and set the extra argument's default value to the object you want preserved.
More verbosely but also more robustly, we can create a new scope for each created lambda:
```
>>> adders = [0,1,2,3]
>>> for i in [0,1,2,3]:
...     adders[i] = (lambda b: lambda a: b + a)(i)
...     
>>> adders[1](3)
4
>>> adders[2](3)
5
```
The scope here is created using a new function (another lambda, for brevity), which binds its argument, and passing the value you want to bind as the argument. In real code, though, you most likely will have an ordinary function instead of the lambda to create the new scope:
```
def createAdder(x):
    return lambda y: y + x
adders = [createAdder(i) for i in range(4)]
```

Python lambda closure scoping

The reason is that closures (lambdas or otherwise) close over names, not values. When you define lambda x: test_fun(n, x), the n is not evaluated, because it is inside the function. It is evaluated when the function is called, at which time the value that is there is the last value from the loop.

You say at the beginning that you want to "use closures to eliminate a variable from a function signature", but it doesn't really work that way. (See below, though, for a way that may satisfy you, depending on what you mean by "eliminate".) Variables inside the function body will not be evaluated when the function is defined. In order to get the function to take a "snapshot" of the variable as it exists at function-definition time, you must pass the variable as an argument. The usual way to do this is to give the function an argument whose default value is the variable from the outer scope. Look at the difference between these two examples:

>>> stuff = [lambda x: n+x for n in [1, 2, 3]]
>>> for f in stuff:
...     print f(1)
4
4
4
>>> stuff = [lambda x, n=n: n+x for n in [1, 2, 3]]
>>> for f in stuff:
...     print f(1)
2
3
4

In the second example, passing n as an argument to the function "locks in" the current value of n to that function. You have to do something like this if you want to lock in the value in this way. (If it didn't work this way, things like global variables wouldn't work at all; it's essential that free variables be looked up at the time of use.)

Note that nothing about this behavior is specific to lambdas. The same scoping rules are in effect if you use def to define a function that references variables from the enclosing scope.

If you really want to, you can avoid adding the extra argument to your returned function, but to do so you must wrap that function in yet another function, like so:

>>> def makeFunc(n):
...     return lambda x: x+n
>>> stuff = [makeFunc(n) for n in [1, 2, 3]]
>>> for f in stuff:
...     print f(1)
2
3
4

Here, the inner lambda still looks up the value of n when it is called. But the n it refers to is no longer a global variable but a local variable inside the enclosing function makeFunc. A new value of this local variable is created every time makeFunc is called, and the returned lambda creates a closure that "saves" the local variable value that was in effect for that invocation of makeFunc. Thus each function created in the loop has its own "private" variable called x. (For this simple case, this can also be done using a lambda for the outer function --- stuff = [(lambda n: lambda x: x+n)(n) for n in [1, 2, 3]] --- but this is less readable.)

Notice that you still have to pass your n as an argument, it's just that, by doing it this way, you don't pass it as an argument to the same function that winds up going into the stuff list; instead you pass it as an argument to a helper function that creates the function you want to put into stuff. The advantage of using this two-function approach is that the returned function is "clean" and doesn't have the extra argument; this could be useful if you were wrapping functions that accepted a lot of arguments, in which case it could become confusing to remember where the n argument was in the list. The disadvantage is that, doing it this way, the process of making the functions is more complicated, since you need another enclosing function.

The upshot is that there is a tradeoff: you can make the function-creation process simpler (i.e., no need for two nested functions), but then you must make the resulting function a bit more complicated (i.e., it has this extra n=n argument). Or you can make the function simpler (i.e., it has no n=n argument), but then you must make the function-creation process more complicated (i.e., you need two nested functions to implement the mechanism).

Python enclosing scope variables with lambda function

Even though i takes multiple values over time, there is effectively only one variable, i. The content of i is being changed during the loop. But the closures captures variables, not values. Nothing is evaluated inside the lambda until you call it. At the time you call the function, you access the current value of i, which happens to be the last one.

As for why i=i solves the problem, this is explained for example in The Hitchhiker's guide to Python (Common Gotchas):

Python’s default arguments are evaluated once when the function is defined, not each time the function is called (like it is in say, Ruby). This means that if you use a mutable default argument and mutate it, you will and have mutated that object for all future calls to the function as well.

And so, each fresh binding that occurs inside the closure you create (and happen to be named i just like the one outside) has its default value being computed when creating the closure. Consequently, you have the "right" value in place, ready to be used when the closure is called.

Python lambda's binding to local values

Change x.append(lambda : pv(v)) to x.append(lambda v=v: pv(v)).

You expect "python lambdas to bind to the reference a local variable is pointing to, behind the scene", but that is not how Python works. Python looks up the variable name at the time the function is called, not when it is created. Using a default argument works because default arguments are evaluated when the function is created, not when it is called.

This is not something special about lambdas. Consider:

x = "before foo defined"
def foo():
    print x
x = "after foo was defined"
foo()

prints

after foo was defined

Scope of lambda functions and their parameters?

The problem here is the m variable (a reference) being taken from the surrounding scope.
Only parameters are held in the lambda scope.

To solve this you have to create another scope for lambda:

def callback(msg):
    print msg

def callback_factory(m):
    return lambda: callback(m)

funcList=[]
for m in ('do', 're', 'mi'):
    funcList.append(callback_factory(m))
for f in funcList:
    f()

In the example above, lambda also uses the surounding scope to find m, but this
time it's callback_factory scope which is created once per every callback_factory
call.

Or with functools.partial:

from functools import partial

def callback(msg):
    print msg

funcList=[partial(callback, m) for m in ('do', 're', 'mi')]
for f in funcList:
    f()

Python lambdas and scoping

This is a frequent question in Python. Basically the scoping is such that when f() is called, it will use the current value of x, not the value of x at the time the lambda is formed. There is a standard workaround:

funcs = []
for x in range(10):
funcs.append(lambda x=x: x)
print [f() for f in funcs]

The use of lambda x = x retrieves and saves the current value of x.

Generating functions inside loop with lambda expression in python

Technically, the lambda expression is closed over the i that's visible in the global scope, which is last set to 9. It's the same i being referred to in all 10 lambdas. For example,

i = 13
print b[3]()

In the makeFun function, the lambda closes on the i that's defined when the function is invoked. Those are ten different is.

How external variables can be passed into the scope of lambda functions?

I am not sure why you are having the problem. In a minimal example like this:

(lambda x: (lambda y: y+x)(3))(2)

You get the expected 5, so there must be something more to what you are doing.