How Does Functools Partial Do What It Does

How does functools partial do what it does?

Roughly, partial does something like this (apart from keyword args support etc):

def partial(func, *part_args):
    def wrapper(*extra_args):
        args = list(part_args)
        args.extend(extra_args)
        return func(*args)

    return wrapper

So, by calling partial(sum2, 4) you create a new function (a callable, to be precise) that behaves like sum2, but has one positional argument less. That missing argument is always substituted by 4, so that partial(sum2, 4)(2) == sum2(4, 2)

As for why it's needed, there's a variety of cases. Just for one, suppose you have to pass a function somewhere where it's expected to have 2 arguments:

class EventNotifier(object):
    def __init__(self):
        self._listeners = []

    def add_listener(self, callback):
        ''' callback should accept two positional arguments, event and params '''
        self._listeners.append(callback)
        # ...

    def notify(self, event, *params):
        for f in self._listeners:
            f(event, params)

But a function you already have needs access to some third context object to do its job:

def log_event(context, event, params):
    context.log_event("Something happened %s, %s", event, params)

So, there are several solutions:

A custom object:

class Listener(object):
   def __init__(self, context):
       self._context = context

   def __call__(self, event, params):
       self._context.log_event("Something happened %s, %s", event, params)

 notifier.add_listener(Listener(context))

Lambda:

log_listener = lambda event, params: log_event(context, event, params)
notifier.add_listener(log_listener)

With partials:

context = get_context()  # whatever
notifier.add_listener(partial(log_event, context))

Of those three, partial is the shortest and the fastest.
(For a more complex case you might want a custom object though).

What exactly is the optimization `functools.partial` is making?

The following arguments actually apply only to CPython, for other Python implementations it could be completely different. You actually said your question is about CPython but nevertheless I think it's important to realize that these in-depth questions almost always depend on implementation details that might be different for different implementations and might even be different between different CPython versions (for example CPython 2.7 could be completely different, but so could be CPython 3.5)!

Timings

First of all, I can't reproduce differences of 15% or even 20%. On my computer the difference is around ~10%. It's even less when you change the lambda so it doesn't have to look up add from the global scope (as already pointed out in the comments you can pass the add function as default argument to the function so the lookup happens in the local scope).

from functools import partial

def add(x, y, z, a):
    return x + y + z + a

def max_lambda_default(lst):
    return max(lst , key=lambda a, add=add: add(10, 20, 30, a))

def max_lambda(lst):
    return max(lst , key=lambda a: add(10, 20, 30, a))

def max_partial(lst):
    return max(lst , key=partial(add, 10, 20, 30))

I actually benchmarked these:

from simple_benchmark import benchmark
from collections import OrderedDict

arguments = OrderedDict((2**i, list(range(2**i))) for i in range(1, 20))
b = benchmark([max_lambda_default, max_lambda, max_partial], arguments, "list size")

%matplotlib notebook
b.plot_difference_percentage(relative_to=max_partial)

Possible explanations

It's very hard to find the exact reason for the difference. However there are a few possible options, assuming you have a CPython version with compiled _functools module (all desktop versions of CPython that I use have it).

As you already found out the Python version of partial will be significantly slower.

• partial is implemented in C and can call the function directly - without intermediate Python layer¹. The lambda on the other hand needs to do a Python level call to the "captured" function.

• partial actually knows how the arguments fit together. So it can create the arguments that are passed to the function more efficiently (it just concatenats the stored argument tuple to the passed in argument tuple) instead of building a completely new argument tuple.

• In more recent Python versions several internals were changed in an effort to optimize function calls (the so called FASTCALL optimization). Victor Stinner has a list of related pull requests on his blog in case you want to find out more about it.

  That probably will affect both the lambda and the partial but again because partial is a C function it knows which one to call directly without having to infer it like lambda does.

However it's very important to realize that creating the partial has some overhead. The break-even point is for ~10 list elements, if the list is shorter, then the lambda will be faster.

Footnotes

¹ If you call a function from Python it uses the OP-code CALL_FUNCTION which is actually a wrapper (that's what I meant with Python layer) around the PyObject_Call* (or FASTCAL) functions. But it also includes creating the argument tuple/dictionary. If you call a function from a C function you can avoid this thin wrapper by directly calling the PyObject_Call* functions.

In case you're interested about the OP-Codes, you can disassemble the function:

import dis
    
dis.dis(max_lambda_default)

 0 LOAD_GLOBAL              0 (max)
 2 LOAD_FAST                0 (lst)
 4 LOAD_GLOBAL              1 (add)
 6 BUILD_TUPLE              1
 8 LOAD_CONST               1 (<code object <lambda>>)
10 LOAD_CONST               2 ('max_lambda_default.<locals>.<lambda>')
12 MAKE_FUNCTION            1 (defaults)
14 LOAD_CONST               3 (('key',))
16 CALL_FUNCTION_KW         2
18 RETURN_VALUE

Disassembly of <code object <lambda>>:
 0 LOAD_FAST                1 (add)      <--- (2)
 2 LOAD_CONST               1 (10)
 4 LOAD_CONST               2 (20)
 6 LOAD_CONST               3 (30)
 8 LOAD_FAST                0 (a)
10 CALL_FUNCTION            4            <--- (1)
12 RETURN_VALUE

As you can see the CALL_FUNCTION op code (1) is actually in there.

As an aside: The LOAD_FAST (2) is responsible for the performance difference between the lambda_default and the lambda without default (which has to resort to a slower lookup). That's because loading a name actually starts by checking the local scope (the function scope), in the case of add=add the add function is in the local scope, so it can make a faster lookup. If you don't have it in the local scope it will check each surrounding scope until it finds the name and it only stops when it reaches the global scope. And that lookup is done every time the lambda is called!

The rationale of `functools.partial` behavior

Using partial with a Positional Argument

f = partial(bar, 3)

By design, upon calling a function, positional arguments are assigned first. Then logically, 3 should be assigned to a with partial. It makes sense to remove it from the signature as there is no way to assign anything to it again!

when you have f(a=2, b=6), you are actually doing

bar(3, a=2, b=6)

when you have f(2, 2), you are actually doing

bar (3, 2, 2)

We never get rid of 3

For the new partial function:

1. We can't give a a different value with another positional argument
2. We can't use the keyword a to assign a different value to it as it is already "filled"

If there is a parameter with the same name as the keyword, then the argument value is assigned to that parameter slot. However, if the parameter slot is already filled, then that is an error.

I recommend reading the function calling behavior section of pep-3102 to get a better grasp of this matter.

Using partial with a Keyword Argument

f = partial(bar, b=3)

This is a different use case. We are applying a keyword argument to bar.

You are functionally turning

def bar(a, b):
    ...

into

def f(a, *, b=3):
    ...

where b becomes a keyword-only argument
instead of

def f(a, b=3):
    ...

inspect.signature correctly reflects a design decision of partial. The keyword arguments passed to partial are designed to append additional positional arguments (source).

Note that this behavior does not necessarily override the keyword arguments supplied with f = partial(bar, b=3), i.e., b=3 will be applied regardless of whether you supply the second positional argument or not (and there will be a TypeError if you do so). This is different from a positional argument with a default value.

>>> f(1, 2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: f() takes 1 positional argument but 2 were given

where f(1, 2) is equivalent to bar(1, 2, b=3)

The only way to override it is with a keyword argument

>>> f(2, b=2)

An argument that can only be assigned with a keyword but positionally? This is a keyword-only argument. Thus (a, *, b=3) instead of (a, b=3).

The Rationale of Non-default Argument follows Default Argument

f = partial(bar, a=3)
assert str(signature(f)) == '(*, a=3, b)'  # whaaa?! non-default argument follows default argument?

1. You can't do def bar(a=3, b). a and b are so called positional-or-keyword arguments.
2. You can do def bar(*, a=3, b). a and b are keyword-only arguments.

Even though semantically, a has a default value and thus it is optional, we can't leave it unassigned because b, which is a positional-or-keyword argument needs to be assigned a value if we want to use b positionally. If we do not supply a value for a, we have to use b as a keyword argument.

Checkmate! There is no way for b to be a positional-or-keyword argument as we intended.

The PEP for positonal-only arguments also kind of shows the rationale behind it.

This also has something to do with the aforementioned "function calling behavior".

partial != Currying & Implementation Details

partial by its implementation wraps the original function while storing the fixed arguments you passed to it.

IT IS NOT IMPLEMENTED WITH CURRYING. It is rather partial application instead of currying in the sense of functional programming. partial is essentially applying the fixed arguments first, then the arguments you called with the wrapper:

def __call__(self, /, *args, **keywords):
    keywords = {**self.keywords, **keywords}
    return self.func(*self.args, *args, **keywords)

This explains f(a=2, b=6) # TypeError: bar() got multiple values for argument 'a'.

See also: Why is partial called partial instead of curry

Under the Hood of inspect

The outputs of inspect is another story.

inspect itself is a tool that produces user-friendly outputs. For partial() in particular (and partialmethod(), similarly), it follows the wrapped function while taking the fixed parameters into account:

if isinstance(obj, functools.partial):
    wrapped_sig = _get_signature_of(obj.func)
    return _signature_get_partial(wrapped_sig, obj)

Do note that it is not inspect.signature's goal to show you the actual signature of the wrapped function in the AST.

def _signature_get_partial(wrapped_sig, partial, extra_args=()):
    """Private helper to calculate how 'wrapped_sig' signature will
    look like after applying a 'functools.partial' object (or alike)
    on it.
    """
    ...

So we have a nice and ideal signature for f = partial(bar, 3)
but get f(a=2, b=6) # TypeError: bar() got multiple values for argument 'a' in reality.

Follow-up

If you want currying so badly, how do you implement it in Python, in the way which gives you the expected TypeError?

Python decorators using functools.partial.. Where does func come from?

First, run_n gets called with func=None and n=2. Since func is None, it will take the first branch of the if statement. partial then creates, and returns, an anonymous function that behaves as if it was defined like this:

def anonymous(*args, **kwargs):
    return run_n(*args, count=2, **kwargs)

Then that function gets called with the function being defined (i.e, the function printing "hello") as its first argument. This immediately calls

run_n(<func-printing-hello>,count=2)

This time, run_n has been given a func argument, and returns the wrapper function that calls func-printing-hello n times. This wrapper, eventually, is what gets assigned to the global func name.

Using functools partial with multiple arguments

Can I recommend the use of lambda for your case.

f1 = lambda u : func(u,v,w,x, alpha = 4, beta = 5) 
f2 = lambda v, x: func(u,v,w,x, alpha = 4, beta = 5) 
f3 = lambda v, alpha: func(u,v,w,x, alpha = alpha, beta = 5)  

The usage is the same as what you wanted:

f1(u) #or f(v),f(w)
f2(v,x) #or all other variable combinations
f3(v,alpha) # a combination of args and kwargs

functools.partial vs normal Python function

Functions do have information about what arguments they accept. The attribute names and data structures are aimed at the interpreter more than the developer, but the info is there. You'd have to introspect the .func_defaults and .func_code.co_varnames structures (and a few more besides) to find those details.

Using the inspect.getargspec() function makes extracting the info a little more straightforward:

>>> import inspect
>>> h = lambda x : int(x,base=2)
>>> inspect.getargspec(h)
ArgSpec(args=['x'], varargs=None, keywords=None, defaults=None)

Note that a lambda produces the exact same object type as a def funcname(): function statement produces.

What this doesn't give you, is what arguments are going to be passed into the wrapped function. That's because a function has a more generic use, while functools.partial() is specialised and thus can easily provide you with that information. As such, the partial tells you that base=2 will be passed into int(), but the lambda can only tell you it receives an argument x.

So, while a functools.partial() object can tell you what arguments are going to be passed into what function, a function object can only tell you what arguments it receives, as it is the job of the (potentially much more complex) expressions that make up the function body to do the call to a wrapped function. And that is ignoring all those functions that don't call other functions at all.

---

Related Topics