Understanding the Python with Statement and Context Managers

Understanding the Python with statement and context managers

with doesn't really replace try/except, but, rather, try/finally. Still, you can make a context manager do something different in exception cases from non-exception ones:

class Mgr(object):
    def __enter__(self): pass
    def __exit__(self, ext, exv, trb):
        if ext is not None: print "no not possible"
        print "OK I caught you"
        return True

with Mgr():
    name='rubicon'/2 #to raise an exception

The return True part is where the context manager decides to suppress the exception (as you do by not re-raising it in your except clause).

What is the purpose of a context manager in python

What are context managers?

Context managers are there to aid with resources (although may be used for much more). The context is basically the resource.

Let's say you opened a file for writing:

f = open(path, "w")

You now have an open file handle. During the handling of your file, no other program can write to it. In order to let other programs write to it, you must close the file handle:

f.close()

But, before closing your file an error occured:

f = open(path, "w")
data = 3/0  # Tried dividing by zero. Raised ZeroDivisionError
f.write(data)
f.close()

What will happen now is that the function or entire program will exit, while leaving your file with an open handle. (CPython cleans handles on termination and handles are freed together with a program but you shouldn't count on that)

A with statement ensures that as soon as you leave it's indentation, it will close the file handle:

with open(path, "w") as f:
    data = 3/0  # Tried dividing by zero. Raised ZeroDivisionError
    f.write(data)
# Here, the file is already closed automatically, no matter what happened.

---

with statements may be used for many more things. For example: threading.Lock()

lock = threading.Lock()
with lock:  # Lock is acquired
   do stuff...
# Lock is automatically released.

Almost everything done with a context manager can be done with try: ... finally: ... but context managers are nicer to use, more comfortable, more readable and by implementing __enter__ and __exit__ provide an easy to use interface.

---

How are context managers implemented?

or What are __enter__ and __exit__?

Creating context managers is done by implementing __enter__() and __exit__() in a normal class.

__enter__() tells what to do when a context manager starts and __exit__() when a context manager exists (giving the exception to the __exit__() method if an exception occurred)

A shortcut for creating context managers can be found in contextlib. It wraps a generator as a context manager like so:

from contextlib import contextmanager

@contextmanager
def tag(name):
    print("<%s>" % name)
    yield
    print("</%s>" % name)

>>> with tag("h1"):
...    print("foo")
...
<h1>
foo
</h1>

Example taken from the Python docs.

Understanding the Python 'with' statement

I don't know why no one has mentioned this yet, because it's fundamental to the way with works. As with many language features in Python, with behind the scenes calls special methods, which are already defined for built-in Python objects and can be overridden by user-defined classes. In with's particular case (and context managers more generally), the methods are __enter__ and __exit__.

Remember that in Python everything is an object -- even literals. This is why you can do things like 'hello'[0]. Thus, it does not matter whether you use the file object directly as returned by open:

with open('filename.txt') as infile:
    for line in infile:
        print(line)

or store it first with a different name (for example to break up a long line):

the_file = open('filename' + some_var + '.txt')
with the_file as infile:
    for line in infile:
        print(line)

Because the end result is that the_file, infile, and the return value of open all point to the same object, and that's what with is calling the __enter__ and __exit__ methods on. The built-in file object's __exit__ method is what closes the file.

Multiple context managers in a with statement in python

To understand this, consider how and works in Python.

The result of the expression x and y is equivalent to y if x else x, except that it only evaluates x once. That is, when bool(x) is True, the and operator results in y, otherwise it results in x (and doesn't evaluate y).

Unless an object defines its own __bool__ dunder method, bool(obj) will generally be True. This is the case for lock objects. So bool(l1) is True, and the expression l1 and l2 and l3 evaluates as l2 and l3. Then since bool(l2) is True, this expression evaluates as l3.

So the with statement ends up managing the lock l3, and therefore that's the one which is locked in the body of the with statement. As you note, if you want to manage multiple locks at once, you should pass them separated by commas.

Understand python3 nested context managers

Why does it print Outer enter and Outer exit at all despite using context manager on already created object in a function?

The with statement you define says that python should call __enter__ before the content of the with statement is executed and __exit__ after that.

It doesn't matter if the object has allready been initialized. The with statement is going to call __enter__ and __exit__ anyway

It raises AttributeError: enter

As you are returning a tuple and the tuple doesn't contain a definition of __enter__ this error is thrown.

I don't get any errors. Neither I get Inner created and Inner enter. Why is that so?

You don't get any errors because you use the with statement correctly with your OuterCtxManager and you don't get the output Inner enter and Inner exit because you don't use the with statement on the InnerCtxManager type.

Explaining Python's '__enter__' and '__exit__'

Using these magic methods (__enter__, __exit__) allows you to implement objects which can be used easily with the with statement.

The idea is that it makes it easy to build code which needs some 'cleandown' code executed (think of it as a try-finally block). Some more explanation here.

A useful example could be a database connection object (which then automagically closes the connection once the corresponding 'with'-statement goes out of scope):

class DatabaseConnection(object):

    def __enter__(self):
        # make a database connection and return it
        ...
        return self.dbconn

    def __exit__(self, exc_type, exc_val, exc_tb):
        # make sure the dbconnection gets closed
        self.dbconn.close()
        ...

As explained above, use this object with the with statement (you may need to do from __future__ import with_statement at the top of the file if you're on Python 2.5).

with DatabaseConnection() as mydbconn:
    # do stuff

PEP343 -- The 'with' statement' has a nice writeup as well.

with, context manager, python: What's going on in simple terms?

File objects are themselves context managers, in that they have __enter__ and __exit__ methods. with notifies the file object when the context is entered and exited (by calling __enter__ and __exit__, respectively), and this is how a file object "knows" to close the file. There is no wrapper object involved here; file objects provide those two methods (in Java terms you could say that file objects implement the context manager interface).

Note that as is not an alias just like import module as altname; instead, the return value of contextmanager.__enter__() is assigned to the target. The fileobject.__enter__() method returns self (so the file object itself), to make it easier to use the syntax:

with open(...) as fileobj:

If fileobject.__enter__() did not do this but either returned None or another object, you couldn't inline the open() call; to keep a reference to the returned file object you'd have to assign the result of open() to a variable first before using it as a context manager:

fileobj = open(...)
with fileobj as something_enter_returned:
    fileobj.write()

or

fileobj = open(...)
with fileobj:  # no as, ignore whatever fileobj.__enter__() produced 
    fileobj.write()

Note that nothing stops you from using the latter pattern in your own code; you don't have to use an as target part here if you already have another reference to the file object, or simply don't need to even access the file object further.

However, other context managers could return something different. Some database connectors return a database cursor:

conn = database.connect(....)
with conn as cursor:
    cursor.execute(...)

and exiting the context causes the transaction to be committed or rolled back (depending on wether or not there was an exception).

---

Related Topics