How to Correctly Clean Up a Python Object

How do I correctly clean up a Python object?

I'd recommend using Python's with statement for managing resources that need to be cleaned up. The problem with using an explicit close() statement is that you have to worry about people forgetting to call it at all or forgetting to place it in a finally block to prevent a resource leak when an exception occurs.

To use the with statement, create a class with the following methods:

def __enter__(self)
def __exit__(self, exc_type, exc_value, traceback)

In your example above, you'd use

class Package:
    def __init__(self):
        self.files = []

    def __enter__(self):
        return self

    # ...

    def __exit__(self, exc_type, exc_value, traceback):
        for file in self.files:
            os.unlink(file)

Then, when someone wanted to use your class, they'd do the following:

with Package() as package_obj:
    # use package_obj

The variable package_obj will be an instance of type Package (it's the value returned by the __enter__ method). Its __exit__ method will automatically be called, regardless of whether or not an exception occurs.

You could even take this approach a step further. In the example above, someone could still instantiate Package using its constructor without using the with clause. You don't want that to happen. You can fix this by creating a PackageResource class that defines the __enter__ and __exit__ methods. Then, the Package class would be defined strictly inside the __enter__ method and returned. That way, the caller never could instantiate the Package class without using a with statement:

class PackageResource:
    def __enter__(self):
        class Package:
            ...
        self.package_obj = Package()
        return self.package_obj

    def __exit__(self, exc_type, exc_value, traceback):
        self.package_obj.cleanup()

You'd use this as follows:

with PackageResource() as package_obj:
    # use package_obj

How to do cleanup reliably in python?

If weakrefs aren't broken, I guess this may work:

from weakref import ref

pointers = set()

class Pointer(object):
    def __init__(self, cfun, ptr):
        pointers.add(self)
        self.ref = ref(ptr, self.cleanup)
        self.data = cast(ptr, c_void_p).value # python cast it so smart, but it can't be smarter than this.
        self.cfun = cfun

    def cleanup(self, obj):
        print 'cleanup 0x%x' % self.data
        self.cfun(self.data)
        pointers.remove(self)

def cleanup(cfun, ptr):
    Pointer(cfun, ptr)

I yet try it. The important piece is that the Pointer doesn't have any strong references to the foreign pointer, except an integer. This should work if ctypes doesn't free memory that I should free with the bindings. Yeah, it's basicly a hack, but I think it may work better than the earlier things I've been trying.

Edit: Tried it, and it seem to work after small finetuning my code. A surprising thing is that even if I got del out from all of my structures, it seem to still fail. Interesting but frustrating.

Neither works, from some weird chance I've been able to drop away cyclic references in places, but things stay broke.

Edit: Well.. weakrefs WERE broken after all! so there's likely no solution for reliable cleanup in python, except than forcing it being explicit.

Is relying on __del__() for cleanup in Python unreliable?

You observe the typical issue with finalizers in garbage collected languages. Java has it, C# has it, and they all provide a scope based cleanup method like the Python with keyword to deal with it.

The main issue is, that the garbage collector is responsible for cleaning up and destroying objects. In C++ an object gets destroyed when it goes out of scope, so you can use RAII and have well defined semantics. In Python the object goes out of scope and lives on as long as the GC likes. Depending on your Python implementation this may be different. CPython with its refcounting based GC is rather benign (so you rarely see issues), while PyPy, IronPython and Jython might keep an object alive for a very long time.

For example:

def bad_code(filename):
    return open(filename, 'r').read()

for i in xrange(10000):
    bad_code('some_file.txt')

bad_code leaks a file handle. In CPython it doesn't matter. The refcount drops to zero and it is deleted right away. In PyPy or IronPython you might get IOErrors or similar issues, as you exhaust all available file descriptors (up to ulimit on Unix or 509 handles on Windows).

Scope based cleanup with a context manager and with is preferable if you need to guarantee cleanup. You know exactly when your objects will be finalized. But sometimes you cannot enforce this kind of scoped cleanup easily. Thats when you might use __del__, atexit or similar constructs to do a best effort at cleaning up. It is not reliable but better than nothing.

You can either burden your users with explicit cleanup or enforcing explicit scopes or you can take the gamble with __del__ and see some oddities now and then (especially interpreter shutdown).

What's the proper way to clean up static python object references in a CPython extension module?

You can't unload C extension modules at all. There is just no way to do it, and I know for sure that most of the standard extension modules would leak like crazy if there was.

How to do some clean when class object is called?

You can always return a proxy object instead of the A instance to get the syntax you want.

class A:

  def __call__(self, **kwargs):
    return B(kwargs)

  def get(self):
    return {}

class B:

    def __init__(self, d):
        self.d = d

    def get(self):
        return self.d

a = A()
print(a(i=1, j=2).get())
print(a.get())

How to specify clean up behavior of an object when it is garbage collected in Python?

The only way to ensure such a method is called if you don't trust users is using __del__ (docs). From the docs:

Called when the instance is about to be destroyed.

Note that there are lots of issues that make using del tricky. For example, at the moment it is called, the interpreter may be shutting down already - meaning other objects and modules may have been destroyed already. See the notes and warnings for details.

---

If you really cannot rely on users to be consenting adults, I would prevent them from implicitly avoiding close - don't give them a public open in the first place. Only supply the methods to support with. If anybody explicitly digs into your code to do otherwise, they probably have a good reason for it.

---

Related Topics