I Don't Understand This Python _Del_ Behaviour

I don't understand this python __del__ behaviour

I'm providing my own answer because, while I appreciate the advice to avoid __del__, my question was how to get it to work properly for the code sample provided.

Short version: The following code uses weakref to avoid the circular reference. I thought I'd tried this before posting the question, but I guess I must have done something wrong.

import types, weakref

class Dummy():
    def __init__(self, name):
        self.name = name
    def __del__(self):
        print "delete",self.name

d2 = Dummy("d2")
def func(self):
    print "func called"
d2.func = types.MethodType(func, weakref.ref(d2)) #This works
#d2.func = func.__get__(weakref.ref(d2), Dummy) #This works too
d2.func()
del d2
d2 = None
print "after d2"

Longer version:
When I posted the question, I did search for similar questions. I know you can use with instead, and that the prevailing sentiment is that __del__ is BAD.

Using with makes sense, but only in certain situations. Opening a file, reading it, and closing it is a good example where with is a perfectly good solution. You've gone a specific block of code where the object is needed, and you want to clean up the object and the end of the block.

A database connection seems to be used often as an example that doesn't work well using with, since you usually need to leave the section of code that creates the connection and have the connection closed in a more event-driven (rather than sequential) timeframe.

If with is not the right solution, I see two alternatives:

1. You make sure __del__ works (see this blog for a better
   description of weakref usage)
2. You use the atexit module to run a callback when your program closes. See this topic for example.

While I tried to provide simplified code, my real problem is more event-driven, so with is not an appropriate solution (with is fine for the simplified code). I also wanted to avoid atexit, as my program can be long-running, and I want to be able to perform the cleanup as soon as possible.

So, in this specific case, I find it to be the best solution to use weakref and prevent circular references that would prevent __del__ from working.

This may be an exception to the rule, but there are use-cases where using weakref and __del__ is the right implementation, IMHO.

Use cases for __del__

Context managers (and try/finally blocks) are somewhat more restrictive than __del__. In general they require you to structure your code in such a way that the lifetime of the resource you need to free doesn't extend beyond a single function call at some level in the call stack, rather than, say, binding it to the lifetime of a class instance that could be destroyed at unpredictable times and places. It's usually a good thing to restrict the lifetime of resources to one scope, but there sometimes edge cases where this pattern is an awkward fit.

The only case where I've used __del__ (aside from for debugging, c.f. @MSeifert's answer) is for freeing memory allocated outside of Python by an external library. Because of the design of the library I was wrapping, it was difficult to avoid having a large number of objects that held pointers to heap-allocated memory. Using a __del__ method to free the pointers was the easiest way to do cleanup, since it would have been impractical to enclose the lifespan of each instance inside a context manager.

How to ensure the __del__ function is called on a Python class as is commonly (but incorrectly) expected?

If you understand all that, why not do it in the Pythonic way? Compare another class where cleanup is important: tempfile.TemporaryDirectory.

with TemporaryDirectory() as tmp:
    # ...
# tmp is deleted

def foo():
    tmp = TemporaryDirectory()
foo()
# tmp is deleted

How do they do this? Here's the relevant bit:

import weakref
class Foo():
    def __init__(self, name):
        self.name = name
        self._finalizer = weakref.finalize(self, self._cleanup, self.name)
        print("%s reporting for duty!" % name)

    @classmethod
    def _cleanup(cls, name):
        print("%s feels forgotten! Bye!" % name)

    def cleanup(self):
        if self._finalizer.detach():
            print("%s told to go away! Bye!" % self.name)

def foo():
    print("Calling Arnold")
    tmpfoo = Foo("Arnold")
    print("Finishing with Arnold")

foo()
# => Calling Arnold
# => Arnold reporting for duty
# => Finishing with Arnold
# => Arnold feels forgotten. Bye!

def bar():
    print("Calling Rocky")
    tmpbar = Foo("Rocky")
    tmpbar.cleanup()
    print("Finishing with Rocky")

bar()
# => Calling Rocky
# => Rocky reporting for duty!
# => Rocky told to go away! Bye!
# => Finishing with Rocky

weakref.finalize will trigger _cleanup when the object is garbage-collected, or at the end of the program if it's still around. We can keep the finaliser around so that we can explicitly kill the object (using detach) and mark it as dead so the finaliser is not called (when we want to manually handle the cleanup).

If you want to support the context usage with with, it is trivial to add __enter__ and __exit__ methods, just invoke cleanup in __exit__ ("manual cleanup" as discussed above).

Python attributeError on __del__

Your __del__ method assumes that the class is still present by the time it is called.

This assumption is incorrect. Groupclass has already been cleared when your Python program exits and is now set to None.

Test if the global reference to the class still exists first:

def __del__(self):
    if Groupclass:
        Groupclass.count -= 1

or use type() to get the local reference:

def __del__(self):
    type(self).count -= 1

but do note that this means that the semantics for count change if Groupclass is subclassed (each subclass gets a .count attribute versus only Groupclass having a .count attribute).

Quoting from the __del__ hook documentation:

Warning: Due to the precarious circumstances under which __del__() methods are invoked, exceptions that occur during their execution are ignored, and a warning is printed to sys.stderr instead. Also, when __del__() is invoked in response to a module being deleted (e.g., when execution of the program is done), other globals referenced by the __del__() method may already have been deleted or in the process of being torn down (e.g. the import machinery shutting down). For this reason, __del__() methods should do the absolute minimum needed to maintain external invariants. Starting with version 1.5, Python guarantees that globals whose name begins with a single underscore are deleted from their module before other globals are deleted; if no other references to such globals exist, this may help in assuring that imported modules are still available at the time when the __del__() method is called.

If you are using Python 3, two additional notes apply:

• CPython 3.3 automatically applies a randomized hash salt to the str keys used in a globals dictionary; this also affects the order in which globals are cleaned up, and it could be that you see the problem on only some of the runs.

• CPython 3.4 no longer sets globals to None (in most cases), as per Safe Object Finalization; see PEP 442.

Python behaviour I can't understand

The correct way to do this is to set a variable when you find a matching row. After the loop is done, check the variable to see if anything was found, and if not you print the message.

found = false
for row in csv_file:
    if code == row[0]:
        found = true
        print("This product is: ", row[1], "\n", "Quantity left in stock: ", row[2], "\n", "Cost: ", row[3], "\n")
        quant = int(input("How many do you you want?: "))
        if quant > int(row[2]):
            print("We don't have that many, try again")
            order()
        else:
            orderrow = row
            orderrow[2] = quant
            orderrow[3] = float(orderrow[3]) * quant
            totalcost = totalcost + orderrow[3]
            orderlist.append(orderrow)
if !found
    print("ITEM NOT FOUND, TRY AGAIN")
    order()

Behaviour of DEL with list taken from a dict

Yes, this is intended behaviour. Python names and entries in dictionaries and lists are mere references to the actual objects, stored in a big pile (the heap) in memory.

Thus, my_dict[2] and chili both refer to the same list object, and list objects are mutable. Deleting an entry from a list object means that all references to that object see the change.

If you want chili to not be the same list object, you must create a copy. You can create a shallow copy with either:

chili = my_dict[2][:]

as slicing from first to last index produces a new list object, or using:

chili = list(my_dict[2])

which produces a new list object, copying all references stored in the original sequence.

These create shallow copies; if anything in my_dict[2] is itself mutable you would still be manipulating an object shared between the chili list and the my_dict[2] list.

You can create a deep copy by using the copy.deepcopy() function, which recursively produces copies of objects.

---

Related Topics