Is the += Operator Thread-Safe in Python

Is the += operator thread-safe in Python?

Single opcodes are thread-safe because of the GIL but nothing else:

import time
class something(object):
    def __init__(self,c):
        self.c=c
    def inc(self):
        new = self.c+1 
        # if the thread is interrupted by another inc() call its result is wrong
        time.sleep(0.001) # sleep makes the os continue another thread
        self.c = new

x = something(0)
import threading

for _ in range(10000):
    threading.Thread(target=x.inc).start()

print x.c # ~900 here, instead of 10000

Every resource shared by multiple threads must have a lock.

Is python's in language construct thread-safe for lists?

I am assuming you are using CPython here.

Provided there is no custom __contains__ or __iter__ hook dropping back into Python or the values you test against contained in the list use custom __eq__ hooks implemented in Python code, the in operator can be handled entirely in C, and is just one opcode.

This makes the operation entirely thread-safe; Python threads only switch between opcodes; the GIL (global interpreter lock) normally only unlocks between opcodes.

That said, if you use in on a custom C type that unlocks the GIL when testing containment would not be thread-safe.

In other words: the in bytecode test is locked, but if the operator needs to call Python code (through __contains__, iterating with __iter__ when no __contains__ implementation is available, or the values are test against __eq__ hooks implemented in Python), then the operation is not thread safe.

For other Python implementations, how threading is handled can vary widely. Certainly, Jython and IronPython have no GIL, and you should assume the operation is not thread safe.

Are Python built-in containers thread-safe?

You need to implement your own locking for all shared variables that will be modified in Python. You don't have to worry about reading from the variables that won't be modified (ie, concurrent reads are ok), so immutable types (frozenset, tuple, str) are probably safe, but it wouldn't hurt. For things you're going to be changing - list, set, dict, and most other objects, you should have your own locking mechanism (while in-place operations are ok on most of these, threads can lead to super-nasty bugs - you might as well implement locking, it's pretty easy).

By the way, I don't know if you know this, but locking is very easy in Python - create a threading.lock object, and then you can acquire/release it like this:

import threading
list1Lock = threading.Lock()

with list1Lock:
    # change or read from the list here
# continue doing other stuff (the lock is released when you leave the with block)

In Python 2.5, do from __future__ import with_statement; Python 2.4 and before don't have this, so you'll want to put the acquire()/release() calls in try:...finally: blocks:

import threading
list1Lock = threading.Lock()

try:
    list1Lock.acquire()
    # change or read from the list here
finally:
    list1Lock.release()
# continue doing other stuff (the lock is released when you leave the with block)

Some very good information about thread synchronization in Python.

Python C API - Is it thread safe?

Python will not release the GIL when you are running C code (unless you either tell it to or cause the execution of Python code - see the warning note at the bottom!). It only releases the GIL just before a bytecode instruction (not during) and from the interpreter's point of view running a C function is part of executing the CALL_FUNCTION bytecode.* (Unfortunately I can't find a reference for this paragraph currently, but I'm almost certain it's right)

Therefore, unless you do anything specific your C code will be the only thread running and thus any operation you do in it should be thread safe.

If you specifically want to release the GIL - for example because you're doing a long calculation which doesn't interfere with Python, reading from a file, or sleeping while waiting for something else to happen - then the easiest way is to do Py_BEGIN_ALLOW_THREADS then Py_END_ALLOW_THREADS when you want to get it back. During this block you cannot use most Python API functions and it's your responsibility to ensure thread safety in C. The easiest way to do this is to only use local variables and not read or write any global state.

If you've already got a C thread running without the GIL (thread A) then simply holding the GIL in thread B does not guarantee that thread A won't modify C global variables. To be safe you need to ensure that you never modify global state without some kind of locking mechanism (either the Python GIL or a C mechanism) in all your C functions.

---

Additional thought

* One place where the GIL can be released in C code is if the C code calls something that causes Python code to be executed. This might be through using PyObject_Call. A less obvious place would be if Py_DECREF caused a destructor to be executed. You'd have the GIL back by the time your C code resumed, but you could no longer guarantee that global objects were unchanged. This obvious doesn't affect simple C like x++.

---

Belated Edit:

It should be emphasised that it's really, really, really easy to cause the execution of Python code. For this reason you shouldn't use the GIL in place of a mutex or actual locking mechanism. You should only consider it for operations that are really atomic (i.e. a single C API call) or entirely on non-Python C objects. You won't lose the GIL unexpected while executing C Code, but a lot of C API calls may release the GIL, do something else, and then regain the GIL before returning to your C code.

The purpose the GIL is to make sure that the Python internals don't get corrupted. The GIL will continue to serve this purpose within an extension module. However race conditions that involve valid Python objects arranged in ways you don't expect are still available to you. For example:

PySequence_SetItem(some_list, 0, some_item);
PyObject* item = PySequence_GetItem(some_list, 0);
assert(item == some_item); // may not be true 
// the destructor of the previous contents of item 0 may have released the GIL

Are Python ints thread-safe?

Like the other have said, Python objects are mostly thread-safe. Although you will need to use Locks in order to protect an object in a place that require it to go through multiple changes before being usable again.

Thread Safety in Python's dictionary

Python's built-in structures are thread-safe for single operations, but it can sometimes be hard to see where a statement really becomes multiple operations.

Your code should be safe. Keep in mind: a lock here will add almost no overhead, and will give you peace of mind.

https://web.archive.org/web/20201108091210/http://effbot.org/pyfaq/what-kinds-of-global-value-mutation-are-thread-safe.htm has more details.

is += in python thread safe?

+= is not threadsafe (source).

Python - How to turn thread into thread-safe

You might want to look into threading.Lock(), it can be used to prevent multiples threads from doing output tasks at the same time and thus mixing the words in the console :

def sendData(list, data):
    with lock:
        logger.log(data)

lock = threading.Lock()
def main():
    ...
    ...
    ...
    data_list.append((list[i], data))

    for index, data in data_list:
         threading.Thread(target=sendData, args=(list, data)).start()

This will prevent multiple threads from running the code in the "with" at the same time.

When a thread X enter in the "with" block, it will claim the lock. If another thread try to claim it (enter the "with" block), it will have to wait until the lock is released by the thread X.

---

Related Topics