How to Fire and Forget a Subprocess

Fire and forget a process from a Python script

Answering my own question: I ended up simply using os.system with & at the end of command as suggested by @kevinsa. This allows the parent process to be terminated without the child being terminated.

Here's some code:

child.py

#!/usr/bin/python

import time
print "Child started"
time.sleep(10)
print "Child finished"

parent.py, using subprocess.Popen:

#!/usr/bin/python

import subprocess
import time

print "Parent started"
subprocess.Popen("./child.py")
print "(child started, sleeping)"

time.sleep(5)

print "Parent finished"

Output:

$ ./parent.py
Parent started
(child started, sleeping)
Child started
^CTraceback (most recent call last):
Traceback (most recent call last):
  File "./child.py", line 5, in <module>
  File "./parent.py", line 13, in <module>
        time.sleep(10)
time.sleep(5)
KeyboardInterrupt
KeyboardInterrupt

• note how the child never finishes if the parent is interrupted with Ctrl-C

parent.py, using os.system and &

#!/usr/bin/python

import os
import time

print "Parent started"
os.system("./child.py &")
print "(child started, sleeping)"

time.sleep(5)

print "Parent finished"

Output:

$ ./parent.py
Parent started
(child started, sleeping)
Child started
^CTraceback (most recent call last):
  File "./parent.py", line 12, in <module>
    time.sleep(5)
KeyboardInterrupt

$ Child finished

Note how the child lives beyond the Ctrl-C.

How to fire and forget a subprocess?

The fork function separates your process in two.

Both processes then receive the result of the function. The child receives a value of zero/nil (and hence knows that it's the child) and the parent receives the PID of the child.

Hence:

exec("something") if fork.nil?

will make the child process start "something", and the parent process will carry on with where it was.

Note that exec() replaces the current process with "something", so the child process will never execute any subsequent Ruby code.

The call to Process.detach() looks like it might be incorrect. I would have expected it to have the child's PID in it, but if I read your code right it's actually detaching the parent process.

How can I fire and forget a process in Perl?

From perlfaq8's answer to How do I start a process in the background?

---

Several modules can start other processes that do not block your Perl
program. You can use IPC::Open3, Parallel::Jobs, IPC::Run, and some of
the POE modules. See CPAN for more details.

You could also use

system("cmd &")

or you could use fork as documented in "fork" in perlfunc, with further
examples in perlipc. Some things to be aware of, if you're on a Unix-
like system:

STDIN, STDOUT, and STDERR are shared

Both the main process and the backgrounded one (the "child"
process) share the same STDIN, STDOUT and STDERR filehandles. If
both try to access them at once, strange things can happen. You
may want to close or reopen these for the child. You can get
around this with "open"ing a pipe (see "open" in perlfunc) but on
some systems this means that the child process cannot outlive the
parent.

Signals

You'll have to catch the SIGCHLD signal, and possibly SIGPIPE too.
SIGCHLD is sent when the backgrounded process finishes. SIGPIPE is
sent when you write to a filehandle whose child process has closed
(an untrapped SIGPIPE can cause your program to silently die).
This is not an issue with "system("cmd&")".

Zombies

You have to be prepared to "reap" the child process when it
finishes.

   $SIG{CHLD} = sub { wait };

   $SIG{CHLD} = 'IGNORE';

You can also use a double fork. You immediately wait() for your
first child, and the init daemon will wait() for your grandchild
once it exits.

   unless ($pid = fork) {
       unless (fork) {
           exec "what you really wanna do";
           die "exec failed!";
       }
       exit 0;
   }
   waitpid($pid, 0);

See "Signals" in perlipc for other examples of code to do this.
Zombies are not an issue with "system("prog &")".

How can I fire and forget a task without blocking main thread?

Your questions are so abstract that I'll try to give common answers to all of them.

How can I "fire and forget" a task without blocking main thread?

It depends on what you mean by saying forget.

• If you are not planning to access that task after running, you can run it in a parallel process.
• If the main application should be able to access a background task, then you should have an event-driven architecture. In that case, the things previously called tasks will be services or microservices.

I don't want to use any task queues (celery, rabbitmq, etc.) here because the tasks I'm thinking of are too small and fast to run. Just want to get them done as out of the way as possible. Would that be an async approach? Throwing them onto another process?

If it contains loops or other CPU-bound operations, then right to use a subprocess. If the task makes a request (async), reads files, logs to stdout, or other I/O bound operations, then it is right to use coroutines or threads.

Does it make sense to have a separate thread that handles background jobs? Like a simple job queue but very lightweight and does not require additional infrastructure?

We can't just use a thread as it can be blocked by another task that uses CPU-bound operations. Instead, we can run a background process and use pipes, queues, and events to communicate between processes. Unfortunately, we cannot provide complex objects between processes, but we can provide basic data structures to handle status changes of the tasks running in the background.

Regarding the Starlette and the BackgroundTask

Starlette is a lightweight ASGI framework/toolkit, which is ideal for building async web services in Python. (README description)

It is based on concurrency. So even this is not a generic solution for all kinds of tasks.
NOTE: Concurrency differs from parallelism.

I'm wondering if we can build something more generic where you can run background tasks in scripts or webservers alike, without sacrificing performance.

The above-mentioned solution suggests use a background process. Still, it will depend on the application design as you must do things (emit an event, add an indicator to the queue, etc.) that are needed for communication and synchronization of running processes (tasks). There is no generic tool for that, but there are situation-dependent solutions.

Situation 1 - The tasks are asynchronous functions

Suppose we have a request function that should call an API without blocking the work of other tasks. Also, we have a sleep function that should not block anything.

import asyncio
import aiohttp

async def request(url):
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as response:
            try:
                return await response.json()
            except aiohttp.ContentTypeError:
                return await response.read()

async def sleep(t):
    await asyncio.sleep(t)

async def main():
    background_task_1 = asyncio.create_task(request("https://google.com/"))
    background_task_2 = asyncio.create_task(sleep(5))

    ...  # here we can do even CPU-bound operations

    result1 = await background_task_1

    ...  # use the 'result1', etc.

    await background_task_2

if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())
    loop.close()

In this situation, we use asyncio.create_task to run a coroutine concurrently (like in the background). Sure we could run it in a subprocess, but there is no reason for that as it would use more resources without improving the performance.

Situation 2 - The tasks are synchronous functions (I/O bound)

Unlike the first situation where the functions were already asynchronous, in this situation, those are synchronous but not CPU-bound (I/O bound). This gives an ability to run them in threads or make them asynchronous (using asyncio.to_thread) and run concurrently.

import time
import asyncio
import requests

def asynchronous(func):
    """
    This decorator converts a synchronous function to an asynchronous
    
    Usage:
        @asynchronous
        def sleep(t):
            time.sleep(t)
            
        async def main():
            await sleep(5)
    """
    
    async def wrapper(*args, **kwargs):
        await asyncio.to_thread(func, *args, **kwargs)

    return wrapper

@asynchronous
def request(url):
    with requests.Session() as session:
        response = session.get(url)
        try:
            return response.json()
        except requests.JSONDecodeError:
            return response.text

@asynchronous
def sleep(t):
    time.sleep(t)

    
async def main():
    background_task_1 = asyncio.create_task(request("https://google.com/"))
    background_task_2 = asyncio.create_task(sleep(5))
    ...

Here we used a decorator to convert a synchronous (I/O bound) function to an asynchronous one and use them like in the first situation.

Situation 3 - The tasks are synchronous functions (CPU-bound)

To run CPU-bound tasks parallelly in the background we have to use multiprocessing. And for ensuring the task is done we use the join method.

import time
import multiprocessing

def task():
    for i in range(10):
        time.sleep(0.3)

def main():
    background_task = multiprocessing.Process(target=task)
    background_task.start()

    ...  # do the rest stuff that does not depend on the background task

    background_task.join()  # wait until the background task is done

    ...  # do stuff that depends on the background task

if __name__ == "__main__":
    main()

Suppose the main application depends on the parts of the background task. In this case, we need an event-driven design as the join cannot be called multiple times.

import multiprocessing

event = multiprocessing.Event()

def task():
    ...  # synchronous operations

    event.set()  # notify the main function that the first part of the task is done

    ...  # synchronous operations

    event.set()  # notify the main function that the second part of the task is also done

    ...  # synchronous operations

def main():
    background_task = multiprocessing.Process(target=task)
    background_task.start()

    ...  # do the rest stuff that does not depend on the background task

    event.wait()  # wait until the first part of the background task is done

    ...  # do stuff that depends on the first part of the background task

    event.wait()  # wait until the second part of the background task is done

    ...  # do stuff that depends on the second part of the background task

    background_task.join()  # wait until the background task is finally done

    ...  # do stuff that depends on the whole background task

if __name__ == "__main__":
    main()

As you already noticed with events we can just provide binary information and those are not effective if the processes are more than two (It will be impossible to know where the event was emitted from). So we use pipes, queues, and manager to provide non-binary information between the processes.

python fire and forget async function in background

Solved my issue by combining multithreading and async tasks as stated by @norbeq here.

Fire and forget python async/await

Upd:

Replace asyncio.ensure_future with asyncio.create_task everywhere if you're using Python >= 3.7 It's a newer, nicer way to spawn tasks.

---

asyncio.Task to "fire and forget"

According to python docs for asyncio.Task it is possible to start some coroutine to execute "in the background". The task created by asyncio.ensure_future won't block the execution (therefore the function will return immediately!). This looks like a way to "fire and forget" as you requested.

import asyncio

async def async_foo():
    print("async_foo started")
    await asyncio.sleep(1)
    print("async_foo done")

async def main():
    asyncio.ensure_future(async_foo())  # fire and forget async_foo()

    # btw, you can also create tasks inside non-async funcs

    print('Do some actions 1')
    await asyncio.sleep(1)
    print('Do some actions 2')
    await asyncio.sleep(1)
    print('Do some actions 3')

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

Output:

Do some actions 1
async_foo started
Do some actions 2
async_foo done
Do some actions 3

What if tasks are executing after the event loop has completed?

Note that asyncio expects tasks to be completed at the moment the event loop completes. So if you'll change main() to:

async def main():
    asyncio.ensure_future(async_foo())  # fire and forget

    print('Do some actions 1')
    await asyncio.sleep(0.1)
    print('Do some actions 2')

You'll get this warning after the program finished:

Task was destroyed but it is pending!
task: <Task pending coro=<async_foo() running at [...]

To prevent that you can just await all pending tasks after the event loop has completed:

async def main():
    asyncio.ensure_future(async_foo())  # fire and forget

    print('Do some actions 1')
    await asyncio.sleep(0.1)
    print('Do some actions 2')

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())
    
    # Let's also finish all running tasks:
    pending = asyncio.Task.all_tasks()
    loop.run_until_complete(asyncio.gather(*pending))

Kill tasks instead of awaiting them

Sometimes you don't want to await tasks to be done (for example, some tasks may be created to run forever). In that case, you can just cancel() them instead of awaiting them:

import asyncio
from contextlib import suppress

async def echo_forever():
    while True:
        print("echo")
        await asyncio.sleep(1)

async def main():
    asyncio.ensure_future(echo_forever())  # fire and forget

    print('Do some actions 1')
    await asyncio.sleep(1)
    print('Do some actions 2')
    await asyncio.sleep(1)
    print('Do some actions 3')

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

    # Let's also cancel all running tasks:
    pending = asyncio.Task.all_tasks()
    for task in pending:
        task.cancel()
        # Now we should await task to execute it's cancellation.
        # Cancelled task raises asyncio.CancelledError that we can suppress:
        with suppress(asyncio.CancelledError):
            loop.run_until_complete(task)

Output:

Do some actions 1
echo
Do some actions 2
echo
Do some actions 3
echo

Asynchronous Python - fire and forget HTTP request

I have answered a rather similar question.

async def main():
    asyncio.ensure_future(fire())

ensure_future schedules coro execution, but does not wait for its completion and run_until_complete does not wait for the completion of all futures.

This should fix it:

async def main():
    await fire()

Stop python sub-process leaving behind dead 'python' process

You should call child.wait at some point.

---

Related Topics