Run Multiple Python Scripts Concurrently

Run multiple python scripts concurrently

With Bash:

python script1.py &
python script2.py &

That's the entire script. It will run the two Python scripts at the same time.

Python could do the same thing itself but it would take a lot more typing and is a bad choice for the problem at hand.

I think it's possible though that you are taking the wrong approach to solving your problem, and I'd like to hear what you're getting at.

how to run multiple python scripts at the same time?

For Ubuntu, there is an answer here that may work. You can run multiple programs through bash. The "&" tells it to run the program in the background.

python program1.py &
python program2.py &

Since it sounds like you are using a remote server that has Ubuntu, I would recommend using tmux instead. It allows you to open up multiple sessions, run a program on each on and keep them running after you have closed your connection. It also allows you to enter back into each session, if you need to enter/read anything from your programs. I found this guide helpful when I had to do something similar a few months ago.

You can run batch files on Ubuntu as well. I'm not as familiar with running batch files on Ubuntu, but something like the following should work for you. You can also add while loops, if statements, etc.. Anything you would normally type into the shell can be put into the batch file to automatically run your programs or navigate directories.

#!/bin/bash
ECHO starting training program
# without the "&", it waits for your training program to finish running
python training_program.py
ECHO training program completed

# Adding the "&" tells the programs to run in the background
# You can also use tmux instead, if you want to navigate the different programs
python program1.py &
python program2.py &
ECHO training programs running in the background

The file should be saved with a ".sh" extension, then make the file executable by running the following through your shell.

chmod +x your_batch_file.sh 

If you are using Windows, you could create a batch file that runs all the programs. Here is an example of the file that you can create with your editor of choice:

# If you don't want to see the outputs/print of your training program, 
#     add @ECHO OFF to the start. If you want to see them, remove the @ECHO OFF
@ECHO OFF

# Without "start" before the script to run your training program,
#     the batch file will wait until the training program finishes
python "path\to\your\training_program.py"
ECHO training program completed

# Adding "start" opens it in a new window, and processes the next line
#     without waiting for the program to finish running
start python "path\to\your\program1.py"
ECHO Running program1
start python "path\to\your\program2.py"
ECHO Running program2

# Adding "PAUSE" makes the script wait for you manually type a key to continue,
#     but it is not required. You can add PAUSE anywhere in the script
PAUSE

"start" runs each program in a new window. After you have configured the text, safe the file with a ".bat" extension. Then all you have to do is click on the file to run the batch file, which will open each program in a separate window.

Similarly, you can just run the following from command prompt and it will open them in separate windows as well.

start python path\to\your\program1.py
start python path\to\your\program2.py

But it sounds like you are performing this more than once, in which case a batch file may be more suitable.

python run multiple scripts

As someone else already suggested, you could make a python file which executes your N python scripts.

Using subprocess as described here: https://stackoverflow.com/a/11230471/11962413

import subprocess

subprocess.call("./test1.py", shell=True)
subprocess.call("./test2.py", shell=True)

Run multiple python scripts in parallel from master script

When you want to spawn a new thread, you need to pass the address of the function you want the thread to execute, and not to call it. What you are doing here is essentially spawning a new thread that immediately calls function_1() which of course runs forever.

Also, you won't be able to reach this line of code:

print("thread finished")

As the threads are executing a while loop - forever, so it is redundent..

from time import sleep
from threading import Thread


def function_1():
    print('function 1 started...')
    while True:
        print('1')
        sleep(1)


def function_2():
    print('function 2 started...')
    while True:
        print('2')
        sleep(1)


thread_1 = Thread(target=function_1)
thread_2 = Thread(target=function_2)
thread_1.start()
thread_2.start()

thread_1.join()
thread_2.join()
# print("thread finished") - redundant

How to run multiple python scripts simultaneously from a wrapper script in such a way that CPU utilization is maximized?

This is a technique I developed for calling many external programs using subprocess.Popen. In this example, I'm calling convert make JPEG images from DICOM files.

In short; it uses manageprocs to keep checking a list of running subprocesses. If one is finished, it is removed and a new one is started as long as unprocesses files remain. After that, the remaining processes are watched until they are all finished.

from datetime import datetime
from functools import partial
import argparse
import logging
import os
import subprocess as sp
import sys
import time


def main():
    """
    Entry point for dicom2jpg.
    """
    args = setup()
    if not args.fn:
        logging.error("no files to process")
        sys.exit(1)
    if args.quality != 80:
        logging.info(f"quality set to {args.quality}")
    if args.level:
        logging.info("applying level correction.")
    start_partial = partial(start_conversion, quality=args.quality, level=args.level)

    starttime = str(datetime.now())[:-7]
    logging.info(f"started at {starttime}.")
    # List of subprocesses
    procs = []
    # Do not launch more processes concurrently than your CPU has cores.
    # That will only lead to the processes fighting over CPU resources.
    maxprocs = os.cpu_count()
    # Launch and mange subprocesses for all files.
    for path in args.fn:
        while len(procs) == maxprocs:
            manageprocs(procs)
        procs.append(start_partial(path))
    # Wait for all subprocesses to finish.
    while len(procs) > 0:
        manageprocs(procs)
    endtime = str(datetime.now())[:-7]
    logging.info(f"completed at {endtime}.")


def start_conversion(filename, quality, level):
    """
    Convert a DICOM file to a JPEG file.

    Removing the blank areas from the Philips detector.

    Arguments:
        filename: name of the file to convert.
        quality: JPEG quality to apply
        level: Boolean to indicate whether level adustment should be done.
    Returns:
        Tuple of (input filename, output filename, subprocess.Popen)
    """
    outname = filename.strip() + ".jpg"
    size = "1574x2048"
    args = [
        "convert",
        filename,
        "-units",
        "PixelsPerInch",
        "-density",
        "300",
        "-depth",
        "8",
        "-crop",
        size + "+232+0",
        "-page",
        size + "+0+0",
        "-auto-gamma",
        "-quality",
        str(quality),
    ]
    if level:
        args += ["-level", "-35%,70%,0.5"]
    args.append(outname)
    proc = sp.Popen(args, stdout=sp.DEVNULL, stderr=sp.DEVNULL)
    return (filename, outname, proc)


def manageprocs(proclist):
    """Check a list of subprocesses for processes that have ended and
    remove them from the list.

    Arguments:
        proclist: List of tuples. The last item in the tuple must be
                  a subprocess.Popen object.
    """
    for item in proclist:
        filename, outname, proc = item
        if proc.poll() is not None:
            logging.info(f"conversion of “{filename}” to “{outname}” finished.")
            proclist.remove(item)
    # since manageprocs is called from a loop, keep CPU usage down.
    time.sleep(0.05)


if __name__ == "__main__":
    main()

I've left out setup(); it's using argparse to deal with command-line arguments.

Here the thing to be processed is just a list of file names.
But it could also be (in your case) a list of tuples of script names and arguments.

run multiple python scripts at the same time

You can always open a terminal terminal window -- with either Python: Create Terminal or Open New Terminal -- and launch the script(s) manually in separate terminals.

How to use batch file to run multiple python scripts simultaneously

Use the start command to initiate a process.

@echo off
start "" foo.py
start "" bar.py
start "" baz.py

Re comment: “is there way to start these minimized?”

You can always ask about how a command works by typing the command name followed by a /?. In this case, start /? tells us its command-line options include:

MIN          Start window minimized.

Hence, to start the application minimized, use:

start "" /MIN quux.py

Run two python files at the same time

I think you are looking for multi-threading

you could merge your both script into another script,
then lauch them using theads

--edit--

from threading import Thread

import cv2
import numpy as np
import os
from playsound import playsound

def play_sound(): 
    # import your script A
    a = (r"C:\Users\A\Desktop\sound.mp3")
    playsound(a)

def CV2_stuff():
    # import your script B
    os.environ['SDL_VIDEO_CENTERED'] = '1'
    cap = cv2.VideoCapture("video.mp4")
    ...


Thread(target = play_sound).start() 
Thread(target = CV2_stuff).start()

hope it helps

this could work too

import ScriptA
import ScriptB

ScriptA.function()
ScriptB.function()

but they wouldn't be exectuted in the same time

---

Related Topics