Threads in bash?
Bash doesn't support threading per se, but you could launch multiple java processes in the background, like:
java myprog &
java myprog &
java myprog &
Anything more than that you might look into Python or Ruby, which have thread management utilities, you could wait for each one to finish and collect output/exit status, etc.
Edit: Borrowing the suggestion from @CédricJulien to use wait
, here's a more thorough example. Given this MyProg.java
program:
public class MyProg {
public static void main(String[] args) {
System.exit(Integer.parseInt(args[0]));
}
}
you could write the following bash-threads.sh
script to launch multiple instances of it in parallel:
#!/bin/bash
set -o errexit
java MyProg 1 &
pid1=$!
java MyProg 0 &
pid2=$!
java MyProg 2 &
pid3=$!
wait $pid1 && echo "pid1 exited normally" || echo "pid1 exited abnormally with status $?"
wait $pid2 && echo "pid2 exited normally" || echo "pid2 exited abnormally with status $?"
wait $pid3 && echo "pid3 exited normally" || echo "pid3 exited abnormally with status $?"
Its output is:
pid1 exited abnormally with status 1
pid2 exited normally
pid3 exited abnormally with status 2
bash while loop threading
You can send tasks to the background by &
If you intend to wait for all of them to finish you can use the wait
command:
process_to_background &
echo Processing ...
wait
echo Done
You can get the pid
of the given task started in the background if you want to wait for one (or few) specific tasks.
important_process_to_background &
important_pid=$!
while i in {1..10}; do
less_important_process_to_background $i &
done
wait $important_pid
echo Important task finished
wait
echo All tasks finished
On note though: the background processes can mess up the output as they will run asynchronously. You might want to use a named pipe to collect the output from them.
edit
As asked in the comments there might be a need for limiting the background processes forked. In this case you can keep track of how many background processes you've started and communicate with them through a named pipe.
mkfifo tmp # creating named pipe
counter=0
while read ip
do
if [ $counter -lt 10 ]; then # we are under the limit
{ check $ip; echo 'done' > tmp; } &
let $[counter++];
else
read x < tmp # waiting for a process to finish
{ check $ip; echo 'done' > tmp; } &
fi
done
cat /tmp > /dev/null # let all the background processes end
rm tmp # remove fifo
Multithreading in Bash
Sure, just add &
after the command:
read_cfg cfgA &
read_cfg cfgB &
read_cfg cfgC &
wait
all those jobs will then run in the background simultaneously. The optional wait
command will then wait for all the jobs to finish.
Each command will run in a separate process, so it's technically not "multithreading", but I believe it solves your problem.
How to create Multiple Threads in Bash Shell Script
That's not a thread, but a background process. They are similar but:
So, effectively we can say that threads and light weight processes are same.
The main difference between a light weight process (LWP) and a normal process is that LWPs share same address space and other resources like open files etc. As some resources are shared so these processes are considered to be light weight as compared to other normal processes and hence the name light weight processes.
NB: Redordered for clarity
What are Linux Processes, Threads, Light Weight Processes, and Process State
You can see the running background process using the jobs
command. E.g.:
nick@nick-lt:~/test/npm-test$ sleep 10000 &
[1] 23648
nick@nick-lt:~/test/npm-test$ jobs
[1]+ Running
You can bring them to the foreground using fg
:
nick@nick-lt:~/test/npm-test$ fg 1
sleep 1000
where the cursor will wait until the sleep time has elapsed. You can pause the job when it's in the foreground (as in the scenario after fg 1
) by pressing CTRL-Z
(SIGTSTP
), which gives something like this:
[1]+ Stopped sleep 1000
and resume it by typing:
bg 1 # Resumes in the background
fg 1 # Resumes in the foreground
and you can kill it by pressing CTRL-C
(SIGINT
) when it's in the foreground, which just ends the process, or through using the kill command with the %
affix to the jobs
ID:
kill %1 # Or kill <PID>
Onto your implementation:
BROWSERS=
for i in "${@}"; do
case $i in
-b)
shift
BROWSERS="$1"
;;
*)
;;
esac
done
IFS=',' read -r -a SPLITBROWSERS <<< "$BROWSERS"
for browser in "${SPLITBROWSERS[@]}"
do
echo "Running ${browser}..."
$browser &
done
Can be called as:
./runtests.sh -b firefox,chrome,ie
Tadaaa.
Bash threading: wait for all job threads to finish doesn't work?
Update: good points in the comment.
So, on a second look, it turns out the problem is the subshell that is created by the pipe to the loop. It's a good way to structure the script but you need to do the final wait in the shell that spun off the background tasks.
So do something like this:
find /home/some/path -type f | (while read filename; do
echo "Creating archive of $filename"
func $somevariable &
done
wait
)
Multi-threaded BASH programming - generalized method?
#adjust these as required
args_per_proc=1 #1 is fine for long running tasks
procs_in_parallel=4
xargs -n$args_per_proc -P$procs_in_parallel povray < list
Note the nproc
command coming soon to coreutils will auto determine
the number of available processing units which can then be passed to -P
How many threads will be started with shell script wait
First, processes are not threads. A process contains one or more threads. It starts with one thread, and may create more if it so chooses. It's a similar idea, but threads within a process share the process's address space (each thread can access/share the program's variables). Processes each get a separate address space, and are unable to access memory outside that assigned to it by the OS.
Any program you run gets a process. When you run a script, the language interpreter--be it shell, Python, whatever--gets invoked to execute the script, and that's a process. The difference between a program and a process is that the process is the running instance. So if you have 3 terminals open running bash
, you have 3 processes running the one program. Note this doesn't necessarily mean windows: my mail program can have several windows open, but it's all still done by one process.
Yes, you can start numerous concurrent processes. Limits are imposed by the OS. 32K is a common limit, but different flavors of Unix/Linux support different process counts. There's usually also a per-user process limit, unless you're root
.
In practice, concurrent process count is also limited by available memory and CPU. If you have 4GB of RAM, and you've got a program where each process/instance takes up 500K, you could run about 6000 copies before you exhaust RAM (500K*6000 copies = 3GB, and the OS needs some for itself). Your system will rely on its swapfile at this point, but you're going to encounter thrashing if all these processes are trying to run. If you do this to your SSD, you will shorten its life.
And, unless you've got a supercomputer with hundreds or thousands of processors, more than a few concurrent, CPU-intensive processes is all that's practical. If you start 100 CPU-intensive ("CPU bound") processes on a 4-core machine, the OS will spread core time over all 100 using time slicing, so each process will run at 4 cores/100 processes = 1/25 the rate it would run had it a core to itself. You won't get more done by forking thousands of concurrent processes, unless you have the hardware to actually do the work.
The flipside of being CPU bound is being I/O bound---suppose you want to mirror a website, so you're going to try downloading all 1000 pages in parallel. It's not going to be any faster than a limited number of parallel connections each grabbing items sequentially, because only so many bits can flow through the network. Once you saturate the network, more concurrency won't make anything faster.
You can use ps
to list your personal processes, or ps -ef
or ps aux
to view all processes. There are many: as I'm writing this, my system has 235, but most of them are idle: terminals I'm not using at the moment, networking support, audio support at-the-ready in case it's called on, the web browser I'm writing in, the compositor that updates the screen when asked to by the web browser. You can learn a lot about your OS by looking through this list, and looking up what various programs do/what services they provide. This is where you see your OS is not one big black box, but a collection of many programs/processes, each providing some limited functionality, but together provide most of the OS services.
Related Topics
Can Ebpf Modify the Return Value or Parameters of a Syscall
Insmod Error: Inserting './Hello.Ko': -1 Invalid Module Format"
How to Self Dlopen an Executable Binary
Execute Command on Remote Server via Ssh
Starting Point for Clock_Monotonic
Replacing a Running Executable in Linux
Why Does Arm-Linux-Gnueabi-G++-4.4 Always Build a "7-A" Binary
Why Is Dd with the 'Direct' (O_Direct) Flag So Dramatically Faster
How to Extract Numbers from a String
Bash: How to Tokenize a String Variable
Why Does Zookeeper Not Use My Log4J.Properties File Log Directory
Linux Dlopen: Can a Library Be "Notified" When It Is Loaded
Call to Daemon in a /Etc/Init.D Script Is Blocking, Not Running in Background
How to Udp Broadcast with C in Linux