Multi-Threaded Ping Script

pinging ~ 100,000 servers, is multithreading or multiprocessing better?

TLDR; MultiThreading is the solution for you-
The threading module uses threads, the multiprocessing module uses processes.
The difference is that threads run in the same memory space, while processes have separate memory.

As for question 1-

For IO tasks, like querying a database or loading a webpage the CPU is just doing nothing but waiting for an answer and that's a waste of resources, thus multithreading is the answer (:

As for question 2-

You can just create pool of threads that will manage them to run simultaneously without you needing to break your head.

Multithreading. How to ping several IPs simultaneously properly?

You don't need threads to execute multiple processes - Just use subprocess.Popen - it doesn't block like os.system, so you can just run it multiple times and they will all run in parallel.

If b is a list that contains all your ips:

import subprocess

while True:
    result = []
    for ip in b:
        p = subprocess.Popen(['ping', '-n', '1', ip]) # runs ping in background
        result.append(p) # store the Popen object for later result retrieval

That will run multiple ping processes in the background! Now you just need to parse the results:

    try_again = []
    for ip, p in zip(b, result):
        if p.wait() == 0:
            print(ip, 'is ok!')
        else:
            print(ip, 'failed!')
            try_again.append(ip)

You can then repeat the ones that failed if you want:

    if not try_again:
        break
    time.sleep(100)
    b = try_again

multiple ping script keeps pinging the same ips from one list several times

It seems the problem is that you're splitting the list of num+i and forward and not grabbing the index value of num+i.
(Also since num is never incremented in your bottom loop, atm it serves no function)

Let's say I have a list that looks like this:

['1.1.1.1','8.8.8.8','8.8.4.4']

Currently you're splitting it like this over the 3 iterations of spawning threads:

['1.1.1.1','8.8.8.8','8.8.4.4']
['8.8.8.8','8.8.4.4']
['8.8.4.4']

This is simply caused by your : operator in your indexing when spawning the thread here:

for i in range(len(ip_list)):
t = Thread(target=ipPinger, args=(ip_list[num + i:],)) #Colon operator in indexing
t.start()

Since your IpPing function takes a list as an argument and a loop through that list it will then loop through the entire list and ping them all, for each thread.

I would also like you to consider that instead of spawning a thread for each IP to ping (in case a big list may be supplied), that you instead may want to spawn either a fixed amount of threads or a variable amount of threads
specified by the user.

Then have each thread pop values from the same list, or handle accessing that list through locks. Since the pinging here takes long and the access to the object is very short.

Good luck with your coding! :-)

Edit: Wording.

how to make ping script, ping a list in parallel

Thanks @mark-setchell i made this:

#!/bin/bash

#Ping function
function pingit () {

    target=${1}

    # send stdout/stderr to /dev/null since all we need is the return code

    ping -c 1 -w 1 ${target} >/dev/null 2>&1 &&
    pingtime=`ping -c 1 "${target}" | tail -1| awk -F '/' '{print $5}'` &&
        echo "`date '+%Y-%m-%d %H:%M:%S,%3N'`;${target};UP;$pingtime" ||
        echo "`date '+%Y-%m-%d %H:%M:%S,%3N'`;${target};Down;Uknown"
}
#input file (<list)
data=$(<./1A/list.txt)

for line in ${data}
do
    # call our function and place the call in the background

    pingit ${line} &
done

# wait for all outstanding background jobs to complete before continuing

wait

# [optional] let operator know we're done.
#echo "Completed."

Old output:

2018-01-13 15:14:53,136;192.168.2.1;UP;0.606
2018-01-13 15:14:56,208;192.168.2.2;UP;0.456
2018-01-13 15:14:59,231;192.168.2.3;UP;1.486
2018-01-13 15:15:00,244;192.168.2.4;Down;Uknown
2018-01-13 15:15:03,284;192.168.2.5;UP;0.359
2018-01-13 15:15:04,291;192.168.2.6;Down;Uknown
2018-01-13 15:15:05,298;192.168.2.7;Down;Uknown
2018-01-13 15:15:06,308;192.168.2.8;Down;Uknown
2018-01-13 15:15:07,316;192.168.2.9;Down;Uknown
2018-01-13 15:15:08,324;192.168.2.10;Down;Uknown
2018-01-13 15:15:09,330;192.168.2.11;Down;Uknown
2018-01-13 15:15:12,357;192.168.2.12;UP;0.709
2018-01-13 15:15:13,366;192.168.2.13;Down;Uknown
2018-01-13 15:15:14,371;192.168.2.14;Down;Uknown
2018-01-13 15:15:17,389;192.168.2.15;UP;1.107
2018-01-13 15:15:18,397;192.168.2.16;Down;Uknown
2018-01-13 15:15:19,410;192.168.2.17;Down;Uknown
2018-01-13 15:15:20,423;192.168.2.18;Down;Uknown
2018-01-13 15:15:21,431;192.168.2.19;Down;Uknown
2018-01-13 15:15:24,452;192.168.2.20;UP;0.260
2018-01-13 15:15:25,461;192.168.2.21;Down;Uknown
2018-01-13 15:15:26,469;192.168.2.22;Down;Uknown
2018-01-13 15:15:27,486;192.168.2.23;Down;Uknown
2018-01-13 15:15:28,497;192.168.2.24;Down;Uknown
2018-01-13 15:15:31,978;192.168.2.25;UP;119.883
2018-01-13 15:15:32,986;192.168.2.26;Down;Uknown
2018-01-13 15:15:34,017;192.168.2.27;Down;Uknown
2018-01-13 15:15:35,024;192.168.2.28;Down;Uknown
2018-01-13 15:15:36,032;192.168.2.29;Down;Uknown
2018-01-13 15:15:37,042;192.168.2.30;Down;Uknown
2018-01-13 15:15:38,050;192.168.2.31;Down;Uknown
2018-01-13 15:15:39,059;192.168.2.32;Down;Uknown
2018-01-13 15:15:40,068;192.168.2.33;Down;Uknown
2018-01-13 15:15:41,077;192.168.2.34;Down;Uknown
2018-01-13 15:15:42,087;192.168.2.35;Down;Uknown
2018-01-13 15:15:43,097;192.168.2.36;Down;Uknown

New output (note the timestamps):

2018-01-13 15:27:43,517;192.168.2.1;UP;1.769
2018-01-13 15:27:43,520;192.168.2.2;UP;1.299
2018-01-13 15:27:43,521;192.168.2.12;UP;0.944
2018-01-13 15:27:43,521;192.168.2.20;UP;0.910
2018-01-13 15:27:43,522;192.168.2.15;UP;0.581
2018-01-13 15:27:43,522;192.168.2.5;UP;2.443
2018-01-13 15:27:43,532;192.168.2.3;UP;10.964
2018-01-13 15:27:43,774;192.168.2.25;UP;1.891
2018-01-13 15:27:44,511;192.168.2.18;Down;Uknown
2018-01-13 15:27:44,512;192.168.2.21;Down;Uknown
2018-01-13 15:27:44,513;192.168.2.16;Down;Uknown
2018-01-13 15:27:44,515;192.168.2.7;Down;Uknown
2018-01-13 15:27:44,509;192.168.2.4;Down;Uknown
2018-01-13 15:27:44,522;192.168.2.27;Down;Uknown
2018-01-13 15:27:44,523;192.168.2.30;Down;Uknown
2018-01-13 15:27:44,525;192.168.2.11;Down;Uknown
2018-01-13 15:27:44,524;192.168.2.22;Down;Uknown
2018-01-13 15:27:44,527;192.168.2.8;Down;Uknown
2018-01-13 15:27:44,519;192.168.2.19;Down;Uknown
2018-01-13 15:27:44,530;192.168.2.28;Down;Uknown
2018-01-13 15:27:44,530;192.168.2.10;Down;Uknown
2018-01-13 15:27:44,531;192.168.2.36;Down;Uknown
2018-01-13 15:27:44,527;192.168.2.14;Down;Uknown
2018-01-13 15:27:44,535;192.168.2.31;Down;Uknown
2018-01-13 15:27:44,527;192.168.2.33;Down;Uknown
2018-01-13 15:27:44,538;192.168.2.6;Down;Uknown
2018-01-13 15:27:44,539;192.168.2.26;Down;Uknown
2018-01-13 15:27:44,535;192.168.2.32;Down;Uknown
2018-01-13 15:27:44,538;192.168.2.29;Down;Uknown
2018-01-13 15:27:44,537;192.168.2.24;Down;Uknown
2018-01-13 15:27:44,538;192.168.2.13;Down;Uknown
2018-01-13 15:27:44,539;192.168.2.9;Down;Uknown
2018-01-13 15:27:44,535;192.168.2.35;Down;Uknown
2018-01-13 15:27:44,541;192.168.2.34;Down;Uknown
2018-01-13 15:27:44,542;192.168.2.17;Down;Uknown
2018-01-13 15:27:44,535;192.168.2.23;Down;Uknown

---

Related Topics