Understanding PHP Curl_Multi_Exec

understanding php curl_multi_exec

You can explore two article that describes this example.

PHP and curl_multi_exec

First, here's the high level. There are two outer loops. The first one is responsible for clearing out the curl buffer right now. The second one is responsible for waiting for more information, and then getting that information. This is an example of what is called blocking I/O. We block execution of the rest of the program until the network I/O is done. While this isn't the most preferable way in general to handle network I/O, it's really our only choice in single-threaded, synchronous PHP.

Doing curl_multi_exec the right way

First the $mrc variable and from the manual we learn that the response is a cURL code defined in the cURL Predefined Constants. In essence it is a regular response and as with any other PHP function curl_multi_exec is no different and only returns a response once it is finished. Which means there should be only ONE response. In a perfect world this single response is 0 (zero) or equal to the predefined constant CURLM_OK.

PHP curl_multi_exec output to array

Solved it by changing the ${'ch_'.$i} to an array $ch[$i] then looping through the array using curl_multi_getcontent as follows:

foreach ($ch as $a)
{
    $result = curl_multi_getcontent($a);
    parse_str($result, $arr);
    print_r($arr);
    echo "--------------------------------------------------\n";
}

This then lets me the data in the array $arr. For example, $arr['EMAIL'];

See as well: Curl Multi and Return Transfer and understanding php curl_multi_exec.

Make curl_multi 'sleep' or wait before sending next request

function asyncCurl($url){
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, $url);
  curl_setopt($ch, CURLOPT_FRESH_CONNECT, true);
  curl_setopt($ch, CURLOPT_TIMEOUT, 1);
  curl_exec($ch);
  curl_close($ch);
}

$timeout = 3; // in seconds
$urls = array(...);

foreach($urls as $url){
  asyncCurl($url);
  sleep($timeout);
}

If you need to get the response, it can still be done by creating a "background process" type of thing on your server. This will require 2 scripts instead of one.

background.php

function curl($url){
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, $url);
  $a = curl_exec($ch);
  curl_close($ch);
  return $a;
}

$response = curl($_GET['url']);

// code here to handle the response

doRequest.php (or whatever, this is the one you will call in your browser)

function asyncCurl($url){
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, "mydomain.com/background.php?url=".urlencode($url));
  curl_setopt($ch, CURLOPT_FRESH_CONNECT, true);
  curl_setopt($ch, CURLOPT_TIMEOUT, 1);
  curl_exec($ch);
  curl_close($ch);
}

$timeout = 3; // in seconds
$urls = array(...);

foreach($urls as $url){
  asyncCurl($url);
  sleep($timeout);
}

The idea here is that PHP is single threaded, but there is no reason you can't have more than one PHP process running at the same time. The only downside is that you have to make the requests on one script and handle the response on another.

Option 3: display the output as soon as it becomes available.

This method is exactly the same as the one above, except that it uses javascript to create a new php process. You didn't have javascript tagged, but this is the only way to accomplish both

asynchronous requests w/ a timeout

and

display the response as soon as it's available

doRequest.php

<?php

$urls = array(); // fill with your urls
$timeout = 3; // in seconds

if (isset($_GET['url'])) {
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $_GET['url']);
    $a = curl_exec($ch);
    curl_close($ch);
    echo $a;
    exit;
}

?><html>
<body>
    <div id='results'></div>
    <script>

        var urls = <?php echo json_encode($urls); ?>;
        var currentIndex = 0;

        function doRequest(url) {
            var xhttp = new XMLHttpRequest();
            xhttp.onreadystatechange = function () {
                if (xhttp.readyState == 4 && xhttp.status == 200) {
                    document.getElementById("results").insertAdjacentHTML("beforeend", "<hr>" + xhttp.responseText);
                }
            };
            xhttp.open("GET", "doRequest.php?url=" + encodeURIComponent(url), true);
            xhttp.send();
        }

        var index=0;
        function startLoop(){
            var url = urls[index];
            doRequest(url);
            setTimeout(function(){
                index++;
                if('undefined' != urls[index]) startLoop();
            }, <?php echo $timeout*1000; ?>);
        }

        startLoop();
    </script>
</body>

What's happening is you server is creating a new request for each url and then using normal curl to get the response, but instead of using curl to create the new process, we use ajax which is async by nature and has the ability to create multiple PHP processes and wait for the response.

Godspeed!

How many times does curl_multi_exec have to be called?

curl_multi_exec() reads and writes data to the socket. It stops when it cannot write or read.
So the number of needed calls depends on the amount of data to be transfered and the speed of the network. curl_multi_select() waits till a socket becomes readable or writable.

You should use the curl_multi_select() and have all hosts inside one multi-handle.
Check curl_getinfo() or curl_multi_info_read() if you need to know the duration of the individual requests.

PHP curl_multi_exec runs once

You're not setting up the curl options correctly:

Currently, you're setting options on $ch which is your array, you need to be setting the options specifically on the current curl handler, which in your loop is $ch[$id]:

   //Initialize and set options 
   curl_setopt($ch[$id], CURLOPT_URL,                $url);
   curl_setopt($ch[$id], CURLOPT_HEADER,             0);
   curl_setopt($ch[$id], CURLOPT_RETURNTRANSFER,     1);
   curl_setopt($ch[$id], CURLOPT_POST,               1);
   curl_setopt($ch[$id], CURLOPT_POSTFIELDS,        $request);

Understanding PHP Curl_Multi_Exec