PHP Parallel Curl Requests

PHP - multiple curl requests curl_multi speed optimizations

current implementation of curl_multi_select() in php doesn't block and doesn't respect timeout parameter, maybe it will be fixed later. the proper way of waiting is not implemented in your code, it have to be 2 loops, i will post some tested code from my bot as an example:

      $running  = 1;
      while ($running)
      {
        # execute request
        if ($a = curl_multi_exec($this->murl, $running)) {
          throw BotError::text("curl_multi_exec[$a]: ".curl_multi_strerror($a));
        }
        # check finished
        if (!$running) {
          break;
        }
        # wait for activity
        while (!$a)
        {
          if (($a = curl_multi_select($this->murl, $wait)) < 0)
          {
            throw BotError::text(
              ($a = curl_multi_errno($this->murl))
                ? "curl_multi_select[$a]: ".curl_multi_strerror($a)
                : 'system select failed'
            );
          }
          usleep($wait * 1000000);# wait for some time <1sec
        }
      }

PHP Multiple Curl Requests

• Reuse the same cURL handler ($ch) without running curl_close. This will speed it up just a little bit.
• Use curl_multi_init to run the processes in parallel. This can have a tremendous effect.

PHP Multi-cURL requests delayed until timeout

My current theory is that there is a bug in Facebook fileserver that means the connection is sometimes not being closed even though the data has been sent, so the connection stays open until the timeout. In the absence of the (optional) content-length header being sent by Facebook's fileserver, cURL can't know if the payload is complete, and so hangs.

My current solution is to 'prime' the fileserver by first making a request for the image without a body, like this:

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_NOBODY, 1);
curl_exec($ch);

This is a pretty quick process, since there is no image being returned. I actually do this in the background using asynchronous multi curl, so I can get on with doing some other processing.

After priming the fileserver, subsequent requests for the files are even quicker than before, as the content-length is known.

This is a bit of a clumsy approach, but in the absence of any response from Facebook so far I'm not sure what else to do.

---

Related Topics