PHP Curl Localhost Is Slow When Making Concurrent Requests

php curl localhost is slow when making concurrent requests

Ok, after so many days of trying to solve this issue, I finally find out why. And It's not name resolving. I can't believe that it takes so many days to track down the root cause which is the number of pm.start_servers in php-fpm's www.conf. Initially, I set the number of pm.start_servers to 3 which is why the ab test to localhost always gets worse after concurrency level 3. While php-cli has no issue of limited number of php process, thus, php-cli always performs great. After increaing pm.start_servers to 5, the ab test result is as fast as php-cli. If this is the reason why your php-fpm is slow, you should also think about changing the number of pm.min_spare_servers, pm.max_spare_servers, pm.max_children and anything related.

CURL with PHP - Very slow

it looks like most of the time is waiting for the server to respond (starttransfer_time - pretransfer_time = 1.620026)... maybe the server is doing some database or other operation that takes time?

Very long loadtime php curl

your loop is not optimized in the slightest, i believe if you optimized your loop code, your code could run A LOT faster. you create and delete the curl handle on each iteration, when you could just keep re-using the same curl handle on each player (this would use less cpu and be faster), you don't use compressed transfer (enabling compression would probably make the transfer faster), and most importantly, you run the api calls sequentially, i believe if you did the api requests in parallel, it would load much faster. also, you don't urlencode psnid, that's probably a bug. try this

<?php
$cmh = curl_multi_init ();
$curls = array ();
$data = $connection->query ( "SELECT * FROM bfplayers" );

while ( ($row = mysqli_fetch_assoc ( $data )) ) {
    $psnid = $row ['psnid'];
    $tmp = array ();
    $tmp [0] = ($ch = curl_init ());
    $tmp [1] = tmpfile ();
    $curls [] = $tmp;
    curl_setopt_array ( $ch, array (
            CURLOPT_URL => "https://battlefieldtracker.com/bf1/api/Stats/BasicStats?platform=2&displayName=" . urlencode ( $psnid ),
            CURLOPT_ENCODING => '',
            CURLOPT_SSL_VERIFYPEER => false,
            CURLOPT_SSL_VERIFYHOST => false,
            CURLOPT_HTTPHEADER => array (
                    'TRN-Api-Key: MYKEY' 
            ),
            CURLOPT_FILE => $tmp [1] 
    ) );
    curl_multi_add_handle ( $cmh, $ch );
    curl_multi_exec ( $cmh, $active );
}
do {
    do {
        $ret = curl_multi_exec ( $cmh, $active );
    } while ( $ret == CURLM_CALL_MULTI_PERFORM );
    curl_multi_select ( $cmh, 1 );
} while ( $active );
foreach ( $curls as $curr ) {
    fseek ( $curr [1], 0, SEEK_SET ); // https://bugs.php.net/bug.php?id=76268
    $response = stream_get_contents ( $curr [1] );
    $result = json_decode ( $response,true );
    print ($result ['profile'] ['displayName']) ;
}
// the rest is just cleanup, the client shouldn't have to wait for this
// OPTIMIZEME: apache version of fastcgi_finish_request() ?
if (is_callable ( 'fastcgi_finish_request' )) {
    fastcgi_finish_request ();
}
foreach ( $curls as $curr ) {
    curl_multi_remove_handle ( $cmh, $curr [0] );
    curl_close ( $curr [0] );
    fclose ( $curr [1] );
}
curl_multi_close ( $cmh );

• it runs all api calls in parallel, and use transfer compression (CURLOPT_ENCODING), and runs api requests in parallel with downloading results from the db, and it tries to disconnect the client before running cleanup routines, it will probably run much faster.

also, if mysqli_fetch_assoc() are causing slow roundtrips to your db, it would probably be even faster to replace it with mysqli_fetch_all()

also, something that would probably be much faster than this, would be to have a cronjob run every minute (or every 10 seconds?) that caches the results, and show a cached result to the client. (even if the api calls lags, the client pageload wouldn't be affected at all.)

Why could Curl be slower than a web browser?

Check your web server logs and try to find any difference between the requests from the normal web browser and the requests from curl

How to speed up cURL in php?

With respect to environment, I've observed in PHP that cURL typically runs very fast in most environments except in places where there is low CPU and there is slower network performance. For example, on localhost on my MAMP installation, curl is fast, on a larger amazon instance, curl is fast. But on a small crappy hosting, i've seen it have performance issues where it is noticeably slower to connect. Though, i'm not sure exactly why that is slower. Also, it sure wasn't 5 seconds slower.

to help determine if its PHP or your environment, you should try interacting with curl via the command line. At least that you'll be able to rule out PHP code being the problem if its still 5 seconds.

---

Related Topics