Make Curl Output Stderr to File (Or String)

Make cURL output STDERR to file (or string)

You are making couple mistakes in your example:

1) you have to call curl_exec() prior to reading from the "verbose log", because curl_setopt() doesn't perform any action, so nothing can be logged prior to the curl_exec().

2) you are opening $curl_log = fopen("curl.txt", 'w'); only for write, so nothing could be read, even after you write to the file and rewind the internal file pointer.

So the correct shortened code should look like:

<?php
$curl = curl_init();
$curl_log = fopen("curl.txt", 'rw'); // open file for READ and write
$url = "http://www.google.com";

curl_setopt_array($curl, array(
    CURLOPT_URL             => $url,
    CURLOPT_VERBOSE         => 1,
    CURLOPT_STDERR          => $curl_log,
    CURLOPT_RETURNTRANSFER  => 1
));

$response = curl_exec($curl);

rewind($curl_log);
$output= fread($curl_log, 2048);
echo "<pre>". print_r($output, 1). "</pre>";
fclose($curl_log);

// ...

?>

NOTE: verbose log could be longer than 2048 bytes, so you could "fclose" the $curl_log after curl_exec() and then read the whole file with for example file_get_contents().
In that case, the point 2) should not be considered as mistake :-)

How do I pipe or redirect the output of curl -v?

add the -s (silent) option to remove the progress meter, then redirect stderr to stdout to get verbose output on the same fd as the response body

curl -vs google.com 2>&1 | less

How to write output of looped curl command to file

You can do this:

while :; do
    curl --location --request POST 'https://api.website.com/Auth/token' \
        --header 'Content-Type: application/x-www-form-urlencoded' \
        --header 'Cookie: blablabla' \
        --data-urlencode 'grant_type=password' \
        --data-urlencode 'username=username' \
        --data-urlencode 'password=password'
    sleep 2 || break
done >> file.txt

cURL including garbage when redirecting stderr to stdout

Put quotes around your $curl_output variable, because it contains * which get interpreted.

% echo "$curl_output"
* Rebuilt URL to: https://www.google.com/
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 2a00:1450:4009:806::2004...
* Connected to www.google.com (2a00:1450:4009:806::2004) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

Whereas:

echo $curl_output

would resolve the * into whatever files names are lying in your current directory.

cURL output to file

This output is sent to stderr. So, to get it all you need is redirect stream 2 (stderr) to a file as

curl -o responseFile.html http://www.somewhere.com 2> informationFile.txt

But, as your capture shows, times are not always included.

The better option to know if the request succeded and how long it took is to ask curl to output some internal variables. This is done with the -w switch. So, your command should look as

curl -o responseFile.html http://www.somewhere.com -w "%{response_code};%{time_total}" > dataFile.txt 2> informationFile.txt

That way, the response will go to responseFile.html (or whatever you need), the progress information (stderr or stream 2) will go to informationFile.txt and the required request response code and time information will go to dataFile.txt

Making curl send errors to stderr and everything else to stdout

After some more experimentation I have come up with the following workaround, but I'm still open to better alternatives.

It works by temporarily storing all output (stdout and stderr) in a temporary file, and then sending the contents of that file to stderr or stdout depending on curl's exit code. If curl failed the entire output will go to stderr (and be e-mailed to me thanks to cron), but if curl succeeded the output will go to stdout instead (which is redirected to a log file in the cron command, resulting in no e-mail.)

# Get a temporary filename
CURL_LOG=`tempfile`

(
  # Run curl, and stick all output in the temp file
  /usr/bin/curl --verbose ... > "$CURL_LOG" 2>&1
) || (
  # If curl exited with a non-zero error code, send its output to stderr so that
  # cron will e-mail it.
  cat "$CURL_LOG" > /dev/stderr
  rm "$CURL_LOG"
  exit 1
)

# Otherwise curl completed successfully, so send the output to stdout (which
# is redirected to a log file in crontab)
cat "$CURL_LOG"
rm "$CURL_LOG"

How to capture stderr in a variable and pipe stdout through

As anubhava correctly diagnosed, the problem is that you're setting HEADERS in a sub-process, not in the main process of your shell.

You can use Bash's process substitution to avoid the problem, without using temporary files which have to be cleaned up in the event of the shell being interrupted:

HEADERS=""
{ HEADERS=$(curl -vs "$URL" 2>&1 1>&3-); } 3> >(grep "regex" > file)
echo "$HEADERS"

The grep sub-process is hidden by the process substitution. The space between 3> and >(…) is necessary.

putting CURL result in a string and not STDOUT?

int http_get_response(void *buffer, size_t size, size_t rxed, char **msg_in)
{
    char *c;

    if (asprintf(&c, "%s%.*s", *msg_in, size * rxed, buffer) == -1) {
        free(*msg_in);
        msg_in = NULL;
        return -1;
    }

    free(*msg_in);
    *msg_in = c;

    return size * rxed;
}

and add the following curl option in your main

char *msg_in = calloc(1,sizeof(char));

curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, http_get_response);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &msg_in);

Then you will get the message in the msg_in

EDIT

do not forget to free msg_in when it become uselless in your program

free(msg_in); msg_in = NULL;

Assign output to variable in Bash

In shell, you don't put a $ in front of a variable you're assigning. You only use $IP when you're referring to the variable.

#!/bin/bash

IP=$(curl automation.whatismyip.com/n09230945.asp)

echo "$IP"

sed "s/IP/$IP/" nsupdate.txt | nsupdate

---

Related Topics