How to Get Curl to Use Https

How do i get cURL to use https

Your curl version wasn't compiled with SSL support. It's actually a flag that is passed during the config phase:

./configure --with-ssl

The quickest and most complete way is to download the curl sources and compile it yourself with the --with-ssl flag. This will also ensure that your curl and SSL libraries won't be vulnerable to any of the nasty, known vulnerabilities as well. There are workarounds for using insecure versions of SSL but of course that's not recommended.

curl: how to specify target hostname for https request

Indeed SNI in TLS does not work like that. SNI, as everything related to TLS, happens before any kind of HTTP traffic, hence the Host header is not taken into account at that step (but will be useful later on for the webserver to know which host you are connecting too).

So to enable SNI you need a specific switch in your HTTP client to tell it to send the appropriate TLS extension during the handshake with the hostname value you need.

In case of curl, you need at least version 7.18.1 (based on https://curl.haxx.se/changes.html) and then it seems to automatically use the value provided in the Host header. It alo depends on which OpenSSL (or equivalent library on your platform) version it is linked to.

See point 1.10 of https://curl.haxx.se/docs/knownbugs.html that speaks about a bug but explains what happens:

When given a URL with a trailing dot for the host name part: "https://example.com./", libcurl will strip off the dot and use the name without a dot internally and send it dot-less in HTTP Host: headers and in the TLS SNI field.

The --connect-to option could also be useful in your case. Or --resolve as a substitute to /etc/hosts, see https://curl.haxx.se/mail/archive-2015-01/0042.html for am example, or https://makandracards.com/makandra/1613-make-an-http-request-to-a-machine-but-fake-the-hostname
You can add --verbose in all cases to see in more details what is happening. See this example: https://www.claudiokuenzler.com/blog/693/curious-case-of-curl-ssl-tls-sni-http-host-header ; you will also see there how to test directly with openssl.

If you have a.example in your /etc/hosts you should just run curl with https://a.example/ and it should take care of the Host header and hence SNI (or use --resolve instead)

So to answer your question directly, replace

curl --header 'Host: a.example' https://x.example

with

curl --connect-to a.example:443:x.example:443 https://a.example

and it should work perfectly.

PHP CURL & HTTPS

Quick fix, add this in your options:

curl_setopt($ch,CURLOPT_SSL_VERIFYPEER, false)

Now you have no idea what host you're actually connecting to, because cURL will not verify the certificate in any way. Hope you enjoy man-in-the-middle attacks!

Or just add it to your current function:

/**
 * Get a web file (HTML, XHTML, XML, image, etc.) from a URL.  Return an
 * array containing the HTTP server response header fields and content.
 */
function get_web_page( $url )
{
    $options = array(
        CURLOPT_RETURNTRANSFER => true,     // return web page
        CURLOPT_HEADER         => false,    // don't return headers
        CURLOPT_FOLLOWLOCATION => true,     // follow redirects
        CURLOPT_ENCODING       => "",       // handle all encodings
        CURLOPT_USERAGENT      => "spider", // who am i
        CURLOPT_AUTOREFERER    => true,     // set referer on redirect
        CURLOPT_CONNECTTIMEOUT => 120,      // timeout on connect
        CURLOPT_TIMEOUT        => 120,      // timeout on response
        CURLOPT_MAXREDIRS      => 10,       // stop after 10 redirects
        CURLOPT_SSL_VERIFYPEER => false     // Disabled SSL Cert checks
    );

    $ch      = curl_init( $url );
    curl_setopt_array( $ch, $options );
    $content = curl_exec( $ch );
    $err     = curl_errno( $ch );
    $errmsg  = curl_error( $ch );
    $header  = curl_getinfo( $ch );
    curl_close( $ch );

    $header['errno']   = $err;
    $header['errmsg']  = $errmsg;
    $header['content'] = $content;
    return $header;
}

How to make an HTTPS request using cURL?

The cacert.pem that you're passing to cURL via the CURLOPT_CAINFO is used to verify certificate authorities, but development servers typically have self signed certificates which are not included in that bundle.

The first step is to generate your own self signed certificate. This article describes the process step-by-step. Make sure that during the CSR generation you're using the intended server name under the Common Name (CN), e.g. ipv4.fiddler.

Once you have configured your web server using the self signed certificate (e.g. server.crt) and key (e.g. server.key), you need to copy the former to a location that your script can access it.

The following bare essentials can be used to verify the whole thing:

$ch = curl_init('https://ipv4.fidler');
curl_setopt_array($ch, array(
    CURLOPT_SSL_VERIFYPEER => true,
    CURLOPT_SSL_VERIFYHOST => 2,
    CURLOPT_VERBOSE => true,
    CURLOPT_CAINFO => '/path/to/server.crt',
));

if (false === curl_exec($ch)) {
    echo "Error while loading page: ", curl_error($ch), "\n";
}

How to get response from https url using curl and shell script

It works exactly the same way for HTTPS.

The only little difference is that HTTPS implies that curl will verify the TLS certificate of the server and unless you use --insecure curl will exit with an error if it can't verify it successfully. In your example here, you use an IP address and not a name and that will most certainly cause a certificate check to fail.

Force cURL to use GET on proxy for HTTPs requests

I don't believe this can be done using some common configuration setting on the client side as it would deny the whole purpose of HTTPS (that no one can eavesdrop your HTTPS traffic).

Your only option is thus to configure your proxy to basically create a man-in-the-middle attack to decrypt the HTTPS traffic going through it. Squid proxy should support this using their "SSL bump" feature. There is a nice intro for it on this wiki and more setup docs here.

What squid does in this regime is that it gets the address of the remote server from the client's CONNECT request and instead of creating just a blind tunnel to the server, it starts a new direct HTTPS request to the server by itself and saves the reply. Thus Squid has access to all the traffic and can cache it or do anything else Squid can do with it.

When sending replies back to the client, it itself needs to present a HTTPS certificate (the client expects HTTPS traffic), so in Squid there is a feature to automatically generate certificates for all proxied domains. To configure it you will essentially have to create a local Certificate Authority. Note that these auto-generated certificates will be simple self-signed certificates so, on the client side, this will look as untrusted certificates and you'll need to switch off peer verification (CURLOPT_SSL_VERIFYPEER = false).

I could not find any similar feature in Apache traffic server. They seem to support only SSL termination in the reverse proxy mode.

Last note: please bear in mind that this is still rather a hack and decrypting HTTPS may bring legal or ethical problems. Never do this without the clients consent!

Can curl default to using https?

HTTPS protocol for URL with missing scheme part (and thus also bypass protocol guessing mentioned in (obsolete) answer by @FatalError) can be set with option

--proto-default https

since version 7.45.0 from October 2015. See also https://github.com/curl/curl/pull/351.

It can be put into ~/.curlrc.

Example:

$ curl -v example.org
*   Trying XXXXIPv6redacted:80...
* Connected to example.org (XXXXIPv6redacted) port 80 (#0)
> GET / HTTP/1.1
...

$ curl --proto-default https -v example.org                                                             
*   Trying XXXXIPv6redacted:443...
* Connected to example.org (XXXXIPv6redacted) port 443 (#0)
* ALPN: offers h2
...

(Note that it's not a magic option to assure security. It e.g. won't affect http proxy, if set, according to the manual.)

---

Related Topics