Using Curl with Commands in Go

using curl with commands in go

In os package you have slice of strings which contains all arguments passed by shell to your program.

os.Args 0th value, i.e., first element is going to be name of the command itself.

If your tool command is mytool, os.Args[0] contains mytool.

Rest are going to be the arguments, which are passed by shell.

package main

import (
    "log"
    "os"
    "os/exec"
)

func main() {

    if len(os.Args) < 2 {
        // If argument is not provided quit
        log.Fatalln("url not provided")
    }
    url := os.Args[1] // URL

    cmd := exec.Command("curl", "-O", url)
    cmd.Run()
}

You can also download multiple URLs concurrently,

var wg *sync.WaitGroup

func main() {
    urls := os.Args[1:]

    wg = new(sync.WaitGroup)
    wg.Add(len(urls))

    for _, url := range urls {
        go download(url)
    }

    wg.Wait()
}

func download(url string) {
    defer wg.Done()
    cmd := exec.Command("curl", "-O", url)
    cmd.Run()
}

Why is the curl command doesn't work in golang

This is happening because quotes are a way for bash to organise arguments, and since you are using Golang's os/exec package, there's no need to escape arguments prior to the command's execution (unless you are using bash in between go and the program you're trying to run):

var body string
var FQDN string
var content string
var url string

content = "Content-Type: application/json"
url = "http://192.168.1.233:5534/current"
DomainName := "test"
IPAddress := "60.60.0.1"

FQDN = DomainName + ".free5gc"
body = `"DNS_ID":"DNS_1","Domain_ID":"Domain_1","Cell_ID":"Cell_1","Device_ID":"IOT1","IMEI":"ims-208930000000003","IPv4":"`+ IPAddress +`","IPv6":":::`+ IPAddress +`","Slice_ID":"121312","FQDN":"`+FQDN  +`"`

fmt.Println(body)

c := exec.Command("curl","-X","POST","-H",content,"-d","{"+ body +"}",url,"-v")

fmt.Println(c)
c.Stdout = os.Stdout
c.Stderr = os.Stderr
err := c.Run()
if err != nil {
    fmt.Println("Error: ", err)
}

tl;dr: drop the quotes, everything should start working :)

Translating CURL to Golang code

Your body is what you think it is, it is just printed in an unexpected format to you.

bytes.NewReader() returns a pointer to the struct type bytes.Reader. When you print it using log.Printf(), it uses the formatting rules defined in the package doc of fmt, that is, pointers to struct are printed as &{} and values of fields enumerated inside of it. This is what you see. bytes.Reader stores the data it provides in a byte slice:

[123 34 114 102 114 73 68 34 58 34 49 49 49 49 49 49 34 125]

And this byte slice is equivalent to the JSON string you expect. To verify:

data := []byte{123, 34, 114, 102, 114, 73, 68, 34, 58, 34, 49, 49, 49, 49, 49, 49, 34, 125}
fmt.Println(string(data))

Output:

{"rfrID":"111111"}

Reading from this reader will provide this exact string, as you can see in this example:

r := bytes.NewReader(data)
fmt.Println(r)
readData, err := ioutil.ReadAll(r)
if err != nil {
    panic(r)
}
fmt.Println(string(readData))

Output:

&{[123 34 114 102 114 73 68 34 58 34 49 49 49 49 49 49 34 125] 0 -1}
{"rfrID":"111111"}

Try these examples on the Go Playground.

Differences

Let's examine the differences between the CURL command and the client you wrote, with this little app which will be the target:

http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
    r.ParseForm()
    log.Println("Method:", r.Method)
    log.Println("Headers:", r.Header)
    log.Println("Form:", r.Form)
    data, err := ioutil.ReadAll(r.Body)
    log.Println("Body:", data, err)
    log.Println("Body string:", string(data))
})
panic(http.ListenAndServe(":8080", nil))

CURL:

2017/10/10 10:17:26 Method: GET
2017/10/10 10:17:26 Headers: map[Content-Length:[19] User-Agent:[curl/7.47.0] Accept:[*/*] Content-Type:[application/json] Web2py-User-Token:[token-string]]
2017/10/10 10:17:26 Form: map[]
2017/10/10 10:17:26 Body: [123 34 114 102 114 73 68 34 58 32 34 49 49 49 49 49 49 34 125] <nil>
2017/10/10 10:17:26 Body string: {"rfrID": "111111"}

Go client:

2017/10/10 10:17:20 Method: GET
2017/10/10 10:17:20 Headers: map[User-Agent:[Go-http-client/1.1] Content-Length:[18] Content-Type:[application/json] Web2py-User-Token:[token-string] Accept-Encoding:[gzip]]
2017/10/10 10:17:20 Form: map[]
2017/10/10 10:17:20 Body: [123 34 114 102 114 73 68 34 58 34 49 49 49 49 49 49 34 125] <nil>
2017/10/10 10:17:20 Body string: {"rfrID":"111111"}

Despite some headers added automatically by the clients, and a space inside the JSON string (which has no effect just formatting), they are identical.

How to pipe curl into Go program?

Regarding OP source code i would consider to change the condition to detect the presence of a pipe.

As already provided in https://stackoverflow.com/a/43947435/4466350 the correct condition does not need to check for the length of the input. Thinking about it, this totally makes sense as you might open stdin without writing data on it.

Besides the proposed solution seems uselessly complex for what it tries to achieve, pretty printing of a json input.

I found out that using the standard library was sufficient to fulfill the goal for the given test case.

About the question ...but how do I optimize my Go program to wait until curl is done?, it seems that OP does not understand the way the file descriptors are working. In fact, the question is not even correct, as the process could theoretically remain alive but actively decided to close Stdin.
The OP is not interested in process liveliness, instead, he should simply look for EOF signal while reading Stdin, indicating that the interesting data was sent correctly.

Anyways, a simple solution look likes this, wrap stdin with a json decoder, loop until eof or an error occur, for each decoded data, encode it to json with a wrapper of stdout, on error break again.

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "log"
    "os"
)

func main() {
    info, err := os.Stdin.Stat()
    if err != nil {
        log.Fatal(err)
    }

    if info.Mode()&os.ModeCharDevice != 0 {
        fmt.Println("The command is intended to work with pipes.")
        fmt.Println("cat file.json | prettyjson")
        return
    }

    dec := json.NewDecoder(os.Stdin)
    enc := json.NewEncoder(os.Stdout)
    enc.SetIndent("", "  ")

    for {
        data := map[string]interface{}{}
        if err := dec.Decode(&data); err != nil {
            if err == io.EOF {
                break
            }
            log.Fatalf("decode error %v", err)
        }
        if err := enc.Encode(data); err != nil {
            log.Fatalf("encod error %v", err)
        }
    }
}

Why is this curl command not working?

exec.Command() doesn't take the whole command line as a single argument. You need to call it as:

exec.Command("curl", "-u", username+":"+password, ...url..., "-d", "status="+status, "-d", "source=API").Run()

How do you know if you get an error? You don't check the return value of Run().

You should actually separate the command creation from running it. This way you can set the process's stdout and stderr to something besides /dev/null, e.g.

c := exec.Command("curl", "-u", username+":"+password, "https://identi.ca/api/statuses/update.xml", "-d", "status="+status, "-d", "source=API")
c.Stdout = os.Stdout
c.Stderr = os.Stderr
err = c.Run()
if err != nil {
    fmt.Println("Error: ", err)
}

Using curl to download golang tarball produces strange result

The issue

https://go.dev/dl/go1.17.7.linux-amd64.tar.gz is performing a redirect to https://dl.google.com/go/go1.17.7.linux-amd64.tar.gz

We can see this in the output you provided above

location: https://dl.google.com/go/go1.17.7.linux-amd64.tar.gz

wget follows redirects by default (up to 20) - see man page

Whereas, curl doesn't follow redirects by default (but has a default max of 50).

Solution

-L will make curl follow redirects

curl  -OL https://go.dev/dl/go1.17.7.linux-amd64.tar.gz

---

Reference

Use -L, --location to follow redirects and use -f, --fail so server data is not written to the file on server error.

Converting curl request to Golang

Note: The credentials provided are valid? Cause i receive a sonorus 401, Authentication Failed.

NOTE: Using the -d in cURL, you are going to send a POST request instead of a GET. Due to this behaviour, your proably want to send a POST request instead of a GET

You can use my little http library: https://github.com/alessiosavi/Requests/

package main

import requests "github.com/alessiosavi/Requests"

func main() {
    req, err := requests.InitRequest("https://postman-echo.com/basic-auth", "GET", []byte{}, false, false)
    if err != nil {
        fmt.Println("ERROR! ", err)
    }
    req.CreateHeaderList("Accept", "application/json", "Accept-Language", "en_US", "Authorization", "postman:password")
    client := &http.Client{}
    resp := req.ExecuteRequest(client)
    fmt.Println(resp.Dump())
}

You can change the data (URL, post data, headers) with the one that you need for authenticate to the service.

In your case, will be something like this:

package main

import requests "github.com/alessiosavi/Requests"

const (
    ID= "Aeit5RskDRN8eUUMB0Ud3RjA_z6feWMUHktwlJZMeQMo9A9ulbKK"
    SECRET= "EAAqyzrOTUWf-OFJCB4BxgXT4xuravL7pnkC8Tn20HYtZExd1mFO"
)

func main() {
    req, err := requests.InitRequest("https://api.sandbox.paypal.com/v1/oauth2/token", "GET", []byte{"grant_type=client_credentials"}, false, false)
    if err != nil {
        fmt.Println("ERROR! ", err)
    }
    req.CreateHeaderList("Accept", "application/json", "Accept-Language", "en_US", "Authorization", ID+":"+SECRET)
    client := &http.Client{}
    resp := req.ExecuteRequest(client)
    fmt.Println(resp.Dump())
}

---

Related Topics