Unable to Pass Wget a Variable with Quotes Inside the Variable

unable to pass wget a variable with quotes inside the variable

Quotes inside of quoted strings or variables are ordinary characters, not quoting characters. There's no way to change that. Use an array instead:

A=(a b 'c d' 'e f')
cmd "${A[@]}"

calls cmd with four arguments a, b, c d, and e f.

(You could achieve a similar effect with eval, but that's a lot more error prone. In your case, using arrays is much more convenient.)

Bash script execute wget with a variable inside it

Four things:

1. Add quotes around your URL: http:://www.myurl.com ==> "http:://www.myurl.com"
2. Remove the double colon: "http:://www.myurl.com" ==> "http://www.myurl.com"
3. Get rid of the extra flags and hyphen on the wget command: "wget -O - -q "$URL/something/something2"" ==> wget "$URL/something/something2"
4. Add curly braces around your variable: "wget "$URL/something/something2"" ==> "wget "${URL}/something/something2""

This works:

#!/bin/bash

URL="http://www.google.com"

echo $(date) 'Running wget...'
wget "${URL}"

Using wget in shell trouble with variable that has \

The following implementation appears to work 100% of the time -- I'm unable to reproduce the claimed sporadic failures:

#!/usr/bin/env bash

set -o pipefail

symbol=$1
today=$(date +%Y%m%d)
tomorrow=$(date --date='1 days' +%Y%m%d)
first_date=$(date -d "$2" '+%s')
last_date=$(date -d "$today" '+%s')

# store complete webpage text in a variable
page_text=$(curl --fail --cookie-jar cookies \
  "https://finance.yahoo.com/quote/$symbol/?p=$symbol") || exit

# extract the JSON used by JavaScript in the page
app_json=$(grep -e 'root.App.main = ' <<<"$page_text" \
           | sed -e 's@^root.App.main = @@' \
                 -e 's@[;]$@@') || exit

# use jq to extract the crumb from that JSON
crumb=$(jq -r \
          '.context.dispatcher.stores.CrumbStore.crumb' \
          <<<"$app_json" | tr -d '\r') || exit

# Perform our actual download
fileloc="https://query1.finance.yahoo.com/v7/finance/download/$symbol?period1=$first_date&period2=$last_date&interval=1d&events=history&crumb=$crumb"
curl --fail --cookie cookies "$fileloc" >"hs$symbol.csv"

Note that the tr -d '\r' is only necessary when using a native-Windows jq mixed with an otherwise native-Cygwin set of tools.

bash quotes in variable treated different when expanded to command

The important thing to keep in mind is that quotes are only removed once, when the command line is originally parsed. A quote which is inserted into the command line as a result of parameter substitution ($foo) or command substitution ($(cmd args)) is not treated as a special character. [Note 1]

That seems different from whitespace and glob metacharacters. Word splitting and pathname expansion happen after parameter/command substitution (unless the substitution occurs inside quotes). [Note 2]

The consequence is that it is almost impossible to create a bash variable $args such that

cmd $args

If $args contains quotes, they are not removed. Words inside $args are delimited by sequences of whitespace, not single whitespace characters.

The only way to do it is to set $IFS to include some non-whitespace character; that character can then be used inside $args as a single-character delimiter. However, there is no way to quote a character inside a value, so once you do that, the character you chose cannot be used other than as a delimiter. This is not usually very satisfactory.

There is a solution, though: bash arrays.

If you make $args into an array variable, then you can expand it with the repeated-quote syntax:

cmd "${args[@]}"

which produces exactly one word per element of $args, and suppresses word-splitting and pathname expansion on those words, so they end up as literals.

So, for example:

actions=(--tags all:)
actions+=(--chapters '')
mkvpropedit "$1" "${actions[@]}"

will probably do what you want. So would:

args=("$1")
args+=(--tags)
args+=(all:)
args+=(--chapters)
args+=('')
mkvpropedit "${args[@]}"

and so would

command=(mkvpropedit "$1" --tags all: --chapters '')
"${command[@]}"

I hope that's semi-clear.

man bash (or the online version) contains a blow-by-blow account of how bash assembles commands, starting at the section "EXPANSION". It's worth reading for a full explanation.

---

Notes:

1. This doesn't apply to eval or commands like bash -c which evaluate their argument again after command line processing. But that's because command-line processing happens twice.

2. Word splitting is not the same as "dividing the command into words", which happens when the command is parsed. For one thing, word-splitting uses as separator characters the value of $IFS, whereas command-line parsing uses whitespace. But neither of these are done inside quotes, so they are similar in that respect. In any case, words are split in one way or another both before and after parameter substitution.

Use bash string to provide multiple arguments to a program, where some argument include whitespace

Do not use a regular variable; use an array.

headers=(--header "Accept-Encoding: gzip, deflate")
wget "${headers[@]}" "$URL"

When should I wrap quotes around a shell variable?

General rule: quote it if it can either be empty or contain spaces (or any whitespace really) or special characters (wildcards). Not quoting strings with spaces often leads to the shell breaking apart a single argument into many.

$? doesn't need quotes since it's a numeric value. Whether $URL needs it depends on what you allow in there and whether you still want an argument if it's empty.

I tend to always quote strings just out of habit since it's safer that way.

Problem with run bash script using cat output ( extra quotes )

For this particular case one idea would be to feed each line to xargs.

For sample data I doubled OP's $THREAD_FILE:

$ cat tfile
--user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:90.0) Gecko/20100101 Firefox/90.0" -np -r -l 1 -A "jpg" --ignore-case -P /var/www/optimazer/public/optimazed -x http://example.com
--user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:90.0) Gecko/20100101 Firefox/90.0" -np -r -l 1 -A "jpg" --ignore-case -P /var/www/optimazer/public/optimazed -x http://example.com

A first pass at xargs:

cat tfile | xargs -r wget

Or we can eliminate the unnecessary cat by feeding the file directly to xargs:

xargs -r -a tfile wget

A few variations on KamilCuk's comment/suggestion:

xargs -r < tfile wget
xargs -r wget < tfile
< tfile xargs -r wget

If we're dealing wih a variable (as with OPs example):

thread=$(head -1 tfile)
xargs -r wget <<< "${thread}"

And expanding on the <<< "${thread}" example ... using this in a loop (eg, need to perform additional processing for each line from a multi-line input file):

while read -r thread
do
    xargs -r wget <<< "${thread}"
done < tfile

All of these generate the following for each line processed:

--2021-07-31 13:50:41--  http://example.com/
Resolving example.com (example.com)... 93.184.216.34, 2606:2800:220:1:248:1893:25c8:1946
Connecting to example.com (example.com)|93.184.216.34|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1256 (1.2K) [text/html]
Saving to: ‘/var/www/optimazer/public/optimazed/example.com/index.html.tmp’

example.com/index.html.tmp               100%[================================================================================>]   1.23K  --.-KB/s    in 0.001s

2021-07-31 13:50:41 (1.25 MB/s) - ‘/var/www/optimazer/public/optimazed/example.com/index.html.tmp’ saved [1256/1256]

Removing /var/www/optimazer/public/optimazed/example.com/index.html.tmp since it should be rejected.

Bash variable not expanding

The value you assign to the COOKIES variable is without the quotes. You must either include the quotes within the variable, or quote the contents afterwards. The easiest solution seems to be:

COOKIES="'cookie:_ga=GA1.2.3865356.1523153047; ....'"
wget --header $COOKIES www.mysite.com

---

Related Topics