Why Does Wget Output to Stderr Rather Than Stdout

Why does wget output to stderr rather than stdout?

It's well known, because it's in the manual.

Reporting messages on stderr is common, because messages are separated from regular output on stdout. This is useful when you combine several tools with a pipe. In this case it would be bad, when regular output and diagnostic messages were mixed up.

Why does wget still print to stderr when there are no error and using -nv?

Use cURL instead:

$ curl -Ss http://www.stackoverflow.com -o /dev/null
(no output)

$ curl -Ss http://www.stackoverflow.invalid -o /dev/null
curl: (6) Couldn't resolve host 'www.stackoverflow.invalid'

If you for whichever reason really need to use wget, you can capture output and only show it on failure:

errors=$(2>&1 wget -nv http://www.stackoverflow.com) || echo "$errors" >&2

Why does wget -O- output to stdout?

By posix standard, if a program takes a file path argument for output, - is used to mean "standard output". The man page describes this under -O.

Why are some programs writing on stderr instead of stdout their output?

When you redirect output from one process to another e.g. via pipes

$ procA | procB | procC

this is traditionally done using stdout. I would expect time and similar commands to output to stderr to avoid corrupting this stream. If you're using time then you're diagnosing issues and you don't want to inadvertently provide extra input to a downstream process.

This article includes some further detail and some history surrounding this.

wget output & error to standard output/standard error

man wget:

-nv

--no-verbose

Turn off verbose without being completely quiet (use -q for that), which means that error messages and basic information still get printed.

For example:

$ wget -nv -O- google.com/doesnotexist
http://google.com/doesnotexist:
2020-09-28 19:57:38 ERROR 404: Not Found.

How do you redirect wget response to standard out?

wget -O - http://whatever.com/page.php > /dev/null

or, if you want to redirect standard error output also:

wget -O - http://whatever.com/page.php > /dev/null 2>&1

or, for codegolf :-)

wget -O-

wget hangs with -r and -O -

Of course:

 wget -r -O file www.blankwebsite.com

works, but the BUG is that:

 wget -r -O - www.blankwebsite.com

hangs!

The same problem is if you create a FIFO

mkfifo /tmp/myfifo
wget -r -O /tmp/myfifo www.blankwebsite.com

wget, when called with -r option, will try to find HTML "a href=..." tags reading the output file. Since the output file is a FIFO or stdout (ex. HYPHEN char '-') it is not able to find any tag and waits for INPUT. Then you will have a wget process waintg forever on a read system call.

To resolve this you can:
1) Patch wget to handle this case
2) Patch wget to not allow "-r -O -" combination... (Just check that the argument of '-O' is a regular file)
3) Use a workaround like:

TMPFILE=$(mktemp /tmp/wget.XXXXXX)
wget -r -O $TMPFILE www.blankwebsite.com
grep STRING $TMPFILE
rm $TMPFILE

---

Related Topics