Bash Sorting Redirection

Bash Sorting Redirection

The result is the same but in the case of -o file2 the resulting file is created by sort directly while in the other case, it is created by bash and filled with the standard output of sort. The xfopen defined in line 450 of sort.c in coreutils treats both cases (stdout and -o filename) equally.

Redirecting the standard output of sort is more generic as it could be redirected to another program with a | in place of a >, which the -o option makes more difficult to do (but not impossible)

The -o option is handy for in place sorting as the redirection to the same file will lead to a truncated file because it is created (and truncated) by the shell prior to the invocation of sort.

bash: sort applied to a file returns right results as terminal output, but does change the file itself

SOLVED

From this thread it turns out that redirecting the output of sort into the same file from which sort reads as source will not work since

the shell is makes the redirections (not the sort(1) program) and the
input file (as being the output also) will be erased just before
giving the sort(1) program the opportunity of reading it.

So I have split my command into two

sort -k1 -n source-g5.txt > tmp-source-g5.txt
mv tmp-source-g5.txt > source-g5.txt

How to sort a file in-place?

You can use the -o, --output=FILE option of sort to indicate the same input and output file:

sort -o file file

Without repeating the filename (with bash brace expansion)

sort -o file{,}

⚠️ Important note: a common mistake is to try to redirect the output to the same input file
(e.g. sort file > file). This does not work as the shell is making the redirections (not the sort(1) program) and the input file (as being the output also) will be erased just before giving the sort(1) program the opportunity of reading it.

Do I need to generate a second file to sort a file?

Try:

sort -o file.txt file.txt

See http://ss64.com/bash/sort.html

`-o OUTPUT-FILE'
     Write output to OUTPUT-FILE instead of standard output.  If
     OUTPUT-FILE is one of the input files, `sort' copies it to a
     temporary file before sorting and writing the output to
     OUTPUT-FILE.

Order of events in Bash shell redirection of standard output and error to the same file

Linux has three default data streams - stdin (0), stdout (1) and stderr (2).

• stdin, or "standard in", is used for reading data from the user.
• stdout, or "standard out", is used for "usual" output.
• stderr, or "standard error" is used for errors.

The character > allows you to redirect your command's output into one of the streams mentioned above, or in to a file.

When you run the command ls -l /bin/usr > ls-output.txt 2>&1, you actually ask from your system two things:

1. Redirect the stdout of the command into the file ls-output.txt.
2. Redirect the stderr of the command into the data stream whose number is 1, which is the stdout stream.

Actually, using > by itself is equivalent to 1>, meaning that you redirect all output from data stream no. 1.

The & char tells your system that you are going to redirect the output into a data stream, instead of a file. Omitting this character will result in your stderr written to a file named 1.

It is important to note that the character & has more roles in Linux, like running commands in the background or concatenating commands (using &&), but it is completely different.

Best regards.

Why does in-place sort of same file fail?

The shell sets up the redirection before the command runs and when the shell opens the file for writing it immediately truncates the contents of the file. sort never has a chance to see them.

From the bash man page:

Redirection

Before a command is executed, its input and output may be redirected
using a special notation interpreted by the shell. Redirection may
also be used to open and close files for the current shell execution
environment. The following redirection operators may precede or appear
anywhere within a simple command or may follow a command. Redirections
are processed in the order they appear, from left to right.

Shell redirection i/o order

This error:

ls: *.xyz: No such file or directory

is being written on stderr by ls binary.

However in this command:

ls -al *.xyz 2>&1 1> files.lst

You're first redirecting stderr to stdout which by default goes to tty (terminal)

And then you're redirecting stdout to a file files.lst, however remember that stderr doesn't redirected to file since you have stderr to stdout redirection before stdout to file redirection. Your stderr still gets written to tty in this case.

However in 2nd case you change the order of redirections (first stdout to file and then stderr to stdout) and that rightly redirects stderr to a file which is also being used by stdout.

bash redirection to files not working

The > yolo.txt shell redirect happens before any of the commands run. In particular, the shell opens yolo.txt for writing and truncates it before executing cat yolo.txt bar.txt. So by the time cat opens yolo.txt, yolo.txt is empty. Therefore the c line in bar.txt is unique, so uniq -u passes it through.

I guess you wanted to use sponge to avoid this problem, since that's what sponge is for. But you used it incorrectly. This is the correct usage:

cat yolo.txt bar.txt | sort | uniq -u | sponge yolo.txt && cat yolo.txt

Note that I just pass the output filename to sponge as a command-line argument, instead of using a shell redirect.

---

Related Topics