Tail Inverse/Printing Everything Except the Last N Lines

What's the opposite of head? I want all but the first N lines of a file

tail --help gives the following:

  -n, --lines=K            output the last K lines, instead of the last 10;
                           or use -n +K to output lines starting with the Kth

So to filter out the first 2 lines, -n +3 should give you the output you are looking for (start from 3rd).

Get last n lines of a file, similar to tail

The code I ended up using. I think this is the best so far:

def tail(f, n, offset=None):
    """Reads a n lines from f with an offset of offset lines.  The return
    value is a tuple in the form ``(lines, has_more)`` where `has_more` is
    an indicator that is `True` if there are more lines in the file.
    """
    avg_line_length = 74
    to_read = n + (offset or 0)

    while 1:
        try:
            f.seek(-(avg_line_length * to_read), 2)
        except IOError:
            # woops.  apparently file is smaller than what we want
            # to step back, go to the beginning instead
            f.seek(0)
        pos = f.tell()
        lines = f.read().splitlines()
        if len(lines) >= to_read or pos == 0:
            return lines[-to_read:offset and -offset or None], \
                   len(lines) > to_read or pos > 0
        avg_line_length *= 1.3

How can I read first n and last n lines from a file?

Chances are you're going to want something like:

... | awk -v OFS='\n' '{a[NR]=$0} END{print a[1], a[2], a[NR-1], a[NR]}'

or if you need to specify a number and taking into account @Wintermute's astute observation that you don't need to buffer the whole file, something like this is what you really want:

... | awk -v n=2 'NR<=n{print;next} {buf[((NR-1)%n)+1]=$0}
         END{for (i=1;i<=n;i++) print buf[((NR+i-1)%n)+1]}'

I think the math is correct on that - hopefully you get the idea to use a rotating buffer indexed by the NR modded by the size of the buffer and adjusted to use indices in the range 1-n instead of 0-(n-1).

To help with comprehension of the modulus operator used in the indexing above, here is an example with intermediate print statements to show the logic as it executes:

$ cat file   
1
2
3
4
5
6
7
8

.

$ cat tst.awk                
BEGIN {
    print "Populating array by index ((NR-1)%n)+1:"
}
{
    buf[((NR-1)%n)+1] = $0

    printf "NR=%d, n=%d: ((NR-1 = %d) %%n = %d) +1 = %d -> buf[%d] = %s\n",
        NR, n, NR-1, (NR-1)%n, ((NR-1)%n)+1, ((NR-1)%n)+1, buf[((NR-1)%n)+1]

}
END { 
    print "\nAccessing array by index ((NR+i-1)%n)+1:"
    for (i=1;i<=n;i++) {
        printf "NR=%d, i=%d, n=%d: (((NR+i = %d) - 1 = %d) %%n = %d) +1 = %d -> buf[%d] = %s\n",
            NR, i, n, NR+i, NR+i-1, (NR+i-1)%n, ((NR+i-1)%n)+1, ((NR+i-1)%n)+1, buf[((NR+i-1)%n)+1]
    }
}
$ 
$ awk -v n=3 -f tst.awk file
Populating array by index ((NR-1)%n)+1:
NR=1, n=3: ((NR-1 = 0) %n = 0) +1 = 1 -> buf[1] = 1
NR=2, n=3: ((NR-1 = 1) %n = 1) +1 = 2 -> buf[2] = 2
NR=3, n=3: ((NR-1 = 2) %n = 2) +1 = 3 -> buf[3] = 3
NR=4, n=3: ((NR-1 = 3) %n = 0) +1 = 1 -> buf[1] = 4
NR=5, n=3: ((NR-1 = 4) %n = 1) +1 = 2 -> buf[2] = 5
NR=6, n=3: ((NR-1 = 5) %n = 2) +1 = 3 -> buf[3] = 6
NR=7, n=3: ((NR-1 = 6) %n = 0) +1 = 1 -> buf[1] = 7
NR=8, n=3: ((NR-1 = 7) %n = 1) +1 = 2 -> buf[2] = 8

Accessing array by index ((NR+i-1)%n)+1:
NR=8, i=1, n=3: (((NR+i = 9) - 1 = 8) %n = 2) +1 = 3 -> buf[3] = 6
NR=8, i=2, n=3: (((NR+i = 10) - 1 = 9) %n = 0) +1 = 1 -> buf[1] = 7
NR=8, i=3, n=3: (((NR+i = 11) - 1 = 10) %n = 1) +1 = 2 -> buf[2] = 8

Copying last n lines to a new file and then removing the n lines from original

Not all versions of head support negative line counts.
The default installed on macOS doesn't.

If you have coreutils installed (If you have Homebrew installed you can do this: brew install coreutils) you should be able to use ghead -n -3000.

Search only last n lines of a file for matching text pattern and print filenames that do not have this pattern

This should work -

for file in `ls slurm-*`;
do
res=`tail -n2 $file | grep "run complete" 1>/dev/null 2>&1; echo $?`;
  if [ $res -ne 0 ];
  then
    echo $file ;
  fi ;
done ;

Explanation -

"echo $?" gives us the return code of the grep command. If grep finds the pattern in the file, it returns 0. Otherwise the return code is non-zero.

We check for this non-zero return code, and only then, "echo" the file name. Since you have not mentioned whether the output of grep is necessary, I have discarded the STD_OUT and STD_ERR by sending it to /dev/null.

Print a file, skipping the first X lines, in Bash

You'll need tail. Some examples:

$ tail great-big-file.log
< Last 10 lines of great-big-file.log >

If you really need to SKIP a particular number of "first" lines, use

$ tail -n +<N+1> <filename>
< filename, excluding first N lines. >

That is, if you want to skip N lines, you start printing line N+1. Example:

$ tail -n +11 /tmp/myfile
< /tmp/myfile, starting at line 11, or skipping the first 10 lines. >

If you want to just see the last so many lines, omit the "+":

$ tail -n <N> <filename>
< last N lines of file. >

Getting head to display all but the last line of a file: command substitution and standard I/O redirection

head is the wrong tool. If you want to see all but the last line, use:

sed \$d

The reason that

# Sample of incorrect code:
echo "hello" | head -n $(wc -l | sed -E -e 's/\s//g')

fails is that wc consumes all of the input and there is nothing left for head to see. wc inherits its stdin from the subshell in which it is running, which is reading from the output of the echo. Once it consumes the input, it returns and then head tries to read the data...but it is all gone. If you want to read the input twice, the data will have to be saved somewhere.

---

Related Topics