How to Gzip Standard in to a File and Also Print Standard in to Standard Out

How can I gzip standard in to a file and also print standard in to standard out?

echo "hey hey, we're the monkees" | tee /dev/tty | gzip --stdout > my_log.gz

As pointed out in the comments, /dev/stdout might work better than /dev/tty in some circumstances.

Compress files while reading data from STDIN

Yes, use gzip for this. The best way is to read data as input and redirect the compressed to output file i.e.

cat test.csv | gzip > test.csv.gz

cat test.csv will send the data as stdout and using pipe-sign gzip will read that data as stdin. Make sure to redirect the gzip output to some file as compressed data will not be written to the terminal.

how to pipe contents of large tar.gz file to STDOUT?

Use this with GNU tar to extract a tgz to stdout:

tar -xOzf large.tar.gz --wildcards '*.html' | grep ...

---

-O, --to-stdout: extract files to standard output

PostgreSQL COPY pipe output to gzip and then to STDOUT

When you use COPY ... TO PROGRAM, the PostgreSQL server process (backend) starts a new process and pipes the file to the process's standard input. The standard output of that process is lost. It only makes sense to use COPY ... TO PROGRAM if the called program writes the data to a file or similar.

If your goal is to compress the data that go across the network, you could use sslmode=require sslcompression=on in your connect string to use the SSL network compression feature I built into PostgreSQL 9.2. Unfortunately this has been deprecated and most OpenSSL binaries are shipped with the feature disabled.

There is currently a native network compression patch under development, but it is questionable whether that will make v14.

Other than that, you cannot get what you want at the moment.

How do you gunzip a file and keep the .gz file?

You're looking for:

gzcat x.txt.gz >x.txt

The gzcat command is equivalent to gunzip -c which simply writes the output stream to stdout. This will leave the compressed file untouched. So you can also use:

gunzip -c x.txt.gz >x.txt

Note that on some systems gzcat is also known as zcat so run like this instead:

zcat x.txt.gz >x.txt

What standard input and output would be if there's no terminal connected to server?

The standard input and standard output (and standard error) streams can point to basically any I/O device. This is commonly a terminal, but it can also be a file, a pipe, a network socket, a printer, etc. What exactly those streams direct their I/O to is usually determined by the process that launches your process, be that a shell or a daemon like cron or apache, but a process can redirect those streams itself it it would like.

I'll use Linux as an example, but the concepts are similar on most other OSes. On Linux, the standard input and standard output stream are represented by file descriptors 0 and 1. The macros STDIN_FILENO and STDOUT_FILENO are just for convenience and clarity. A file descriptor is just a number that matches up to some file description that the OS kernel maintains that tells it how to write to that device. That means that from a user-space process's perspective, you write to pretty much anything the same way: write(some_file_descriptor, some_string, some_string_length) (higher-level I/O functions like printf or cout are just wrappers around one or more calls to write). To the process, it doesn't matter what type of device some_file_descriptor represents. The OS kernel will figure that out for you and pass your data to the appropriate device driver.

---

The standard way to launch a new process is to call fork to duplicate the parent process, and then later to call one of the exec family of functions in the child process to start executing some new program. In between, it will often close the standard streams it inherited from its parent and open new ones to redirect the child process's output somewhere new. For instance, to have the child pipe its output back to the parent, you could do something like this in C++:

int main()
{
    // create a pipe for the child process to use for its
    // standard output stream
    int pipefds[2];
    pipe(pipefds);

    // spawn a child process that's a copy of this process
    pid_t pid = fork();
    if (pid == 0)
    {
        // we're now in the child process

        // we won't be reading from this pipe, so close its read end
        close(pipefds[0]);
        // we won't be reading anything
        close(STDIN_FILENO);
        // close the stdout stream we inherited from our parent
        close(STDOUT_FILENO);
        // make stdout's file descriptor refer to the write end of our pipe
        dup2(pipefds[1], STDOUT_FILENO);
        // we don't need the old file descriptor anymore.
        // stdout points to this pipe now
        close(pipefds[1]);
        // replace this process's code with another program
        execlp("ls", "ls", nullptr);
    } else {
        // we're still in the parent process

        // we won't be writing to this pipe, so close its write end
        close(pipefds[1]);

        // now we can read from the pipe that the
        // child is using for its standard output stream
        std::string read_from_child;
        ssize_t count;
        constexpr size_t BUF_SIZE = 100;
        char buf[BUF_SIZE];
        while((count = read(pipefds[0], buf, BUF_SIZE)) > 0) {
            std::cout << "Read " << count << " bytes from child process\n";
            read_from_child.append(buf, count);
        }

        std::cout << "Read output from child:\n" << read_from_child << '\n';
        return EXIT_SUCCESS;
    }
}

Note: I've omitted error handling for clarity

This example creates a child process and redirects its output to a pipe. The program run in the child process (ls) can treat the standard output stream just as it would if it were referencing a terminal (though ls changes some behaviors if it detects its standard output isn't a terminal).

---

This sort of redirection can also be done from a terminal. When you run a command you can use the redirection operators to tell your shell to redirect that commands standard streams to some other location than the terminal. For instance, here's a convoluted way to copy a file from one machine to another using an sh-like shell:

gzip < some_file | ssh some_server 'zcat > some_file'

This does the following:

1. create a pipe
2. run gzip redirecting its standard input stream to read from "some_file" and redirecting its standard output stream to write to the pipe
3. run ssh and redirect its standard input stream to read from the pipe
4. on the server, run zcat with its standard input redirected from the data read from the ssh connection and its standard output redirected to write to "some_file"

Can you use gzip to just compress text that you type into the terminal?

Yes, you can, but it's not a common thing to do.

For example, this is perfectly legal (I typed "Hello, world.", then Enter, then Ctrl-D).

$ gzip > input.gz
Hello, world.
$ ls -l input.gz 
-rw-r--r-- 1 kst kst 34 Oct  5 23:18 input.gz
$ gzip -d < input.gz 
Hello, world.

(The compressed file is actually bigger than the input; this is common for very small inputs.)

But remember that standard input doesn't have to come from the keyboard. For example, the du command prints a summary of disk usage, and can generate a lot of output if you run it on a large directory. If you want to save the output directly in compressed form, you might type:

du -a / | gzip > du-a.out.gz

The standard input of the gzip command comes from the output of the du -a / command; its standard output is written to the new file du-a.out.gz.

The tar command is another common example:

tar cf - some_dir | gzip > some_dir.tar.gz

(but many versions of tar have options to do this without having to explicitly invoke the gzip command).

As for the last part of your question:

Also, when I type gzip -f in my terminal, cursor goes to newline and starts accepting input but I just have to Ctrl+C it out and then nothing happens.

Typing Ctrl-C kills the gzip process, and may not allow it to finish writing its output. You need to type Ctrl-D to trigger an end-of-file condition; this lets the gzip -f command finish normally. But you don't generally want to do that; it will write the compressed output to your terminal, which can include control characters that can mess up your terminal settings.

Perl to parse text file which may or may not be gzipped

open $FILE, "zcat -f $file|" or die $!;

man zcat

-f --force

If the input data is not in a format recognized by gzip, and if the option --stdout is also given, copy the input data without change to the standard output: let zcat behave as cat.

Zip file and print to stdout

The zip program from Info-ZIP (the one usually found on Linux systems) allows generating ZIP files into the standard output, when you use - as the name of the file. For example you could send a zip file to port 8787 on a remote host with this command:

zip -r - files_to_be_archived | nc remotehost 8787

All of this is documented on the zip command manual page.

---

Related Topics