(Un/De)Compress a String in Bash

(Un/De)compress a string in bash?

When you do:

hey=$(echo "hello world" | gzip -cf)

You don't have same same bytes in variable hey as you have in /tmp/myfile created by:

echo "hello world" | gzip -cf > /tmp/myfile

You get "gzip: stdin is a multi-part gzip file -- not supported" error simply because you have broken compressed data which cannot be uncompressed.

The VAR=$(...) construction is designed for working with text. This is why you get extra trailing trim for example.

gzip encode/deflate STRING in bash, then decode/inflate STRING in php

The gzipped string is not safe in a bash string, if i run

FOO="$(echo "Hello world" | gzip -c | base64)"
FOO="$(echo "$FOO" | base64 -d)"

I get an error

-bash: warning: command substitution: ignored null byte in input

But it works if I leave the string base64 encoded like this using base64_decode() / gzdecode():

FOO="$(echo "Hello world" | gzip -c | base64)"
php -r "echo gzdecode(base64_decode('$FOO'));"

Output:

Hello world

---

It also works if you pipe the output of the decoded string to PHP:

FOO="$(echo "Hello world" | gzip -c | base64)"
echo "$FOO" | base64 -d | php -r 'echo gzdecode(file_get_contents("php://stdin"));'

(example to read from stdin taken from here)

compression and decompression of string data in java

This is because of

String outStr = obj.toString("UTF-8");

Send the byte[] which you can get from your ByteArrayOutputStream and use it as such in your ByteArrayInputStream to construct your GZIPInputStream. Following are the changes which need to be done in your code.

byte[] compressed = compress(string); //In the main method

public static byte[] compress(String str) throws Exception {
    ...
    ...
    return obj.toByteArray();
}

public static String decompress(byte[] bytes) throws Exception {
    ...
    GZIPInputStream gis = new GZIPInputStream(new ByteArrayInputStream(bytes));
    ...
}

Conversion hex string into ascii in bash command line

This worked for me.

$ echo 54657374696e672031203220330 | xxd -r -p
Testing 1 2 3$

-r tells it to convert hex to ascii as opposed to its normal mode of doing the opposite

-p tells it to use a plain format.

Decompress gzip file to specific directory

Given the fact that you have a .tar.gz file, the first way you tried, with the -C option, will work just fine:

tar xvzf /dir/to/file.tar.gz -C /dir/to/output/

tar calls gzip to decompress, and then extracts the files from the tar stream. gzip can only decompress, so gunzip file.tar.gz would simply leave with the decompressed file.tar, on which you would then need to tar xvf file.tar. The z option of tar is simply a shortcut to do the decompression with gzip along with extracting the files.

If you want to just decompress a .gz file (as opposed to extracting a .tar.gz file) to a different directory, simply pipe there. E.g. gzip -dc < file.gz > /somewhere/file.

How can I Zip and Unzip a string using GZIPOutputStream that is compatible with .Net?

The GZIP methods:

public static byte[] compress(String string) throws IOException {
    ByteArrayOutputStream os = new ByteArrayOutputStream(string.length());
    GZIPOutputStream gos = new GZIPOutputStream(os);
    gos.write(string.getBytes());
    gos.close();
    byte[] compressed = os.toByteArray();
    os.close();
    return compressed;
}

public static String decompress(byte[] compressed) throws IOException {
    final int BUFFER_SIZE = 32;
    ByteArrayInputStream is = new ByteArrayInputStream(compressed);
    GZIPInputStream gis = new GZIPInputStream(is, BUFFER_SIZE);
    StringBuilder string = new StringBuilder();
    byte[] data = new byte[BUFFER_SIZE];
    int bytesRead;
    while ((bytesRead = gis.read(data)) != -1) {
        string.append(new String(data, 0, bytesRead));
    }
    gis.close();
    is.close();
    return string.toString();
}

And a test:

final String text = "hello";
try {
    byte[] compressed = compress(text);
    for (byte character : compressed) {
        Log.d("test", String.valueOf(character));
    }
    String decompressed = decompress(compressed);
    Log.d("test", decompressed);
} catch (IOException e) {
    e.printStackTrace();
}

=== Update ===

If you need .Net compability my code has to be changed a little:

public static byte[] compress(String string) throws IOException {
    byte[] blockcopy = ByteBuffer
        .allocate(4)
        .order(java.nio.ByteOrder.LITTLE_ENDIAN)
        .putInt(string.length())
        .array();
    ByteArrayOutputStream os = new ByteArrayOutputStream(string.length());
    GZIPOutputStream gos = new GZIPOutputStream(os);
    gos.write(string.getBytes());
    gos.close();
    os.close();
    byte[] compressed = new byte[4 + os.toByteArray().length];
    System.arraycopy(blockcopy, 0, compressed, 0, 4);
    System.arraycopy(os.toByteArray(), 0, compressed, 4, os.toByteArray().length);
    return compressed;
}

public static String decompress(byte[] compressed) throws IOException {
    final int BUFFER_SIZE = 32;
    ByteArrayInputStream is = new ByteArrayInputStream(compressed, 4, compressed.length - 4);
    GZIPInputStream gis = new GZIPInputStream(is, BUFFER_SIZE);
    StringBuilder string = new StringBuilder();
    byte[] data = new byte[BUFFER_SIZE];
    int bytesRead;
    while ((bytesRead = gis.read(data)) != -1) {
        string.append(new String(data, 0, bytesRead));
    }
    gis.close();
    is.close();
    return string.toString();
}

You can use the same test script.

How to base64 encode image in linux bash / shell

You need to use cat to get the contents of the file named 'DSC_0251.JPG', rather than the filename itself.

test="$(cat DSC_0251.JPG | base64)"

However, base64 can read from the file itself:

test=$( base64 DSC_0251.JPG )

Tool to Unminify / Decompress JavaScript

You can use this : http://jsbeautifier.org/
But it depends on the minify method you are using, this one only formats the code, it doesn't change variable names, nor uncompress base62 encoding.

edit: in fact it can unpack "packed" scripts (packed with Dean Edward's packer : http://dean.edwards.name/packer/)

---

Related Topics