Bash: /Bin/Tar: Argument List Too Long When Compressing Many Files with Tar

When compressing files (zip, tar, ect...) in SSH what determines the 'sort order' in which files are compressed?

If you give tar a list of directory names, the order of the entries in the tar file will match the order that readdir returns filenames from the filesystem. The fact that you are compressing the tar file has no bearing on the order.

Here is a quick example to illustrate what happens on a Linux ext4 filesystem. Other filesystems may behave differently.

First create a new directory with three files, a1, a2 and a3

$ mkdir fred
$ cd fred
$ touch a1 a2 a3

Now lets see the order that readdir returns the files. The -U option will make ls return the filenames unsorted in the order they are stored in the directory.

$ ls -U
a3  a1  a2

As you can see, on my Linux setup the files are returned in an apparently random order.

Now stick the files in a tar file. Note I'm giving tar a directory name for the input file ("." in this instance) to make sure it has to call readdir behind the scenes.

$ tar cf xxx.tar .

And finally, lets see the order that tar has stored the files.

$ tar tf xxx.tar 
./
./a3
./a1
./a2

The order of the files a1, a2 and a3 matches the order that readdir returned the filenames from the filesystem. The . filename is present because it was explicitly included on the command line passed to tar.

If you want to force an order you will have to give tar a sorted list of filenames. The example below shows how to get tar to read the list of filenames from stdin, using the -T - command line option.

$ ls a* | tar  cvf yyy.tar -T -
a1
a2
a3

In this toy example the list of filenames will be automatically sorted because the shell sorts the filenames that match the wildcard a*.

And just to confirm, this is what is in the tar file.

$ tar tf yyy.tar 
a1
a2
a3

In your use-case a combination of the find and sort commands piped into tar should allow you to create a sorted tar file with as many entries as you like.

Something like this as a starting point.

find | sort | tar -cvzf _backup.tar.gz -T -

Shell command to tar directory excluding certain files/folders

You can have multiple exclude options for tar so

$ tar --exclude='./folder' --exclude='./upload/folder2' -zcvf /backup/filename.tgz .

etc will work. Make sure to put --exclude before the source and destination items.

Tar archiving that takes input from a list of files

Yes:

tar -cvf allfiles.tar -T mylist.txt

Create a .tar.bz2 file Linux

You are not indicating what to include in the archive.

Go one level outside your folder and try:

sudo tar -cvjSf folder.tar.bz2 folder

Or from the same folder try

sudo tar -cvjSf folder.tar.bz2 *

Cheers!

---

Related Topics