Most Efficient Way to Concatenate Thousands of Files in Perl

perl, unix: fastest way to merge thousands of small files into one file

The cat command works nicely:

cat *someglob* > output.txt

It's name (short for concatenate) even gives away its purpose.

If your argument list is too long (i.e. too many files are matched by the glob) you can always use the find command and pipe the arguments to xargs.

find . -name \*someglob\* -print0 | xargs -0 cat > output.txt

How can I merge some columns of two files using perl?

Another way to get the requested output is to use one while loop instead of two:

mod.pl

#!/usr/bin/perl 

use strict;
use warnings;

open my $input1, '<', "input1.txt" or die qq{Failed to open "input1.txt" for writing: $!};

open my $input2, '<', "input2.txt" or die qq{Failed to open "input2.txt" for writing: $!};

open my $outfile, '>', "outfile.txt" or die qq{Failed to open "outfile.txt" for writing: $!};

while(my $l1 = <$input1>){ 
 my $l2 = <$input2>;
 chomp $l1;
 chomp $l2;
  my @columns1 = split(/ /, $l1);
  my @columns2 = split(/ /, $l2);
  print $outfile join("\t", $columns1[1-1], $columns2[3-1]),"\n";
}

  close($input1);
  close($input2);
  close($outfile);

---

#$ perl mod.pl 
#$ cat outfile.txt 
1   8
2   7
3   4
4   6

Running thousands of commands

Almost identical to Miller's answer except that this will actually run the command and print the corresponding output:

for my $i ('AAA'..'ZZZ') {
    my $output = `command.exe -X="$i"`;
    print "$i: $output\n";
}

Concatenating multiple text files into a single file in Bash

This appends the output to all.txt

cat *.txt >> all.txt

This overwrites all.txt

cat *.txt > all.txt

What's the fastest way to merge multiple csv files by column?

[...] transposing each csv file and re-saving to disk, and then using the command line to concatenate them [...]

Sounds like Transpose-Cat-Transpose. Use paste for joining files horizontally.

paste -d ',' a.csv b.csv c.csv ... > result.csv

---

Related Topics