How to Randomize the Lines in a File Using Standard Tools on Red Hat Linux

How can I randomize the lines in a file using standard tools on Red Hat Linux?

And a Perl one-liner you get!

perl -MList::Util -e 'print List::Util::shuffle <>'

It uses a module, but the module is part of the Perl code distribution. If that's not good enough, you may consider rolling your own.

I tried using this with the -i flag ("edit-in-place") to have it edit the file. The documentation suggests it should work, but it doesn't. It still displays the shuffled file to stdout, but this time it deletes the original. I suggest you don't use it.

Consider a shell script:

#!/bin/sh

if [[ $# -eq 0 ]]
then
echo "Usage: $0 [file ...]"
exit 1
fi

for i in "$@"
do
perl -MList::Util -e 'print List::Util::shuffle <>' $i > $i.new
if [[ `wc -c $i` -eq `wc -c $i.new` ]]
then
mv $i.new $i
else
echo "Error for file $i!"
fi
done

Untested, but hopefully works.

How can I shuffle the lines of a text file on the Unix command line or in a shell script?

You can use shuf. On some systems at least (doesn't appear to be in POSIX).

As jleedev pointed out: sort -R might also be an option. On some systems at least; well, you get the picture. It has been pointed out that sort -R doesn't really shuffle but instead sort items according to their hash value.

[Editor's note: sort -R almost shuffles, except that duplicate lines / sort keys always end up next to each other. In other words: only with unique input lines / keys is it a true shuffle. While it's true that the output order is determined by hash values, the randomness comes from choosing a random hash function - see manual.]

How can I print the lines in STDIN in random order in Perl?

I bet real Perl hackers will tear this apart, but here it goes nonetheless.

use strict;
use warnings;
use List::Util 'shuffle';

my @lines = ();
my $bufsize = 512;
while(<STDIN>) {
push @lines, $_;
if (@lines == $bufsize) {
print shuffle(@lines);
undef @lines;
}
}
print shuffle(@lines);

Difference between this and the other solution:

  • Will not consume all the input and then randomize it (memory hog), but will randomize every $bufsize lines (not truly random and slow as a dog compared to the other option).
  • Uses a module which returns a new list instead of a in place editing Fisher - Yates implementation. They are interchangeable (except that you would have to separate the print from the shuffle). For more information type perldoc -q rand on your shell.

Shuf - To Shuffle the contents of a file in linux

Try your package manager or sudo apt-get install shuf

How to shuffle the lines in a file without reading the whole file in advance?

I cannot think of a way to randomly do the entire file without somehow maintaining a list of what has already been written. I think if I had to do a memory efficient shuffle, I would scan the file, building a list of offsets for the new lines. Once I have this list of new line offsets, I would randomly pick one of them, write it to stdout, and then remove it from the list of offsets.

I am not familiar with perl, or python, but can demonstrate with php.

<?php
$offsets = array();

$f = fopen("file.txt", "r");
$offsets[] = ftell($f);
while (! feof($f))
{
if (fgetc($f) == "\n") $offsets[] = ftell($f);
}

shuffle($offsets);
foreach ($offsets as $offset)
{
fseek($f, $offset);
echo fgets($f);
}
fclose($f);
?>

The only other option I can think of, if scanning the file for new lines is absolutely unacceptable, would be (I am not going to code this one out):

  1. Determine the filesize
  2. Create a list of offsets and lengths already written to stdout
  3. Loop until bytes_written == filesize
  4. Seek to a random offset that is not already in your list of already written values
  5. Back up from that seek to the previous newline or start of file
  6. Display that line, and add it to the list of offsets and lengths written
  7. Go to 3.

Bash language randomize order of words in .txt file

If there's one word per line, you can use shuf yourfile to output them in random order, or

shuf yourfile > tmpfile && mv tmpfile yourfile

to write the shuffled contents back into yourfile.

How to randomize a list and iterate through the randomized list (bash)

You can shuffle the lines of a file using the shuf command.

Edit: Your code using shuf would look

while read -r -a array
do
python make_move.py "${array[@]}"
done < <(shuf game_commands.dat)


Related Topics



Leave a reply



Submit