How can I randomize the lines in a file using standard tools on Red Hat Linux?
And a Perl one-liner you get!
perl -MList::Util -e 'print List::Util::shuffle <>'
It uses a module, but the module is part of the Perl code distribution. If that's not good enough, you may consider rolling your own.
I tried using this with the -i
flag ("edit-in-place") to have it edit the file. The documentation suggests it should work, but it doesn't. It still displays the shuffled file to stdout, but this time it deletes the original. I suggest you don't use it.
Consider a shell script:
#!/bin/sh
if [[ $# -eq 0 ]]
then
echo "Usage: $0 [file ...]"
exit 1
fi
for i in "$@"
do
perl -MList::Util -e 'print List::Util::shuffle <>' $i > $i.new
if [[ `wc -c $i` -eq `wc -c $i.new` ]]
then
mv $i.new $i
else
echo "Error for file $i!"
fi
done
Untested, but hopefully works.
How can I shuffle the lines of a text file on the Unix command line or in a shell script?
You can use shuf
. On some systems at least (doesn't appear to be in POSIX).
As jleedev pointed out: sort -R
might also be an option. On some systems at least; well, you get the picture. It has been pointed out that sort -R
doesn't really shuffle but instead sort items according to their hash value.
[Editor's note: sort -R
almost shuffles, except that duplicate lines / sort keys always end up next to each other. In other words: only with unique input lines / keys is it a true shuffle. While it's true that the output order is determined by hash values, the randomness comes from choosing a random hash function - see manual.]
How can I print the lines in STDIN in random order in Perl?
I bet real Perl hackers will tear this apart, but here it goes nonetheless.
use strict;
use warnings;
use List::Util 'shuffle';
my @lines = ();
my $bufsize = 512;
while(<STDIN>) {
push @lines, $_;
if (@lines == $bufsize) {
print shuffle(@lines);
undef @lines;
}
}
print shuffle(@lines);
Difference between this and the other solution:
- Will not consume all the input and then randomize it (memory hog), but will randomize every $bufsize lines (not truly random and slow as a dog compared to the other option).
- Uses a module which returns a new list instead of a in place editing Fisher - Yates implementation. They are interchangeable (except that you would have to separate the print from the shuffle). For more information type perldoc -q rand on your shell.
Shuf - To Shuffle the contents of a file in linux
Try your package manager or sudo apt-get install shuf
How to shuffle the lines in a file without reading the whole file in advance?
I cannot think of a way to randomly do the entire file without somehow maintaining a list of what has already been written. I think if I had to do a memory efficient shuffle, I would scan the file, building a list of offsets for the new lines. Once I have this list of new line offsets, I would randomly pick one of them, write it to stdout, and then remove it from the list of offsets.
I am not familiar with perl, or python, but can demonstrate with php.
<?php
$offsets = array();
$f = fopen("file.txt", "r");
$offsets[] = ftell($f);
while (! feof($f))
{
if (fgetc($f) == "\n") $offsets[] = ftell($f);
}
shuffle($offsets);
foreach ($offsets as $offset)
{
fseek($f, $offset);
echo fgets($f);
}
fclose($f);
?>
The only other option I can think of, if scanning the file for new lines is absolutely unacceptable, would be (I am not going to code this one out):
- Determine the filesize
- Create a list of offsets and lengths already written to stdout
- Loop until bytes_written == filesize
- Seek to a random offset that is not already in your list of already written values
- Back up from that seek to the previous newline or start of file
- Display that line, and add it to the list of offsets and lengths written
- Go to 3.
Bash language randomize order of words in .txt file
If there's one word per line, you can use shuf yourfile
to output them in random order, or
shuf yourfile > tmpfile && mv tmpfile yourfile
to write the shuffled contents back into yourfile
.
How to randomize a list and iterate through the randomized list (bash)
You can shuffle the lines of a file using the shuf
command.
Edit: Your code using shuf
would look
while read -r -a array
do
python make_move.py "${array[@]}"
done < <(shuf game_commands.dat)
Related Topics
Syntax Error in Shell Script With Process Substitution
Lost Httpd.Conf File Located Apache
How to Quickly Sum All Numbers in a File
How to Get Docker Linux Container Information from Within the Container Itself
How to Generate New Variable Names on the Fly in a Shell Script
Why Do You Need to Put #!/Bin/Bash At the Beginning of a Script File
Bash Script Processing Limited Number of Commands in Parallel
Automatically Capture Output of Last Command into a Variable Using Bash
How to Set Environment Variable For Everyone Under My Linux System
Split One File into Multiple Files Based on Delimiter
How to Escape Single Quotes in Bash/Grep
Quickly Create a Large File on a Linux System
Why Are the Backslash and Semicolon Required With the Find Command'S -Exec Option
Exit Code of Variable Assignment to Command Substitution in Bash
&&' Vs. '&' With the 'Test' Command in Bash