Appending a Line to a File Only If It Does Not Already Exist

Appending a line to a file only if it does not already exist

Just keep it simple :)

grep + echo should suffice:

grep -qxF 'include "/configs/projectname.conf"' foo.bar || echo 'include "/configs/projectname.conf"' >> foo.bar

• -q be quiet
• -x match the whole line
• -F pattern is a plain string
• https://linux.die.net/man/1/grep

Edit:
incorporated @cerin and @thijs-wouters suggestions.

Fastest way -- Appending a line to a file only if it does not already exist

Few suggested improvements:

1. Try using awk instead of grep so that you can both detect the string and write it in one action;
2. If you do use grep don't use a Bash loop to feed each potential match to grep and then append that one word to the file. Instead, read all the potential lines into grep as matches (using -f file_name) and print the matches. Then invert the matches and append the inverted match. See last pipeline here;
3. Exit as soon as you see the string (for a single string) rather than continuing to loop over a big file;
4. Don't call the script millions of times with one or just a few lines -- organize the glue script (in Bash I suppose) so that the core script is called once or a few times with all the lines instead;
5. Perhaps use multicores since the files are not dependent on each other. Maybe with GNU Parallel (or you could use Python or Ruby or Perl that has support for threads).

Consider this awk for a single line to add:

$ awk -v line=line_to_append 'FNR==NR && line==$0{f=1; exit} 
                              END{if (!f) print line >> FILENAME}' file

Or for multiple lines:

$ awk 'FNR==NR {lines[$0]; next} 
       $0 in lines{delete lines[$0]} 
       END{for (e in lines) print e >> FILENAME}' lines file

---

Some timings using a copy of the Unix words file (235,886 lines) with a five line lines file that has two overlaps:

$ echo "frob
knob
kabbob
stew
big slob" > lines
$ time awk 'FNR==NR {lines[$0]; next} 
   $0 in lines{delete lines[$0]} 
   END{for (e in lines) print e >> FILENAME}' lines words
real    0m0.056s
user    0m0.051s
sys 0m0.003s
$ tail words
zythum
Zyzomys
Zyzzogeton
frob
kabbob
big slob

---

Edit 2

Try this as being the best of both:

$ time grep -x -f lines words | 
       awk 'FNR==NR{a[$0]; next} !($0 in a)' - lines >> words
real    0m0.012s
user    0m0.010s
sys     0m0.003s

Explanation:

1. grep -x -f lines words find the lines that ARE in words
2. awk 'FNR==NR{a[$0]; next} !($0 in a)' - lines invert those into lines that are NOT in words
3. >> words append those to the file

add line to a file ONLY if it is not in file already

Assuming you want it at the end of the file:

LINE="nohup java -jar /mnt/fusion/nfs/labStats/LabInfoAutoLog.jar > /dev/null &"
FILE=/etc/rc.d/rc.local
grep -q "$LINE" "$FILE" || echo "$LINE" >> "$FILE"

Append to file only if it exists

Assuming the file is either nonexistent or both readable and writable, you can try to open it for reading first to determine whether it exists or not, e.g.:

command 3<file 3<&- >>file

3<&- may be omitted in most cases as it's unexpected for a program to start reading from file descriptor 3 without redirecting it first.

Proof of concept:

$ echo hello 3<file 3<&- >>file
bash: file: No such file or directory
$ ls file
ls: cannot access 'file': No such file or directory
$ touch file
$ echo hello 3<file 3<&- >>file
$ cat file
hello
$

This works because redirections are processed from left to right, and a redirection error causes the execution of a command to halt. So if file doesn't exist (or is not readable), 3<file fails, the shell prints an error message and stops processing this command. Otherwise, 3<&- closes the descriptor (3) associated with file in previous step, >>file reopens file for appending and redirects standard output to it.

Add lines to a document if they do not already exist within the document

If the file is initially empty, you'll never enter the loop, and thus never add the line. If the file is not empty, you'd add your line once for every non-matching line anyway. Try this: set a flag to indicate whether or not to add the line, then read through the file. If you ever find a matching line, clear the flag to prevent the line from being added after the loop.

do_it=true
while read lines
do 
    if [[ $lines = $site/$name ]]
    then 
        do_it=false
        break
    fi
done < "$doc"
if [[ $do_it = true ]]; then
    echo "$site/$name" >> "$doc"
fi

search and add the line if not present in shell

1,

grep -q -F 'abc.def.*, \' filename || tac filename|sed '4aabc.def.*, \\'|tac

If match the pattern abc.def.*, \, it will not execute the command after ||, the first tac will reverse the file lines for append the pattern in 5th line by sed '4a, the last tac will restore the file lines.

2,

grep -q -F 'abc.def.*, \' filename || sed "$(( $( wc -l < filename) -4 )) a abc.def.*, \\\\" filename

Something as the case 1, but this will use wc to index the 5th line and append to that, wc -l < filename will only get the file lines number. The \\\\ for escape and print / in double quotes.

Add a string to a file ONLY if it doesn't exist

Strictly said I might not use exists() after all, just go with the exception-path:

File file = new File("s.txt");  // this is a file handle, s.txt may or may not exist
boolean found=false;  // flag for target txt being present
try(BufferedReader br=new BufferedReader(new FileReader(file))){
  String line;
  while((line=br.readLine())!=null)  // classic way of reading a file line-by-line
    if(line.equals("something")){
      found=true;
      break;  // if the text is present, we do not have to read the rest after all
    }
} catch(FileNotFoundException fnfe){}

if(!found){  // if the text is not found, it has to be written
  try(PrintWriter pw=new PrintWriter(new FileWriter(file,true))){  // it works with
                                                                   // non-existing files too
    bw.println("something");
  }
}

---

Related Topics