Best Way to Overwrite File with Itself

Best way to overwrite file with itself

You cannot generically do this in one step because writing to the file can interfere with reading from it. If you try this on a regular file, it's likely the > redirection will blank the file out before anything is even read.

Your safest bet is to split it into two steps. You can hide it behind a function call if you want.

rewrite() {
    local file=$1

    local contents=$(< "$file")
    cat <<< "$contents" > "$file"
}

...

rewrite /sys/misc/whatever

Is there a better way to overwrite file contents?

The short answer: write both and profile.

The longer answer with considerable hand-waving:

Overwriting a file will involve the following system calls:

open
write
close

Creating a new file, deleting the old file, and renaming the new file will involve the following system calls:

open
write
close
unlink
rename

System calls are often the slowest part of programs; in general, reducing system calls is a good way to speed a program. Overwriting the one file will re-use the operating system's internal directory entry data; this will probably also lead to some speed improvements. (They may be difficult to measure in a language with VM overhead...)

Your files are small enough that each write() should be handled atomically, assuming you're updating the entire 1K in a single write. (Since you care about performance, this seems like a safe assumption.) This does mean that other processes should not see partial writes except in the case of catastrophic powerfailures and lossy mount options. (Not common.) The file re-name approach does give consistent files even in the face of multiple writes.

However, 1K files are a pretty inefficient storage mechanism; many filesystems will write files along 4k blocks. If these data blocks exist only in your application it might make sense to write them in containers of some sort, several at a time. (Quake-derived systems do this for reading their maps, textures, and so forth, out of zip files, because giant streaming IO requests are far faster than thousands of smaller IO requests.) Of course, this is harder if your application is writing these files for other applications to work with, but it might still be worth investigating if the files are rarely shared.

How to make program to overwrite itself during execution in go

A workaround can be (because it doesn't overwrite itself, it just creates an other file):

1. copy all content of the original executable
2. modify what I need
3. rename di original executable to a fixed name "old version"
4. write the modified bytes to "original name" (the modified executable)
5. launch the new executable just created
6. either have the original executable self delete or delete it from the modified executable just created

I think this gets the job done even if not on the cleanest way (the program has to start from beginning but i guess this is unavoidable)...

If someone still know a better way you are more the welcome to write your idea.

How To Overwrite A File If It Already Exists?

WriteAllText

File.WriteAllText should do what you want.

Creates a new file, writes the specified string to the file, and then
closes the file. If the target file already exists, it is overwritten.

StreamWriter

The StreamWriter class also has an option to overwrite/append:

Initializes a new instance of the StreamWriter class for the specified
file by using the default encoding and buffer size. If the file
exists, it can be either overwritten or appended to.

public StreamWriter(
    string path,
    bool append
)

Example:

using (StreamWriter writer = new StreamWriter("test.txt", false)){ 
    writer.Write(textToAdd);
}

Looking at your code, you're passing in true which means append.

sw = new StreamWriter(@"C:\Users\Me\Desktop\JAM_MACHINE\record.txt", true);
sw.WriteLine(line);

.NET Compact Framework

If you're stuck on a .NET version that doesn't support anything (e.g. compact framework), you can also implement WriteAllText yourself:

static void WriteAllText(string path, string txt) {
    var bytes = Encoding.UTF8.GetBytes(txt);
    using (var f = File.Open(path, FileMode.Create)) {
        f.Write(bytes, 0, bytes.Length);
    }
}

Why does my tool output overwrite itself and how do I fix it?

The problem is that your input file uses DOS line endings of CRLF instead of UNIX line endings of just LF and you are running a UNIX tool on it so the CR remains part of the data being operated on by the UNIX tool. CR is commonly denoted by \r and can be seen as a control-M (^M) when you run cat -vE on the file while LF is \n and appears as $ with cat -vE.

So your input file wasn't really just:

what isgoingon

it was actually:

what isgoingon\r\n

as you can see with cat -v:

$ cat -vE file
what isgoingon^M$

and od -c:

$ od -c file
0000000   w   h   a   t       i   s   g   o   i   n   g   o   n  \r  \n
0000020

so when you run a UNIX tool like awk (which treats \n as the line ending) on the file, the \n is consumed by the act of reading the line, but that leaves the 2 fields as:

<what> <isgoingon\r>

Note the \r at the end of the second field. \r means Carriage Return which is literally an instruction to return the cursor to the start of the line so when you do:

print $2, $1

awk will print isgoingon and then will return the cursor to the start of the line before printing what which is why the what appears to overwrite the start of isgoingon.

To fix the problem, do either of these:

dos2unix file
sed 's/\r$//' file
awk '{sub(/\r$/,"")}1' file
perl -pe 's/\r$//' file

Apparently dos2unix is aka frodos in some UNIX variants (e.g. Ubuntu).

Be careful if you decide to use tr -d '\r' as is often suggested as that will delete all \rs in your file, not just those at the end of each line.

Note that GNU awk will let you parse files that have DOS line endings by simply setting RS appropriately:

gawk -v RS='\r\n' '...' file

but other awks will not allow that as POSIX only requires awks to support a single character RS and most other awks will quietly truncate RS='\r\n' to RS='\r'. You may need to add -v BINMODE=3 for gawk to even see the \rs though as the underlying C primitives will strip them on some platforms, e.g. cygwin.

One thing to watch out for is that CSVs created by Windows tools like Excel will use CRLF as the line endings but can have LFs embedded inside a specific field of the CSV, e.g.:

"field1","field2.1
field2.2","field3"

is really:

"field1","field2.1\nfield2.2","field3"\r\n

so if you just convert \r\ns to \ns then you can no longer tell linefeeds within fields from linefeeds as line endings so if you want to do that I recommend converting all of the intra-field linefeeds to something else first, e.g. this would convert all intra-field LFs to tabs and convert all line ending CRLFs to LFs:

gawk -v RS='\r\n' '{gsub(/\n/,"\t")}1' file

Doing similar without GNU awk left as an exercise but with other awks it involves combining lines that do not end in CR as they're read.

Also note that though CR is part of the [[:space:]] POSIX character class, it is not one of the whitespace characters included as separating fields when the default FS of " " is used, whose whitespace characters are only tab, blank, and newline. This can lead to confusing results if your input can have blanks before CRLF:

$ printf 'x y \n'
x y
$ printf 'x y \n' | awk '{print $NF}'
y
$

$ printf 'x y \r\n'
x y
$ printf 'x y \r\n' | awk '{print $NF}'

$

That's because trailing field separator white space is ignored at the beginning/end of a line that has LF line endings, but \r is the final field on a line with CRLF line endings if the character before it was whitespace:

$ printf 'x y \r\n' | awk '{print $NF}' | cat -Ev
^M$

Find and replace in file and overwrite file doesn't work, it empties the file

When the shell sees > index.html in the command line it opens the file index.html for writing, wiping off all its previous contents.

To fix this you need to pass the -i option to sed to make the changes inline and create a backup of the original file before it does the changes in-place:

sed -i.bak s/STRING_TO_REPLACE/STRING_TO_REPLACE_IT/g index.html

Without the .bak the command will fail on some platforms, such as Mac OSX.

How to move (and overwrite) all files from one directory to another?

It's just mv srcdir/* targetdir/.

If there are too many files in srcdir you might want to try something like the following approach:

cd srcdir
find -exec mv {} targetdir/ +

In contrast to \; the final + collects arguments in an xargs like manner instead of executing mv once for every file.

Are there better ways to overwrite a file then check for changes (in C#)?

One solution could be to generate some sort of checksum from the contents of the file. Then when you generate a new contents you only need to compare the checksum values to see if the files have changed.

Store the checksum as the first record in the file (or at least fairly near the start of the file) to minimise the amount of data you have to read.

If you could somehow store the checksum as an attribute of the file (rather than in the file itself) you wouldn't even need to open the old file. Another alternative would be to store the checksum and the file it referred to in another central file or database, but there is the danger that could get out of step.

php - What will happen if I overwrite the file itself when it is executing (using ZipArchive)

On Linux files are not usually locked (see https://unix.stackexchange.com/questions/147392/what-is-advisory-locking-on-files-that-unix-systems-typically-employs) so you can do whatever you want with that file. PHP works with that file in memory so you can overwrite it during it's execution.

But if you will run the script multiple times while the first one is in progress it might load incomplete version and then it will throw some error so it might be wise to make sure that won't happen (using locks) or try to do some more atomic approach.

Windows locks files so I assume you won't be able to extract files the same way there.

---

Related Topics