How to Make File Creation an Atomic Operation

How to make file creation an atomic operation?

Write data to a temporary file and when data has been successfully written, rename the file to the correct destination file e.g

with open(tmpFile, 'w') as f:
    f.write(text)
    # make sure that all data is on disk
    # see http://stackoverflow.com/questions/7433057/is-rename-without-fsync-safe
    f.flush()
    os.fsync(f.fileno())    
os.replace(tmpFile, myFile)  # os.rename pre-3.3, but os.rename won't work on Windows

According to doc http://docs.python.org/library/os.html#os.replace

Rename the file or directory src to dst. If dst is a non-empty directory, OSError will be raised. If dst exists and is a file, it will be replaced silently if the user has permission. The operation may fail if src and dst are on different filesystems. If successful, the renaming will be an atomic operation (this is a POSIX requirement).

Note:

• It may not be atomic operation if src and dest locations are not on same filesystem

• os.fsync step may be skipped if performance/responsiveness is more important than the data integrity in cases like power failure, system crash etc

Atomic file write operations (cross platform)

AFAIK no.

And the reason is that for such an atomic operation to be possible, there has to be OS support in the form of a transactional file system. And none of the mainstream operating system offer a transactional file system.

EDIT - I'm wrong for POSIX-compliant systems at least. The POSIX rename syscall performs an atomic replace if a file with the target name already exists ... as pointed out by @janneb. That should be sufficient to do the OP's operation atomically.

However, the fact remains that the Java File.renameTo() method is explicitly not guaranteed to be atomic, so it does not provide a cross-platform solution to the OP's problem.

EDIT 2 - With Java 7 you can use java.nio.file.Files.move(Path source, Path target, CopyOption... options) with copyOptions and ATOMIC_MOVE. If this is not supported (by the OS / file system) you should get an exception.

Create and write a file in one atomic operation on OS level

I'd suggest you to write temporary file and then rename it to your file. I am not sure this operation is implemented in java as atomic for all operating system but at least on Unix you have a chance because I think it uses the same call as mv that is atomic.

It will not be truly atomic on windows, I guess. It will be "almost atomic" that is enough for most applications.

Atomically creating a file if it doesn't exist in Python

You can use os.open with os.O_CREAT | os.O_EXCL flags which will fail if the file exists, they are according to the docs available on Unix and Windows but I am not sure if atomic file creation exists on windows or not:

os.open("filename", os.O_CREAT | os.O_EXCL)

From the linux open man page:

O_EXCL
If O_CREAT and O_EXCL are set, open() shall fail if the file exists. The check for the existence of the file and the creation of the file if it does not exist shall be atomic with respect to other threads executing open() naming the same filename in the same directory with O_EXCL and O_CREAT set. If O_EXCL and O_CREAT are set, and path names a symbolic link, open() shall fail and set errno to [EEXIST], regardless of the contents of the symbolic link. If O_EXCL is set and O_CREAT is not set, the result is undefined.

Not sure what you want to do if the file exists but you just need to catch a FileExistsError when the file does already exist:

import os

def try_make_file(filename):
    try:
        os.open(filename,  os.O_CREAT | os.O_EXCL)
        return True
    except FileExistsError:
        return False

Implementing atomic file writes in a nontransactional filesystem

Your steps can be simplified further:

1. Write new content to a temporary file New
2. Delete Original file
3. Move New to Original

On startup:

1. If Original does not exist but New does, the process was interrupted before step 3, move New to Original.
2. If Original and New both exist, the process was interrupted before step 2, delete New.

I have used this in managing configuration files and have never encountered a problem from this process.

Atomically write byte[] to file

Atomic writes to files are not possible. Operating systems don't support it, and since they don't, programming language libraries can't do it either.

The best you are going to get with a files in a conventional file system is atomic file renaming; i.e.

1. write new file into same file system as the old one

2. use FileDescriptor.sync() to ensure that new file is written

3. rename the new file over the old one; e.g. using

   java.nio.file.Files.move(Path source, Path target, 
                            CopyOption... options) 
   

   with CopyOptions ATOMIC_MOVE. According to the javadocs, this may not be supported, but if it isn't supported you should get an exception.

But note that the atomicity is implemented in the OS, and if the OS cannot give strong enough guarantees, you are out of luck.

(One issue is what might happen in the event of a hard disk error. If the disk dies completely, then atomicity is moot. But if the OS is still able to read data from the disk after the failure, then the outcome may depend on the OS'es ability to repair a possibly inconsistent file system.)

atomic operations on a file from different processes

You can use a Named Mutex to share a lock between processes:

const int MAX_RETRY = 50;
const int DELAY_MS = 200;
bool Success = false;
int Retry = 0;

// Will return an existing mutex if one with the same name already exists
Mutex mutex = new Mutex(false, "MutexName"); 
mutex.WaitOne();

try
{
    while (!Success && Retry < MAX_RETRY)
    {
        using (StreamWriter Wr = new StreamWriter(ConfPath))
        {
            Wr.WriteLine("My content");
        }
        Success = true;
    }
}
catch (IOException)
{
  Thread.Sleep(DELAY_MS);
  Retry++;
}
finally
{
    mutex.ReleaseMutex();
}

---

Related Topics