How to Open a File for Exclusive Access in Python

What is the best way to open a file for exclusive access in Python?

I don't think there is a fully crossplatform way. On unix, the fcntl module will do this for you. However on windows (which I assume you are by the paths), you'll need to use the win32file module.

Fortunately, there is a portable implementation (portalocker) using the platform appropriate method at the python cookbook.

To use it, open the file, and then call:

portalocker.lock(file, flags)

where flags are portalocker.LOCK_EX for exclusive write access, or LOCK_SH for shared, read access.

Opening a file on Windows with exclusive locking in Python

For those who are interested in a Windows specific solution:

import os
import ctypes
import msvcrt
import pathlib

# Windows constants for file operations
NULL = 0x00000000
CREATE_ALWAYS = 0x00000002
OPEN_EXISTING = 0x00000003
FILE_SHARE_READ = 0x00000001
FILE_ATTRIBUTE_READONLY = 0x00000001  # strictly for file reading
FILE_ATTRIBUTE_NORMAL = 0x00000080  # strictly for file writing
FILE_FLAG_SEQUENTIAL_SCAN = 0x08000000
GENERIC_READ = 0x80000000
GENERIC_WRITE = 0x40000000

_ACCESS_MASK = os.O_RDONLY | os.O_WRONLY
_ACCESS_MAP = {os.O_RDONLY: GENERIC_READ,
               os.O_WRONLY: GENERIC_WRITE
               }

_CREATE_MASK = os.O_CREAT | os.O_TRUNC
_CREATE_MAP = {NULL: OPEN_EXISTING,
               os.O_CREAT | os.O_TRUNC: CREATE_ALWAYS
               }

win32 = ctypes.WinDLL('kernel32.dll', use_last_error=True)
win32.CreateFileW.restype = ctypes.c_void_p
INVALID_FILE_HANDLE = ctypes.c_void_p(-1).value

def _opener(path: pathlib.Path, flags: int) -> int:

    access_flags = _ACCESS_MAP[flags & _ACCESS_MASK]
    create_flags = _CREATE_MAP[flags & _CREATE_MASK]

    if flags & os.O_WRONLY:
        share_flags = NULL
        attr_flags = FILE_ATTRIBUTE_NORMAL
    else:
        share_flags = FILE_SHARE_READ
        attr_flags = FILE_ATTRIBUTE_READONLY

    attr_flags |= FILE_FLAG_SEQUENTIAL_SCAN

    h = win32.CreateFileW(path, access_flags, share_flags, NULL, create_flags, attr_flags, NULL)

    if h == INVALID_FILE_HANDLE:
        raise ctypes.WinError(ctypes.get_last_error())

    return msvcrt.open_osfhandle(h, flags)

class _FileControlAccessor(pathlib._NormalAccessor):

    open = staticmethod(_opener)

_control_accessor = _FileControlAccessor()

class Path(pathlib.WindowsPath):

    def _init(self) -> None:

        self._closed = False
        self._accessor = _control_accessor

    def _opener(self, name, flags) -> int:

        return self._accessor.open(name, flags)

how to open (create if not exists) a file while acquiring exclusive lock avoiding races

No, it is not possible as a basic operation supported by Linux/UNIX.

The O_CREAT|O_EXCL technique in the answer you referenced can work here. Instead of exclusively creating the target file, you exclusively create a lockfile whose name is predictably derived from the target file. E.g., os.path.join("/tmp", hashlib.md5(target_filename).hexdigest() + ".lock").

However, as others have suggested, it's not clear that you need to protect both the target file creation and its checksumming + possible replacement. An fcntl advisory lock will suit your needs.

What does python3 open x mode do?

As @Martjin has already said, you have already answered your own question. I would only be amplifying on the explanation in the manual so as to get a better understanding of the text

'x': open for exclusive creation, failing if the file already exists

When you specify exclusive creation, it clearly means, you would use this mode for exclusively creating the file. The need for this is required when you won't accidentally truncate/append an existing file with either of the modes w or a.

In absence of this, developers should be cautious to check for the existence of the file before leaping to open the file for updation.

With this mode, your code would be simply be written as

try:
    with open("fname", "x") as fout:
        #Work with your open file
except FileExistsError:
    # Your error handling goes here

Previously though your code might had been written as

import os.path
if os.path.isfile(fname):
    # Your error handling goes here
else:
    with open("fname", "w") as fout:
        # Work with your open file

How to open a file with exclusive file access in node?

The module add-on fs-ext will work on Windows if you can build it successfully. I had to install Python on my system and put it in the path before it would install and build successfully. Once I did that, then I was able to run this test which verified that a child process was not allowed to read or write to a file that I had an flock on:

// open-exclusive.js
const fs = require('fs');
const fsp = fs.promises;
const child = require('child_process');
const { promisify } = require('util');
const execP = promisify(child.exec);
const flock = promisify(require('fs-ext').flock);

const exclusive = fs.constants.S_IRUSR | fs.constants.S_IWUSR | fs.constants.S_IXUSR;

async function run() {
    const fHandle = await fsp.open("./temp.txt", "r+", exclusive);
    await flock(fHandle.fd, 'ex');
    await fHandle.write("Goodbye", 0);
    console.log('parent finished write');

    const { stdout, stderr } = await execP("node open-exclusive-child.js");
    console.log('stdout:', stdout);
    console.error('stderr:', stderr);
    console.log('parent done exec');
    await fHandle.close();
    return "good";
}

run().then(result => {
    console.log(result);
}).catch(err => {
    console.log(err);
});

And the child process file open-exclusive-child.js

const fs = require('fs');
const fsp = fs.promises;

async function run() {
    const fHandle = await fsp.open("./temp.txt", "r");
    let buf = Buffer.alloc(10);
    await fHandle.read(buf, 0, 10, 0);
    await fHandle.write("Hello", 0);
    console.log('child: finished write');
    await fHandle.close();
    return "child good"
}

run().then(result => {
    console.log(result);
}).catch(err => {
    console.log(err);
});

When I run node open-exclusive.js, the parent will open it, flock it successfully and then the child will not be able to read or write to the file. The child is apparently allowed to "open" the file, but gets an EBUSY error when trying to read or write to it.

In fact, the flock works in the same process too:

const fs = require('fs');
const fsp = fs.promises;
const { promisify } = require('util');
const execP = promisify(child.exec);
const flock = promisify(require('fs-ext').flock);

const exclusive = fs.constants.S_IRUSR | fs.constants.S_IWUSR | fs.constants.S_IXUSR;

async function run() {
    const fHandle = await fsp.open("./temp.txt", "r+", exclusive);
    await flock(fHandle.fd, 'ex');
    await fHandle.write("Goodbye", 0);
    console.log('parent finished write');

    const fHandle2 = await fsp.open("./temp.txt", "r");
    const buf = Buffer.alloc(10);
    console.log("About to read from fHandle2");
    let bytes = await fHandle2.read(buf, 0, 10, 0);
    console.log(bytes);
    await fHandle2.close();
    await fHandle.close();
    return "good";
}

run().then(result => {
    console.log(result);
}).catch(err => {
    console.log(err);
});

When I run this, I get this in the console:

node open-exclusive.js
parent finished write
About to read from fHandle2
[Error: EBUSY: resource busy or locked, read] {
  errno: -4082,
  code: 'EBUSY',
  syscall: 'read'
}

Lock file for access on windows

You could use subprocess to open the file in notepad or excel:

import subprocess, time

subprocess.call('start excel.exe "\lockThisFile.txt\"', shell = True)

time.sleep(10) # if you need the file locked before executing the next commands, you may need to sleep it for a few seconds

or

subprocess.call('notepad > lockThisFile.txt', shell = True)

As written you need shell = True, otherwise windows will give you a syntax error.

(subprocess.Popen() works as well)

You can then close the process later using:

subprocess.call('taskkill /f /im notepad.exe') # or excel.exe

Other options include

-write some C++ code and call it from python (https://msdn.microsoft.com/en-us/library/windows/desktop/aa365203(v=vs.85).aspx)

-call 3rd party programs with subprocess.call():

FileLocker http://www.jensscheffler.de/filelocker (https://superuser.com/questions/294826/how-to-purposefully-exclusively-lock-a-file)

Easy File Locker http://www.xoslab.com/efl.html and Dispatch (from win32com.client import Dispatch), although last choice is the most complex

Locking a file in Python

Alright, so I ended up going with the code I wrote here, on my website ^{link is dead, view on archive.org} (also available on GitHub). I can use it in the following fashion:

from filelock import FileLock

with FileLock("myfile.txt"):
    # work with the file as it is now locked
    print("Lock acquired.")

Python open() fails on Windows when another process is opening the file

As suggested in this answer, open() throws an IOError either if the process doesn't have permission to access the file or if the file is locked by another process.

To verify that it's the second and not the first, you can use os.access() to check whether the process has permissions to access the file.

For example:

if not os.access(FILE_PATH, os.R_OK):
  print "Failure opening file: No read access for %s" % FILE_PATH
  return

try:                                                                                                                                                      
  fd = open(FILE_PATH)
  print "Success opening file"
except IOError:
  print "Failure opening file: Another process holds lock on %s" % FILE_PATH

With flock on Unix systems, opening and locking files are separate concepts. LOCK_FILE_EX on Windows systems, however, actually blocks other processes from opening that portion of the file for read or write access (source).

---

Related Topics