Write-Only Mapping a O_Wronly Opened File Supposed to Work

Write-only mapping a O_WRONLY opened file supposed to work?

The IEEE Std 1003.1, 2004 Edition (POSIX.1 2004) appears to forbid it.

An implementation may permit accesses other than those specified by prot; however, if the Memory Protection option is supported, the implementation shall not permit a write to succeed where PROT_WRITE has not been set or shall not permit any access where PROT_NONE alone has been set. The implementation shall support at least the following values of prot: PROT_NONE, PROT_READ, PROT_WRITE, and the bitwise-inclusive OR of PROT_READ and PROT_WRITE. If the Memory Protection option is not supported, the result of any access that conflicts with the specified protection is undefined. The file descriptor fildes shall have been opened with read permission, regardless of the protection options specified. If PROT_WRITE is specified, the application shall ensure that it has opened the file descriptor fildes with write permission unless MAP_PRIVATE is specified in the flags parameter as described below.

(emphasis added)

Also, on x86, it is not possible to have write-only memory, and this is a limitation of the page table entries. Pages may be marked read-only or read-write and independently may be executable or non-executable, but cannot be write-only. Moreover the man-page for mprotect() says:

Whether PROT_EXEC has any effect different from PROT_READ is architecture- and kernel version-dependent. On some hardware architectures (e.g., i386), PROT_WRITE implies PROT_READ.

This being the case, you've opened a file descriptor without read access, but mmap() would be bypassing the O_WRONLY by giving you PROT_READ rights. Instead, it will refuse outright with EACCESS.

How to open (read-write) or create a file with truncation allowed?

According to OpenGroup:

O_TRUNC

If the file exists and is a regular file, and the file is successfully
opened O_RDWR or O_WRONLY, its length is truncated to 0 and the mode
and owner are unchanged. It will have no effect on FIFO special files
or terminal device files. Its effect on other file types is
implementation-dependent. The result of using O_TRUNC with O_RDONLY is
undefined.

So, O_TRUNC is probably passed when opening a file with "w" or "w+". This gives "truncation" a different meaning, not what I want.

With python the solution seems to open file at low-level I/O with os.open() function.

The following python function:

def touchopen(filename, *args, **kwargs):
    # Open the file in R/W and create if it doesn't exist. *Don't* pass O_TRUNC
    fd = os.open(filename, os.O_RDWR | os.O_CREAT)

    # Encapsulate the low-level file descriptor in a python file object
    return os.fdopen(fd, *args, **kwargs)

has the behavior I wanted. You can use it like this (it's in fact my use case):

# Open an existing file or create if it doesn't exist
with touchopen("./tool.run", "r+") as doing_fd:

    # Acquire a non-blocking exclusive lock
    fcntl.lockf(doing_fd, fcntl.LOCK_EX)

    # Read a previous value if present
    previous_value = doing_fd.read()
    print previous_value 

    # Write the new value and truncate
    doing_fd.seek(0)
    doing_fd.write("new value")
    doing_fd.truncate()

Android Studio file mapping problem, not recognize files

It happens because of wrong File Encoding and you can see my answer about it in here.

Solution: The solution will be reinstalling Android Studio or, go to:

File -> Other Settings -> Default Settings, find File Encodings, change Project Encoding to System-Default or UTF-8 format.

mmap: will the mapped file be loaded into memory immediately?

No, yes, maybe. It depends.

Calling mmap generally only means that to your application, the mapped file's contents are mapped to its address space as if the file was loaded there. Or, as if the file really existed in memory, as if they were one and the same (which includes changes being written back to disk, assuming you have write access).

No more, no less. It has no notion of loading something, nor does the application know what this means.

An application does not truly have knowledge of any such thing as memory, although the virtual memory system makes it appear like that. The memory that an application can "see" (and access) may or may not correspond to actual physical memory, and this can in principle change at any time, without prior warning, and without an obvious reason (obvious to your application).

Other than possibly experiencing a small delay due to a page fault, an application is (in principle) entirely unaware of any such thing happening and has little or no control over it¹.

Applications will, generally, load pages from mapped files (including the main executable!) on demand, as a consequence of encountering a fault. However, an operating system will usually try to speculatively prefetch data to optimize performance.

In practice, calling mmap will immediately begin to (asynchronously) prefetch pages from the beginning of the mapping, up to a certain implementation-specified size. Which means, in principle, for small files the answer would be "yes", and for larger files it would be "no".

However, mmap does not block to wait for completion of the readahead, which means that you have no guarantee that any of the file is in RAM immediately after mmap returns (not that you have that guarantee at any time anyway!). Insofar, the answer is "maybe".

Under Linux, last time I looked, the default prefetch size was 31 blocks (~127k) -- but this may have changed, plus it's a tuneable parameter. As pages near or at the end of the prefetched area are touched, more pages are being prefetched asynchronously.

If you have hinted MADV_RANDOM to madvise, prefetching is "less likely to happen", under Linux this completely disables prefetch.

On the other hand, giving the MADV_SEQUENTIAL hint will asynchronously prefetch "more aggressively" beginning from the beginning of the mapping (and may discard accessed pages quicker). Under Linux, "more aggressively" means twice the normal amount.

Giving the MADV_WILLNEED hint suggests (but does not guarantee) that all pages in the given range are loaded as soon as possible (since you're saying you're going to access them). The OS may ignore this, but under Linux, it is treated rather as an order than a hint, up to the process' maximum RSS limit, and an implementation-specified limit (if I remember correctly, 1/2 the amount of physical RAM).

Note that MADV_DONTNEED is arguably implemented wrongly under Linux. The hint is not interpreted in the way specified by POSIX, i.e. you're OK with pages being paged out for the moment, but rather that you mean to discard them. Which makes no big difference for readonly mapped pages (other than a small delay, which you said would be OK), but it sure does matter for everything else.

In particular, using MADV_DONTNEED thinking Linux will release unneeded pages after the OS has written them lazily to disk is not how things work! You must explicitly sync, or prepare for a surprise.

Having called readahead on the file descriptor prior to calling mmap (or alternatively, having had read/written the file previously), the file's contents will in practice indeed be in RAM immediately.

This is, however, only an implementation detail (unified virtual memory system), and subject to memory pressure on the system.

Calling mlock will -- assuming it succeeds² -- immediately load the requested pages into RAM. It blocks until all pages are physically present, and you have the guarantee that the pages will stay in RAM until you unlock them.

---

¹ There exist functionality to query (mincore) whether any or all of the pages in a particular range are actually present at the very moment, and functionality to hint the OS about what you would like to see happening without any hard guarantees (madvise), and finally functionality to force a limited subset of pages to be present in memory (mlock) for privilegued processes.

² It might not, both for lack of privilegues and for exceeding quotas or the amount of physical RAM present.

Sorting file using system and library functions

I put fseek(file, 0, SEEK_CUR); after every fread() and fwrite() functions and that worked for me. I don't know why.

---

Related Topics