Windows: Moving a File That Was Previously Mapped in Memory Fails

Windows: moving a file that was previously mapped in memory fails

In Java, file mappings are garbage collected, and there is no supported way to forcibly destroy a mapping.

From the FileChannel.map() documentation:

The buffer and the mapping that it represents will remain valid until
the buffer itself is garbage-collected.

A mapping, once established, is not dependent upon the file channel
that was used to create it. Closing the channel, in particular, has
no effect upon the validity of the mapping.

In Sun's JDK, you can test that this is indeed the culprit by forcibly destroying the mapping just before doing the file move:

import sun.nio.ch.DirectBuffer;
import sun.misc.Cleaner;
[...]
if (byteBuffer.isDirect()) {
    Cleaner cleaner = ((DirectBuffer) byteBuffer).cleaner();
    cleaner.clean();
}
// move file

Why does Java FileChannel.truncate fail predictably on Windows only?

There doesn't appear to be a proper answer / solution to my question, but at least I came across some details which inform the problem, thanks to the link @diginoise posted as a comment.

I was getting the exception because I was trying to truncate a file for which there was a memory mapped byte buffer still "active". Apparently there is no way for a user to manually unmap / release Windows' handle on that memory (see JDK bug in oracle's bug database).

In the end, I think the answer is simply to NOT try truncate / delete these files when I'm done with them but rather keep track of them and process them next time the application starts, that way they're guaranteed to not have active memory maps.

By the way, a File.deleteOnExit() also fails under these memory mapped file conditions.

Until I implement the more permanent fix of handling the files on next startup, I've elected to manually unmap the memory via reflection.

This mitigation does function without throwing errors, it does allow truncation, and does allow deleteOnExit, but this is a short-term unsustainable workaround. It's not good practice to be using reflection to invoke a method that may not be there depending on the FileChannel implementation. With another implementation of FilChannel I might have broken temp files laying around that never get cleaned up.

Python writing into a mapped file - strange behaviour

I do not have Windows so I cannot quite test the behaviour, but I believe this happens because Windows does not quite follow the POSIX semantics and open(name, 'wb') will, instead of truncating the existing file, open it with CREATE_ALWAYS which would conflict with the file being mapped in another process. The ab mode could work too, but... as the Python documentation says

'a' for appending (which on some Unix systems, means that all writes append to the end of the file regardless of the current seek position).

Unfortunately the open function uses C semantics for the flag, and hence it is not possible to specify the desired mode as "open the file for random access writing only without truncating the file", so the best you can do is "open it for both reading and writing without truncating the file", which would be r+b.

---

As user Eryk Sun pointed out, the problem is that Windows does not support file truncation at all if there are any existing memory mappings:

If
CreateFileMapping is called to create a file mapping object for hFile, UnmapViewOfFile must be called first to unmap all views and call CloseHandle to close the file mapping object before you can call SetEndOfFile.

Likewise there is no mapping of space larger than the existing file - if the mapping is longer than the file size, the file is extended instead...

---

On POSIXly correct platforms, a program like

#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <sys/mman.h>
#include <time.h>
#include <unistd.h>


int main(int argc, char const *argv[])
{
    struct stat s;
    int fd = open("a.bin", O_RDWR);
    fstat(fd, &s);
    unsigned char *d = mmap(0, s.st_size, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
    getchar();

    for (int i = 0; i < s.st_size; i++) {
        putchar(d[i]);
    }

    putchar('\n');
    fflush(stdout);
    getchar();
}

will not interfere with running the said Python program. However if this C program accesses the file within the within the window while it has been truncated to 0, SIGBUS signal will be raised in the C process.

FILE_FLAG_DELETE_ON_CLOSE and memory mapped files

FILE_FLAG_DELETE_ON_CLOSE follows the unfortunate Windows tradition of referring to an unlink operation as "delete". In fact, the flag only causes the file to be unlinked from the specified directory when the file is closed.

Like other operating systems, Windows only gives ordinary user code the ability to unlink a file from a particular directory. Deleting is always the operating system's decision, taking place when the file can no longer be referenced in any way.

If you look, you'll see the file has in fact been unlinked from the directory, but it will not actually be deleted (and the space the data takes on disk available for re-use) until its reference count drops to zero. The mapping holds a reference.

Trying to create a global file mapping object from a user-mode process fails

Only administrators, and services running in session 0, can gain the SeCreateGlobalPrivilege privilege needed to create file mappings in the Global namespace. Assuming you do not want to re-write your code into a service, you will have to spawn a separate elevated process to create the file mapping.

Is it possible to create smaller memory-mapped views on a 2GB+ file?

I can't see your pastebin link, but I can suggest a simple solution with a c++ class declaration. I think the implementation should be obvious from the comments:

class ShiftingMemMap
{
public:
    // constructs a dynamically shifting memory map of a file...
    ShiftingMemMap ( const char* fileName, size_t view_size = 4096 );

    // retrieve/set a byte at the given file offset. If the offset is not currently in-view,
    // shift the view to encompass the offset. The reference should not be stored for later
    // access because the view may need to shift again...
    byte& operator [] ( unsigned int_64_t offset );

private:
    int_64_t current_offset;
    size_t current_size;
};

All that being said, you could write a class that returns multiple views of a file to allow saving a reference for later and also editing different parts of the file simultaneously without having to shift the view back and forth repeatedly.

class MemMap
{
public:
    MemMap ( const char* filename );

    shared_ptr<MemMapView> View ( unsigned int_64_t offset, size_t size = 4096 );
};

class MemMapView
{
public:
    char& operator[] ( size_t offset );
};

File.Move fails when preceded by a File.Delete

Can you reverse the logic?

File.Copy (source, target, true) 

to overwrite the target then

File.Delete(source)

---

Related Topics