How to Create a Virtual File

How to create a virtual file?

You can store the data in an NTFS stream. That way you can get a real path pointing to your data that you can give to your dll in the form of

x:\myfile.exe:mystreamname

This works precisely like a normal file, however it only works if the file system used is NTFS. This is standard under Windows nowadays, but is of course not an option if you want to support older systems or would like to be able to run this from a usb-stick or similar. Note that any streams present in a file will be lost if the file is sent as an attachment in mail or simply copied from a NTFS partition to a FAT32 partition.

I'd say that the most compatible way would be to write your data to an actual file, but you can of course do it one way on NTFS systems and another on FAT systems. I do recommend against it because of the added complexity. The appropriate way would be to distribute your files separately of course, but since you've indicated that you don't want this, you should in that case write it to a temporary file and give the dll the path to that file. Make sure you write the temporary file to the users' temp directory (you can find the path using GetTempPath in C/C++).

Your other option would be to write a filesystem filter driver, but that is a road that I strongly advise against. That sort of defeats the purpose of using a single file as well...

Also, in case you want only a single file for distribution, how about using a zip file or an installer?

How to construct an in-memory virtual file system and then write this structure to disk

You can use fs.copy.copy_fs() to copy from one filesystem to another, or fs.move.move_fs() to move the filesystem altogether.

Given that PyFilesystem also abstracts around the underlying system filesystem - OSFS - in fact, it's the default protocol, all you need is to copy your in-memory filesystem (MemoryFS) to it and, in effect, you'll have it written to the disk:

import fs
import fs.copy

mem_fs = fs.open_fs('mem://')
mem_fs.makedirs('fruit')
mem_fs.makedirs('vegetables')
with mem_fs.open('fruit/apple.txt', 'w') as apple:
    apple.write('braeburn')

# write to the CWD for testing...
with fs.open_fs(".") as os_fs:  # use a custom path if you want, i.e. osfs://<base_path>
    fs.copy.copy_fs(mem_fs, os_fs)

How to create a virtual file system with offline files (similar to OneDrive) in .NET?

To build an OneDrive-like client you need to create the Sync Provider using cloud filter API, which is mostly programmable via Win32 (with some exceptions, see below), but you can build a complete OneDrive analog in .NET, including "Always keep on this device" menu and on-demand loading.

Here is the functionality that is available via .NET, so you do not need to PInvoke it:

• Sync root registration can be done in .NET, via the StorageProviderSyncRootManager and StorageProviderSyncRootInfo classes.
• You can create properties in .NET using StorageProviderItemProperty class and IStorageProviderItemPropertySource interface.
• Icons in Windows File Manager on files and folders can be set via StorageProviderItemProperties.SetAsync() method and StorageFile/StorageFolder class.
• You can read and set file attributes in .NET using File.GetAttributes() / File.SetAttributes(). You will need them to read offline attribute, pinned and unpinned attributes.
• File system path to URI mapping can be done via implementing IStorageProviderUriSource interface.

Here is functionality available via Win32 functions only, you will need to call it via PInvoke:

• Folders on-demand enumeration, on-demand file content loading is available via Win32 callback only. You will need to register callback using CfConnectSyncRoot().
• Files hydration/dehydration via Windows Explorer menus. You will need to monitor the file system and call CfHydratePlaceholder() and CfUpdatePlaceholder().
• Download progress reporting, error reporting, notifications to Windows, sync provider status reporting. To report progress use the CfReportProviderProgress(). Errors are reported via CfExecute().
• Detection is the file is a regular file/folder or a placeholder file/folder. Can be done via CfGetPlaceholderStateFromFileInfo() call.
• Creation of placeholders and update of placeholder's info. Can be done using CfCreatePlaceholders() and CfUpdatePlaceholder() functions.
• Converting files/folders into placeholders and back. Can be done using CfConvertToPlaceholder() and CfRevertPlaceholder().
• Setting and reading placeholder statuses, such as in-sync/not in-sync. Can be done using CfGetPlaceholderStateFromFileInfo() and CfGetPlaceholderInfo().

Here is what you can do to write all code in .NET:

• One option is to import all required Win32 functions using extern to .NET, for example:

   [DllImport("cldapi.dll", SetLastError = true, ExactSpelling = true)]
   public static extern int CfGetPlaceholderStateFromFileInfo(IntPtr infoBuffer, FILE_INFO_BY_HANDLE_CLASS infoClass);
  
   [DllImport("cldapi.dll", SetLastError = true, ExactSpelling = true)]
   public static extern HRESULT CfSetPinState(IntPtr fileHandle, int pinState, int pinFlags, IntPtr overlapped);
  

• Another option could be using this sample.

Some more notes on building an OneDrive-like file system:

• The "Always keep on this device"/"Free up space" shows automatically on sync root registration. But it only sets Pinned and Unpinned file attributes. You need to monitor the file system, examine the pinned/unpinned attributes, and hydrate/dehydrate each file.
• The on-demand folders listing is done inside the CF_CALLBACK_TYPE_FETCH_PLACEHOLDERS callback. You need to register callback using CfConnectSyncRoot() and listen to it in .NET code.
• The on-demand file content download (hydration) is done inside the CF_CALLBACK_TYPE_FETCH_DATA callback. You need to register and listen to it in .NET code.

How to do virtual file processing?

You might want to consider using a tempfile.SpooledTemporaryFile which gives you the best of both worlds in the sense that it will create a temporary memory-based virtual file initially but will automatically switch to a physical disk-based file if the data held in memory exceeds a specified size.

Another nice feature is that (when using memory) it will automatically use either an io.BytesIO or io.StringIO depending on what mode is being used—allowing you to either read and write Unicode strings or binary data (bytes) to it.

The only tricky part might be the fact that you'll need to avoid closing the file between steps because doing so would cause it to be deleted from memory or disk. Instead you can just rewind it back to the beginning with a file seek(0) method call.

When you are completely done with the file and close it, it will automatically be deleted from disk if the amount of data in it caused it to be rolled-over to a physical file.

How to create a fake / virtual file?

It looks like you just need to implement a public url_stat() method on VirtualFileWrapper for it to pass the file_exists() check.

To silence warnings and errors from include and require, you must implement stream_read() and stream_eof() methods:

class VirtualFileWrapper
{
    public $context;

    public function stream_open( $path, $mode, $options, &$opened_path )
    {
        return TRUE;
    }

    public function stream_stat()
    {
        var_dump( __METHOD__ );
        return [];
    }

    public function url_stat()
    {
        return array (
          0 => 0,
          1 => 0,
          2 => 0,
          3 => 0,
          4 => 0,
          5 => 0,
          6 => 0,
          7 => 0,
          8 => 0,
          9 => 0,
          10 => 0,
          11 => 0,
          12 => 0,
          'dev' => 0,
          'ino' => 0,
          'mode' => 0,
          'nlink' => 0,
          'uid' => 0,
          'gid' => 0,
          'rdev' => 0,
          'size' => 0,
          'atime' => 0,
          'mtime' => 0,
          'ctime' => 0,
          'blksize' => 0,
          'blocks' => 0
        );
    }

    public function stream_read(){
        return '';
    }

    public function stream_eof(){
        return '';
    }

}

stream_wrapper_register( 'virtual', 'VirtualFileWrapper' );

$file = fopen( "virtual://foo", 'r+' );

// Executes VirtualFileWrapper::stream_stat()
fstat( $file );

// Executes no VirtualFileWrapper method
file_exists("virtual://foo");

//Still no errors :-)!
require "virtual://foo";
include "virtual://foo";

Take care to pass file_exists() a string, rather than the resource you created with fopen().

How to load a zip file with pyscript and save into the virtual file system

You were very close with your code. The problem was in converting the file data to the correct data type. The requirement is to convert the arrayBuffer to Uint8Array and then to a bytearray.

Import the required function:

from js import Uint8Array

Read the file data into an arrayBuffer and copy it to a new Uint8Array

data = Uint8Array.new(await f.arrayBuffer())

Convert the Uint8Array to a bytearray that BytesIO expects

mf = io.BytesIO(bytearray(data))

---

Related Topics