How to Do Virtual File Processing

How to do virtual file processing?

You might want to consider using a tempfile.SpooledTemporaryFile which gives you the best of both worlds in the sense that it will create a temporary memory-based virtual file initially but will automatically switch to a physical disk-based file if the data held in memory exceeds a specified size.

Another nice feature is that (when using memory) it will automatically use either an io.BytesIO or io.StringIO depending on what mode is being used—allowing you to either read and write Unicode strings or binary data (bytes) to it.

The only tricky part might be the fact that you'll need to avoid closing the file between steps because doing so would cause it to be deleted from memory or disk. Instead you can just rewind it back to the beginning with a file seek(0) method call.

When you are completely done with the file and close it, it will automatically be deleted from disk if the amount of data in it caused it to be rolled-over to a physical file.

Running a process inside a virtual file system?

I think I'm going to stick with Window's mklink command. It seems to suit my needs the best.

What I'm going to do is use QFile::link() on all operating systems that aren't Windows, and QProcess with mklink on windows. This should work on every operating system.

For a good example look here: https://stackoverflow.com/a/21013935/979732

Passing virtual file to a Process in Java

If you have the data already, and the external process can accept the data from standard input, you can stream the data directly in to the process, skipping the file writing entirely.

You can get the input stream from the Process, and look into Pipe[Input|Output]Stream in Java to tie it all together.

Is there a library in Python which allows virtual file system management in single file?

You can use SVFS package.

SVFS allows to create virtual filesystem inside file on real filesystem. It can be used to store multiple files inside single file (with directory structure). Unlike archives, SVFS allows to modify files in-place. SVFS files use file-like interface, so they can be used (pretty much) like regular Python file objects. Finally, it’s implemented in pure python and doesn’t use any 3rd party modules, so it should be very portable. Tests show write speed to be around 10-12 MB/s and read speed to be around 26-28 MB/s.

c preprocessor with virtual file system

Here is snipped for hooking file loading in boost::wave (version 1.53)

class custom_directives_hooks
   : public boost::wave::context_policies::default_preprocessing_hooks
{
public:
#if BOOST_WAVE_USE_DEPRECIATED_PREPROCESSING_HOOKS != 0
   void found_include_directive(std::string const& filename, bool include_next)
   {}
#else
   template <typename ContextT>
   bool found_include_directive(ContextT const& ctx, std::string const& filename, bool include_next)
   {
      return false; // ok to include this file
   }
#endif

   template <typename ContextT>
   bool locate_include_file(ContextT& ctx, std::string &file_path,
      bool is_system, char const *current_name, std::string &dir_path,
      std::string &native_name)
   {
      //write code here to locate file
      return true; //or false if file is not found
   }
}

void main()
{
   //...
   //typedef for boost::wave context with hooks for file loading
   typedef boost::wave::cpplexer::lex_iterator< boost::wave::cpplexer::lex_token<> >
      lex_iterator_type;
   typedef boost::wave::context<
      std::string::iterator, lex_iterator_type,
      boost::wave::iteration_context_policies::load_file_to_string,
      custom_directives_hooks>
     context_type;
   //...
}

---

Related Topics