Prevent Python from Caching the Imported Modules

Prevent Python from caching the imported modules

Quitting and restarting the interpreter is the best solution. Any sort of live reloading or no-caching strategy will not work seamlessly because objects from no-longer-existing modules can exist and because modules sometimes store state and because even if your use case really does allow hot reloading it's too complicated to think about to be worth it.

How to cache imported modules once off for multiple runs of a python script?

No, you can't. You would have to use persistent storage, like shelve, or something in-memory such as SQLite, where you'd store any expensive computations that persist between sessions, and subsequently you'd just read those results from memory/disk, depending on your chosen storage.

Moreover, do note modules are, in fact, being cached upon import in order to improve load time, however, not in memory, but rather on disk, as .pyc files under __pychache__ and the import time per se is insignificant in general, so your imports take that long not because of the import itself, but because of the computations inside those modules, so you might want to optimise those.

The reason you can't do what you want is because in order to keep data in memory, the process must keep running. Memory belongs to the process running the script, and once that script finished, the memory is freed. See here for additional details regarding your issue.

You can't just run a script and fill the memory with whatever computations you have until you might run it another time, because, firstly, the memory wouldn't know when that other time would be (it might be 1 min later, it might be 1 year later) and secondly, if you would be able to do that, then imagine how shortly you'd run out of memory when different scripts from different applications across the OS (it's not just your program out there) would fill the memory with the results of their computations.

So you can either run your code in an indefinite loop with sleep (and keep the process active) or you can use a crontab and store your previous results somewhere.

Does python optimize modules when they are imported multiple times?

Python modules could be considered as singletons... no matter how many times you import them they get initialized only once, so it's better to do:

import MyLib
import ReallyBigLib

Relevant documentation on the import statement:

https://docs.python.org/2/reference/simple_stmts.html#the-import-statement

Once the name of the module is known (unless otherwise specified, the term “module” will refer to both packages and modules), searching for the module or package can begin. The first place checked is sys.modules, the cache of all modules that have been imported previously. If the module is found there then it is used in step (2) of import.

The imported modules are cached in sys.modules:

This is a dictionary that maps module names to modules which have already been loaded. This can be manipulated to force reloading of modules and other tricks. Note that removing a module from this dictionary is not the same as calling reload() on the corresponding module object.

Does Python cache imported files?

Both a and b are loaded once. When you import a module, its content is cached so when you load the same module again, you're not calling upon the original script for the import, done using a "finder":

• https://www.python.org/dev/peps/pep-0451/#finder
• https://docs.python.org/3/library/importlib.html#importlib.abc.MetaPathFinder

This works across modules so if you had a d.py of which import b, it will bind to the same cache as an import within c.py.

---

Some interesting builtin modules can help understand what happens during an import:

• https://docs.python.org/3/reference/import.html#importsystem

  When a module is first imported, Python searches for the module and if found, it creates a module object 1, initializing it.

  Notably here the first import, all imports after follow the __import__. Internal caches of finders are stored at sys.meta_path.

• https://docs.python.org/3/library/functions.html#import

You can leverage the import system to invalidate those caches for example:

https://docs.python.org/3/library/importlib.html#importlib.import_module

If you are dynamically importing a module that was created since the interpreter began execution (e.g., created a Python source file), you may need to call invalidate_caches() in order for the new module to be noticed by the import system.

---

The imp (and importlib py3.4+) allows the recompilation of a module after import:

import imp
import a
imp.reload(a)

• https://docs.python.org/3/library/importlib.html#importlib.reload
  
  

  Python module’s code is recompiled and the module-level code re-executed, defining a new set of objects which are bound to names in the module’s dictionary by reusing the loader which originally loaded the module.

• https://docs.python.org/3/library/imp.html

Stop Python3 creating module cache in system directory

You can start python with the -B command-line flag to prevent it from writing cached bytecode.

$ ls
bar.py  foo.py
$ cat foo.py 
import bar
$ python -B foo.py; ls
bar.py  foo.py
$ python foo.py; ls
bar.py  foo.py  __pycache__

Setting the PYTHONDONTWRITEBYTECODE environment variable to a non-empty string or the sys.dont_write_bytecode to True will have the same effect.

Of course, I'd say that the benefits in this case (faster loading times for your app, for free) vastly outweigh the perceived unsightliness you were talking about - but if you really want to disable caching, here's how.

Source: man python

---

Related Topics