How to Avoid .Pyc Files

How to avoid .pyc files?

From "What’s New in Python 2.6 - Interpreter Changes":

Python can now be prevented from
writing .pyc or .pyo files by
supplying the -B switch to the Python
interpreter, or by setting the
PYTHONDONTWRITEBYTECODE environment
variable before running the
interpreter. This setting is available
to Python programs as the
sys.dont_write_bytecode variable, and
Python code can change the value to
modify the interpreter’s behaviour.

So run your program as python -B prog.py.

Update 2010-11-27: Python 3.2 addresses the issue of cluttering source folders with .pyc files by introducing a special __pycache__ subfolder, see What's New in Python 3.2 - PYC Repository Directories.

NOTE: The default behavior is to generate the bytecode and is done for "performance" reasons (for more information see here for python2 and see here for python3).

• The generation of bytecode .pyc files is a form of caching (i.e. greatly improves average performance).
• Configuring python with PYTHONDONTWRITEBYTECODE=1 can be bad for python performance (for python2 see https://www.python.org/dev/peps/pep-0304/ and for python3 see https://www.python.org/dev/peps/pep-3147/ ).
• If you are interested in the performance impact please see here https://github.com/python/cpython .

Disabling pyc files in production

Disabling the writing and use of compiled bytecode comes with a performance cost at startup -- your Python code is loaded into memory when the process spawns, and non-.pyc/.pyo files means the interpreter is forced to parse every imported file on startup. If you're currently removing all .pyc and .pyo files from your code directory and your imports, you're forcing a version of this already.

I'm curious as to where they've caused grief in the past (are you disabling modified times on the filesystem?) as if the .py file is newer than the .pyc file, the Python interpreter (at least in common implementations, including CPython), will re-compile the source, but that's not very important for the scope of this question.

If you're importing behind a function, e.g.:

def my_database_call(budget_id):
    import some_expensive_module_to_import
    some_orm(...).get(budget_id)

that import cost won't be incurred until that function is called (and perhaps, every subsequent time as well). This is not markedly different than if you were to allow bytecode (.pyc or "optimized" .pyo files), but at least in that case, you're paying to that import/parse/compile price only once.

In the case you brought up in the comments, if you're "shelling out" to the OS to call a Python script, and are not allowing bytecode to be written in that call, every call incurs the compilation price.

Avoid .pyc Files in Python 3.x?

I believe that environment variable must be set before the Python interpreter starts. Setting it from within a Python script seems to be too late (the interpreter has already loaded the script and generated the bytecode file).

Ignore .pyc files in git repository

Put it in .gitignore. But from the gitignore(5) man page:

  ·   If the pattern does not contain a slash /, git treats it as a shell
       glob pattern and checks for a match against the pathname relative
       to the location of the .gitignore file (relative to the toplevel of
       the work tree if not from a .gitignore file).

  ·   Otherwise, git treats the pattern as a shell glob suitable for
       consumption by fnmatch(3) with the FNM_PATHNAME flag: wildcards in
       the pattern will not match a / in the pathname. For example,
       "Documentation/*.html" matches "Documentation/git.html" but not
       "Documentation/ppc/ppc.html" or
       "tools/perf/Documentation/perf.html".

So, either specify the full path to the appropriate *.pyc entry, or put it in a .gitignore file in any of the directories leading from the repository root (inclusive).

How to avoid .pyc files using selenium webdriver/python while running test suites?

pyc files are created any time you import a module, but not when you run a module directly as a script. That's why you're seeing them when you import the modules with the test code but don't see them created when you run the modules separately.

If you're invoking Python from the command line, you can suppress the creating of pyc files by using the -B argument. You can also set the environment variable PYTHONDONTWRITEBYTECODE with the same effect. I don't believe there's a way to change that setting with Python code after the interpreter is running.

In Python 3.2 and later, the pyc files get put into a separate __pycache__ folder, which might be visually nicer. It also allows multiple pyc files to exist simultaneously for different interpreter versions that have incompatible bytecode (a "tag" is added to the file name indicating which interpreter uses each file).

But even in earlier versions of Python, I think that saying the pyc files are "trashing" the directory is a bit hyperbolic. Usually you can exempt the created files from source control (e.g. by listing .pyc in a .gitignore or equivalent file), and otherwise ignore them. Having pyc files around speeds up repeated imports of the file, since the interpreter doesn't need to recompile the source to bytecode if the pyc file already has the bytecode available.

Best way to avoid stale *.pyc files?

There is a useful environment variable for that: PYTHONDONTWRITEBYTECODE

export PYTHONDONTWRITEBYTECODE=true

Make Python ignore .pyc files

You could use the standard Python library's imp module to reimplement __builtins__.__import__, which is the hook function called by import and from statement. In particular, the imp.load_module function can be used to load a .py even when the corresponding .pyc is present. Be sure to study carefully all the docs in the page I've pointed to, plus those for import, as it's kind of a delicate job. The docs themselves suggest using import hooks instead (per PEP 302) but for this particular task I suspect that would be even harder.

BTW, likely causes for your observed problems include race conditions between different computers trying to write .pyc files at the same time -- NFS locking is notoriously flaky and has always been;-). As long as every Python compiler you're using is at the same version (if not, you're in big trouble anyway;-), I'd rather precompile all of those .py files into .pyc and make their directories read-only; the latter seems the simplest approach anyway (rather than hacking __import__), even if for some reason you can't precompile.

Python3 project remove __pycache__ folders and .pyc files

I found the answer myself when I mistyped pyclean as pycclean:

    No command 'pycclean' found, did you mean:
     Command 'py3clean' from package 'python3-minimal' (main)
     Command 'pyclean' from package 'python-minimal' (main)
    pycclean: command not found

Running py3clean . cleaned it up very nicely.

---

Related Topics