Is It Feasible to Compile Python to MAChine Code

Is it feasible to compile Python to machine code?

Try ShedSkin Python-to-C++ compiler, but it is far from perfect. Also there is Psyco - Python JIT if only speedup is needed. But IMHO this is not worth the effort. For speed-critical parts of code best solution would be to write them as C/C++ extensions.

Compiling Python to native code?

Shed Skin can compile Python to C++, but only a restricted subset of it. Some aspects of Python are very difficult to compile to native code.

How does Python Compiler and Virtual Machine handle eval expressions?

The Python compiler doesn't care where the code it's compiling comes from. It can compile code that comes from a file before execution when you run python filename, it can compile files during execution when you use import, and it can compile code from a string expression when you call eval() or exec(). These functions invoke the compiler dynamically.

What's preventing python from being compiled?

Python, the language, like any programming language, is not in itself compiled or interpreted. The standard Python implementation, called CPython, compiles Python source to bytecode automatically and executes that via a virtual machine, which is not what is usually meant by "interpreted".

There are implementations of Python which compile to native code. For example, the PyPy project uses JIT compilation to get the benefits of CPython's ease of use combined with native code performance.

Cython is another hybrid approach, generating and compiling C code on the fly from a dialect of Python.

However, because Python is dynamically typed, it's not generally practical to completely precompile all possible code paths, and it won't ever be as fast as mainstream statically typed languages, even if JIT-compiled.

How can I compile a python file executable and different architectures

I began expanding this answer today to provide sample code. In the course of doing so I stumbled over a recent blog post by Gregory Szorc for his PyOxidizer tool, which on the surface appears to do more of what you were looking for in the OP. It uses Rust to boil Python down to machine code, and while it still technically requires a runtime, it embeds that runtime in the executable so that you don't need to bundle files. I'd be interested in your thoughts if you try this out.

Since I still think my previous answer is potentially useful, here's the update:

General Answer

I don't know of a 100% coverage solution for this. I think it depends a lot on your code, complexity, dependencies, and goals.

You mentioned trying Cython. I think there's a little more to it than what you're seeing. I'm not familiar with Cython; it's just an option that comes up when I read about the topic. But I gather that you can use it to build a native executable, subject to runtime requirements.

See Is it feasible to compile Python to machine code? for some other options.

You also can build an excutable yourself, by writing a small C program that instantiates a python interpreter and loads compiled python bytecode from a separate file or a compiled-in bytestring. Details for this are buried in the Python embedding/extending documentation.

Runtime issues

It is easy to build Python code into a native machine executable (this is called embedding), but you're right (in your comment) that you still need a Python runtime. But this is a complexity problem. For a small enough use case, e.g. your simple hello world, you can bundle the elements that you actually depend on into your code. (For example, link statically to the python library.)

If you're looking for an arbitrarily-scalable solution, then one common solution — I've seen this from commercial games to backup software — is to bundle a sufficient python binary runtime pre-compiled for your target platform, with an installation prefix that you can predict or control (i.e. not /usr). This bundle usually includes any machine code (shared libraries) and compiled python bytecode as needed (.pyc or .pyo), without any .py files. This doesn't necessarily include the totality of the python installation — just the machine code and the bytecode required for the application.

Embedding Example

Here's a really simple hello world, written in Python but embedded into a C skeleton:

#include <Python.h>

int
main(int argc, char *argv[])
{
    wchar_t *name = Py_DecodeLocale(argv[0], NULL);
    if (name == NULL) {
        fprintf(stderr, "error: cannot decode program name\n");
        exit(1);
    }

    /* Setup */
    Py_SetProgramName(name);
    Py_Initialize();

    /* Load and run code */
    if (argc > 1)
        PyRun_SimpleString(argv[1]);
    else
        PyRun_SimpleString("print('Hello, world')");

    /* Importing and running code from a file is also possible.
     * See https://docs.python.org/3/c-api/veryhigh.html
     * and https://docs.python.org/3/c-api/import.html
     */

    /* Clean up */
    Py_Finalize();
    PyMem_RawFree(name);

    /* Ideally we should get a result from the interpreter
     * and evaluate it, returning conditionally. Py_FinalizeEx()
     * is a good way but it's only in Python 3.6+, so I'm leaving
     * it out of this example. */
    return 0;
}

A makefile for this could go as follows:

PY=python3
PC=$(PY)-config

all: py

py: py.o
        $(CC) $$($(PC) --ldflags) -o $@ $< $$($(PC) --libs)

py.o: py.c
        $(CC) $$($(PC) --cflags) -c $<

Build and run:

$ make
gcc $(python3-config --cflags) -c py.c
gcc $(python3-config --ldflags) -o py py.o $(python3-config --libs)
$ ./py
Hello, world
$ ./py 'print("hi")'
hi

You can run this under strace to see what the runtime requirements are:

$ strace -e trace=open ./py 2>&1 | grep py

... and then you know what files need to be bundled. If you were to pick up each of those in a cpio file, you'd have a portable application. (It just would need to be installed in the same location. Workarounds for that include building python from source with --prefix as mentioned above, and playing odd games with ldconfig and PYTHON_PATH.)

Compiling Python to run on another machine

If you have Python installed on your Raspberry Pi, then from the shell, you just need to run:

# This installs pip (Python installer) as well as the requests library
sudo apt-get install python-pip

Once that is installed, run:

# To install the ImgurClient
pip install imgurpython

Then you can just run your script at the shell by typing:

python your_script_name.py

---

If you do not already have Python installed, just run the following command to install it before the others:

sudo apt-get install python

---

Related Topics