Python: Can Executable Zip Files Include Data Files

python: can executable zip files include data files?

You could use pkg_resources functions to access files:

# __main__.py
import pkg_resources
from PIL import Image

print pkg_resources.resource_string(__name__, 'README.txt')

im = Image.open(pkg_resources.resource_stream('app', 'im.png'))
im.rotate(45).show()

Where zipfile contains:

.
|-- app
|   |-- im.png
|   `-- __init__.py
|-- README.txt
`-- __main__.py

To make zipfile executable, run:

$ echo '#!/usr/bin/env python' | cat - zipfile > program-name
$ chmod +x program-name

To test it:

$ cp program-name /another-dir/
$ cd /another-dir && ./program-name

Distributing an executable zip file with __main__.py, how to access extra data?

I figured out by myself, It's sufficient to use pkgutil.get_data to access the data inside a package.

I want to bundle a zip file into my python executable

So after some more trial and error I got it working.
In your script, where you access a file, in my case data.zip. do it like this:

with zipfile.ZipFile(os.path.join(sys._MEIPASS, "data.zip"), 'r') as zip_ref:
                zip_ref.extractall(pte2)

How can you bundle all your python code into a single zip file?

You can automate most of the work with regular python tools. Let's start with clean virtualenv.

[zart@feena ~]$ mkdir ziplib-demo
[zart@feena ~]$ cd ziplib-demo
[zart@feena ziplib-demo]$ virtualenv .
New python executable in ./bin/python
Installing setuptools.............done.
Installing pip...............done.

Now let's install set of packages that will go into zipped library. The trick is to force installing them into specific directory.

(Note: don't use --egg option either on command-line or in pip.conf/pip.ini because it will break file layout making it non-importable in zip)

[zart@feena ziplib-demo]$ bin/pip install --install-option --install-lib=$PWD/unpacked waitress
Downloading/unpacking waitress
  Downloading waitress-0.8.5.tar.gz (112kB): 112kB downloaded
  Running setup.py egg_info for package waitress

Requirement already satisfied (use --upgrade to upgrade): setuptools in ./lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg (from waitress)
Installing collected packages: waitress
  Running setup.py install for waitress

    Installing waitress-serve script to /home/zart/ziplib-demo/bin
Successfully installed waitress
Cleaning up...

Update: pip now has -t <path> switch, that does the same thing as --install-option --install-lib=.

Now let's pack all of them into one zip

[zart@feena ziplib-demo]$ cd unpacked
[zart@feena unpacked]$ ls
waitress  waitress-0.8.5-py2.7.egg-info
[zart@feena unpacked]$ zip -r9 ../library.zip *
  adding: waitress/ (stored 0%)
  adding: waitress/receiver.py (deflated 71%)
  adding: waitress/server.pyc (deflated 64%)
  adding: waitress/utilities.py (deflated 62%)
  adding: waitress/trigger.pyc (deflated 63%)
  adding: waitress/trigger.py (deflated 61%)
  adding: waitress/receiver.pyc (deflated 60%)
  adding: waitress/adjustments.pyc (deflated 51%)
  adding: waitress/compat.pyc (deflated 56%)
  adding: waitress/adjustments.py (deflated 60%)
  adding: waitress/server.py (deflated 68%)
  adding: waitress/channel.py (deflated 72%)
  adding: waitress/task.pyc (deflated 57%)
  adding: waitress/tests/ (stored 0%)
  adding: waitress/tests/test_regression.py (deflated 63%)
  adding: waitress/tests/test_functional.py (deflated 88%)
  adding: waitress/tests/test_parser.pyc (deflated 76%)
  adding: waitress/tests/test_trigger.pyc (deflated 73%)
  adding: waitress/tests/test_init.py (deflated 72%)
  adding: waitress/tests/test_utilities.pyc (deflated 78%)
  adding: waitress/tests/test_buffers.pyc (deflated 79%)
  adding: waitress/tests/test_trigger.py (deflated 82%)
  adding: waitress/tests/test_buffers.py (deflated 86%)
  adding: waitress/tests/test_runner.py (deflated 75%)
  adding: waitress/tests/test_init.pyc (deflated 69%)
  adding: waitress/tests/__init__.pyc (deflated 21%)
  adding: waitress/tests/support.pyc (deflated 48%)
  adding: waitress/tests/test_utilities.py (deflated 73%)
  adding: waitress/tests/test_channel.py (deflated 87%)
  adding: waitress/tests/test_task.py (deflated 87%)
  adding: waitress/tests/test_functional.pyc (deflated 82%)
  adding: waitress/tests/__init__.py (deflated 5%)
  adding: waitress/tests/test_compat.pyc (deflated 53%)
  adding: waitress/tests/test_receiver.pyc (deflated 79%)
  adding: waitress/tests/test_adjustments.py (deflated 78%)
  adding: waitress/tests/test_adjustments.pyc (deflated 74%)
  adding: waitress/tests/test_server.pyc (deflated 73%)
  adding: waitress/tests/fixtureapps/ (stored 0%)
  adding: waitress/tests/fixtureapps/filewrapper.pyc (deflated 59%)
  adding: waitress/tests/fixtureapps/getline.py (deflated 37%)
  adding: waitress/tests/fixtureapps/nocl.py (deflated 47%)
  adding: waitress/tests/fixtureapps/sleepy.pyc (deflated 44%)
  adding: waitress/tests/fixtureapps/echo.py (deflated 40%)
  adding: waitress/tests/fixtureapps/error.py (deflated 52%)
  adding: waitress/tests/fixtureapps/nocl.pyc (deflated 48%)
  adding: waitress/tests/fixtureapps/getline.pyc (deflated 32%)
  adding: waitress/tests/fixtureapps/writecb.pyc (deflated 42%)
  adding: waitress/tests/fixtureapps/toolarge.py (deflated 37%)
  adding: waitress/tests/fixtureapps/__init__.pyc (deflated 20%)
  adding: waitress/tests/fixtureapps/writecb.py (deflated 50%)
  adding: waitress/tests/fixtureapps/badcl.pyc (deflated 44%)
  adding: waitress/tests/fixtureapps/runner.pyc (deflated 58%)
  adding: waitress/tests/fixtureapps/__init__.py (stored 0%)
  adding: waitress/tests/fixtureapps/filewrapper.py (deflated 74%)
  adding: waitress/tests/fixtureapps/runner.py (deflated 41%)
  adding: waitress/tests/fixtureapps/echo.pyc (deflated 42%)
  adding: waitress/tests/fixtureapps/groundhog1.jpg (deflated 24%)
  adding: waitress/tests/fixtureapps/error.pyc (deflated 48%)
  adding: waitress/tests/fixtureapps/sleepy.py (deflated 42%)
  adding: waitress/tests/fixtureapps/toolarge.pyc (deflated 43%)
  adding: waitress/tests/fixtureapps/badcl.py (deflated 45%)
  adding: waitress/tests/support.py (deflated 52%)
  adding: waitress/tests/test_task.pyc (deflated 78%)
  adding: waitress/tests/test_channel.pyc (deflated 78%)
  adding: waitress/tests/test_regression.pyc (deflated 68%)
  adding: waitress/tests/test_parser.py (deflated 80%)
  adding: waitress/tests/test_server.py (deflated 78%)
  adding: waitress/tests/test_receiver.py (deflated 87%)
  adding: waitress/tests/test_compat.py (deflated 51%)
  adding: waitress/tests/test_runner.pyc (deflated 72%)
  adding: waitress/__init__.pyc (deflated 50%)
  adding: waitress/channel.pyc (deflated 58%)
  adding: waitress/runner.pyc (deflated 54%)
  adding: waitress/buffers.py (deflated 74%)
  adding: waitress/__init__.py (deflated 61%)
  adding: waitress/runner.py (deflated 58%)
  adding: waitress/parser.py (deflated 69%)
  adding: waitress/compat.py (deflated 69%)
  adding: waitress/buffers.pyc (deflated 69%)
  adding: waitress/utilities.pyc (deflated 60%)
  adding: waitress/parser.pyc (deflated 53%)
  adding: waitress/task.py (deflated 72%)
  adding: waitress-0.8.5-py2.7.egg-info/ (stored 0%)
  adding: waitress-0.8.5-py2.7.egg-info/dependency_links.txt (stored 0%)
  adding: waitress-0.8.5-py2.7.egg-info/installed-files.txt (deflated 83%)
  adding: waitress-0.8.5-py2.7.egg-info/top_level.txt (stored 0%)
  adding: waitress-0.8.5-py2.7.egg-info/PKG-INFO (deflated 65%)
  adding: waitress-0.8.5-py2.7.egg-info/not-zip-safe (stored 0%)
  adding: waitress-0.8.5-py2.7.egg-info/SOURCES.txt (deflated 71%)
  adding: waitress-0.8.5-py2.7.egg-info/entry_points.txt (deflated 33%)
  adding: waitress-0.8.5-py2.7.egg-info/requires.txt (deflated 5%)
[zart@feena unpacked]$ cd ..

Note that those files should be at top of zip, you can't just zip -r9 library.zip unpacked

Checking the result:

[zart@feena ziplib-demo]$ PYTHONPATH=library.zip python
Python 2.7.1 (r271:86832, Apr 12 2011, 16:15:16)
[GCC 4.6.0 20110331 (Red Hat 4.6.0-2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import waitress
>>> waitress
<module 'waitress' from '/home/zart/ziplib-demo/library.zip/waitress/__init__.pyc'>
>>>
>>> from wsgiref.simple_server import demo_app
>>> waitress.serve(demo_app)
serving on http://0.0.0.0:8080
^C>>>

Update: since python 3.5 there is also zipapp module which can help with bundling the whole package into .pyz file. For more complex needs pyinstaller, py2exe or py2app might better fit the bill.

How does the Python interpreter detect that it was called with a ZIP archive instead of a source file?

I wondered the same and found:

You are correct that prepended data is allowed, and this is explicitly mentioned in the docs:

Python has been able to execute zip files which contain a __main__.py file since version 2.6. In order to be executed by Python, an application archive simply has to be a standard zip file containing a __main__.py file [...]

The zip file format allows arbitrary data to be prepended to a zip file.

You are also correct in guessing that Python looks for a ZIP central directory. This happens in zipimport.py, which looks for STRING_END_ARCHIVE = b'PK\x05\x06' near the end of the file.

The contents of the archive, such as uncompressed Python code files, does therefore not affect the detection of the zip file.

A demonstation:

$ echo 'print("hello")' > script.py
$ python script.py
hello
$ echo 'print("hi")' > __main__.py
$ zip app.zip __main__.py
  adding: __main__.py (stored 0%)
$ dd if=app.zip >> script.py
0+1 records in
0+1 records out
184 bytes transferred in 0.000066 secs (2786108 bytes/sec)
$ zip -A script.py
Zip entry offsets appear off by 15 bytes - correcting...
$ head -n 1 script.py 
print("hello")
$ unzip -l script.py 
Archive:  script.py
  Length      Date    Time    Name
---------  ---------- -----   ----
       12  08-04-2022 23:02   __main__.py
---------                     -------
       12                     1 file
$ python script.py 
hi

Having py2exe include my data files (like include_package_data)

I ended up solving it by giving py2exe the option skip_archive=True. This caused it to put the Python files not in library.zip but simply as plain files. Then I used data_files to put the data files right inside the Python packages.

---

Related Topics