A Simple Python Deployment Problem - a Whole World of Pain

A simple Python deployment problem - a whole world of pain

Development and deployment of Python code is made much easier by setuptools in combination with virtualenv and pip.

Core ideas

The trickiest part, I've found, is running a development environment that mirrors the deployed setup as closely as possible, while at the same time respecting Pythonic tools and idioms. But it turns out that this is very easy to achieve with pip and setuptools, which together allow you to "install" a development tree into a Python environment without moving the files around. (Actually setuptools does this all by itself, but pip acts as a front end handle dependencies better.)

Another key issue is preparing a clean environment with a known package set across both environments. Python's virtualenv is a god-send in this respect, allowing you to configure a completely customised Python environment with your own choice of packages, without requiring root access, or OS packages (rpm or dpkg), and without being constrained by whatever packages and versions thereof that happen to be installed on your distro.

Finally, one annoying bug-bear is the difficulty of creating command-line scripts that play nice with PYTHON_PATH. This is also dealt with quite elegantly by setuptools.

Setting up

(To keep things simple, this is fairly prescriptive. Feel free to diverge as appropriate.)

1. Prepare a working directory that your Python stuff can call home.
2. Grab the latest virtualenv from the bottom of this page and untar it (doesn't matter where).

3. From the working directory, setup a new Python virtual environment:

   $ python <untarred_directory>/virtualenv.py venv
   

4. You'll want to do most of your work from inside this virtual environment. Use this command to do so (. is a shortcut for source):

   $ . venv/bin/activate
   

5. Install pip:

   $ easy_install pip
   

6. Create directories for each installable package you want to create.

7. In each directory, you'll need a setup.py that defines the content and structure of the package. The setuptools documentation is a very good resource for getting started on this. It's worth taking the time absorb large chunks of it.

Development

Once your tree structure is ready, you are almost ready to begin coding. But right now, packages that depend on each other can't see each other as they will in the deployed environment. This problem is resolved with a neat little trick that setuptools offers, and which pip makes use of. For each package you are developing, run the following command (make sure you are in the virtual environment for your project, as per step 3, above):

$ pip install -e pkg1

This command will install pkg1 into your virtual environment, and it does so without copying any of your files. It simply adds a link to the site-packages directory pointing to the package's development root and creates an egg-info directory in that root. You can also do this without pip, as follows:

$ cd pkg1
$ python setup.py develop

And it will usually work, but if you have third-party dependencies (which should be listed in setup.py, as explained here in the setuptools documentation), pip is smarter about finding them.

One caveat to note is that neither setuptools nor pip have any smarts about finding dependencies amongst your own packages. If PkgB in directory B, depends on PkgA in directory A, then pip install -e B will fail, because pip has no way of knowing that PkgA can be found in directory A; it will instead try, and fail, to download PkgA from its online repository sources. The workaround is simply to install each package after its dependencies.

At this point, you can start python, load up one of your modules and start toying with it. You can edit code, and it will be immediately available the next time you import it.

Finally, if you want to create command-line tools with your packages. Don't write them by hand. You'll end up with a horrible mess of PYTHON_PATH hacks that never quite works properly. Just read up on automatic script creation in the setuptools documentation. This will spare you a lot of grief.

Deployment

When your packages are ready for action, you can use setup.py to create deployment packages. There are way too many options to go into here, but the following should get you started:

$ cd pkg1
$ python setup.py --help
$ python setup.py --help-commands

Loose ends

Due to the broad nature of the question, this answer is necessarily incomplete. I haven't dealt with long-running servers, web frameworks or the actual deployment process itself (in particular, use of pip install's --index-url to manage a private repository of third-party and internal packages and -e vcs+..., which will pull packages out of svn, git, hg or bzr). But I hope I've given you enough rope to tie it all together (just don't hang yourself with it :-).

Easy Deploying of Python and application in one Bundle , For Linux

Most Python packages can be deployed by creating a lib or similar directory in your deployment, and adding it to sys.path in Python, or PYTHONPATH outside, then copying the package directory (usually inside the directory you unzipped) into that directory. This lets you keep the package with your deployed code, say, in a Mercurial repository.

Deploying Python itself is going to be a bit more hassle, but if you can control where it's installed (such as /usr/local or /opt), then it's just a matter of ./configure --prefix=..., make, and sudo make install. Then you can point your scripts to that Python by starting them with a line like #!/usr/local/bin/python, as long as the script is marked executable.

For example, if you were deploying code that needs docutils, then you'd do something like:

cd projectDir
mkdir -p lib
tar xzvf ~/Downloads/docutils-0.8.tgz
mv docutils-0.8/docutils lib
rm -r docutils-0.8

Then a Python module in this directory would just add the following at the start:

#!/usr/local/bin/python

import os
import sys
sys.path(os.path.join(os.path.dirname(sys.argv[0]), "lib"))
import docutils

file layout and setuptools configuration for the python bit of a multi-language library

The short answer is that none of the Python distribution tools is going to do what you want, the exact way you want it. Even if you use distutils' data_files feature, you're still going to have to have your javascript files copied into your Python project directory (i.e., somewhere under the same directory as your setup.py.)

Given that, you might as well just copy the .js files to your package (i.e. alongside mongofulltextsearch/init.py) as part of your build process, and use package_data or include_package_data=True.

(Or alternatively, you could possibly use symlinks, externals, or some such, if your revision control system supports those. I believe that when building source distributions, the Python distribution tools convert symlinks to real files. At least, you could give that a try.)

any way to composite configuration/.ini files in pylons?

You can use the ConfigParser module.

Deploying Django Application

Can I plug the company I work for, PythonAnywhere? Making Python application deployment easy is pretty much what we're there for :-)

Postgres support (including PosGIS) is currently in beta, but is pretty solid -- if you sign up, just drop us a line and we can enable it for your account.

Which lightweight python web framework has applications and hot-deployed code

No mature python framework that I'm aware of allows you to map urls to python modules, and frankly, for good reason. You can do this with CGI, but it's definitely not the recommended way to deploy python apps. Setting that requirement aside, flask and bottle are both lightweight micro web-frameworks with similar approaches, both allow you to reload automatically when changes are detected (this is only wise during development).

Flask app doesn't start successfully after deploying on IBM Toolchain

If your app takes a long time to start, you may want to consider setting timeout in your manifest.yml.

Also note that VCAP_APP_PORT is deprecated and you should use PORT.

Try using python instead of python3 in your Procfile. And try to add a runtime.txt file like https://github.com/IBM-Cloud/cloud-sql-database/blob/master/runtime.txt

---

Related Topics