How to Pack Python Libs I'm Using So I Can Distribute Them with My App and Have as Few Dependencies as Possible

How to pack and distribute python program (.py source code) so that other developers can easily install all required dependencies?

In the not so old days I used this guide to learn how to package and distribute my python code, then some good people created flit which allows me to do the whole process in three steps.

$pip install flit

Create my metadata file:

[metadata]
author=Some guy
author-email=some-email@nowhere.com
home-page=https://github.com/someuser/somepackage
requires=requests
requires-python= >=3
description-file=README.rst
classifiers=Intended Audience :: Developers
    License :: OSI Approved :: BSD License
    Programming Language :: Python :: 3
    Topic :: Software Development :: Libraries :: Python Modules

Publish my package:

$pip flit publish

And done!!!

How can I distribute python programs?

The normal way of distributing Python applications is with distutils. It's made both for distributing library type python modules, and python applications, although I don't know how it works on Windows. You would on Windows have to install Python separately if you use distutils, in any case.

I'd probably recommend that you distribute it with disutils for Linux, and Py2exe or something similar for Windows. For OS X I don't know. If it's an end user application you would probably want an disk image type of thing, I don't know how to do that. But read this post for more information on the user experience of it. For an application made for programmers you are probably OK with a distutils type install on OS X too.

Best practice for bundling third party libraries for distribution in Python 3

There are no best practices, but there are a few different tracks people follow. With regard to commercial product distribution there are the following:

Manage Your Own Package Server

With regard to your development process, it is typical to either have your dev boxes update from a local package server. That allows you to "freeze" the dependency list (i.e. just stop getting upstream updates) so that everyone is on the same version. You can update at particular times and have the developers update as well, keeping everyone in lockstep.

For customer installs you usually write an install script. You can collect all the packages and install your libs, as well as the other at the same time. There can be issues with trying to install a new Python, or even any standard library because the customer may already depend on a different version. Usually you can install in a sandbox to separate your packages from the systems packages. This is more of a problem on Linux than Windows.

Toolchain

The other option is to create a toolchain for each supported OS. A toolchain is all the dependencies (up to, but not including base OS libs like glibc). This toolchain gets packaged up and distributed for both developers AND customers. Best practice for a toolchain is:

change the executable to prevent confusion. (ie. python -> pkg_python)
don't install in .../bin directories to prevent accidental usage. (ie. on Linux you can install under .../libexec. /opt is also used although personally I detest it.)
install your libs in the correct location under lib/python/site-packages so you don't have to use PYTHONPATH.
Distribute the source .py files for the executables so the install script can relocate them appropriately.
The package format should be an OS native package (RedHat -> RPM, Debian -> DEB, Win -> MSI)

How can I make a Python script standalone executable to run without ANY dependency?

You can use py2exe as already answered and use Cython to convert your key .py files in .pyc, C compiled files, like .dll in Windows and .so on Linux.

It is much harder to revert than common .pyo and .pyc files (and also gain in performance!).

Packaging and shipping a python library and scripts, the professional way

This is not a complete answer but just a bunch of ideas. I wrote an installer for a client that incorporated some ideas that might be useful to you.

It was Linux only so I focussed on just that. We needed to ship specific custom versions of mySQL, lighttpd, python, memcached, a few 3rd party Python modules and some custom scripts. We needed to launch all these services without any problems and let the user control them using regular initscripts. It should work fine on a bunch of popular distros and therefore shouldn't rely on distro specific stuff.

What I did was as follows.

Created a 500MB (I'm don't recollect the size) file and formatted it as an ext3fs file system.
Mounted it at a point using a loopback device.
Ran deb-bootstrap on the mountpoint to create a custom Debian install.
Chrooted inside the partition and then ran a bunch of scripts which did an apt-get install on all our dependencies, installed all the eggs and other packages which were necessary for the app, installed the app itself in /opt (inside the chroot), installed supervisord (to do process management) and set things up. Now, this partition was a completely self contained Linux filesystem that contained the application and everything needed to run it. You could dump it anywhere, chroot inside it and launch the app. The only dependency it had with the outside world were the ports it would use for its services and the supervisord control socket. This was the main point. We were able to include exactly what we needed (compiled files, .pycs only etc.) for a few of the applications and didn't have to bother with any limitations in standard installation tools.
After this, we packaged a few extra scripts that would go into the external operating system. These were custom made for each distro that we would have to support. This part was distro specific. There were scripts that would go into /etc/init.d and some scripts that would setup the database and stuff at the beginning.
We then created an archive of the entire filesystem using makeself. It would checksum stuff and all that and provide a self extracting archive which if run would untar the whole thing into /opt on the host machine, chroot inside the directory and run a setup script that would ask the user a few questions like db username/password etc. and set things up. After that, it would fetch the scripts I mentioned in step 5 and put them on the host OS.

The initscripts would simply chroot into the partition and start supervisord. It would then take care of launching all the services we cared about. Shutting down the application was simply a matter of connecting to running supervisord and running a command. We wrapped this in the initscript so that the user experience was UNIX like.

Now, we'd give clients the self extracting .run file. They'd run it, get asked a few questions and it would create a directory under /opt which contained our app and all it's dependencies. The init scripts would be modified to start our app on bootup and things would work as expected.

I think step 4 gives you the freedom to install whatever you want, however you want so that things would work fine.

How to deploy a Python application with libraries as source with no further dependencies?

I sometimes use the approach I describe below, for the exact same reason that @Boris states: I would prefer that the use of some code is as easy as a) svn checkout/update - b) go.

But for the record:

I use virtualenv/easy_install most of the time.
I agree to a certain extent to the critisisms by @Ali A and @S.Lott

Anyway, the approach I use depends on modifying sys.path, and works like this:

Require python and setuptools (to enable loading code from eggs) on all computers that will use your software.
Organize your directory structure this:


project/
    *.py
    scriptcustomize.py
    file.pth

    thirdparty/
        eggs/
            mako-vNNN.egg
            ... .egg
        code/
            elementtree\
                *.py
            ...

In your top-level script(s) include the following code at the top:


from scriptcustomize import apply_pth_files
apply_pth_files(__file__)

Add scriptcustomize.py to your project folder:


import os
from glob import glob
import fileinput
import sys

def apply_pth_files(scriptfilename, at_beginning=False):
    """At the top of your script:
    from scriptcustomize import apply_pth_files
    apply_pth_files(__file__)

    """
    directory = os.path.dirname(scriptfilename)
    files = glob(os.path.join(directory, '*.pth'))
    if not files:
        return
    for line in fileinput.input(files):
        line = line.strip()
        if line and line[0] != '#':
            path = os.path.join(directory, line)
            if at_beginning:
                sys.path.insert(0, path)
            else:
                sys.path.append(path)

Add one or more *.pth file(s) to your project folder. On each line, put a reference to a directory with packages. For instance:


# contents of *.pth file
thirdparty/code
thirdparty/eggs/mako-vNNN.egg

I "kind-of" like this approach. What I like: it is similar to how *.pth files work, but for individual programs instead of your entire site-packages. What I do not like: having to add the two lines at the beginning of the top-level scripts.
Again: I use virtualenv most of the time. But I tend to use virtualenv for projects where I have tight control of the deployment scenario. In cases where I do not have tight control, I tend to use the approach I describe above. It makes it really easy to package a project as a zip and have the end user "install" it (by unzipping).

How to include and install local dependencies in setup.py in Python?

it is possible but not sure what setuptools version you should use.
steps:

in setup.py

setup(
  ...,
  install_requires=['my-package'],
  dependency_links=[
    # location to your egg file
    os.path.join(os.getcwd(), 'deps', 'my_package-1.0.0-py3.5.egg')
  ]
)

important thing is that your location should not pass URL pattern test and egg file name should have structure <package_name_with_no_hyphens>-<version>-<py_version>.egg