What's the Difference Between Dist-Packages and Site-Packages

What's the difference between dist-packages and site-packages?

dist-packages is a Debian-specific convention that is also present in its derivatives, like Ubuntu. Modules are installed to dist-packages when they come from the Debian package manager into this location:

/usr/lib/python2.7/dist-packages

Since easy_install and pip are installed from the package manager, they also use dist-packages, but they put packages here:

/usr/local/lib/python2.7/dist-packages

From the Debian Python Wiki:

dist-packages instead of site-packages. Third party Python software
installed from Debian packages goes into dist-packages, not
site-packages. This is to reduce conflict between the system Python,
and any from-source Python build you might install manually.

This means that if you manually compile and install Python interpreter from source, it uses the site-packages directory. This allows you to keep the two installations separate, especially since Debian and Ubuntu rely on the system version of Python for many system utilities.

Python module in 'dist-packages' vs. 'site-packages'

From what I learnt from IRC is that I should install the modules in 'dist-packages' only, assuming that the admin would have installed the python provided by Ubuntu repo only.

What is python's site-packages directory?

site-packages is the target directory of manually built Python packages. When you build and install Python packages from source (using distutils, probably by executing python setup.py install), you will find the installed modules in site-packages by default.

There are standard locations:

Unix (pure)¹: prefix/lib/pythonX.Y/site-packages
Unix (non-pure): exec-prefix/lib/pythonX.Y/site-packages
Windows: prefix\Lib\site-packages

¹ Pure means that the module uses only Python code. Non-pure can contain C/C++ code as well.

site-packages is by default part of the Python search path, so modules installed there can be imported easily afterwards.

Useful reading

Installing Python Modules (for Python 2)
Installing Python Modules (for Python 3)

The reason cause different location of python packages

Your question is mainly about the Linux filesystem layout. You can read a lot about that, for example on Wikipedia and more specific for the different lib locations in this askubuntu question.

I'll try to answer (1) and (2) by summarizing how the three given folders are conventionally used:

/usr/lib/python3/dist-packages contains non-host-specific modules installed by the system with the package manager, for example on Ubuntu with sudo apt-get install python-numpy.
/usr/local/lib/python3.6/dist-packages contains modules that you installed yourself system-wide through a package manager, for example with sudo pip install numpy. (Of cause using sudo pip can cause problems as you rightly mentioned.)
/home/twotwo/.local/lib/python3.6/site-packages contains modules that the user twotwo has installed in his own user directory, for example by using pip in user-mode. Those modules can of cause only be imported by twotwo, because they don't appear in other user's PATH variables and might not even be readable by another user.

Also note that the naming dist-packages is a Debian (and derivates) specific convention for python packages installed using debian packages, as explained here. In a manual python installation, the respective folders would be named site-packages, as is standard for pip.

As to question (3): The details about this can be read in the Python 3 docs. Basically, after looking for the module in the folder from which your python script is run, the folders in your sys.path variable are looked up in the same order in which they are listed there. As soon as a module of matching name is found, it is imported.

How do I find the location of my Python site-packages directory?

There are two types of site-packages directories, global and per user.

Global site-packages ("dist-packages") directories are listed in sys.path when you run:
```
 python -m site
```
For a more concise list run getsitepackages from the site module in Python code:
```
 python -c 'import site; print(site.getsitepackages())'
```
Caution: In virtual environments getsitepackages is not available with older versions of virtualenv, sys.path from above will list the virtualenv's site-packages directory correctly, though. In Python 3, you may use the sysconfig module instead:
```
 python3 -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])'
```
The per user site-packages directory (PEP 370) is where Python installs your local packages:
```
 python -m site --user-site
```
If this points to a non-existing directory check the exit status of Python and see python -m site --help for explanations.
Hint: Running pip list --user or pip freeze --user gives you a list of all installed per user site-packages.

Practical Tips

<package>.__path__ lets you identify the location(s) of a specific package: (details)

  $ python -c "import setuptools as _; print(_.__path__)"
  ['/usr/lib/python2.7/dist-packages/setuptools']

<module>.__file__ lets you identify the location of a specific module: (difference)
```
  $ python3 -c "import os as _; print(_.__file__)"
  /usr/lib/python3.6/os.py
```

Run pip show <package> to show Debian-style package information:

  $ pip show pytest
  Name: pytest
  Version: 3.8.2
  Summary: pytest: simple powerful testing with Python
  Home-page: https://docs.pytest.org/en/latest/
  Author: Holger Krekel, Bruno Oliveira, Ronny Pfannschmidt, Floris Bruynooghe, Brianna Laugher, Florian Bruhin and others
  Author-email: None
  License: MIT license
  Location: /home/peter/.local/lib/python3.4/site-packages
  Requires: more-itertools, atomicwrites, setuptools, attrs, pathlib2, six, py, pluggy

Virtualenv: global site-packages vs the site-packages in the virtual environment

The previous answer wraps up question 1 but ignores question 2.

The general best practice I've seen for which packages to put globally:

First, the core Python packages, as these don't change with backwards-incompatible issues unless you're upgrading a major version, and you'll want whatever security fixes from a python upgrade to apply automatically to your virtualenvs.

Second, packages that are a pain to easy_install or pip install into each individual virtualenv but that don't change very often -- MySQLdb/psycopg and PIL, for example.

Pretty much everything else should go into your virtualenv's packages (I highly recommend using pip requirements files and virtualenvwrapper to make this minimally painful and easy to set up on other machines).

What's the Difference Between Dist-Packages and Site-Packages