Including Non-Python Files with Setup.Py

Including non-Python files with setup.py

Probably the best way to do this is to use the setuptools package_data directive. This does mean using setuptools (or distribute) instead of distutils, but this is a very seamless "upgrade".

Here's a full (but untested) example:

from setuptools import setup, find_packages

setup(
    name='your_project_name',
    version='0.1',
    description='A description.',
    packages=find_packages(exclude=['ez_setup', 'tests', 'tests.*']),
    package_data={'': ['license.txt']},
    include_package_data=True,
    install_requires=[],
)

Note the specific lines that are critical here:

package_data={'': ['license.txt']},
include_package_data=True,

package_data is a dict of package names (empty = all packages) to a list of patterns (can include globs). For example, if you want to only specify files within your package, you can do that too:

package_data={'yourpackage': ['*.txt', 'path/to/resources/*.txt']}

The solution here is definitely not to rename your non-py files with a .py extension.

See Ian Bicking's presentation for more info.

UPDATE: Another [Better] Approach

Another approach that works well if you just want to control the contents of the source distribution (sdist) and have files outside of the package (e.g. top-level directory) is to add a MANIFEST.in file. See the Python documentation for the format of this file.

Since writing this response, I have found that using MANIFEST.in is typically a less frustrating approach to just make sure your source distribution (tar.gz) has the files you need.

For example, if you wanted to include the requirements.txt from top-level, recursively include the top-level "data" directory:

include requirements.txt
recursive-include data *

Nevertheless, in order for these files to be copied at install time to the package’s folder inside site-packages, you’ll need to supply include_package_data=True to the setup() function. See Adding Non-Code Files for more information.

Including non-python files in a python package

The solution is a combination of @m.rp and @vin's answers:

Instead of from distutils.core import setup
use

from setuptools import setup

include the following argument to the setup() call

include_package_data=True,

and ONLY use the MANIFEST.in to include files, where the directory is specified relative from the location where the MANIFEST.in file is.

recursive-include gimpscm *.txt

What a massive PITA this was. Thank you for your answers!

Adding non-python files to setup.py

I got it. Make sure you have the complete path in the MANIFEST.in file. Something like:

include smlgui/gui/*.ui
include smlgui/gui/assets/*.png

Then test it by creating a wheel, by doing

python setup.py bdist_wheel

Under build folder you should be able to see all the contents.

Accessing non-Python files from another package in PEP518 environment (pyproject.toml)

Do you really want to use data_files?

I would recommend using package_data instead. This way files are part of the Python package itself (installed in site-packages for example) and are very easy to retrieve once installed.

Resources:

• https://sinoroc.gitlab.io/kb/python/package_data.html
• https://docs.python.org/3/distutils/setupscript.html#installing-package-data
• https://setuptools.readthedocs.io/en/latest/setuptools.html#including-data-files
• https://packaging.python.org/guides/distributing-packages-using-setuptools/#package-data

Access the installed package data with either one of those:

1. pkgutil

import pkgutil
pkgutil.get_data('my_top_level_package.my_sub_package', 'file.bin')

2. importlib.resources

importlib.resources.read_binary('my_top_level_package.my_sub_package', 'file.bin')

3. pkg_resources

Including non-Python files with setup.py

Probably the best way to do this is to use the setuptools package_data directive. This does mean using setuptools (or distribute) instead of distutils, but this is a very seamless "upgrade".

Here's a full (but untested) example:

from setuptools import setup, find_packages

setup(
    name='your_project_name',
    version='0.1',
    description='A description.',
    packages=find_packages(exclude=['ez_setup', 'tests', 'tests.*']),
    package_data={'': ['license.txt']},
    include_package_data=True,
    install_requires=[],
)

Note the specific lines that are critical here:

package_data={'': ['license.txt']},
include_package_data=True,

package_data is a dict of package names (empty = all packages) to a list of patterns (can include globs). For example, if you want to only specify files within your package, you can do that too:

package_data={'yourpackage': ['*.txt', 'path/to/resources/*.txt']}

The solution here is definitely not to rename your non-py files with a .py extension.

See Ian Bicking's presentation for more info.

UPDATE: Another [Better] Approach

Another approach that works well if you just want to control the contents of the source distribution (sdist) and have files outside of the package (e.g. top-level directory) is to add a MANIFEST.in file. See the Python documentation for the format of this file.

Since writing this response, I have found that using MANIFEST.in is typically a less frustrating approach to just make sure your source distribution (tar.gz) has the files you need.

For example, if you wanted to include the requirements.txt from top-level, recursively include the top-level "data" directory:

include requirements.txt
recursive-include data *

Nevertheless, in order for these files to be copied at install time to the package’s folder inside site-packages, you’ll need to supply include_package_data=True to the setup() function. See Adding Non-Code Files for more information.

Using setuptools to copy non .py files

1. Add include_package_data=True to your setup-function (you already did that).
2. Create a file MANIFEST.in in the same directory as setup.py

MANIFEST.in can look as follows:

include indictrans/models/ben-eng/*
include indictrans/models/ben-guj/*

You don't need setup.cfg for doing this.

Source: This great writeup of python packaging

EDIT about recursive-include:
According to the documentation this should also work:

recursive-include indictrans/models *.npy *.vec

Copy a non-Python file to specific directory during Pip Install

After a lot of research I have figure out how to resolve this issue. Let me summarize my findings, it might be helpful for other who wants to do post_pip_install processing.

setup.py

• Different options to install package: 1) pip install pkg_name, 2) python -m setup.py sdist

• If you want to make them work in either ways, need to have install, egg_info and develop all 3 options repeated as shown in setup.py

• If you create *.whl file by python -m setup.py bdist_wheel , post pip install processing won't be executed! Please upload .tar.gz format generated usingsdist to PyPi/Artifacts to make post pip install processing work. Again, Please note: It will not work when installing from a binary wheel

• upload the pip package: twine upload dist/*.tar.gz

    from setuptools import setup, find_packages
    from setuptools.command.install import install
    from setuptools.command.egg_info import egg_info
    from setuptools.command.develop import develop
  
    rootDir = os.path.abspath(os.path.dirname(__file__))
  
    def run_post_processing():
  
        print("--------Start running custom command -------")
        # One can Run any Post Processing here that will be executed post pip install 
  
    class PostInstallCommand(install):
        def run(self):
            print("***********Custom run from install********")
            install.run(self)
            run_post_processing()
  
    class PostEggCommand(egg_info):
        def run(self):
            print("***********Custom run from Egg********")
            egg_info.run(self)
            run_post_processing()
  
    class PostDevelopCommand(develop):
        def run(self):
            print("***********Custom run from Develop********")
            develop.run(self)
            run_post_processing()
  
    ver = "0.0.0"
    setup(
        name='my_pkg',
        version=ver,
        packages=find_packages(),
        python_requires='>=3.6.0',
        install_requires = getRequirements(),
        include_package_data= True,
        cmdclass={
                'install' : PostInstallCommand,
                'egg_info': PostEggCommand,
                'develop': PostDevelopCommand
                }
         )
  

Few More Things from my research:

1. If you want to do pre-processing instead of post-processing, need to move install.run(self) at the end
2. while pip installing, if you want to see custom messages of pre/post instllation, use -vvv. Example: pip install -vvv my_pkg

Adding non-python files to colcon build

I solved it. I am not sure if this is the right way of doing this or is there like another better/proper way or not, but here we go.

In the setup.py file, I added the line
(os.path.join('lib/python3.8/site-packages/package_name/sub_package/OSM'),glob(package_name+'/sub_package_name/OSM/*.osm')), in the data_files variable.

The first part of the new line which is os.path.join('lib/python3.8/site-packages/package_name/sub_package_name/OSM') determines the new location of the files in the install folder after building the workspace.

The second part which is glob(package_name+'/sub_package_name/OSM/*.osm') determines the files original location in the project workspace.

So the result is that it takes the files from the location mentioned in the second part and puts them in the location mentioned in the first part.

The resulting block is:

setup(
    name=package_name,
    version='0.0.0',
    packages=[package_name, submodules, osm],
    data_files=[
        ('share/ament_index/resource_index/packages',
            ['resource/' + package_name]),
        ('share/' + package_name, ['package.xml']),
        (os.path.join('share', package_name), glob('launch/*.launch.py')),
        (os.path.join('lib/python3.8/site-packages/package_name/sub_package_name/OSM'), glob(package_name+'/sub_package_name/OSM/*.osm')),
    ],

.
.
.
)

---

Related Topics