How to Deal with Linux/Python Dependencies

How to deal with Linux/Python dependencies?

I see two separate problems here:

1. Keeping track of all the python modules you need for your project.

2. Keeping track of all the dynamic libraries you need for the python modules in your project.

For the first problem, I have found that buildout is good help, althought it takes a litle while to grasp.

In your case, I would start by creating a directory for my new project. I would then go into that directory and download bootstrap.py

wget http://python-distribute.org/bootstrap.py 

I would then create a buildout.cfg file:

[buildout]
parts = qrproject
        python
eggs = pyqrencode

[qrproject]
recipe = z3c.recipe.scripts
eggs = ${buildout:eggs}
entry-points= qrproject=qrprojectmodule:run
extra-paths = ${buildout:directory}

# This is a simple way of creating an interpreter that will have
# access to all the eggs / modules that this project uses.
[python]
recipe = z3c.recipe.scripts
interpreter = python
eggs = ${buildout:eggs}
extra-paths = ${buildout:directory}

In this buildout.cfg I'm referencing the module qrprojectmodule (in entry-points under [qrproject]. This will create a bin/qrproject that runs the function run in the module qrprojectmodule. So I will also create the file qrprojectmodule.py

import qrencode

def run():
    print "Entry point for qrproject. Happily imports qrencode module"

Now it's time to run bootstrap.py with the python binary you want to use:

python bootstrap.py

Then run the generated bin/buildout

bin/buildout

This will create two additional binaries in the bin/ directory - bin/qrproject and bin/python. The former is your project's main binary. It will be created automatically each time you run buildout and will have all the modules and eggs you want loaded. The second is simply a convenient way to get a python prompt where all your modules and eggs are loaded, for easy debugging. The fine thing here is that bin/buildout will automatically install any python eggs that the eggs (in your case pyqrencode) have specified as dependencies.

Actually, you will probably get a compilation error in the step where you run bin/buildout. This is because you need to address problem 2: All dynamic libraries being available on your system. On Linux, it's usually best to get help from your packaging system. I'm going to assume you're using a Debian derivate such as Ubuntu here.

The pyqrencode web site specifies that you need the libqrencode library for pyqrencode to work. So I used my package manager to search for that:

$ apt-cache search libqrencode
libqrencode-dev - QR Code encoding library -- development
libqrencode3 - QR Code encoding library
qrencode - QR Code encoder into PNG image

In this case, I want the -dev package, as that installs linkable libraries and header files required to compile python C-modules. Also, the dependency system in the package manager will make sure that if I install libqrencode-dev, I will also get libqrencode3, as that is required at runtime, i.e. after compilation of the module.

So, I install the package:

sudo apt-get install libqrencode-dev

Once that has completed, rerun bin/buildout and the pyqrencode module will (hopefully) compile and install successfully. Now try to run bin/qrproject

$ bin/qrproject 
Entry point for qrproject. Happily imports qrencode module

Success! :-)

So, in summary:

1. Use buildout to automatically download and install all the python modules/eggs you need for your project.

2. Use your system's package manager to install any dynamic (C) libraries required by the python modules you use.

Be aware that in many cases there are already packaged versions of your python modules available in the package system. For example, pil is available by installing the python-imaging package on Ubuntu. In this case, you don't need to install it via buildout, and you don't need to worry about libraries being available - the package manager will install all dependencies required for the module to run. Doing it via buildout can however make it easier to distribute your project and make it run on other systems.

Best practice to install dependencies?

The most important thing to help you decide is to consider your audience.

Are they technically-inclined and likely to be comfortable following instructions specifying how to build the dependencies themselves? If so, go with (3). If not, writing a python or shell script, or a makefile to automate the task may be the way to go. Pick whichever you feel most comfortable writing.

How to deal with missing libraries on Linux? [No Admin privileges and PIP is blocked]

This won't answer your question about the missing libraries, but I hope it helps solve your current issue with PySide.

I've had a similar problem before, you should always develop on the target platform to get comparable results.

This means that you theoretically have to write, compile and package your program on the RHEL machine. You also need to always develop on the older platform. Forward compatibility is not always guaranteed. I therefore suggest, that you install CentOS 7 in a virtual machine and if your program is not too complicated try to use PySide2 instead of PySide6.

Installing Python Dependencies locally in project

use this command

pip install -r requirements.txt -t <path-to-the-lib-directory>

python packages with dependencies in windows system

You're basically asking how to quickly set up a development environment to compile a given project. You're generally at the mercy of the project developers and how well they documented the build process.

On linux, you often have a package manager to make the installing and resolving of dependencies easy.

Since Windows doesn't have a package manager, many popular projects with lots of dependencies will include a download link to a Libraries zip file that contains all the dependencies necessary to compile the source.

Instead of running pip each time, it may be faster to just download the source to those python projects and run the setup.py manually, resolving dependencies until it succeeds.

In general, for python libraries that wrap C/C++ libraries, you're not going to be able to build the python library if you can't build the corresponding C/C++ library. So, you may want to download the ffmpeg source and try compiling it first.

Also, for some compiled python libraries, you may be able to find python wheels, which will contain pre-compiled binaries for your system, making the compile step unnecessary. If the python library wraps another C/C++ library, you'll still need to download and install the appropriate version of the library that it wraps (e.g. ffmpeg)

How can I stop a python dependency from exiting?

As noted in the comments above, there isn't really a way to do this without creating a subprocess. In the end, I ended up (ab)using the python multiprocessing library to solve this problem. The solution looks like:

import multiprocessing as mp
import unsafe_module  # my external dependency that hard exits unexpectedly

def access_unsafe_module_safely(unsafe_args, pipe):
    unsafe_obj = unsafe_module.UnsafeClass()
    
    # the next line causes the process to exit when certain args are passed
    results = unsafe_obj.do_unsafe_thing(unsafe_args)
    
    # report the results using multiprocessing.Pipe
    pipe.send(results)

# unsafe_args are passed in from the user
def main(unsafe_args):
    (receive_end, send_end) = mp.Pipe(False)  # False gives non-duplex mode

    # the target callable is invoked with args, which must be iterable
    process = mp.Process(target=access_unsafe_module_safely, args=(unsafe_args, send_end))
    process.start()
    process.join()  # waits until the subprocess is complete

    # if you know your module's exit codes, you can be smarter here
    if process.exitcode != 0:  # generally signals error
        raise RuntimeError("something bad happened in unsafe_module")
    
    # gets the returned results from the subprocess
    results = receive_end.recv()

    # (Python 3.7+) cleans up the subprocess resources
    process.close()
    
    # continue on with results from here...

Unfortunately there isn't much of a point in going to the library maintainers; it's Python bindings for a scientific C/C++ application. exit makes perfect sense for their design case.

---

Related Topics