Combining Conda Environment.Yml with Pip Requirements.Txt

Combining conda environment.yml with pip requirements.txt

Pip dependencies can be included in the environment.yml file like this (docs):

# run: conda env create --file environment.yml
name: test-env
dependencies:
- python>=3.5
- anaconda
- pip
- numpy=1.13.3  # pin version for conda
- pip:
  # works for regular pip packages
  - docx
  - gooey
  - matplotlib==2.0.0  # pin version for pip
  # and for wheels
  - http://www.lfd.uci.edu/~gohlke/pythonlibs/bofhrmxk/opencv_python-3.1.0-cp35-none-win_amd64.whl

It also works for .whl files in the same directory (see Dengar's answer) as well as with common pip packages.

Is it safe to declare pip dependencies in a conda environment yml

I don't think it's quite fair to say it's not safe to use them in combination, people do it all the time. It's just that you have to be careful. The blog post is a little outdated (in my opinion due for a refresh) but some of the important details have not changed. As long as you install and update from a conda environment file and prefer installing packages via conda and not pip where possible, you should be fine. Or at least this is what I've been doing for a year across several projects. The problems I run into with setup are always due to other issues in the ecosystem, not the pip: feature of conda environment files.

Is this feature safe to use?

Yes. I'm sure there are ways to botch it, and I'm not quite sure how you'd defined "safe" here, but a configuration like below is used commonly in production.

name: my-dev-env
channels:
  - conda-forge
dependencies:
  - python
  - ...
  - some_other_conda_package
  - pip:
    - some_pypi_package

This does more or less what it would do without the pip: section, but calls pip and the end to install some_pypi_package.

Can it lead to broken environments?

Yes, but you'd have to try to do so. This bit

Most of these issues stem from that fact that conda, like other package managers, has limited abilities to control packages it did not install. Running conda after pip has the potential to overwrite and potentially break packages installed via pip. Similarly, pip may upgrade or remove a package which a conda-installed package requires. In some cases these breakages are cosmetic, where a few files are present that should have been removed, but in other cases the environment may evolve into an unusable state.

still holds true, and it tends to happens if after setting up an environment, conda install xyz and pip install xyz are called interchangeably and haphazardly. This is not the same thing as, and much more dangerous than, the pip: section in your environment YAML.

What should one do to ensure this doesn't happen?

In general, if you need to pull down something from PyPI but keep it safely managed in your conda environment:

• Avoid manual conda install and pip install calls. Instead, update from your environment file, i.e. conda env update --file env.yaml. This will update your conda packages and re-install whatever you need from PyPI in a (relatively) safe way. The fact that you're asking about installing via pip from a conda environment file means you're already on the right track.
• Only install from pip if something is only available on PyPI or for another reason can only be installed via pip. For example I use it for its git+ feature to install development builds before conda packages are made - though I wouldn't recommend doing this much.
• Try moving some packages to conda - the blog post mentions this a few times. conda-forge also particular tries to make this easy.

Strategies To Convert requirements.txt to a Conda Environment File

So, it turns out there were already good answers to help me accomplish what I needed. This answer shows how to get the packages without any locked versions.

pip freeze | sed s/=.*// > packages.txt

This answer shows a great loop that will install everything via conda when it can and use pip otherwise.

FOR /F "delims=~" %f in (requirements.txt) DO conda install --yes "%f" || pip install "%f"

Then to generate the actual conda file that can be kept in source control so that new environments can be created from it, you can run

conda env export

How to specify pip --extra-index-url in environment.yml?

This configuration should work, see the advanced-pip-example for other options.

name: foo
channels:
  - defaults
dependencies:
  - python
  - pip
  - pip:
    - --extra-index-url https://download.pytorch.org/whl/cu116
    - torch==1.12.1+cu116

See also

• Combining conda environment.yml with pip requirements.txt
• Can conda be configured to use a private pypi repo?

When combining Conda environment files and Pip, is Conda evironment active when Pip install is run?

In the environment.yml file itself you can menton the pip packages as well.You don't need to create a seperate requirements.txt file for pip packages.

Usually an environment.yml file contains the following fields:

name : the conda environment name

channels: the channels from which the dependencies needs to be installed

dependencie:: list of package. In that you can mention the pip dependencies as well

For more details, you can refer the following urls:

https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#create-env-file-manually

https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#create-env-from-file

The pip dependencies will be installed in the conda environment and you don't need to activate the environment and install pip dependencies manually.
Command to create conda environment:

conda env create -f environment.yml

Once the above command completes successfully, you can activate the environment and check the installed packages as below:

conda activate <env_name>
conda list

Using a pip requirements file in a conda yml file throws AttributeError: 'FileNotFoundError'

Changes to Pip Behavior in 21.2.1

A recent change in the Pip code has changed its behavior to be more strict with respect to file: URI syntax. As pointed out by a PyPA member and Pip developer, the syntax file:requirements.txt is not a valid URI according to the RFC8089 specification.

Instead, one must either drop the file: scheme altogether:

name: test
dependencies:
  - python>=3
  - pip
  - pip:
    - -r requirements.txt

or provide a valid URI, which means using an absolute path (or a local file server):

name: test
dependencies:
  - python>=3
  - pip
  - pip:
    - -r file:/full/path/to/requirements.txt
    # - -r file:///full/path/to/requirements.txt # alternate syntax

From conda create requirements.txt for pip3

As the comment at the top indicates, the output of

conda list -e > requirements.txt

can be used to create a conda virtual environment with

conda create --name <env> --file requirements.txt

but this output isn't in the right format for pip.

If you want a file which you can use to create a pip virtual environment (i.e. a requirements.txt in the right format)
you can install pip within the conda environment, then use pip to create requirements.txt.

conda activate <env>
conda install pip
pip freeze > requirements.txt

Then use the resulting requirements.txt to create a pip virtual environment:

python3 -m venv env
source env/bin/activate
pip install -r requirements.txt

When I tested this, the packages weren't identical across the outputs (pip included fewer packages) but it was sufficient to set up a functional environment.

For those getting odd path references in requirements.txt, use:

pip list --format=freeze > requirements.txt

Transform conda-generated requirements.txt file to a format that works with pip

No and no. The relation between a Conda package name to the PyPI package name is effectively arbitrary since there are no formal rules for mapping them. Sometimes they are equal; sometimes they involve prefixes/suffixes like py or python, with and without hyphens. Further, many Conda packages are not Python packages. A trivial example is python itself. Since there is nothing in the metadata of the packages to indicate the PyPI equivalent (if any), it is impossible to determine this without an actual brute force iteration over all the packages to extract this mapping. Perhaps someone might generate such a database someday, but for now I am unaware of its existence.

If you must have a pip requirements file, then export that from the environment, as in this thread. If you only have the file from Conda, but no environment, then the easiest path is probably to recreate the environment using Conda, then do the pip list --format=freeze.

I'd also note that I agree with the comments: it sounds like the question is solving the wrong problem. Conda is fully operational without elevated privileges and is almost ubiquitously used on HPC servers I use. Look into installing Miniforge.

---

Related Topics