Python - Activate Conda Env Through Shell Script

Python - Activate conda env through shell script

The error message is rather helpful - it's telling you that conda is not properly set up from within the subshell that your script is running in. To be able to use conda within a script, you will need to (as the error message says) run conda init bash (or whatever your shell is) first. The behaviour of conda and how it's set up depends on your conda version, but the reason for the version 4.4+ behaviour is that conda is dependent on certain environment variables that are normally set up by the conda shell itself. Most importantly, this changelog entry explains why your conda activate and deactivate commands no longer behave as you expect in versions 4.4 and above.

For more discussion of this, see the official conda issue on GitHub.

---

Edit: Some more research tells me that the conda init function mentioned in the error message is actually a new v4.6.0 feature that allows a quick environment setup so that you can use conda activate instead of the old source activate. However, the reason why this works is that it adds/changes several environment variables of your current shell and also makes changes to your RC file (e.g.: .bashrc), and RC file changes are never picked up in the current shell - only in newly created shells. (Unless of course you source .bashrc again). In fact, conda init --help says as much:

IMPORTANT: After running conda init, most shells will need to be closed and restarted for changes to take effect

However, you've clearly already run conda init, because you are able to use conda activate interactively. In fact, if you open up your .bashrc, you should be able to see a few lines added by conda teaching your shell where to look for conda commands. The problem with your script, though, lies in the fact that the .bashrc is not sourced by the subshell that runs shell scripts (see this answer for more info). This means that even though your non-login interactive shell sees the conda commands, your non-interactive script subshells won't - no matter how many times you call conda init.

This leads to a conjecture (I don't have conda on Linux myself, so I can't test it) that by running your script like so:

bash -i shell_script.sh

you should see conda activate work correctly. Why? -i is a bash flag that tells the shell you're starting to run in interactive mode, which means it will automatically source your .bashrc. This should be enough to enable you to use conda within your script as if you were using it normally.

Using conda activate or specifying python path in bash script?

Programmatic execution with an environment is usually better done through the conda run subcommand. E.g.,

my_slurm_script.sh

#!/bin/bash -l
conda run -n my_env python my_script.py

Read the conda run --help for details.

How to activate a Conda environment within a Python script?

conda activate is really intended to be used in interactive settings (shells/command prompts). But you can use conda run to execute a particular python script within a given environment.

Try this instead:

conda run -n <environment> <script_name>

How does `conda activate` change *current* bash environment?

conda itself is a python script

I don't know where you have read this. conda is set up as a bash function, which you can verify yourself. In the .bashrc you can see that the file /miniconda3/etc/profile.d/conda.sh is called to set up the conda command, which looks like this:

conda() {
    if [ "$#" -lt 1 ]; then
        "$CONDA_EXE" $_CE_M $_CE_CONDA
    else
        \local cmd="$1"
        shift
        case "$cmd" in
            activate|deactivate)
                __conda_activate "$cmd" "$@"
                ;;
            install|update|upgrade|remove|uninstall)
                CONDA_INTERNAL_OLDPATH="${PATH}"
                __add_sys_prefix_to_path
                "$CONDA_EXE" $_CE_M $_CE_CONDA "$cmd" "$@"
                \local t1=$?
                PATH="${CONDA_INTERNAL_OLDPATH}"
                if [ $t1 = 0 ]; then
                    __conda_reactivate
                else
                    return $t1
                fi
                ;;
            *)
                CONDA_INTERNAL_OLDPATH="${PATH}"
                __add_sys_prefix_to_path
                "$CONDA_EXE" $_CE_M $_CE_CONDA "$cmd" "$@"
                \local t1=$?
                PATH="${CONDA_INTERNAL_OLDPATH}"
                return $t1
                ;;
        esac
    fi
}

and calls the __add_sys_prefix_to_path function to modify the PATH:

__add_sys_prefix_to_path() {
    # In dev-mode CONDA_EXE is python.exe and on Windows
    # it is in a different relative location to condabin.
    if [ -n "${_CE_CONDA}" ] && [ -n "${WINDIR+x}" ]; then
        SYSP=$(\dirname "${CONDA_EXE}")
    else
        SYSP=$(\dirname "${CONDA_EXE}")
        SYSP=$(\dirname "${SYSP}")
    fi

    if [ -n "${WINDIR+x}" ]; then
        PATH="${SYSP}/bin:${PATH}"
        PATH="${SYSP}/Scripts:${PATH}"
        PATH="${SYSP}/Library/bin:${PATH}"
        PATH="${SYSP}/Library/usr/bin:${PATH}"
        PATH="${SYSP}/Library/mingw-w64/bin:${PATH}"
        PATH="${SYSP}:${PATH}"
    else
        PATH="${SYSP}/bin:${PATH}"
    fi
    \export PATH
}

so conda doesn't launch any sub bash shells, but is just a (series) of bash functions, which simply edit the current env variable in place

Create and activate Conda environment in same script

Running bash in login mode should work, e.g.,

bash -l setup-environment.sh

Note, the conda init in the script is superfluous - it edits the .bash_profile but doesn't actually initialize a current bash session; it only needs to be executed once for a user. Hence, the script should be changed to

#!/usr/bin/env bash -l
set -euo pipefail

conda create -y --name myenv python=3.6 pip
conda activate myenv
# Perform pip-based installation here.

and you could just run it like ./setup-environment.sh.

---

Alternate Solution: Use YAML Environment Definition

Admittedly, I am missing the other parts you might have planned in your script, but everything that is shown could be more succinctly done using a Conda YAML environment definition. For example, if you write a YAML file like:

myenv.yaml

name: myenv
channels:
 - defaults
dependencies:
 - python=3.6
 - pip
 - pip:
   - some_pkg

and then run

conda env create -f myenv.yaml

it would do exactly what your script is doing, including all the pip installations. All commands that you can run in a pip requirements.txt can be included in a YAML. See the Advanced Pip Example in the Conda GitHub.

---

Related Topics