Python - Activate conda env through shell script
The error message is rather helpful - it's telling you that conda is not properly set up from within the subshell that your script is running in. To be able to use conda within a script, you will need to (as the error message says) run conda init bash
(or whatever your shell is) first. The behaviour of conda and how it's set up depends on your conda version, but the reason for the version 4.4+ behaviour is that conda
is dependent on certain environment variables that are normally set up by the conda shell itself. Most importantly, this changelog entry explains why your conda activate
and deactivate
commands no longer behave as you expect in versions 4.4 and above.
For more discussion of this, see the official conda issue on GitHub.
Edit: Some more research tells me that the conda init
function mentioned in the error message is actually a new v4.6.0 feature that allows a quick environment setup so that you can use conda activate
instead of the old source activate
. However, the reason why this works is that it adds/changes several environment variables of your current shell and also makes changes to your RC file (e.g.: .bashrc
), and RC file changes are never picked up in the current shell - only in newly created shells. (Unless of course you source .bashrc again). In fact, conda init --help
says as much:
IMPORTANT: After running
conda init
, most shells will need to be closed and restarted for changes to take effect
However, you've clearly already run conda init
, because you are able to use conda activate
interactively. In fact, if you open up your .bashrc, you should be able to see a few lines added by conda teaching your shell where to look for conda commands. The problem with your script, though, lies in the fact that the .bashrc is not sourced by the subshell that runs shell scripts (see this answer for more info). This means that even though your non-login interactive shell sees the conda commands, your non-interactive script subshells won't - no matter how many times you call conda init
.
This leads to a conjecture (I don't have conda on Linux myself, so I can't test it) that by running your script like so:
bash -i shell_script.sh
you should see conda activate
work correctly. Why? -i
is a bash flag that tells the shell you're starting to run in interactive mode, which means it will automatically source your .bashrc. This should be enough to enable you to use conda within your script as if you were using it normally.
Using conda activate or specifying python path in bash script?
Programmatic execution with an environment is usually better done through the conda run
subcommand. E.g.,
my_slurm_script.sh
#!/bin/bash -l
conda run -n my_env python my_script.py
Read the conda run --help
for details.
How to activate a Conda environment within a Python script?
conda activate
is really intended to be used in interactive settings (shells/command prompts). But you can use conda run
to execute a particular python script within a given environment.
Try this instead:
conda run -n <environment> <script_name>
How does `conda activate` change *current* bash environment?
conda itself is a python script
I don't know where you have read this. conda
is set up as a bash function, which you can verify yourself. In the .bashrc
you can see that the file /miniconda3/etc/profile.d/conda.sh
is called to set up the conda
command, which looks like this:
conda() {
if [ "$#" -lt 1 ]; then
"$CONDA_EXE" $_CE_M $_CE_CONDA
else
\local cmd="$1"
shift
case "$cmd" in
activate|deactivate)
__conda_activate "$cmd" "$@"
;;
install|update|upgrade|remove|uninstall)
CONDA_INTERNAL_OLDPATH="${PATH}"
__add_sys_prefix_to_path
"$CONDA_EXE" $_CE_M $_CE_CONDA "$cmd" "$@"
\local t1=$?
PATH="${CONDA_INTERNAL_OLDPATH}"
if [ $t1 = 0 ]; then
__conda_reactivate
else
return $t1
fi
;;
*)
CONDA_INTERNAL_OLDPATH="${PATH}"
__add_sys_prefix_to_path
"$CONDA_EXE" $_CE_M $_CE_CONDA "$cmd" "$@"
\local t1=$?
PATH="${CONDA_INTERNAL_OLDPATH}"
return $t1
;;
esac
fi
}
and calls the __add_sys_prefix_to_path
function to modify the PATH:
__add_sys_prefix_to_path() {
# In dev-mode CONDA_EXE is python.exe and on Windows
# it is in a different relative location to condabin.
if [ -n "${_CE_CONDA}" ] && [ -n "${WINDIR+x}" ]; then
SYSP=$(\dirname "${CONDA_EXE}")
else
SYSP=$(\dirname "${CONDA_EXE}")
SYSP=$(\dirname "${SYSP}")
fi
if [ -n "${WINDIR+x}" ]; then
PATH="${SYSP}/bin:${PATH}"
PATH="${SYSP}/Scripts:${PATH}"
PATH="${SYSP}/Library/bin:${PATH}"
PATH="${SYSP}/Library/usr/bin:${PATH}"
PATH="${SYSP}/Library/mingw-w64/bin:${PATH}"
PATH="${SYSP}:${PATH}"
else
PATH="${SYSP}/bin:${PATH}"
fi
\export PATH
}
so conda
doesn't launch any sub bash shells, but is just a (series) of bash functions, which simply edit the current env variable in place
Create and activate Conda environment in same script
Running bash in login mode should work, e.g.,
bash -l setup-environment.sh
Note, the conda init
in the script is superfluous - it edits the .bash_profile
but doesn't actually initialize a current bash session; it only needs to be executed once for a user. Hence, the script should be changed to
#!/usr/bin/env bash -l
set -euo pipefail
conda create -y --name myenv python=3.6 pip
conda activate myenv
# Perform pip-based installation here.
and you could just run it like ./setup-environment.sh
.
Alternate Solution: Use YAML Environment Definition
Admittedly, I am missing the other parts you might have planned in your script, but everything that is shown could be more succinctly done using a Conda YAML environment definition. For example, if you write a YAML file like:
myenv.yaml
name: myenv
channels:
- defaults
dependencies:
- python=3.6
- pip
- pip:
- some_pkg
and then run
conda env create -f myenv.yaml
it would do exactly what your script is doing, including all the pip installations. All commands that you can run in a pip requirements.txt
can be included in a YAML. See the Advanced Pip Example in the Conda GitHub.
Related Topics
Enable Python to Connect to MySQL via Ssh Tunnelling
Importerror When Importing Tkinter in Python
Calling Dot Products and Linear Algebra Operations in Cython
How to Remove Item from a Python List in a Loop
Accessing a Value in a Tuple That Is in a List
Extract Email Sub-Strings from Large Document
Find and Replace Values in Xml Using Python
Link Several Popen Commands with Pipes
Find Nearest Indices for One Array Against All Values in Another Array - Python/Numpy
Does a Slicing Operation Give Me a Deep or Shallow Copy
Subsampling Every Nth Entry in a Numpy Array
Case-Insensitive List Sorting, Without Lowercasing the Result
How to Find Out Whether a File Is at Its 'Eof'
How to Make a Barplot and a Lineplot in the Same Seaborn Plot with Different Y Axes Nicely