Modiyfing conda configuration file does not reflect changes in environment
Background: Conda's configuration priorities
As documented in "The Conda Configuration Engine for Power Users" post, Conda sources configuration values from four sources, listed from lowest to highest priority:
- Default values in the Python code
.condarc
configuration files (system < user < environment < working directory)- Environment variables (
CONDA_*
variables) - Command-line specifications
Problem: Environment variable prioritized
We can observe how this plays out in OP's case, with the --show-sources
result. Specifically, there are three places where envs_dirs
is defined:
- System level configuration file at
/util/opt/anaconda/4.9.2/.condarc
- User-level configuration file at
/home/helikarlab/joshl/.condarc
- Environment variable
CONDA_ENVS_PATH
1
And since the environment variable takes priority and defines the preferred directory to be /home/helikarlab/joshl/.conda/envs
, that will take precedence no matter what is set with conda config
and .condarc
files.
Workarounds
All the following workarounds involve manipulating the environment variable. It is unclear when the variable is set (probably via a system-level shell configuration file). It should be reliable to manipulate the variable by appending any of the following workarounds to user-level shell configuration file (e.g., ~/.bashrc
, ~/.bash_profile
, ~/.zshrc
).
Option 1: Unset variable
One could completely remove the variable with
unset CONDA_ENVS_PATH
This would then allow the user-level .condarc
to take priority.
However, this variable also appears to provide locations for several system-level shared environments. It is unclear how integral these shared environments are for normal functionality. So, removing the variable altogether could have additional consequences.
Option 2: Replace value
Conveniently, the location default and desired locations differ only by replacing /home
with /work
. This could be changed directly in the variable with:
export CONDA_ENVS_PATH=${CONDA_ENVS_PATH/\/home/\/work}
Option 3: Prepend desired default
The most general override would be to prepend the desired default path to the environment variable:
export CONDA_ENVS_PATH="/work/helikarlab/joshl/.conda/envs/:${CONDA_ENVS_PATH}"
This is probably the most robust, since it assumes nothing about the inherited value.
Additional Note
Users with small disk quotas in default locations should also consider moving the package cache (pkgs_dirs
) to coordinate with the environments directory. Details in this answer.
[1]: CONDA_ENVS_DIRS
and CONDA_ENVS_PATH
are interchangeable, however only one can be defined at time. The former is the contemporary usage, so I believe the latter is likely supported for backward compatibility.
How to move .conda from one folder to another at the moment of creating the environment
Configure Environment and Package Default Locations
I'd guess that, despite your efforts to put your environments on the large partition, there is still a default user-level package cache and that is filling up the home partition. At minimum, set up a new package cache and a default environments directory on the large partition:
# create a new pkgs_dirs (wherever, doesn't have to be hidden)
mkdir -p /big_partition/users/user/.conda/pkgs
# add it to Conda as your default
conda config --add pkgs_dirs /big_partition/users/user/.conda/pkgs
# create a new envs_dirs (again wherever)
mkdir -p /big_partition/users/user/.conda/envs
# add it to Conda as your default
conda config --add envs_dirs /big_partition/users/user/.conda/envs
Now you don't have to fuss around with using the --prefix
flag any more - your named environments (conda create -n foo
) will by default be created inside this directory and you can activate by name instead of directory (conda activate foo
).
Transferring Previous Environments and Package Cache
Unfortunately, there's not a great way to move Conda environments across filesystems without destroying the hardlinks. Instead, you'll need to recreate your environments. Since you may or may not want to bother with this, I'm only going to outline it. I can elaborate if needed.
- Archive environments. Use
conda env export -n foo > foo.yaml
(One per environment.) - Move package cache. Copy contents of old package cache (
/home/users/user_name/.conda/envs/.pkgs/
) to new package cache. - Recreate environments. Use
conda env create -n foo -f foo.yaml
.
Again, you could just skip this altogether. This is mainly if you want to be very thorough about transferring and not having to redownload stuff for environments you already created.
After this you can delete some the stuff under the old ~/.conda/envs/pkgs
folder.
conda prefix (-p) still create a .conda directory on home
The ~/.conda/environments.txt
is unavoidable. The location of environments and packages is controlled by the envs_dirs
and pkgs_dirs
variables respectively. See docs:
conda config --describe envs_dirs pkgs_dirs
# # envs_dirs (sequence: primitive)
# # aliases: envs_path
# # env var string delimiter: ':'
# # The list of directories to search for named environments. When
# # creating a new named environment, the environment will be placed in
# # the first writable location.
# #
# envs_dirs: []
# # pkgs_dirs (sequence: primitive)
# # env var string delimiter: ','
# # The list of directories where locally-available packages are linked
# # from at install time. Packages not locally available are downloaded
# # and extracted into the first writable directory.
# #
# pkgs_dirs: []
Recommendation
Provide a pkgs_dirs
directory that isn't in under your home directory. Also, since Conda uses hardlinks to conserve disk usage, it is recommended you also provide an envs_dirs
that is on the same volume. This also gives you the advantage that you should no longer need to specify a --prefix,-p
argument to avoid environments being created under the user home, but you can instead use a --name,-n
argument and specify your environments by name.
How to change location of conda's environments.txt file
Not currently possible. The pertinent Conda code is
def get_user_environments_txt_file(userhome='~'):
return expand(join(userhome, '.conda', 'environments.txt'))
and all the references to this call it with no argument. So there isn't any way in Conda to avoid this.
If you would like this to change, file an Issue on the repository. This existing Issue seems like a good place to chime it and propose some functionality to manage it.
Symlinking
If you really don't want it tracked, you might get away with a symlink to /dev/null
:
ln -s /dev/null ~/.conda/environments.txt
Alternatively, you could symlink to a real file at another location. Not sure that has any advantage though - you still have the symlink in your user home.
Related Topics
Linux Umask for Sudo and Apache
How to Determine The Files Corresponding to a UInput Device
Firefox Not Closing After Selenium Tests Are Run
How to Ping Other Containers in a Docker Network Through Their Hostname
Bash: Getting Percentage from a Frequency Table
Print Stdout/Stderr and Write Them to a File in Bash
Bash Script to Rename All Files in a Directory
Sending from The Same Udp Socket in Multiple Threads
Golang Os/Exec, Realtime Memory Usage
Pyinstaller on 32-Bit Linux - Importerror: The 'six' Package Is Required
What Is The Correct Way to Define a Netfilter Hook Function
Need Explanation on Pri Standard Format Specifier for Ps - Possible Bug in Documentation
Get X/Y Position of Caret (Input Text Cursor) Under Xorg
Rtnetlink Answers: No Such File or Directory
What's The Meaning of 'C' in Result of Command "Ls -L /Dev/Tty'
Different Utf-8 Signature for Same Diacritics (Umlauts) - 2 Binary Ways to Write Umlauts