Change Conda Default Pkgs_Dirs and Envs Dirs

Modiyfing conda configuration file does not reflect changes in environment

Background: Conda's configuration priorities

As documented in "The Conda Configuration Engine for Power Users" post, Conda sources configuration values from four sources, listed from lowest to highest priority:

1. Default values in the Python code
2. .condarc configuration files (system < user < environment < working directory)
3. Environment variables (CONDA_* variables)
4. Command-line specifications

Problem: Environment variable prioritized

We can observe how this plays out in OP's case, with the --show-sources result. Specifically, there are three places where envs_dirs is defined:

1. System level configuration file at /util/opt/anaconda/4.9.2/.condarc
2. User-level configuration file at /home/helikarlab/joshl/.condarc
3. Environment variable CONDA_ENVS_PATH¹

And since the environment variable takes priority and defines the preferred directory to be /home/helikarlab/joshl/.conda/envs, that will take precedence no matter what is set with conda config and .condarc files.

Workarounds

All the following workarounds involve manipulating the environment variable. It is unclear when the variable is set (probably via a system-level shell configuration file). It should be reliable to manipulate the variable by appending any of the following workarounds to user-level shell configuration file (e.g., ~/.bashrc, ~/.bash_profile, ~/.zshrc).

Option 1: Unset variable

One could completely remove the variable with

unset CONDA_ENVS_PATH

This would then allow the user-level .condarc to take priority.

However, this variable also appears to provide locations for several system-level shared environments. It is unclear how integral these shared environments are for normal functionality. So, removing the variable altogether could have additional consequences.

Option 2: Replace value

Conveniently, the location default and desired locations differ only by replacing /home with /work. This could be changed directly in the variable with:

export CONDA_ENVS_PATH=${CONDA_ENVS_PATH/\/home/\/work}

Option 3: Prepend desired default

The most general override would be to prepend the desired default path to the environment variable:

export CONDA_ENVS_PATH="/work/helikarlab/joshl/.conda/envs/:${CONDA_ENVS_PATH}"

This is probably the most robust, since it assumes nothing about the inherited value.

Additional Note

Users with small disk quotas in default locations should also consider moving the package cache (pkgs_dirs) to coordinate with the environments directory. Details in this answer.

---

_{[1]: CONDA_ENVS_DIRS and CONDA_ENVS_PATH are interchangeable, however only one can be defined at time. The former is the contemporary usage, so I believe the latter is likely supported for backward compatibility.}

How to move .conda from one folder to another at the moment of creating the environment

Configure Environment and Package Default Locations

I'd guess that, despite your efforts to put your environments on the large partition, there is still a default user-level package cache and that is filling up the home partition. At minimum, set up a new package cache and a default environments directory on the large partition:

# create a new pkgs_dirs (wherever, doesn't have to be hidden)
mkdir -p /big_partition/users/user/.conda/pkgs

# add it to Conda as your default
conda config --add pkgs_dirs /big_partition/users/user/.conda/pkgs

# create a new envs_dirs (again wherever)
mkdir -p /big_partition/users/user/.conda/envs

# add it to Conda as your default
conda config --add envs_dirs /big_partition/users/user/.conda/envs

Now you don't have to fuss around with using the --prefix flag any more - your named environments (conda create -n foo) will by default be created inside this directory and you can activate by name instead of directory (conda activate foo).

Transferring Previous Environments and Package Cache

Unfortunately, there's not a great way to move Conda environments across filesystems without destroying the hardlinks. Instead, you'll need to recreate your environments. Since you may or may not want to bother with this, I'm only going to outline it. I can elaborate if needed.

1. Archive environments. Use conda env export -n foo > foo.yaml (One per environment.)
2. Move package cache. Copy contents of old package cache (/home/users/user_name/.conda/envs/.pkgs/) to new package cache.
3. Recreate environments. Use conda env create -n foo -f foo.yaml.

Again, you could just skip this altogether. This is mainly if you want to be very thorough about transferring and not having to redownload stuff for environments you already created.

After this you can delete some the stuff under the old ~/.conda/envs/pkgs folder.

conda prefix (-p) still create a .conda directory on home

The ~/.conda/environments.txt is unavoidable. The location of environments and packages is controlled by the envs_dirs and pkgs_dirs variables respectively. See docs:

conda config --describe envs_dirs pkgs_dirs
# # envs_dirs (sequence: primitive)
# #   aliases: envs_path
# #   env var string delimiter: ':'
# #   The list of directories to search for named environments. When
# #   creating a new named environment, the environment will be placed in
# #   the first writable location.
# # 
# envs_dirs: []

# # pkgs_dirs (sequence: primitive)
# #   env var string delimiter: ','
# #   The list of directories where locally-available packages are linked
# #   from at install time. Packages not locally available are downloaded
# #   and extracted into the first writable directory.
# # 
# pkgs_dirs: []

Recommendation

Provide a pkgs_dirs directory that isn't in under your home directory. Also, since Conda uses hardlinks to conserve disk usage, it is recommended you also provide an envs_dirs that is on the same volume. This also gives you the advantage that you should no longer need to specify a --prefix,-p argument to avoid environments being created under the user home, but you can instead use a --name,-n argument and specify your environments by name.

How to change location of conda's environments.txt file

Not currently possible. The pertinent Conda code is

def get_user_environments_txt_file(userhome='~'):
    return expand(join(userhome, '.conda', 'environments.txt'))

and all the references to this call it with no argument. So there isn't any way in Conda to avoid this.

If you would like this to change, file an Issue on the repository. This existing Issue seems like a good place to chime it and propose some functionality to manage it.

Symlinking

If you really don't want it tracked, you might get away with a symlink to /dev/null:

ln -s /dev/null ~/.conda/environments.txt

Alternatively, you could symlink to a real file at another location. Not sure that has any advantage though - you still have the symlink in your user home.

---

Related Topics