Copy/Move One Environment to Another

Copy/move one environment to another

There seem to be at least 3 different things you can do:

1. Clone an environment (create an exact duplicate)
2. Copy the content of one environment to another environment
3. Share the same environment

To clone:

# Make the source env
e1 <- new.env()
e1$foo <- 1
e1$.bar <- 2   # a hidden name
ls(e1) # only shows "foo"

# This will clone e1
e2 <- as.environment(as.list(e1, all.names=TRUE))

# Check it...
identical(e1, e2) # FALSE
e2$foo
e2$.bar

To copy the content, you can do what @gsk showed. But again, the all.names flag is useful:

# e1 is source env, e2 is dest env
for(n in ls(e1, all.names=TRUE)) assign(n, get(n, e1), e2)

To share the environment is what @koshke did. This is probably often much more useful. The result is the same as if creating a local function:

f2 <- function() {
  v1 <- 1 
  v2 <- 2

  # This local function has access to v1 and v2
  flocal <- function() {
    print(v1)
    print(v2)
  }

  return(flocal)
} 

f1 <- f2()
f1() # prints 1 and 2 

Move a value to a different environment

Perhaps write to disk, delete, read from disk? The only potential problem I can foresee with this approach is that any relationships between parent/child environments will be lost. But if you're simply trying to copy the values from one environment to another, maybe this isn't a problem?

Update:

I cannot replicate what you say about the copy approach. The code below shows that the maximum memory used (as reported by gc) does not increase. This is because the values are "promised", not deep-copied. A copy will be made, however, if you change an object in the new environment before you delete it from the old environment.

R> e1 <- new.env()
R> e1$x <- numeric(5e7)
R> e1$y <- numeric(5e7)
R> gc()
            used  (Mb) gc trigger  (Mb)  max used  (Mb)
Ncells    171022   9.2     350000  18.7    350000  18.7
Vcells 100271746 765.1  110886821 846.0 100272535 765.1
R> e2 <- new.env()
R> for(n in ls(e1, all.names=TRUE))
+   assign(n, get(n, e1), e2)
R> gc()
            used  (Mb) gc trigger  (Mb)  max used  (Mb)
Ncells    171038   9.2     350000  18.7    350000  18.7
Vcells 100271788 765.1  116511162 889.0 100272535 765.1
R> identical(e1$x,e2$x)
[1] TRUE
R> identical(e1$y,e2$y)
[1] TRUE

How to 'copy' Conda environment to another machine after modifying some packages' source code?

Have you tried conda-pack?

Seems to solve exactly the problem you have.

Quote from the docs:

A tool like conda-pack is necessary because conda environments are not relocatable. Simply moving an environment to a different directory can render it partially or completely inoperable. conda-pack addresses this challenge by building archives from original conda package sources and reproducing conda’s own relocation logic.

How can you clone a conda environment into the root environment?

There are options to copy dependency names/urls/versions to files.

Recommendation

Normally it is safer to work from a new environment rather than changing root. However, consider backing up your existing environments before attempting changes. Verify the desired outcome by testing these commands in a demo environment. To backup your root env for example:

λ conda activate root
λ conda env export > environment_root.yml
λ conda list --explicit > spec_file_root.txt

Options

Option 1 - YAML file

Within the second environment (e.g. myenv), export names⁺ to a yaml file:

λ activate myenv
λ conda env export > environment.yml  

then update the first environment⁺ (e.g. root) with the yaml file:

λ conda env update --name root --file environment.yml     

---

Option 2 - Cloning an environment

Use the --clone flag to clone environments (see @DevC's post):

λ conda create --name myclone --clone root

This basically creates a direct copy of an environment.

---

Option 3 - Spec file

Make a spec-file⁺⁺ to append dependencies from an env (see @Ormetrom):

λ activate myenv
λ conda list --explicit > spec_file.txt
λ conda install --name root --file spec_file.txt

Alternatively, replicate a new environment (recommended):

λ conda create --name myenv2 --file spec_file.txt

See Also

• conda env for more details on the env sub-commands.
• Anaconada Navigator desktop program for a more graphical experience.
• Docs on updated commands. With older conda versions use activate (Windows) and source activate (Linux/Mac OS). Newer versions of conda can use conda activate (this may require some setup with your shell configuration via conda init).
• Discussion on keeping conda env

---

Extras

There appears to be an undocumented conda run option to help execute commands in specific environments.

# New command
λ conda run --name myenv conda list --explicit > spec_file.txt

The latter command is effective at running commands in environments without the activation/deactivation steps. See the equivalent command below:

# Equivalent
λ activate myenv
λ conda list --explicit > spec_file.txt
λ deactivate

Note, this is likely an experimental feature, so this may not be appropriate in production until official adoption into the public API.

^{+Conda docs have changed since the original post; links updated.
++Spec-files only work with environments created on the same OS. Unlike the first two options, spec-files only capture links to conda dependencies; pip dependencies are not included.}

How to move .conda from one folder to another at the moment of creating the environment

Configure Environment and Package Default Locations

I'd guess that, despite your efforts to put your environments on the large partition, there is still a default user-level package cache and that is filling up the home partition. At minimum, set up a new package cache and a default environments directory on the large partition:

# create a new pkgs_dirs (wherever, doesn't have to be hidden)
mkdir -p /big_partition/users/user/.conda/pkgs

# add it to Conda as your default
conda config --add pkgs_dirs /big_partition/users/user/.conda/pkgs

# create a new envs_dirs (again wherever)
mkdir -p /big_partition/users/user/.conda/envs

# add it to Conda as your default
conda config --add envs_dirs /big_partition/users/user/.conda/envs

Now you don't have to fuss around with using the --prefix flag any more - your named environments (conda create -n foo) will by default be created inside this directory and you can activate by name instead of directory (conda activate foo).

Transferring Previous Environments and Package Cache

Unfortunately, there's not a great way to move Conda environments across filesystems without destroying the hardlinks. Instead, you'll need to recreate your environments. Since you may or may not want to bother with this, I'm only going to outline it. I can elaborate if needed.

1. Archive environments. Use conda env export -n foo > foo.yaml (One per environment.)
2. Move package cache. Copy contents of old package cache (/home/users/user_name/.conda/envs/.pkgs/) to new package cache.
3. Recreate environments. Use conda env create -n foo -f foo.yaml.

Again, you could just skip this altogether. This is mainly if you want to be very thorough about transferring and not having to redownload stuff for environments you already created.

After this you can delete some the stuff under the old ~/.conda/envs/pkgs folder.

---

Related Topics