How to connect R conda env to jupyter notebook
Jupyter does not automatically recognize Conda environments, activated or not.
Kernel Module
First, for an environment to run as a kernel, it must have the appropriate kernel package installed. For R environments, that is r-irkernel
, so that one needs to run
conda install -n r_venv_conda r-irkernel
For Python kernels, it's ipykernel
.
Kernel Registration
Second, kernels need to be registered with Jupyter. If one has Jupyter installed via Conda (say in an Anaconda base env), then I recommend using the nb_conda_kernels
package, which enables auto-discovery of kernel-ready Conda environments. This must be installed in the environment that has jupyter
installed (only one installation is needed!), for example, if this is base, then
conda install -n base nb_conda_kernels
Please read the documentation for details.
If using a system-level installation of Jupyter (i.e., not installed by Conda), then one needs to manually register the kernel. For example, something like
conda run -n r_venv_conda Rscript -e 'IRkernel::installspec(name="ir33", displayname="R 3.3")'
where one can set arbitrary values for name
and displayname
. See IRkernel for details.
Running Jupyter
If using a Conda-installed Jupyter, again, it only needs to be installed in a single env. This is the environment that should be activated before running jupyter notebook
. The kernel will be available to select from within Jupyter.
Is it possible to create an r conda environment from an existing version of r?
No, a non-Conda version of R cannot be used as a basis for a Conda environment.
However, it is possible to use any R installation as a kernel in Jupyter, no matter where either are installed. The key to doing so is to install IRkernel package in the R environment.
Steps for Registering R as a Jupyter Kernel
If the Jupyter is installed in a Conda env, e.g.,
my_jupyter_env
, first activate that env:$ conda activate my_jupyter_env
(my_jupyter_env) $Otherwise, i.e.,
jupyter
is onPATH
, then just continue without any Conda envs activated.Start an R session using the native R:
(my_jypyter_env) $ path/to/native/R
Install
IRkernel
:> install.packages("IRkernel")
Note that if this R is in a Conda env, instead use
conda install r-irkernel
outside of the R session.From the R session, register R as a kernel. For example, if I had an R 3.6.0 natively installed I might call it:
> IRkernel::installspec(name="ir36_native", displayname="R 3.6.0 (native)")
but the arguments are whatever you want.
Quit the session and start Jupyter.
> q()
(my_jupyter_env) $ jupyter notebookUse the "New" button to create a new notebook and select the R kernel from the dropdown.
Link Conda environment with Jupyter Notebook
For Anaconda I suggest you a much easier and proper solution;
just give a look at the nb_conda_kernels package.
It allows you to "manage your conda environment-based kernels inside the Jupyter Notebook".
Is should be included since Anaconda version 4.1.0, otherwise simply use
conda install nb_conda
Now you should be able to manage all direcly from the Notebook interface.
Note that only environments that have a Jupyter kernel installed (in the case of Python, the ipykernel
package). Quote from the nb_conda_kernels
GitHub page:
Any other environments you wish to access in your notebooks must have an appropriate kernel package installed. For instance, to access a Python environment, it must have the ipykernel package; e.g.
conda install -n python_env ipykernel
How to use Jupyter notebooks in a conda environment?
Disclaimer: ATM tested only in Ubuntu and Windows (see comments to this answer).
Jupyter runs the user's code in a separate process called kernel. The kernel can be a different Python installation (in a different conda environment or virtualenv or Python 2 instead of Python 3) or even an interpreter for a different language (e.g. Julia or R). Kernels are configured by specifying the interpreter and a name and some other parameters (see Jupyter documentation) and configuration can be stored system-wide, for the active environment (or virtualenv) or per user. If nb_conda_kernels
is used, additional to statically configured kernels, a separate kernel for each conda environment with ipykernel
installed will be available in Jupyter notebooks.
In short, there are three options how to use a conda environment and Jupyter:
Option 1: Run Jupyter server and kernel inside the conda environment
Do something like:
conda create -n my-conda-env # creates new virtual env
conda activate my-conda-env # activate environment in terminal
conda install jupyter # install jupyter + notebook
jupyter notebook # start server + kernel inside my-conda-env
Jupyter will be completely installed in the conda environment. Different versions of Jupyter can be used
for different conda environments, but this option might be a bit of overkill. It is enough to
include the kernel in the environment, which is the component wrapping Python which runs the code.
The rest of Jupyter notebook can be considered as editor or viewer and it is not necessary to
install this separately for every environment and include it in every env.yml
file. Therefore one
of the next two options might be preferable, but this one is the simplest one and definitely fine.
Option 2: Create special kernel for the conda environment
Do something like:
conda create -n my-conda-env # creates new virtual env
conda activate my-conda-env # activate environment in terminal
conda install ipykernel # install Python kernel in new conda env
ipython kernel install --user --name=my-conda-env-kernel # configure Jupyter to use Python kernel
Then run jupyter from the system installation or a different conda environment:
conda deactivate # this step can be omitted by using a different terminal window than before
conda install jupyter # optional, might be installed already in system e.g. by 'apt install jupyter' on debian-based systems
jupyter notebook # run jupyter from system
Name of the kernel and the conda environment are independent from each other, but it might make sense to use a similar name.
Only the Python kernel will be run inside the conda environment, Jupyter from system or a different conda environment will be used - it is not installed in the conda environment. By calling ipython kernel install
the jupyter is configured to use the conda environment as kernel, see Jupyter documentation and IPython documentation for more information. In most Linux installations this configuration is a *.json
file in ~/.local/share/jupyter/kernels/my-conda-env-kernel/kernel.json
:
{
"argv": [
"/opt/miniconda3/envs/my-conda-env/bin/python",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
],
"display_name": "my-conda-env-kernel",
"language": "python"
}
Option 3: Use nb_conda_kernels to use a kernel in the conda environment
When the package nb_conda_kernels
is installed, a separate kernel is available automatically for each
conda environment containing the conda package ipykernel
or a different kernel (R, Julia, ...).
conda activate my-conda-env # this is the environment for your project and code
conda install ipykernel
conda deactivate
conda activate base # could be also some other environment
conda install nb_conda_kernels
jupyter notebook
You should be able to choose the Kernel Python [conda env:my-conda-env]
. Note that nb_conda_kernels
seems to be available only via conda and not via pip or other package managers like apt.
Troubleshooting
Using Linux/Mac the command which
on the command line will tell you which jupyter is used, if you
are using option 1 (running Jupyter from inside the conda environment), it should be an executable
from your conda environment:
$ which jupyter
/opt/miniconda3/envs/my-conda-env/bin/jupyter
$ which jupyter-notebook # this might be different than 'which jupyter'! (see below)
/opt/miniconda3/envs/my-conda-env/bin/jupyter-notebook
Inside the notebook you should see that Python uses Python paths from the conda environment:
[1] !which python
/opt/miniconda3/envs/my-conda-env/bin/python
[2] import sys; sys.executable
'/opt/miniconda3/envs/my-conda-env/bin/python'
['/home/my_user',
'/opt/miniconda3/envs/my-conda-env/lib/python37.zip',
'/opt/miniconda3/envs/my-conda-env/lib/python3.7',
'/opt/miniconda3/envs/my-conda-env/lib/python3.7/lib-dynload',
'',
'/opt/miniconda3/envs/my-conda-env/lib/python3.7/site-packages',
'/opt/miniconda3/envs/my-conda-env/lib/python3.7/site-packages/IPython/extensions',
'/home/my_user/.ipython']
Jupyter provides the command jupyter-troubleshoot
or in a Jupyter notebook:
!jupyter-troubleshoot
This will print a lot of helpful information about including the outputs mentioned above as well as installed libraries and others. When
asking for help regarding Jupyter installations questions, it might be good idea to provide this information in bug reports or questions.
To list all configured Jupyter kernels run:
jupyter kernelspec list
Common errors and traps
Jupyter notebook not installed in conda environment
Note: symptoms are not unique to the issue described here.
Symptoms: ImportError in Jupyter notebooks for modules installed in the conda environment (but
not installed system wide), but no error when importing in a Python terminal
Explaination: You tried to run jupyter notebook from inside your conda environment
(option 1, see above), there is no configuration for a kernel for this conda environment (this
would be option 2) and nb_conda_kernels is not installed (option 3), but jupyter notebook is not (fully)
installed in the conda environment, even if which jupyter
might make you believe it was.
In GNU/Linux you can type which jupyter
to check which executable of Jupyter is run.
This means that system's Jupyter is used, probably because Jupyter is not installed:
(my-conda-env) $ which jupyter-notebook
/usr/bin/jupyter
If the path points to a file in your conda environment, Jupyter is run from inside Jupyter:
(my-conda-env) $ which jupyter-notebook
/opt/miniconda3/envs/my-conda-env/bin/jupyter-notebook
Note that when the conda package ipykernel
is installed, an executable jupyter
is shipped, but
no executable jupyter-notebook
. This means that which jupyter
will return a path to the conda
environment but jupyter notebook
will start system's jupyter-nootebook
(see also here):
$ conda create -n my-conda-env
$ conda activate my-conda-env
$ conda install ipykernel
$ which jupyter # this looks good, but is misleading!
/opt/miniconda3/envs/my-conda-env/bin/jupyter
$ which jupyter-notebook # jupyter simply runs jupyter-notebook from system...
/usr/bin/jupyter-notebook
This happens because jupyter notebook
searches for jupyter-notebook
, finds/usr/bin/jupyter-notebook
and
calls it
starting a new Python process. The shebang in /usr/bin/jupyter-notebook
is #!/usr/bin/python3
and not a dynamic#!/usr/bin/env python
.
Therefore Python manages to break out of the conda environment. I guess jupyter could callpython /usr/bin/jupyter-notebook
instead to overrule the shebang, but mixing
system's bin files and the environment's python path can't work well anyway.
Solution: Install jupyter notebook inside the conda environment:
conda activate my-conda-env
conda install jupyter
jupyter notebook
Wrong kernel configuration: Kernel is configured to use system Python
Note: symptoms are not unique to the issue described here.
Symptoms: ImportError in Jupyter notebooks for modules installed in the conda environment (but
not installed system wide), but no error when importing in a Python terminal
Explanation: Typically the system provides a kernel called python3 (display name "Python 3")
configured to use /usr/bin/python3
, see e.g. /usr/share/jupyter/kernels/python3/kernel.json
.
This is usually overridden by a kernel in the conda environment, which points to the environments
python binary /opt/miniconda3/envs/my-conda-env/bin/python
. Both are generated by the packageipykernel
(see here
and here).
A user kernel specification in ~/.local/share/jupyter/kernels/python3/kernel.json
might override
the system-wide and environment kernel. If the environment kernel is missing or the user kernel
points to a python installation outside the environment option 1 (installation of jupyter in the
environment) will fail.
For occurrences and discussions of this problem and variants see here,
here,
here
and also here,
here and
here.
Solution: Use jupyter kernelspec list
to list the location active kernel locations.
$ conda activate my-conda-env
$ jupyter kernelspec list
Available kernels:
python3 /opt/miniconda3/envs/my-conda-env/share/jupyter/kernels/python3
If the kernel in the environment is missing, you can try creating it manually usingipython kernel install --sys-prefix
in the activated environment, but it is probably better to
check your installation, because conda install ipykernel
should have created the environment
(maybe try re-crate the environment and re-install all packages?).
If a user kernel specification is blocking the environment kernel specification, you can either
remove it or use a relative python path which will use $PATH
to figure out which python
to use.
So something like this, should be totally fine:
$ cat ~/.local/share/jupyter/kernels/python3/kernel.json
{
"argv": [
"python",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
],
"display_name": "Python 3",
"language": "python"
}
Correct conda environment not activated
Symptoms: ImportError for modules installed in the conda environment (but not installed system
wide) in Jupyter notebooks and Python terminals
Explanation: Each terminal has a set of environment variables, which are lost when the terminal
is closed. In order to use a conda environment certain environment variables need to be set, which
is done by activating it using conda activate my-conda-env
. If you attempted to run Jupyter
notebook from inside the conda environment (option 1), but did not activate the conda environment
before running it, it might run the system's jupyter.
Solution: Activate conda environment before running Jupyter.
conda activate my-conda-env
jupyter notebook
Broken kernel configuration
Symptoms: Strange things happening. Maybe similar symptoms as above, e.g. ImportError
Explanation: If you attempted to use option 2, i.e. running Jupyter from system and the Jupyter
kernel inside the conda environment by using an explicit configuration for the kernel, but it does
not behave as you expect, the configuration might be corrupted in some way.
Solution: Check configuration in ~/.local/share/jupyter/kernels/my-kernel-name/kernel.json
and fix mistakes manually or remove the entire directory and re-create it using the command
provided above for option 2. If you can't find the kernel configuration there runjupyter kernelspec list
.
Python 2 vs 3
Symptoms: ImportError due to wrong Python version of the Jupyter kernel or other problems
with Python 2/3
Explanation: The kernel configuration can have all sorts of confusing and misleading effects.
For example the default Python 3 kernel configuration will allow me to launch a Jupyter notebook
running on Python 2:
conda create -n my-conda-env
conda activate my-conda-env
conda install python=2
conda install jupyter
jupyter notebook
The default Python 3 kernel:
$ cat ~/.local/share/jupyter/kernels/python3/kernel.json
{
"argv": [
"python",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
],
"display_name": "Python 3",
"language": "python"
}
After creating a new Jupyter Notebook with the Python 3 kernel, Python 2 from the conda
environment will be used even if "Python 3" is displayed by Jupyter.
Solution: Don't use Python 2 ;-)
setting up R in conda environment with jupiter notebook
lme4 isn't in the conda package so you need to install.
https://conda.io/docs/commands/build/conda-skeleton-cran.html
Try installing the package through cran:
./bin/conda skeleton cran lme4
And then run:
./bin/conda build r-lme4/
Related Topics
Django Gunicorn Not Load Static Files
How Is the Feature Score(/Importance) in the Xgboost Package Calculated
Does Python Have an "Or Equals" Function Like ||= in Ruby
Which of These Scripting Languages Is More Appropriate for Pen-Testing
Difference Between Multiple If's and Elif'S
How to Get Reproducible Results in Keras
What's the Best Way to Return Multiple Values from a Function
Pip Freeze Creates Some Weird Path Instead of the Package Version
Pyqt Showing Video Stream from Opencv
How to Rotate the Sprite and Shoot the Bullets Towards the Mouse Position
Style Active Navigation Element with a Flask/Jinja2 MACro
R, Python: Install Packages on Rpy2
Benchmarks: Does Python Have a Faster Way of Walking a Network Folder
Simple File Server to Serve Current Directory