Running Jupyter with multiple Python and IPython paths
This is fairly straightforward to fix, but it involves understanding three different concepts:
- How Unix/Linux/OSX use
$PATH
to find executables (%PATH%
in Windows) - How Python installs and finds packages
- How Jupyter knows what Python to use
1. Unix/Linux/OSX $PATH
When you type any command at the prompt (say, python
), the system has a well-defined sequence of places that it looks for the executable. This sequence is defined in a system variable called PATH
, which the user can specify. To see your PATH
, you can type echo $PATH
.
The result is a list of directories on your computer, which will be searched in order for the desired executable. From your output above, I assume that it contains this:
$ echo $PATH
/usr/bin/:/Library/Frameworks/Python.framework/Versions/3.5/bin/:/usr/local/bin/
In windows echo %path%
Probably with some other paths interspersed as well. What this means is that when you type python
, the system will go to /usr/bin/python
. When you type ipython
, in this example, the system will go to /Library/Frameworks/Python.framework/Versions/3.5/bin/ipython
, because there is no ipython
in /usr/bin/
.
It's always important to know what executable you're using, particularly when you have so many installations of the same program on your system. Changing the path is not too complicated; see e.g. How to permanently set $PATH on Linux?.
Windows - How to set environment variables in Windows 10
2. How Python finds packages
When you run python and do something like import matplotlib
, Python has to play a similar game to find the package you have in mind. Similar to $PATH
in unix, Python has sys.path
that specifies these:
$ python
>>> import sys
>>> sys.path
['',
'/Users/jakevdp/anaconda/lib/python3.5',
'/Users/jakevdp/anaconda/lib/python3.5/site-packages',
...]
Some important things: by default, the first entry in sys.path
is the current directory. Also, unless you modify this (which you shouldn't do unless you know exactly what you're doing) you'll usually find something called site-packages
in the path: this is the default place that Python puts packages when you install them using python setup.py install
, or pip
, or conda
, or a similar means.The important thing to note is that each python installation has its own site-packages, where packages are installed for that specific Python version. In other words, if you install something for, e.g. /usr/bin/python
, then ~/anaconda/bin/python
can't use that package, because it was installed on a different Python! This is why in our twitter exchange I recommended you focus on one Python installation, and fix your$PATH
so that you're only using the one you want to use.
There's another component to this: some Python packages come bundled with stand-alone scripts that you can run from the command line (examples are pip
, ipython
, jupyter
, pep8
, etc.) By default, these executables will be put in the same directory path as the Python used to install them, and are designed to work only with that Python installation.
That means that, as your system is set-up, when you run python
, you get /usr/bin/python
, but when you run ipython
, you get /Library/Frameworks/Python.framework/Versions/3.5/bin/ipython
which is associated with the Python version at /Library/Frameworks/Python.framework/Versions/3.5/bin/python
! Further, this means that the packages you can import when running python
are entirely separate from the packages you can import when running ipython
or a Jupyter notebook: you're using two completely independent Python installations.
So how to fix this? Well, first make sure your $PATH
variable is doing what you want it to. You likely have a startup script called something like ~/.bash_profile
or ~/.bashrc
that sets this $PATH
variable. On Windows, you can modify the user specific environment variables. You can manually modify that if you want your system to search things in a different order. When you first install anaconda/miniconda, there will be an option to do this automatically (add Python to the PATH): say yes to that, and then python
will always point to ~/anaconda/python
, which is probably what you want.
3. How Jupyter knows what Python to use
We're not totally out of the water yet. You mentioned that in the Jupyter notebook, you're getting a kernel error: this indicates that Jupyter is looking for a non-existent Python version.
Jupyter is set-up to be able to use a wide range of "kernels", or execution engines for the code. These can be Python 2, Python 3, R, Julia, Ruby... there are dozens of possible kernels to use. But in order for this to happen, Jupyter needs to know where to look for the associated executable: that is, it needs to know which path the python
sits in.
These paths are specified in jupyter's kernelspec
, and it's possible for the user to adjust them to their desires. For example, here's the list of kernels that I have on my system:
$ jupyter kernelspec list
Available kernels:
python2.7 /Users/jakevdp/.ipython/kernels/python2.7
python3.3 /Users/jakevdp/.ipython/kernels/python3.3
python3.4 /Users/jakevdp/.ipython/kernels/python3.4
python3.5 /Users/jakevdp/.ipython/kernels/python3.5
python2 /Users/jakevdp/Library/Jupyter/kernels/python2
python3 /Users/jakevdp/Library/Jupyter/kernels/python3
Each of these is a directory containing some metadata that specifies the kernel name, the path to the executable, and other relevant info.You can adjust kernels manually, editing the metadata inside the directories listed above.
The command to install a kernel can change depending on the kernel. IPython relies on the ipykernel package which contains a command to install a python kernel: for example
$ python -m ipykernel install
It will create a kernelspec associated with the Python executable you use to run this command. You can then choose this kernel in the Jupyter notebook to run your code with that Python.You can see other options that ipykernel provides using the help command:
$ python -m ipykernel install --help
usage: ipython-kernel-install [-h] [--user] [--name NAME]
[--display-name DISPLAY_NAME] [--prefix PREFIX]
[--sys-prefix]
Install the IPython kernel spec.
optional arguments:
-h, --help show this help message and exit
--user Install for the current user instead of system-wide
--name NAME Specify a name for the kernelspec. This is needed to
have multiple IPython kernels at the same time.
--display-name DISPLAY_NAME
Specify the display name for the kernelspec. This is
helpful when you have multiple IPython kernels.
--prefix PREFIX Specify an install prefix for the kernelspec. This is
needed to install into a non-default location, such as
a conda/virtual-env.
--sys-prefix Install to Python's sys.prefix. Shorthand for
--prefix='/Users/bussonniermatthias/anaconda'. For use
in conda/virtual-envs.
Note: the recent version of anaconda ships with an extension for the notebook that should automatically detect your various conda environments if the ipykernel
package is installed in it.Wrap-up: Fixing your Issue
So with that background, your issue is quite easy to fix:
Set your
PATH
so that the desired Python version is first. For example, you could runexport PATH="/path/to/python/bin:$PATH"
to specify (one time) which Python you'd like to use. To do this permanently, add that line to your.bash_profile
/.bashrc
(note that anaconda can do this automatically for you when you install it). I'd recommend using the Python that comes with anaconda or miniconda: this will allow you toconda install
all the tools you need.Make sure the packages you want to use are installed for that python. If you're using conda, you can type, e.g.
conda install jupyter matplotlib scikit-learn
to install those packages foranaconda/bin/python
.Make sure that your Jupyter kernels point to the Python versions you want to use. When you
conda install jupyter
it should set this up foranaconda/bin/python
automatically. Otherwise you can use thejupyter kernelspec
command orpython -m ipykernel install
command to adjust existing kernels or install new ones.For installing modules into other Python Jupyter kernels not managed by Anaconda, you need to copy the path to the Python executable for the kernel and run
/path/to/python -m pip install <package>
How to run Jupyter Notebook with a different version of Python?
Found it myself, the hard way. Let me share anyway, in case this helps anyone.
I guess, the problem was that, jupyter notebook installed through pacman searches for python binary files in the PATH variable and not in the path specified by the virtual environment. Since I installed Python 3.7 locally in my home directory, Jupyter can't find it and it might have defaulted to the default python version.
So the possible solutions are:
- Install Jupyter Notebook through pip (instead of pacman) within the virtual environment set on Python 3.7 (This is not at all recommended for Arch Linux users, as installing packages through pip can probably cause issues in future)
> wget https://www.python.org/ftp/python/3.7.4/Python-3.7.4.tgz
> tar -xvf Python-3.7.4.tgz
> cd Python-3.5.1/
> ./configure --prefix=$HOME/Python37
> make
> make install
> virtualenv -p ~/Python3.7/bin/python3 py37
> source py37/bin/activate
(py37) > pip install notebook
(py37) > python -m notebook
- Install Python 3.7 within default directory (instead of specifying somewhere else). Create a new IPython kernel using the suitable virtual environment and use jupyter-notebook installed through pacman. (Recommended for Arch Linux users)
Note 1:> python
points to the updated global Python 3.8 version and> python3
or> python3.7
points to newly installed Python 3.7
Note 2: Once the required kernel is created, you might even be able to use that python version outside the virtual environment.
> wget https://www.python.org/ftp/python/3.7.4/Python-3.7.4.tgz
> tar -xvf Python-3.7.4.tgz
> cd Python-3.5.1/
> ./configure
> make
> sudo make install
> virtualenv -p $(which python3.7) py37
> source py37/bin/activate
(py37) > ipython kernel install --user --name py37 --display-name "Python 3.7"
(py37) > jupyter notebook
- Add the path of the directory where you have locally installed the new Python version to the $PATH variable, create an IPython kernel and run Jupyter Notebook within suitable virtual environment. (Haven't yet tried this one personally. Just felt that this should work. So no guarantee. Also I don't think this is a good solution)
> wget https://www.python.org/ftp/python/3.7.4/Python-3.7.4.tgz
> tar -xvf Python-3.7.4.tgz
> cd Python-3.5.1/
> ./configure --prefix=$HOME/Python37
> make
> make install
> export PATH="$HOME/Python37/bin:$PATH"
> virtualenv -p py37
> source py37/bin/activate
(py37) > ipython kernel install --user --name py37 --display-name "Python 3.7"
(py37) > jupyter notebook
Using both Python 2.x and Python 3.x in IPython Notebook
A solution is available that allows me to keep my MacPorts installation by configuring the Ipython kernelspec.
Requirements:
- MacPorts is installed in the usual /opt directory
- python 2.7 is installed through macports
- python 3.4 is installed through macports
- Ipython is installed for python 2.7
- Ipython is installed for python 3.4
$ cd /opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin
$ sudo ./ipython kernelspec install-self
For python 3.x:$ cd /opt/local/Library/Frameworks/Python.framework/Versions/3.4/bin
$ sudo ./ipython kernelspec install-self
Now you can open an Ipython notebook and then choose a python 2.x or a python 3.x notebook. How to make Jupyter notebook use PYTHONPATH in system variables without hacking sys.path directly?
Jupyter uses its own JUPYTER_PATH environment variable.
Related Topics
Why Do We Need to Call Zero_Grad() in Pytorch
Pandas Interpolate Within a Groupby
String with 'F' Prefix in Python-3.6
Cannot Redirect Output When I Run Python Script on Windows Using Just Script's Name
Crontab Not Executing a Python Script
Pandas Latitude-Longitude to Distance Between Successive Rows
Syntax Error: Invalid Syntax' for No Apparent Reason
List Sorting with Multiple Attributes and Mixed Order
Turning Off Logging in Selenium (From Python)
Serve Image Stored in SQLalchemy Largebinary Column
Which Classes Cannot Be Subclassed
Improve Current Implementation of a Setinterval
Python-Pandas: the Truth Value of a Series Is Ambiguous
Python & Selenium: Difference Between Driver.Implicitly_Wait() and Time.Sleep()
Why Isn't Assigning to an Empty List (E.G. [] = "") an Error
Lookup Values by Corresponding Column Header in Pandas 1.2.0 or Newer