Broken References in Virtualenvs

Broken references in Virtualenvs

I found the solution to the problem here, so all credit goes to the author.

The gist is that when you create a virtualenv, many symlinks are created to the Homebrew installed Python.

Here is one example:

$ ls -la ~/.virtualenvs/my-virtual-env
...
lrwxr-xr-x  1 ryan staff   78 Jun 25 13:21 .Python -> /usr/local/Cellar/python/2.7.7/Frameworks/Python.framework/Versions/2.7/Python
...

When you upgrade Python using Homebrew and then run brew cleanup, the symlinks in the virtualenv point to paths that no longer exist (because Homebrew deleted them).

The symlinks needs to point to the newly installed Python:

lrwxr-xr-x  1 ryan staff   78 Jun 25 13:21 .Python -> /usr/local/Cellar/python/2.7.8_1/Frameworks/Python.framework/Versions/2.7/Python

The solution is to remove the symlinks in the virtualenv and then recreate them:

find ~/.virtualenvs/my-virtual-env/ -type l -delete
virtualenv ~/.virtualenvs/my-virtual-env

It's probably best to check what links will be deleted first before deleting them:

find ~/.virtualenvs/my-virtual-env/ -type l

In my opinion, it's even better to only delete broken symlinks. You can do this using GNU find:

gfind ~/.virtualenvs/my-virtual-env/ -type l -xtype l -delete

You can install GNU find with Homebrew if you don't already have it:

brew install findutils

Notice that by default, GNU programs installed with Homebrew tend to be prefixed with the letter g. This is to avoid shadowing the find binary that ships with OS X.

Homebrew broken link to Python in a virtualenv

I'll put my comment as an answer, for clarity for future visitors with the same problem.

The related links you gave tell you exactly what's the problem: because virtualenv creates symbolic links to files, and homebrew replaces those files when upgrading with differently named files, you end up with broken links.
Both related links give suggestions how to solve that: one to create a new virtualenv, the other to fix the broken links.

All in all though, when Python upgrades again in homebrew, you may end up with the same situation.

This was apparently realised by the virtualenv developers, and there is an option --always-copy to avoid problems like this:

$ virtualenv --help
Usage: virtualenv-3.4 [OPTIONS] DEST_DIR

Options:
  ...

  --always-copy         Always copy files rather than symlinking.

  ...

This should prevent problems when upgrading Python through Homebrew in the future. Though then, of course, your virtualenv will have an older version of Python. Which may at times be exactly what you want.

New Pipenv cause my previous virtualenvwrapper to break

So thanks to @ryan-marvin's comment I found the answer. It's in here - Broken references in Virtualenvs.

But the thing is, it's not exactly the same situation because for me, this command:
ls -la ~/.virtualenvs/my-virtual-env pointed to the correct python location (Because I didn't change my python).

But still I needed to run the same commands to fix the broken links:

gfind ~/.virtualenvs/my-virtual-env/ -type l -xtype l -delete
virtualenv ~/.virtualenvs/my-virtual-env

(You can install gfind with Homebrew)

And of course credit to the author.

PyCharm cannot find the packages in virtualenv

The problem may lay in PyCharm picking up faulty 'Interpreter Paths' for your virtual environment. Go here:

PyCharm (menu) -> Preferences (Menu option)
               -> Project: <name> (Dropdown) 
               -> Project Interpreter (Menu option)
               -> 'Settings' button (Looks like a gear)
               -> More (Menu option)
               -> Select your virtualenv interpreter
               -> Click 'Show paths for interpreter' button (on bottom of list window)

Now that you're in this (admittedly tortuously found) location, you should see paths being used by this interpreter. If my theory is correct, these are pointing to global system locations. To add the virtual environment paths, you should click the + button and add corresponding paths that exist inside your virtual environment. Once you're done with this, it's a good idea to select the global system paths and click - to remove them. Click apply, and go to File -> Invalidate caches / Restart to reload PyCharm.

This should get your interpreter to be pointed to the correct location for the libraries you've installed into your virtualenv, and you should no longer be getting the import error. Note that even with this fix you will not see your libraries under the Project Interpreter, but they should be being loaded.

Pip installation in virtualenv broken

Turns out that virtualenv hard links the python executable to the system python, so that when I upgraded python, it got out of line with the virtual env's pip installation.

Posting here for other to find, in case this happens to them.

Solution is to wipe the pyenv folder, and reinstall a fresh virtualenv (if you've been using a requirements.txt file, this is what the virtual env is designed to do!

---

Related Topics