Solving problems with virtualenvwrapper using mismatched python versions

TLDR

• understand the precedence of PATH
• understand where PATH is usually set
• make sure your python version is using the corresponding pip shell command

Although this is geared specifically towards virtualenvwrapper this could help you to figure out why a shell command you’ve installed isn’t working for some reason.

If you’ve ever used virtualenvwrapper you know how useful it is to mix and match python versions and have separate dependency silos for each project so that you don’t use the same pip for each project. On some rare occasions virtualenvwrapper might give you problems if you have multiple versions of python installed.

In my case, mkvirtualenv would fail when specifying a different python interpreter than what was default. I wanted to use python 3.7 and it was picking pip packages from python 2.7. The command that I was using was something like mkvirtualenv -p python3 my-env.

After analyzing my setup there was a couple of problems. I had several versions of python from different sources, some from macports, some from homebrew, and the built-in python that came pre-installed with Mac OS.

In order to solve what was going on I had to understand the problem. The ultimate problem was that python 3.7 was trying to use pip for python 2.7. This was apparent from the python exceptions where it was obvious that the interpreter being used was python 3.7, but the stack trace was showing paths referencing python 2.7.

What I had to was to make sure that the correct pip was installed for python 3.7. Homebrew, at least on my machine, took the last entry the macports bin path was taking precedence (showed up first in the PATH) over the homebrew bin path. That is important to understand because the way PATH works is very simple. Whatever path is defined first will be used first to find the shell command you want to run. So if you run $ python and /opt/local/bin is defined first in PATH, then it will resolve to /opt/local/bin/python.

I thought that I could remove the macports python out of the picture to see if it was the one causing problems. To do that I used port select python none. As far as I can tell, what that did was remove the symlink from the macports bin path. For some reason, that didn’t do anything which was strange. So I needed to investigate more.

I remembered to keep an eye on what in PATH as that basically controls what shell command gets picked up first. When I did echo $PATH I noticed there was still a reference to macports python. That definitely shouldn’t have happened. I had already ran port select python none. The $PATH environment gets set in multiple places. In general here’s a couple of places to look with the highest precedence being listed first:

• /etc/profile
• /etc/bashrc
• ~/.bash_profile
• ~/.profile
• ~/.bashrc

The culprit was in ~/.profile. There was an entry that prepended $PATH with macports python version export PATH=/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin:$PATH. After I commented that out my shell ended up picking up the homebrew version as I intended to. I validated this by running which python and seeing that it was picking up from the homebrew bin path /usr/local/bin and not from macports bin path /opt/local/bin. There was a problem though, the homebrew python command was actually python 2.7 and not the 3.7 I wanted to use.

I found out that homebrew installs python 3.7 as python3. I could have just made a symbolic link from python3 python, but then I would need to keep tabs that I actually did that. Macports has a port select command where I could swap out different versions of the same program, but homebrew doesn’t have that. The closest thing is brew link and unlink, but it doesn’t work the same way as in macports. All it does it remove the program from the homebrew bin path. So that didn’t really help me.

When I backtracked I remembered that for some reason python 3.7 was trying to use pip installed for python 2.7. When I ran brew link -vd to show me verbose output there weren't any references to pip which I found very suspicious. After some Google searching, I found that someone recommended doing brew reinstall python. That ended up being the crucial command that installed pip for python 3.7. It ended up using installing pip, but brew names the binary pip3 instead of pip. In homebrew, pip is for python 2.7.

The next part is very important, because it’s not just as straightforward as specifying the python interpreter version you’d like to use in mkvirtualenv -p python3. Seems like virtualenvwrapper by default uses the python version and pip version of whatever’s currently installed. As I’ve mentioned before it’s python 2.7 from homebrew that’s installed. So to get around that virtualenvwrapper has an environment variable that you could set to the python interpreter that you’d like to use for virtualenvwrapper itself. It’s called VIRTUALENVWRAPPER_PYTHON. You set it to the absolute path of the python interpreter you’d like to use. In my case I did
export VIRTUALENVWRAPPER_PYTHON=`which python3`. Now when I run mkvirtualenv -p python3, it doesn’t throw any python exceptions trying to use pip for python 2.7.