Setting up virtual environments for Python in OSX

One of the things I wish I had known more about when I first started working with Python (and, really, any time before I finally learned about it a couple years ago) is the power of virtual environments for Python using the virtualenv tool.

The basic idea with virtual environments is that you have a project that might need a specific version of Python (2.7, 3, whatever), and specific versions of whichever packages you are using. For a developer working on multiple projects, this specifically allows you to avoid version hell where upgrading a package for one project breaks an unrelated project. This also ensures everyone on the team is using the same version of every package without having to add all the package source to the team's repository. All the virtual Python environments will exist in one place and you can switch between them with a single command.

This page will guide you through setting up virtual environments in OSX. The steps are similar for Linux but very different for Windows. Here are the highlights:

  1. install the virtualenvwrapper package at the system level
  2. set up your environment
    1. disable pip for the system-level Python installation so that all packages are only ever installed in virtual environments (optional)
  3. set up your top-level virtual environment directory
  4. create your first virtual environment
  5. learn the commands for using virtual environments

Prerequisites

You will need to have pip installed for this to work properly. In a terminal window, type pip. If you get a dump of how to use the command, you're all set. If you see a 'command not found' error, type the following:

$ sudo easy_install pip

You will need to enter your password. Accept the defaults for any options it gives you.

Install the virtualenvwrapper package

mkvirtualenv is the tool you will use to quickly create a new virtual environment or remove one you are no longer using. It's part of a package called virtualenvwrapper. To install this package, open a Terminal window and type:

$ sudo pip install virtualenvwrapper

You will need to enter your password. Accept the defaults for any options it gives you.

Set up your environment

You will need to set a couple environment variables for virtualenvwrapper to work properly. Open your .profile with TextEdit (or your favorite editor):

$ open ~/.profile

... and put the following three lines anywhere in the file:

export WORKON_HOME=~/dev/virtualenv
export PIP_REQUIRE_VIRTUALENV=true
source /usr/local/bin/virtualenvwrapper.sh

The first line tells the virtualenvwrapper set of commands where to find your virtual environments. I have a directory in my home directory called 'dev' and I put all external libraries and SDKs there... it's a reasonable place to put this as well. Some people put it at ~/.virtualenv, so you might see instructions elsewhere recommending this location. It doesn't really matter where it goes, but these two choices are probably better than most others if you have no strong reason to put it somewhere else.

The second line tells pip to refuse to install packages unless you are working in a virtual environment. This is a good thing. You might even consider removing all the packages you've previously installed so that you don't accidentally use one when you think you are in a virtual environment.

The third line tells bash (your terminal shell) to always turn on the virtualenvwrapper tools automatically.

You need to make these changes active in your terminal window because they weren't there when you started it:

$ source ~/.profile
$ echo $WORKON_HOME
/Users/seawolf/dev/virtualenv

You can also just close and restart the Terminal window and type the echo $WORKON_HOME command to make sure .profile is being loaded automatically.

Set up your top-level virtual environment directory

You need to create the directory where your virtual environments will live:

$ mkdir -p $WORKON_HOME

That's it!

Create your first virtual environment

I have a general virtual environment I call 'system' where I keep a hodge-podge collection of packages for testing, exploring, and just playing around.

$ mkvirtualenv system
New python executable in system/bin/python2.7
Also creating executable in system/bin/python
Installing setuptools, pip...done.

That was easy. You might see more lines of output as a few low-level tools are installed for the first time.

Commands for using virtual environments

The commands you will use the most are the commands to:

  1. start a virtual environment
  2. switch to a different virtual environment
  3. install a package
  4. uninstall a package
  5. list all the installed packages
  6. create a requirements.txt file
  7. install all packages listed in a requirements.txt file
  8. exit the virtual environment

Starting a virtual environment is easy: just type workon <environment name> (insert the correct name in place of the placeholder, obviously). Your command prompt will change to show you the name of the environment you are currently in:

$ workon system
(system)$

To switch between environments, you use the same command:

(system)$ workon my_happy_project
(my_happy_project)$

Tab works in all the commands to auto-complete where possible. In my case, system is the only virtualenv that starts with an 's', so I can just type workon s<tab> and it will auto-fill in the rest of the command for me.

Installing a package works just like you would expect using pip:

(system)$ pip install numpy
Downloading/unpacking numpy
  Downloading numpy-1.9.2-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x
  Downloading numpy-1.9.2-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x
  ... (lines removed) ...
86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (3.7MB): 3.7MB downloaded
Installing collected packages: numpy
Successfully installed numpy
Cleaning up...
(system)$

Uninstalling is equally easy:

(system)$ pip uninstall numpy
Uninstalling numpy:
  /Users/seawolf/dev/py_virtualenv/my_happy_project/bin/f2py
  /Users/seawolf/dev/py_virtualenv/my_happy_project/lib/python2.7/site-packages/numpy-1.9.2.dist-info/DESCRIPTION.rst
  /Users/seawolf/dev/py_virtualenv/my_happy_project/lib/python2.7/site-packages/numpy-1.9.2.dist-info/METADATA
  ... (lines removed) ...
Proceed (y/n)? y
  Successfully uninstalled numpy
(system)$

Listing all installed packages is done with pip freeze:

(system)$ pip freeze
numpy==1.9.2
wsgiref==0.1.2
(system)$

Notice that version numbers are shown, which makes it easy to see (and recreate) the combination of packages you are using.

By redirecting the output of pip freeze into a separate file, you capture the list for later use:

(system)$ pip freeze > requirements.txt
(system)$ more requirements.txt
numpy==1.9.2
wsgiref==0.1.2
(system)$

You can later use this file to recreate your environment on another machine (or in a different virtualenv):

(system)$ pip install -r requirements.txt
... (output as all packages are installed) ...
(system)$

Pip is smart enough to tell you if a particular package is already installed and will skip those, so you can just install from requirements any time someone else changes the file.

Finally, to exit the virtual environment altogether when you are finished:

(system)$ deactivate
$

That's it! Happy programming in your new virtual environments!