Home » Odeon Blogs » Calvin, Weaver »

Virtualenv for multiple python projects

Virtualenv for multiple python projects


Virtualenv is a simple and effective way of isolating a Python environment on a specific machine. 

Why? 

Why do we need an isolated Python environment? 

The need for virtualenv begins innocently enough after your first django or other types of Python project. Each distinct project you are working on may depend on different python libraries.

For example, for "project A" that you started say sometime in 2009 could very well be using Python Imaging Library (commonly known as PIL) version 1.1.6.  How do we know? We check it like this in our Python shell:

  1. $ python

  2. Python 2.7.2 (default, Aug 23 2011, 20:21:01) 

  3. [GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.9)] on darwin

  4. Type "help", "copyright", "credits" or "license" for more information.

  5. >>> from PIL import Image

  6. >>> Image.VERSION

  7. '1.1.6'

  8. >>>

It so happens that PIL 1.1.6 does not support processing of interlaced PNG files when you were working on Project A some time ago.

Fortunately, you have made a great choice in using PIL and it is an active Python library with a community of developers constantly improving it.  So discovering that interlaced PNG support has now been implemented -  http://hg.effbot.org/pil-2009-raclette/issue/2/interlaced-png-support-patch (Hip Hip Hooray!) - you now happily decide to upgrade your PIL to 1.1.7 and use PIL 1.1.7 for the new Projects B that you have just started.

Unfortunately, the code which you wrote in Project A conflicts with your OS-wide (system wide) PIL upgrade from 1.1.6 to 1.1.7 and now you have to waste time refactoring your code in Project A.

This is a common scenario of what could transpire when you are working on multiple Python projects on the same machine - you may be forced to update some code you wrote a long time ago (even though there isn't a real need to)  simply because the open source Python libraries you have installed are kept in your system wide "site-packages" directory.

A Brief Note on site-packages and Python Path

Now, what is this "site-packages" directory, you ask?

On your terminal, run:

  1. $ python -c "from distutils.sysconfig import get_python_lib; print get_python_lib()"

  2. /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages

As you can see, when I first installed Python 2.7 on my Mac OS, the "site-packages" directory is an automatically created directory during the installation process and it is where all the additional open source/3rd party Python libraries are stored when we additionally install these open source/3rd party Python libraries.

This seemingly magical site-packages directory is in your Python search path (simply known as "Python Path") by default after installation.

What that simply means is that when we start a Python shell or run a Python script, stating 

  1. from PIL import Image

will automatically work and we now are able to write image processing functionality in our Python code as provided by PIL's "Image" class.   And the PIL source code has been placed in the site-packages directory during the installation process.

So, back to our Problem

So in our described scenario, what we are now facing is that Project A was using PIL 1.1.6 and we want our new project B to use PIL 1.1.7 with its new interlacing PNG capability (feature)!

Because of practical time constraints, we do not have time to upgrade Project A at the current moment and we want to be able to continue working on project B without delay.

Virtualenv solves this problem for us by allowing us to create an isolated group of directories (the environment) which ignores our OS wide (system wide) PIL 1.1.6 installation once we activate this environment.

A new set of directories with its own "site-packages" subdirectory will be created and when we create this virtual python environment.  With a simple virtualenv command, we can tell our virtual python environment to completely ignore the system wide site-packages directory and install PIL 1.1.7 specifically in that environment.

Let's begin our step-by-step

If you haven't installed the virtualenv software on your machine, do so.  

If you are using macports, it is a simple one-liner to get virtualenv installed on your Mac OS:

  1. $ sudo port -v install py27-virtualenv

To check that we have gotten our installation done correctly, simply run

  1. $ virtualenv --version

  2. 1.6.1

And to take a peek at all the available subcommands which virtualenv provides us with, run

  1. $ virtual --help

  2. Usage: virtualenv [OPTIONS] DEST_DIR

  3. 

  4. Options:

  5.   --version             show program's version number and exit

  6.   -h, --help            show this help message and exit

  7.   -v, --verbose         Increase verbosity

  8.   -q, --quiet           Decrease verbosity

  9.   -p PYTHON_EXE, --python=PYTHON_EXE

  10.                         The Python interpreter to use, e.g.,

  11.                         --python=python2.5 will use the python2.5 interpreter

  12.                         to create the new environment.  The default is the

  13.                         interpreter that virtualenv was installed with (/opt/l

  14.                         ocal/Library/Frameworks/Python.framework/Versions/2.7/

  15.                         Resources/Python.app/Contents/MacOS/Python)

  16.   --clear               Clear out the non-root install and start from scratch

  17.   --no-site-packages    Don't give access to the global site-packages dir to

  18.                         the virtual environment

  19.   --unzip-setuptools    Unzip Setuptools or Distribute when installing it

  20.   --relocatable         Make an EXISTING virtualenv environment relocatable.

  21.                         This fixes up scripts and makes all .pth files

  22.                         relative

  23.   --distribute          Use Distribute instead of Setuptools. Set environ

  24.                         variable VIRTUALENV_USE_DISTRIBUTE to make it the

  25.                         default

  26.   --extra-search-dir=SEARCH_DIRS

  27.                         Directory to look for setuptools/distribute/pip

  28.                         distributions in. You can add any number of additional

  29.                         --extra-search-dir paths.

  30.   --never-download      Never download anything from the network.  Instead,

  31.                         virtualenv will fail if local distributions of

  32.                         setuptools/distribute/pip are not present.

  33.   --prompt==PROMPT      Provides an alternative prompt prefix for this

  34.                         environment

Creating our isolated Python virtual environment

Now, to create our isolated virtual environment for us to continue working with Project B using PIL 1.1.7, we run

  1. $ virtualenv -p python2.7 --no-site-packages --distribute projectbenv

  2. Running virtualenv with interpreter /opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7

  3. New python executable in projectb/bin/python

  4. Installing distribute.....................................................................................................................................................................................done.

  5. Installing pip...............done.

Notes:
  • The option -p is where we specify the Python version we want to use in the environment named "projectbenv".
  • When we specify the option "--no-site-packages", we are saying that the environment we are creating will ignore the system wide site-packages.
  • --distribute is where we tell our environment to use a python library called "python distribute" instead of the default "python setuptools".  We will discuss distribute versus setuptools as a separate topic.  Just take my word for it at the moment that distribute is a lot better than setuptools :-)

  1. $ ls -la projectbenv/

  2. total 8

  3. drwxrwxrwx    6 calvin  staff   204 Aug 23 23:42 .

  4. drwxrwxrwx  103 calvin  staff  3502 Aug 23 23:42 ..

  5. lrwxrwxrwx    1 calvin  staff    66 Aug 23 23:42 .Python -> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/Python

  6. drwxrwxrwx   12 calvin  staff   408 Aug 23 23:42 bin

  7. drwxrwxrwx    3 calvin  staff   102 Aug 23 23:42 include

  8. drwxrwxrwx    3 calvin  staff   102 Aug 23 23:42 lib

As you can see, a bunch of directories have been created. If you drill down the directories further, you will find a "site-packages" directory under the "lib" directory.  This is the place where all your open source libraries (for instance, when you install PIL 1.1.7 in our above example scenario) will be kept when you install those libraries when you are in the environment.

How Do We Get In (and out of) this Python Virtual Environment?

So that's the million dollar question - after we have created such an environment, how do we begin hopping into this "4th dimension"? :-)

In the bin directory that was created above, there is a command named "activate".

So let's say we want to create a project directory separately. Yes - there's no necessity to keep your project directory inside the project environment directory.

  1. $ mkdir project

  2. $ ls -la

  3. total 2

  4. drwxrwxrwx    6 calvin  staff   204 Aug 23 23:42 .

  5. drwxrwxrwx  103 calvin  staff  3502 Aug 23 23:42 ..

  6. drwxrwxrwx    3 calvin  staff   102 Aug 23 23:42 projectb

  7. drwxrwxrwx    3 calvin  staff   102 Aug 23 23:42 projectbenv
 And we get into the virtual "projectbenv" environment by simply running the activate command (which happens to be kept in the bin directory).

  1. $ source projectbenv/bin/activate

You will now notice that you have a "(projectbenv)" string right above your command line prompt ($).

That tells you that whatever you do from this point onwards, you are in that virtual environment and any open source Python packages which you install using a PIP command, will be installed into projectbenv's site-packages.

To get out of this virtual environment, simply run

  1. $ deactivate

Now you are equipped with the basic skills to manage multiple Python projects where each project can depend on different versions of open source Python libaries! Hack away! ;-)




Category: Python


Tagged as: distribute site-packages virtualenv



Leave a Comment :

(required)


(required)




(required)




(required)






Leave a Comment


Page generated in: 0.62s