Python Virtual Environments

Table of Contents


Python virtual environments are a useful tool that allow a user to configure a python environment for a specific purpose without introducing conflicts between packages or package versions for other applications. If a user had two applications that required different versions of a python package, they would be able to install each version in a virtual environment and avoid conflicts with the applications. More can be read about Python virtual environments here:

Creating a Python Virtual Environment

The module venv is used to create a virtual environment. Cluster users will have to load the version of python they need first with a module load command.

module load python

Once they selected the version of python they want, they can create their virtual environment. These environments can be large, depending on what packages are installed, so it is recommended that they are created in a space other than the home directory. To create a python3 environment named Test1 in the current directory, a user would run:

python3 -m venv Test1

 This will create the directory Test1 as well as subdirectories with the files for running Python in the current directory.

Activating a Python Virtual Environment

Once the environment is created, it needs to be activated to install packages and be used. Activating the environment can be done by running the source command with the path to the activate script that was created with the environment. For the Test1 example created above, the command would be:

source Test1/bin/activate

Running this will change the shell prompt to indicate that the environment is active by including the python environment name in round brackets. For the Test1 example it would display:

(Test1) $

Once the environment is active, packages can be installed in it and it can be used.

Installing Packages in a Python Virtual Environment

In an active environment, packages can be installed with pip. To install NumPy with pip in an active environment, the following command would be run:

python3 -m pip --no-cache-dir install numpy

To uninstall NumPy from an active environment the following command would be run:

python3 -m pip --no-cache-dir uninstall numpy

Note: The --no-cache-dir option is specified to prevent the default behavior of a cache files being created in the user's home directory, which can fill the directory. Documentation for pip's caching behavior can be found here

Deactivating a Python Virtual Environment

To stop using a virtual environment, a user just has to run deactivate from their shell.


Using a Python Virtual Environment with ISU Clusters

Python virtual environments can be created and used both interactively and in slurm jobs. To use the environment in a slurm job, a user just has to include the source command. From the Test1 example above, the command would be:

### Script Above ###

source /path/to/Test1/bin/activate

### Script Below ###

Using Python Virtual Environments with OOD Jupyter Notebooks

Virtual environments can be used with Jupyter notebooks on the Open OnDemand platform with ISU clusters. This requires that the ipykernel package be installed and the config placed in the user's home directory. To do this, first:

Start an interactive session to prevent excessive load on the login node:

salloc #desired options

 As above, choose a version of python, create a Python virtual environment, and activate it. Note: Home directories should be avoided, as they will fill rapidly. 

module load python
python3 -m venv JupyterTest1
source JupyterTest1/bin/activate

Install the ipykernel package. 

 python3 -m pip install ipykernel --no-cache-dir

Create the ipykernel config in the user's home directory. This is not an issue since the config is small. 

python3 -m ipykernel install --user --name "JupyterTest1"

Once the config is created, the environment can be used by:

  • Logging in to the appropriate cluster's Open OnDemand. See guide here
  • Creating a Jupyter Notebook Session
    • Select Interactive Apps
    • Select Jupyter Notebook from the list of apps
    • Select desired compute and partition settings. 
    • Select Launch
    • Once it starts, select Connect to Jupyter
  • In the Jupyter Notebook Launcher, select JupyterTest1 from the Notebook or Console section. It can also be selected as the kernel when choosing the kernel for a new notebook or console.