Python Virtual Environments

Table of Contents

• Overview
• Creating a Python Virtual Environment
• Activating a Python Virtual Environment
• Installing Packages in a Python Virtual Environment
• Deactivating a Python Virtual Environment
• Using a Python Virtual Environment
• Using Python Virtual Environments with Jupyter

Python virtual environments are a useful tool that allow a user to configure a python environment for a specific purpose without introducing conflicts between packages or package versions for other applications. If a user had two applications that required different versions of a python package, they would be able to install each version in a virtual environment and avoid conflicts with the applications. More can be read about Python virtual environments here: https://docs.python.org/3.9/tutorial/venv.html

The module venv is used to create a virtual environment. Cluster users will have to load the version of python they need first with a module load command.

Once they selected the version of python they want, they can create their virtual environment. These environments can be large, depending on what packages are installed, so it is recommended that they are created in a space other than the home directory. To create a python3 environment named Test1 in the current directory, a user would run:

 This will create the directory Test1 as well as subdirectories with the files for running Python in the current directory.

Once the environment is created, it needs to be activated to install packages and be used. Activating the environment can be done by running the source command with the path to the activate script that was created with the environment. For the Test1 example created above, the command would be:

Running this will change the shell prompt to indicate that the environment is active by including the python environment name in round brackets. For the Test1 example it would display:

Once the environment is active, packages can be installed in it and it can be used.

In an active environment, packages can be installed with pip. To install NumPy with pip in an active environment, the following command would be run:

To uninstall NumPy from an active environment the following command would be run:

Note: The --no-cache-dir option is specified to prevent the default behavior of a cache files being created in the user's home directory, which can fill the directory. Documentation for pip's caching behavior can be found here. 

To stop using a virtual environment, a user just has to run deactivate from their shell.

Python virtual environments can be created and used both interactively and in slurm jobs. To use the environment in a slurm job, a user just has to include the source command. From the Test1 example above, the command would be:

Virtual environments can be used with Jupyter notebooks on the Open OnDemand platform with ISU clusters. This requires that the ipykernel package be installed and the config placed in the user's home directory. To do this, first:

Start an interactive session to prevent excessive load on the login node:

 As above, choose a version of python, create a Python virtual environment, and activate it. Note: Home directories should be avoided, as they will fill rapidly. 

Install the ipykernel package. 

Create the ipykernel config in the user's home directory. This is not an issue since the config is small. 

Once the config is created, the environment can be used by:

• Logging in to the appropriate cluster's Open OnDemand. See guide here
• Creating a Jupyter Notebook Session
  • Select Interactive Apps
  • Select Jupyter Notebook from the list of apps
  • Select desired compute and partition settings. 
  • Select Launch
  • Once it starts, select Connect to Jupyter
• In the Jupyter Notebook Launcher, select JupyterTest1 from the Notebook or Console section. It can also be selected as the kernel when choosing the kernel for a new notebook or console.