Virtual Environments: Construction and Use

There are several issues for constructing and use virtual environments (VEs). The high level categories are here with links to further details.

Constructing Virtual Environments

There are multiple ways to generate virtual environments (VEs). You must build a VE on the cluster and on the compute node on which you intend to use the VE (unless a head node has the same architecture as the target compute nodes).

Two are listed below and there are ARC pages for how to build these:

• Making a Conda Virtual Environment (CVE)

• Making a pip-venv Virtual Environment (VE)

ARC recommends using the first of the two methods, which produces CVEs using Miniforge.

Simple Testing of Virtual Environments

When using Python with VEs, it is useful to perform simple tests to ensure that the import statements in your Python code are satisfied by the contents of the VE.

Doing this is straight-forward. First, you can do this before you deactivate the new VE in each of the instruction sets above, or you can re-request resources from slurm via salloc just as you do in the procedures above, log into the provided compute node, and activate the VE. After completing one of these two steps, you are now ready to do the evaluation. You:

• enter python, which takes you to the python command interpreter.

• successively import each package that is in your code, by using the exact syntax in your Python source code, and usually has the form (a) import <package_name> or (b) import <package_name> as <abbrev>

If you execute all of the imports while in the Python interpretor, and no errors occur, then your VE is ready for use.

Using Virtual Environments in Jupyter Notebooks Through Open OnDemand (OOD)

Many users frequently work with Jupyter Notebooks on Open OnDemand and need to use their custom VEs within these notebooks. Users may encounter issues integrating their VEs. The steps in Using Virtual Environments and Jupyter Notebooks in OnDemand provide a reliable method to ensure that their VEs work seamlessly in Jupyter Notebooks on OnDemand.