Tools

This directory contains some information that might be useful to set up the tool stack required for this training session.

What is it?

  1. pipx_install.md: instructions to install dvc using pipx (includes instructions to install pipx).

  2. ollama.py: HPCCM specification to build a container with ollama installed.

  3. ollama.recipe: Singularity/Apptainer recipe to build a container with ollama installed.

Installing DVC on HPC systems

The most convenient way to install DVC on HPC systems is by using pipx, which allows you to install and run Python applications in isolated environments. This method avoids conflicts with system packages and other Python applications.

If pipx can not be installed on your HPC system, a virtual environment can be used as an alternative.

Prerequisites

On most HPC systems, Python and pip will be available as part of the system image. However, it is likely that it is an older version. In that case, you can install DVC, but it will be an older version as well.

Load a Python module if available:

$ module load python

Create a virtual environment to install pipx. (This step is not required if pipx is already available on your system, or it can be installed usring python -m pip install --user pipx.)

$ python3 -m venv ~/pipx_venv --system-site-packages
$ source ~/pipx_venv/bin/activate

Install pipx using pip and initialize::

$ python -m pip install pipx
$ python -m pipx ensurepath

Install DVC using pipx:

$ pipx install dvc[ssh]

Now you can deactivate the virtual environment and unload the Python module if needed:

$ deactivate
$ module unload python

Now DVC is installed and ready to use on your HPC system.

Troubleshooting

Although applications installed with pipx are isolated from the system Python packages, things can go south if PYTHONPATH is set. If that is the case, you can run the applications nevertheless, e.g.,

$ PYTHONPATH= dvc --version

Alternatively, you can unset the PYTHONPATH variable in the following way:

$ env -u PYTHONPATH dvc --version

More conveniently, you can define an alias in your shell configuration file:

alias dvc='env -u PYTHONPATH dvc'