Software installation on HPC systems

Questions

  • Should the APP need to be installed on HPC ?

  • Should I install the APP by myself ?

A not very accurate metaphor: using HPC looks like renting a room with a shared kitchen, and the landlord has very strict rules.

Caution

as a normal user on HPC system

  • no root access

    • no package manager like: apt,yum,dfn …

    • only user’s home and project directory

Always Start from Reading the Document.

LUMI

Location Country

Finland

CPU Architecture

AMD EPYC

GPU Type

AMD Instinct MI250X

User Document/Guide Link

https://docs.lumi-supercomputer.eu/

Support Contact

https://docs.lumi-supercomputer.eu/support/

LEONARDO

Location Country

Italy

CPU Architecture

Intel Xeon ICE LAKE

GPU Type

NVIDIA A100 SXM4 64GB

User Document/Guide Link

https://docs.hpc.cineca.it/hpc/leonardo.html

Support Contact

superc [AT] cineca.it

Vega

Location Country

Slovenia

CPU Architecture

AMD EPYC

GPU Type

NVIDIA A100

User Document/Guide Link

https://doc.vega.izum.si/

Support Contact

support [at] sling.si

Karolina

Location Country

Czechia

CPU Architecture

AMD EPYC

GPU Type

NVIDIA A100

User Document/Guide Link

https://docs.it4i.cz/en/docs/clusters/karolina/introduction/

Support Contact

support [at] it4i.cz

MeluXina

Location Country

Luxembourg

CPU Architecture

AMD EPYC

GPU Type

NVIDIA A100

User Document/Guide Link

https://docs.meluxina.eu/

Support Contact

servicedesk [at] lxp.lu

Deucalion

Location Country

Portugal

CPU Architecture

ARM A64FX, AMD EPYC

GPU Type

NVIDIA A100

User Document/Guide Link

https://docs.macc.fccn.pt/

Support Contact

deucalion [at] support.macc.fccn.pt

Discoverer

Location Country

Bulgaria

CPU Architecture

AMD EPYC

GPU Type

NVIDIA A100

User Document/Guide Link

https://docs.discoverer.bg/

Support Contact

helpdesk [at] discoverer.bg

MareNostrum 5

Location Country

Spain

CPU Architecture

Intel Xeon Sapphire Rapids

GPU Type

NVIDIA Hopper

User Document/Guide Link

https://www.bsc.es/supportkc/docs/MareNostrum5/intro/

Support Contact

eurohpc_support [at] bsc.es

Check What is Already Installed

Objectives

  • learn to how to use the module tool

  • understand what does module tool do for you

If you would like to cook in the shared kitchen, it is a good idea to start with checking what kind of cooking wares are provided.

In HPC (High-Performance Computing) systems, the most common tool used to manage installed libraries and applications is Environment module systems: using command with module or ml.

  • List all or matching available modules

    module avail
    

    or e.g you want to list all modules which match with “intel”

    module avail intel
    
  • Load modules

    e.g load the gcc with version 12.2.0

    module load gcc/12.2.0
    
  • List loaded modules

    module list
    
  • Remove/unload the modules

    e.g remove the module gcc with version 12.2.0

    module remove gcc/12.2.0
    
    • remove all modules

    module purge
    
  • show what is in the module

    module show gcc/12.2.0
    

A Quick Revisit of How to Compile Source Code into Library and Executable

Here we use a simple example as demo:

  • mylib: click to download mylib.tar

    mylib is library contains only 1 function of naive matrix multiplication \(C_{ij} = \sum_k A_{ik} \times B_{kj} + C_{ij}\)

    mylib
    ├── include
    │   └── mymath.h
    └── src
        └── mymath.c
    
  • myapp: click to download myapp.tar

    myapp benchmark the naive matrix multiplication from mylib and also gemm from cblas library.

    myapp
    └── src
        └── main.c
    

manually compile the source code

Objectives

  • understand

    • compiler flag -I and env var C_INCLUDE_PATH or CPATH

    • compiler flag -L and env var LIBRARY_PATH

    • environment variable LD_LIBRARY_PATH

    • compiler flag -O3

    • use compiler preprocessor flag -D

  • use ldd check linked dynamic libraries

  • create shared library

    gcc --shared mymath.c -o libmylib.so
    
  • compile main.c to “obj”

    gcc -c myapp.c
    

    you may get error message:

    output

    main.c:5:10: fatal error: mymath.h: No such file or directory
     #include "mymath.h"
              ^~~~~~~~~~
    compilation terminated.
    

    you can choose from:

    • use env variable C_INCLUDE_PATH or CPATH

      C_INCLUDE_PATH=../../mylib/include gcc -c main.c
      
      CPATH=../../mylib/include gcc -c main.c
      
    • use compiler flag -I

      gcc -I ../../mylib/include -c main.c
      
  • create the myapp.exe with linking main.o and libmylib.so

       gcc -o myapp.exe main.o -lmylib
    

    you may get error message:

    output

    /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: cannot find -lmylib: No such file or directory
    collect2: error: ld returned 1 exit status
    

    you can choose from:

    • use env variable LIBRARY_PATH

      LIBRARY_PATH=../../mylib/ gcc -o myapp.exe main.o -lmylib
      
    • use compiler flag -L

      gcc -o myapp.exe main.o -L ../../mylib/ -lmylib
      
  • run the myapp.exe

    ./myapp.exe
    

    you may get error message:

    output

    ./myapp.exe: error while loading shared libraries: libmylib.so: cannot open shared object file: No such file or directory
    

    let’s check what libraries myapp.exe linked with

    ldd ./myapp.exe
    

    output:

      linux-vdso.so.1 (0x00007ffe281bb000)
      libmylib.so => not found
      libc.so.6 => /lib64/libc.so.6 (0x0000146264f43000)
      /lib64/ld-linux-x86-64.so.2 (0x0000146265170000)
    

    libmylib.so => not found

    use env variable LD_LIBRARY_PATH

    LD_LIBRARY_PATH=../../mylib/ ldd ./myapp.exe
    

    output:

    linux-vdso.so.1 (0x00007fff71de5000)
    libmylib.so => ../../mylib/libmylib.so (0x000015105741f000)
    libc.so.6 => /lib64/libc.so.6 (0x00001510571f4000)
    /lib64/ld-linux-x86-64.so.2 (0x0000151057424000)
    

    now run with:

    LD_LIBRARY_PATH=../../mylib/ ./myapp.exe
    

    output:

    Benchmarking GEMM (Size: 512x512, Trials: 100)
    --------------------------------------------------
    Manual GEMM (Avg): 0.616966 seconds, checksum= 5171200.000000
    
  • compile mylib with -O3

    1cd mylib/src
    2gcc -O3 --shared mymath.c -o ../libmylib.so
    3cd -
    4cd myapp/src
    5gcc -I ../../mylib/include -c main.c
    6gcc -o myapp.exe main.o -L ../../mylib/ -lmylib
    7LD_LIBRARY_PATH=../../mylib/ ./myapp.exe
    

    output:

    Benchmarking GEMM (Size: 512x512, Trials: 100)
    --------------------------------------------------
    Manual GEMM (Avg): 0.062695 seconds, checksum= 5171200.000000
    

    compare to time without -O3, myapp.exe almost has 10x speed

  • with -D CBLAS to call cblas_dgemm

    43#ifdef CBLAS
    44    // 1. Warm-up (Ensures CPU is out of power-saving mode)
    45    cblas_dgemm(CblasRowMajor, CblasNoTrans, CblasNoTrans,
    46                    N, N, N, 1.0, A, N, B, N, 0.0, C, N);
    47    for (int i = 0; i < N * N; i++) { C[i] = 0.0; }
    48
    49    checksum = 0.0;
    50    // 2. Benchmark BLAS DGEMM
    51    start = get_time();
    52    for(int t = 0; t < TAIALS; t++) {
    53        cblas_dgemm(CblasRowMajor, CblasNoTrans, CblasNoTrans,
    54                    N, N, N, 1.0, A, N, B, N, 1.0, C, N);
    55        checksum += C[t];
    56    }
    57    end = get_time();
    58    printf("System BLAS (Avg): %f seconds, checksum= %f\n", (end - start) / TAIALS, checksum);
    59#endif
    

    compile the main.c same as above, simply add -D CBLAS. we also need let the compiler find the header file: cblas.h and link to the BLAS library.

    ml purge
    ml LUMI/25.03 partition/C PrgEnv-gnu
    cd mylib/src
    cc -O3 --shared mymath.c -o ../libmylib.so
    cd -
    cd myapp/src
    cc -I ../../mylib/include -c -DCBLAS main.c
    cc -o myapp.exe main.o -L ../../mylib/ -lmylib
    LD_LIBRARY_PATH=../../mylib/ ./myapp.exe
    

    output:

    Benchmarking GEMM (Size: 512x512, Trials: 100)
    --------------------------------------------------
    Manual GEMM (Avg): 0.060575 seconds, checksum= 5171200.000000
    System BLAS (Avg): 0.005845 seconds, checksum= 5171200.000000
    

    with dgemm from highly optimized BLAS library provided by Cray libsci, we get another 10x speed

    LD_LIBRARY_PATH=../../mylib/ ldd myapp.exe
      linux-vdso.so.1 (0x00007ffe7ddf1000)
      libmylib.so => ../../mylib/libmylib.so (0x0000154025ba7000)
      libsci_gnu.so.6 => /opt/cray/pe/lib64/libsci_gnu.so.6 (0x0000154023522000)
      libxpmem.so.0 => /usr/lib64/libxpmem.so.0 (0x000015402351f000)
      libc.so.6 => /lib64/libc.so.6 (0x0000154023308000)
      libgfortran.so.5 => /usr/lib64/libgfortran.so.5 (0x0000154023003000)
      libm.so.6 => /lib64/libm.so.6 (0x0000154022f19000)
      /lib64/ld-linux-x86-64.so.2 (0x0000154025bac000)
      libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000154022eec000)
    
    ml purge
    ml gcc/12.2.0 intel-oneapi-mkl/2024.0.0
    cd mylib/src
    gcc -O3 --shared mymath.c -o ../libmylib.so
    cd -
    cd myapp/src; sed -i 's/<cblas.h>/<mkl_cblas.h>/' main.c
    gcc -I ../../mylib/include -c -DCBLAS main.c
    gcc -o myapp.exe main.o -L ../../mylib/ -lmylib -Wl,--no-as-needed -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -ldl
    LD_LIBRARY_PATH=../../mylib/:$LD_LIBRARY_PATH ./myapp.exe
    

    output:

    Benchmarking GEMM (Size: 512x512, Trials: 100)
    --------------------------------------------------
    Manual GEMM (Avg): 0.033518 seconds, checksum= 5171200.000000
    System BLAS (Avg): 0.003465 seconds, checksum= 5171200.000000
    

    with dgemm from highly optimized BLAS library provided by MKL, we get another 10x speed

    LD_LIBRARY_PATH=../../mylib/:$LD_LIBRARY_PATH ldd ./myapp.exe
      linux-vdso.so.1 (0x00007ffc23bb9000)
      libmylib.so => ../../mylib/libmylib.so (0x00007f39bfb08000)
      libmkl_intel_lp64.so.2 => /leonardo/prod/spack/06/install/0.22/linux-rhel8-icelake/gcc-8.5.0/intel-oneapi-mkl-2024.0.0-vexzfithc53dsedryb625yywykakdg7m/mkl/2024.0/lib/libmkl_intel_lp64.so.2 (0x00007f39be3d1000)
      libmkl_sequential.so.2 => /leonardo/prod/spack/06/install/0.22/linux-rhel8-icelake/gcc-8.5.0/intel-oneapi-mkl-2024.0.0-vexzfithc53dsedryb625yywykakdg7m/mkl/2024.0/lib/libmkl_sequential.so.2 (0x00007f39bcfc9000)
      libmkl_core.so.2 => /leonardo/prod/spack/06/install/0.22/linux-rhel8-icelake/gcc-8.5.0/intel-oneapi-mkl-2024.0.0-vexzfithc53dsedryb625yywykakdg7m/mkl/2024.0/lib/libmkl_core.so.2 (0x00007f39b8e97000)
      libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f39b8c77000)
      libm.so.6 => /lib64/libm.so.6 (0x00007f39b88f5000)
      libdl.so.2 => /lib64/libdl.so.2 (0x00007f39b86f1000)
      libc.so.6 => /lib64/libc.so.6 (0x00007f39b831a000)
      /lib64/ld-linux-x86-64.so.2 (0x00007f39bf8e0000)
    

compile the source code with makefile or cmake

Objectives

  • briefly understand

    • how to use makefile to build your app

    • how to use cmake to build your app

  • use module file to manage your lib and apps for different version

check out the make branch of mylib

cd mylib
git checkout make
tree
.
├── CMakeLists.txt
├── include
│   └── mymath.h
├── makefile
├── modulefile
└── src
    └── mymath.c

2 directories, 5 files

demo on Leonardo Booster

  • use gcc to build and install to APP/mylib

    make PREFIX=../APP/mylib/1.0.0 install
    make clean
    make PREFIX=../APP/mylib/2.0.0 CFLAGS=-O3 install
    
  • create modulefiles in APP/modules/mylib

    sed "s:XXXXXX*:$(dirname $(pwd))/APP/mylib/1.0.0/:" modulefile > ../APP/modules/mylib/1.0.0
    sed "s:XXXXXX*:$(dirname $(pwd))/APP/mylib/2.0.0/:" modulefile > ../APP/modules/mylib/2.0.0
    
  • use APP/modules as module path

    cd ../APP/modules
    module use `pwd`
    module avail mylib
    

demo on Leonardo Booster

  • use gcc to build and install to APP/mylib

    cmake -B build01 -DCMAKE_INSTALL_PREFIX=$(dirname $PWD)/APP/mylib/1.0.0
    cd build01
    make install
    cd -
    cmake -B build02 -DCMAKE_C_FLAGS='-O3' -DCMAKE_INSTALL_PREFIX=$(dirname $PWD)/APP/mylib/2.0.0
    cd build02
    make install
    
  • create modulefiles in APP/modules/mylib

    sed "s:XXXXXX*:$(dirname $(pwd))/APP/mylib/1.0.0/:" modulefile > ../APP/modules/mylib/1.0.0
    sed "s:XXXXXX*:$(dirname $(pwd))/APP/mylib/2.0.0/:" modulefile > ../APP/modules/mylib/2.0.0
    
  • use APP/modules as module path

    cd ../APP/modules
    module use `pwd`
    module avail mylib
    

check out the make branch of myapp

cd myapp
git checkout make
tree
.
├── CMakeLists.txt
├── makefile
├── modulefile
└── src
    └── main.c

1 directory, 4 files

demo on Leonardo Booster

  • load mylib/2.0.0

    module load mylib/2.0.0
    
  • chose to compile with openblas

    module load openblas/0.3.26--gcc--12.2.0
    make clean
    make PREFIX=../APP/myapp/gcc-12.2.0-openblas-0.3.26 USE_CBLAS=1 BLAS=openblas install
    
  • chose to compile with mkl

    module load intel-oneapi-mkl/2024.0.0
    make clean
    make PREFIX=../APP/myapp/gcc-12.2.0-intel-oneapi-mkl-2024.0.0 USE_CBLAS=1 BLAS=mkl install
    
  • create modulefiles in APP/modules/myapp

    cp modulefile ../APP/modules/myapp/gcc-12.2.0-openblas-0.3.26 
    cp modulefile ../APP/modules/myapp/gcc-12.2.0-intel-oneapi-mkl-2024.0.0
    # need modify these 2 files
    
    module avail myapp
    

demo on Leonardo Booster

  • load mylib/2.0.0

    module load mylib/2.0.0
    
  • chose to compile with openblas

    module load openblas/0.3.26--gcc--12.2.0
    cmake -B build01 -DUSE_CBLAS=on \
    -DCMAKE_INSTALL_PREFIX=$(dirname $PWD)/APP/myapp/gcc-12.2.0-openblas-0.3.26
    cd build01
    make install
    
  • chose to compile with mkl

    module load intel-oneapi-mkl/2024.0.0
    cmake -B build02 -DUSE_CBLAS=on \
    -DCMAKE_INSTALL_PREFIX=$(dirname $PWD)/APP/myapp/gcc-12.2.0-intel-oneapi-mkl-2024.0.0
    cd build02
    make install
    
  • create modulefiles in APP/modules/myapp

    cp modulefile ../APP/modules/myapp/gcc-12.2.0-openblas-0.3.26 
    cp modulefile ../APP/modules/myapp/gcc-12.2.0-intel-oneapi-mkl-2024.0.0
    # need modify these 2 files
    
    module avail myapp
    

spack: a package manager for supercomputers

useful links:

quick start:

git clone --depth=2 https://github.com/spack/spack.git
cd spack/bin
./spack install libelf

demo on Leonardo

module avail spack
module load spack/0.22-06
  • check which env vars set for the spack

    env | grep -i ^spack
    
  • check which compilers already installed/setup

    spack compilers
    
  • check which packages are already installed

    spack find
    
  • list the installed package you want to know more about

    spack find openblas
    

    more information about it (show difference)

    spack find -Lv openblas
    
  • show what spack did for the package e.g the one

    sfcv7zah7u4wzwdlcplnwqmalq6c4fda openblas@0.3.26~bignuma~consistent_fpcsr+dynamic_dispatch+fortran~ilp64+locking+pic+shared build_system=makefile symbol_suffix=none threads=openmp
    

    use /+ hash

    spack spec /sfcv7zah7u4wzwdlcplnwqmalq6c4fda
    
  • list information about a package which could be installed

    spack info openblas
    
  • Basic structure of a Spack install spec spack install @%@^

    e.g install a older version 0.3.25 of openblas with gcc 12.2.0

    spack install openblas@0.3.25%gcc@12.2.0
    

    it takes time …

    generate module file for it

    spack module tcl refresh openblas@0.3.25
    

Apptainer/Singularity: container on HPC

Caution

  • different with docker

singularity pull ubuntu_22.04.sif docker://ubuntu:22.04
  • check what default map/mount into the container

    singularity shell ubuntu_22.04.sif
    

    then use df -h also check your id and compare to host

  • simple example with binary which is not work on the host OS of HPC

    aflow release

    • download and install

      wget https://github.com/aflow-org/aflow/releases/download/v4.1/aflow-4.1-ubuntu22-amd64.sh
      mkdir aflow
      bash ./aflow-4.1-ubuntu22-amd64.sh --help
      bash ./aflow-4.1-ubuntu22-amd64.sh --prefix=$PWD/aflow --skip-license --exclude-subdir
      ./aflow/bin/aflow --help
      

      error:

    • run it with ubuntu 22.04 image

      singularity exec ubuntu_22.04.sif aflow/bin/aflow --help