Episode 1: Building JDFTx on HPC systems

Choose a recipe for your system: a CPU build on LUMI-C, or a CUDA build on Dardel GH200 or Leonardo Booster. The recipes include regression-test commands and the results from the original build notes.

Before you start

Prerequisites

  • An account on the chosen system and an allocation for submitting jobs.

  • Familiarity with shell commands, environment modules, CMake, and Slurm.

  • A working directory with enough space for the source, build, and test output.

Recorded configurations

Component

LUMI-C

Dardel GH200

Leonardo Booster

Execution

CPU

NVIDIA GPU

NVIDIA GPU

JDFTx revision in configure log

049b2702

a9eec715

049b2702

Compiler

CrayClang 19.0.0

NVHPC 24.11.0

GCC 12.2.0

Cray programming environment

25.03 stack; craype 2.7.34

PDC/25.03; craype 2.7.34

Not used

CUDA compiler

Not used

12.6.77

12.2.91

GSL

2.8, cpeCray-25.03

2.8

2.7.1

FFTW

Cray FFTW 3.3.10.10

Cray FFTW 3.3.10.10

3.3.10

BLAS/LAPACK

Implicitly linked through Cray wrappers

Implicitly linked through Cray wrappers

oneAPI MKL 2024.0.0

MPI

Cray wrappers

Cray wrappers

Open MPI 4.1.6

Reported regression result

10/10 passed

11/11 passed

10/10 passed

The table records earlier builds; the current module stacks have not been retested. The clone command below selects the default branch, so it may fetch a different revision. Test counts and timings also differ by revision and configuration; use them as build records rather than a comparison of systems.

1. Prepare the source directory

Start in a fresh working directory on your chosen system:

mkdir jdftx
cd jdftx
git clone https://github.com/shankar1729/jdftx.git jdftx-git
mkdir build
cd build

Run one of the following recipes from this build directory. Keep separate source and build directories when retaining multiple configurations.

2. Build and test on LUMI-C

Load modules and configure

module purge
module load LUMI
module load partition/C
module load PrgEnv-cray
module load cray-fftw
module load GSL/2.8-cpeCray-25.03-sequential

CC=cc CXX=CC cmake ../jdftx-git/jdftx

CMake found GSL, threaded FFTW, MPI, and implicitly linked BLAS/LAPACK. The build notes identify LibSci as the BLAS/LAPACK library supplied by the Cray wrappers. Doxygen was absent, but configuration completed.

Compile

make -j 32

Run the regression suite

Request an eight-task interactive allocation from the build directory. Replace project_XXX with your allocation account:

srun -N 1 -n 8 -c 1 -A project_XXX -t 30:00 -p debug --pty /bin/bash

Inside the allocation, run:

make test

The allocation reserves eight tasks, but this invocation leaves JDFTX_LAUNCH unset and does not request an eight-rank test. The test summary was:

100% tests passed, 0 tests failed out of 10
Total Test time (real) = 861.44 sec

3. Build and test on Dardel GH200

Load modules and configure

CMake 3.20 failed to detect the CUDA toolkit layout on this Arm system. The build notes specify a version newer than 3.27, without giving the exact version or installation command. Load or install that CMake before configuring.

module purge
module load PDC/25.03
module load PrgEnv-nvidia
module load cray-fftw
module load gsl

CC=cc CXX=CC \
cmake ../jdftx-git/jdftx -D EnableCUDA=yes -D CMAKE_CUDA_ARCHITECTURES=90

CMake found NVHPC 24.11.0, CUDA 12.6.77, GSL 2.8, FFTW, and MPI. Configuration completed despite failed checks for two optional C++ compiler flags.

Compile

make -j 32

Submit the GPU tests

Save the following as job.sb in the build directory. Replace project_XXX with an account that can access the gpugh partition. Submit from the loaded build environment so the job can find the same libraries.

#!/bin/bash
#SBATCH -A project_XXX
#SBATCH -t 30:00
#SBATCH -p gpugh
#SBATCH -N 1
#SBATCH -n 4
#SBATCH -c 8
#SBATCH --ntasks-per-node=4
#SBATCH --gpus-per-task=1

export SLURM_CPU_BIND="cores"

make testclean
export JDFTX_LAUNCH="srun -n %d"
export JDFTX_SUFFIX="_gpu"
make test

Submit and inspect the Slurm output when the job finishes:

sbatch job.sb

The suite included the checkpoint test and reported:

100% tests passed, 0 tests failed out of 11
Total Test time (real) = 106.72 sec

4. Build and test on Leonardo Booster

Load modules and configure

module purge
module load gcc/12.2.0
module load cuda/12.2
module load openmpi/4.1.6--gcc--12.2.0-cuda-12.2
module load fftw/3.3.10--openmpi--4.1.6--gcc--12.2.0-spack0.22
module load gsl/2.7.1--gcc--12.2.0-spack0.22
module load intel-oneapi-mkl/2024.0.0

CC=gcc CXX=g++ cmake \
   -D EnableCUDA=yes \
   -D EnableMKL=yes \
   -D ForceFFTW=yes \
   -D MKL_PATH="$MKLROOT" \
   ../jdftx-git/jdftx

CMake found CUDA 12.2.91, MKL, FFTW, GSL, and MPI. It completed with a CMP0104 developer warning: CUDA_ARCHITECTURES was empty for jdftxlib_gpu. The recipe has no explicit CUDA architecture setting.

Compile

make -j 32

Submit the GPU tests

Save this as job.sb in the build directory. Replace EUHPC_XXX_XXX with your allocation account and submit from the loaded build environment:

#!/bin/bash
#SBATCH -A EUHPC_XXX_XXX
#SBATCH -t 30:00
#SBATCH -p boost_usr_prod
#SBATCH -N 1
#SBATCH -n 4
#SBATCH -c 8
#SBATCH --ntasks-per-node=4
#SBATCH --gpus-per-task=1

export SLURM_CPU_BIND="cores"
export JDFTX_MEMPOOL_SIZE=8192      # adjust as needed (in MB)

make testclean
export JDFTX_LAUNCH="srun -n %d"
export JDFTX_SUFFIX="_gpu"
make test

Submit the job and inspect its output for the test summary:

sbatch job.sb

The test summary was:

100% tests passed, 0 tests failed out of 10
Total Test time (real) = 101.69 sec

Completion checklist

  • CMake finds the required libraries and the build completes.

  • Tests run in the allocation with the intended CPU or GPU executable.

  • Test summaries show no failures; check any change in test count against the source revision.

Further reading

Continue with Episode 2: JDFTx with HIP on LUMI for the JDFTx HIP development port on LUMI AMD GPUs.