Episode 1: Building JDFTx on HPC systems¶
Choose a recipe for your system: a CPU build on LUMI-C, or a CUDA build on Dardel GH200 or Leonardo Booster. The recipes include regression-test commands and the results from the original build notes.
Before you start¶
Prerequisites
An account on the chosen system and an allocation for submitting jobs.
Familiarity with shell commands, environment modules, CMake, and Slurm.
A working directory with enough space for the source, build, and test output.
Recorded configurations¶
Component |
LUMI-C |
Dardel GH200 |
Leonardo Booster |
|---|---|---|---|
Execution |
CPU |
NVIDIA GPU |
NVIDIA GPU |
JDFTx revision in configure log |
|
|
|
Compiler |
CrayClang 19.0.0 |
NVHPC 24.11.0 |
GCC 12.2.0 |
Cray programming environment |
25.03 stack; craype 2.7.34 |
PDC/25.03; craype 2.7.34 |
Not used |
CUDA compiler |
Not used |
12.6.77 |
12.2.91 |
GSL |
2.8, cpeCray-25.03 |
2.8 |
2.7.1 |
FFTW |
Cray FFTW 3.3.10.10 |
Cray FFTW 3.3.10.10 |
3.3.10 |
BLAS/LAPACK |
Implicitly linked through Cray wrappers |
Implicitly linked through Cray wrappers |
oneAPI MKL 2024.0.0 |
MPI |
Cray wrappers |
Cray wrappers |
Open MPI 4.1.6 |
Reported regression result |
10/10 passed |
11/11 passed |
10/10 passed |
The table records earlier builds; the current module stacks have not been retested. The clone command below selects the default branch, so it may fetch a different revision. Test counts and timings also differ by revision and configuration; use them as build records rather than a comparison of systems.
1. Prepare the source directory¶
Start in a fresh working directory on your chosen system:
mkdir jdftx
cd jdftx
git clone https://github.com/shankar1729/jdftx.git jdftx-git
mkdir build
cd build
Run one of the following recipes from this build directory. Keep separate
source and build directories when retaining multiple configurations.
2. Build and test on LUMI-C¶
Load modules and configure¶
module purge
module load LUMI
module load partition/C
module load PrgEnv-cray
module load cray-fftw
module load GSL/2.8-cpeCray-25.03-sequential
CC=cc CXX=CC cmake ../jdftx-git/jdftx
CMake found GSL, threaded FFTW, MPI, and implicitly linked BLAS/LAPACK. The build notes identify LibSci as the BLAS/LAPACK library supplied by the Cray wrappers. Doxygen was absent, but configuration completed.
Compile¶
make -j 32
Run the regression suite¶
Request an eight-task interactive allocation from the build directory. Replace project_XXX with your allocation account:
srun -N 1 -n 8 -c 1 -A project_XXX -t 30:00 -p debug --pty /bin/bash
Inside the allocation, run:
make test
The allocation reserves eight tasks, but this invocation leaves
JDFTX_LAUNCH unset and does not request an eight-rank test. The test summary was:
100% tests passed, 0 tests failed out of 10
Total Test time (real) = 861.44 sec
3. Build and test on Dardel GH200¶
Load modules and configure¶
CMake 3.20 failed to detect the CUDA toolkit layout on this Arm system. The build notes specify a version newer than 3.27, without giving the exact version or installation command. Load or install that CMake before configuring.
module purge
module load PDC/25.03
module load PrgEnv-nvidia
module load cray-fftw
module load gsl
CC=cc CXX=CC \
cmake ../jdftx-git/jdftx -D EnableCUDA=yes -D CMAKE_CUDA_ARCHITECTURES=90
CMake found NVHPC 24.11.0, CUDA 12.6.77, GSL 2.8, FFTW, and MPI. Configuration completed despite failed checks for two optional C++ compiler flags.
Compile¶
make -j 32
Submit the GPU tests¶
Save the following as job.sb in the build directory. Replace project_XXX
with an account that can access the gpugh partition. Submit from the loaded
build environment so the job can find the same libraries.
#!/bin/bash
#SBATCH -A project_XXX
#SBATCH -t 30:00
#SBATCH -p gpugh
#SBATCH -N 1
#SBATCH -n 4
#SBATCH -c 8
#SBATCH --ntasks-per-node=4
#SBATCH --gpus-per-task=1
export SLURM_CPU_BIND="cores"
make testclean
export JDFTX_LAUNCH="srun -n %d"
export JDFTX_SUFFIX="_gpu"
make test
Submit and inspect the Slurm output when the job finishes:
sbatch job.sb
The suite included the checkpoint test and reported:
100% tests passed, 0 tests failed out of 11
Total Test time (real) = 106.72 sec
4. Build and test on Leonardo Booster¶
Load modules and configure¶
module purge
module load gcc/12.2.0
module load cuda/12.2
module load openmpi/4.1.6--gcc--12.2.0-cuda-12.2
module load fftw/3.3.10--openmpi--4.1.6--gcc--12.2.0-spack0.22
module load gsl/2.7.1--gcc--12.2.0-spack0.22
module load intel-oneapi-mkl/2024.0.0
CC=gcc CXX=g++ cmake \
-D EnableCUDA=yes \
-D EnableMKL=yes \
-D ForceFFTW=yes \
-D MKL_PATH="$MKLROOT" \
../jdftx-git/jdftx
CMake found CUDA 12.2.91, MKL, FFTW, GSL, and MPI. It completed with a
CMP0104 developer warning: CUDA_ARCHITECTURES was empty for jdftxlib_gpu.
The recipe has no explicit CUDA architecture setting.
Compile¶
make -j 32
Submit the GPU tests¶
Save this as job.sb in the build directory. Replace EUHPC_XXX_XXX with
your allocation account and submit from the loaded build environment:
#!/bin/bash
#SBATCH -A EUHPC_XXX_XXX
#SBATCH -t 30:00
#SBATCH -p boost_usr_prod
#SBATCH -N 1
#SBATCH -n 4
#SBATCH -c 8
#SBATCH --ntasks-per-node=4
#SBATCH --gpus-per-task=1
export SLURM_CPU_BIND="cores"
export JDFTX_MEMPOOL_SIZE=8192 # adjust as needed (in MB)
make testclean
export JDFTX_LAUNCH="srun -n %d"
export JDFTX_SUFFIX="_gpu"
make test
Submit the job and inspect its output for the test summary:
sbatch job.sb
The test summary was:
100% tests passed, 0 tests failed out of 10
Total Test time (real) = 101.69 sec
Completion checklist¶
CMake finds the required libraries and the build completes.
Tests run in the allocation with the intended CPU or GPU executable.
Test summaries show no failures; check any change in test count against the source revision.
Further reading¶
Continue with Episode 2: JDFTx with HIP on LUMI for the JDFTx HIP development port on LUMI AMD GPUs.