Episode 3: JDFTx HIP benchmarks and profiling on LUMI (draft)¶
Note
Draft — work in progress. The results and supporting evidence are still under review.
These GPU timings and CPU profiles come from the HIP validation draft. They use the original port from Episode 2: JDFTx with HIP on LUMI with the solver fix listed below.
Before you start¶
Prerequisites
The environment from Episode 2: JDFTx with HIP on LUMI and the development revision listed below.
Access to a LUMI AMD GPU allocation with the resources for the chosen case.
The original benchmark inputs, pseudopotentials, and timing wrappers.
The benchmark inputs, wrappers, raw logs, and profiling report are not bundled with the tutorial. Results below are taken from the draft; the calculations have not been rerun for this revision.
Recorded configuration¶
Component |
Recorded version or setting |
|---|---|
Validation date in source draft |
8 October 2026 |
Source branch for recorded fix run |
|
Original HIP port |
|
Solver cutoff fix |
|
Historical optimized build used for comparison |
|
GPU target |
MI250X GCD, |
Programming environment / ROCm |
CPE 25.09 / ROCm 6.4.4 |
Compiler |
CCE 20.0.0; ROCm Clang 19 for HIP |
Standard Pt45 timing resources |
One MPI rank, 16 CPU threads, one GCD |
Prepare the workspace variables¶
From the workspace used in Episode 2, set:
export TASK_ROOT="$PWD"
export SOURCE_ROOT="$TASK_ROOT/source"
source "$TASK_ROOT/env.sh"
The commands use the original build/ directory. Its executable must include
the solver fix below; Episode 2’s unmodified port is a different build. Prepare
the benchmark directories and timing wrappers before running the examples.
Build used for these results¶
Solver fix¶
The fix changes the HIP solver cutoff from 32 to 96 in
jdftx/core/matrixLinalg.cpp. Small matrices use CPU LAPACK; larger matrices
remain eligible for hipSOLVER. The Hermitian eigensolver uses the GPU at
N >= 96; other routines have their own inequalities around the same cutoff.
CUDA keeps its cutoff of 32.
The cutoff is a matrix dimension, not a plane-wave energy cutoff. This
branch uses a compile-time value and does not implement the later
JDFTX_GPU_SOLVER_CUTOFF environment override.
This change avoided the charged-metalSurface allocation failure on ROCm 6.4.4; its underlying cause remains unresolved. The branch contains the original port and this fix, without the later launch-policy, batching, or FFT optimizations.
Recorded rebuild and regression tests¶
The validation used an incremental rebuild:
cd "$TASK_ROOT"
source ./env.sh
cmake --build build --target jdftx_gpu PseudopotentialLibrary -j8
The binaries were built before the patch was committed, so their banner
still says 9573a3e5. According to the draft, the tested patch was committed
unchanged. The GPU library SHA-256 was:
1d94ed2d25383781ba2832e63e8919a4c83a9bdd198bf94e193b6090394538a6
A newly configured build of the fix should carry the fix revision in its banner.
The regression run used the loop from Episode 2: JDFTx with HIP on LUMI, with BUILD_ROOT set
to the original build/ and a fresh tests-fix/ directory. All 11 test scripts
passed, including neutral and charged metalSurface. The build-and-test driver took 374.809 seconds.
The run included 45 numerical checks. Two optional graphene band checks were skipped because Octave was absent; the graphene energy and chemical-potential checks passed.
1. Larger GPU calculations¶
256-band metalSurface¶
The neutral metalSurface input used the original three-Pt-atom geometry,
48 electrons, ten reduced k-points, and elec-n-bands 256. From the prepared
benchmark directory, the timing wrapper launched:
source "$TASK_ROOT/env.sh"
cd "$TASK_ROOT/benchmark-256"
export SRCDIR="$PWD"
"$TASK_ROOT/build/jdftx_gpu" -i neutral.in -d -o neutral.out
It passed on one GPU with eight threads: 68.7955 seconds wall time and 61.86 seconds internal duration.
45-atom Pt(111) slab¶
The five-layer, 3×3 lateral vacuum slab had 45 atoms, 720 electrons, 540 bands, and four reduced k-points. The input and Pt pseudopotential hashes matched the retained historical benchmark. Each calculation ran to SCF convergence.
The first interactive run used:
source "$TASK_ROOT/env.sh"
export PSEUDODIR="$TASK_ROOT/build/pseudopotentials"
cd "$TASK_ROOT/benchmark-pt111-45"
python3 time-run.py
It converged in 17 cycles: 163.2131 seconds wall time and 133.46 seconds internal duration. This was one fresh run with eight threads, without an excluded warmup.
A separate batch job used the historical timing protocol:
Resource or setting |
Value |
|---|---|
Nodes / MPI tasks |
1 / 1 |
CPUs per task / OpenMP threads |
16 / 16 |
GPUs |
One MI250X GCD |
Memory / wall-time limit |
128 GiB / 20 minutes |
Sampling |
One excluded warmup, three fresh measured runs |
Stack |
CPE 25.09 / ROCm 6.4.4 |
The payload sourced env.sh, set OMP_NUM_THREADS=16 and PSEUDODIR, and
ran the Python timing wrapper. It checked hashes, linked ROCm libraries,
allocated resources, convergence, normal termination, and energy. Submission
used the validation workspace’s broker, not a standalone sbatch script;
its profile and payload are not provided here.
Measurement |
Recorded result |
|---|---|
Measured wall times |
116.8959, 115.9475, 116.8030 s |
Mean wall time |
116.5488 s |
Sample standard deviation |
0.5228 s |
Mean internal duration |
113.2233 s |
SCF cycles |
17 in every run |
Largest energy difference from historical pilot |
1.2e-10 Hartree |
Job exit status |
|
The historical optimized 994a542b build averaged 100.4734 seconds with
matching input, stack, resources, and warmup protocol. The cutoff-only build
took 16.0% longer. Different allocations were used, so this was a historical
comparison rather than a source A/B test within one allocation.
The first batch wrapper failed before launching JDFTx because Python 3.6 does
not accept subprocess.check_output(text=True). The corrected wrapper used
universal_newlines=True. The timings cover the whole GPU-enabled calculation.
No kernel timings or runtime hipSOLVER API trace were collected.
2. Understand the HIP port¶
The manual port at 9573a3e5 shares kernels and numerical algorithms between
CUDA and HIP. Episode 2: JDFTx with HIP on LUMI shows the API header and CMake excerpts.
GPU API boundary¶
jdftx/core/GpuApi.h retains CUDA names in the implementation and maps them
to HIP when GPU_BACKEND_HIP is defined:
Existing interface |
HIP implementation |
|---|---|
CUDA runtime and memory operations |
HIP runtime, such as |
cuBLAS |
hipBLAS |
cuFFT |
hipFFT |
cuSOLVER, when enabled |
hipSOLVER |
The CUDA branch includes the original CUDA libraries. GPU utility headers include this shared header instead of CUDA headers directly. As a result, CUDA names remain in source and logs; a message mentioning a “cuda device” can describe an AMD device in the HIP build.
Build and device compilation¶
EnableHIP compiles the existing .cu files as HIP and links hip::host,
roc::hipblas, and hip::hipfft. EnableHipSolver adds roc::hipsolver.
The _gpu executable names and CUSOLVER_ENABLED guard stay shared. HIP
defines GPU_ENABLED and GPU_BACKEND_HIP; CUDA uses GPU_BACKEND_CUDA.
The port requires CMake 3.21 or newer. CMake rejects CUDA and HIP together, a solver without its backend, and CUDA LibXC with HIP.
GPU_DEVICE_COMPILE covers both CUDA’s __CUDA_ARCH__ and HIP’s
__HIP_DEVICE_COMPILE__. The radial-function and spline code uses that guard
for device coefficients and shared-memory operations. Other changes route
CUDA includes through GpuApi.h, adjust the function pointer passed to HIP’s
kernel-attribute query, and add an explicit __global__ qualifier to the slab
Coulomb kernel specialization.
The cutoff fix is a separate change to CPU/GPU solver dispatch. To inspect the two changes in the supplied source tree:
git -C "$SOURCE_ROOT" show 9573a3e5 -- \
jdftx/CMakeLists.txt jdftx/core/GpuApi.h
git -C "$SOURCE_ROOT" show fa9be096 -- jdftx/core/matrixLinalg.cpp
3. CPU profiling follow-up¶
The first CPU executable failed during Pt45 LCAO initialization, while the GPU run converged. A CPU timing comparison therefore required a separate build.
The follow-up used separate CPU builds with threaded LibSci, without changing
the source. On the recorded CPE stack, -fopenmp selected
libsci_cray_mp.so.6, and -DTHREADED_BLAS told JDFTx to let BLAS own its
parallelism. The draft notes that the CMake ThreadedBLAS option alone did
not add that definition for LibSci in this source.
The profiling build used:
source "$TASK_ROOT/env.sh"
cmake -S "$SOURCE_ROOT/jdftx" \
-B "$TASK_ROOT/profile-pt111-45/cpu-threaded-profile/build" \
-DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=cc -DCMAKE_CXX_COMPILER=CC \
-DEnableHIP=OFF -DEnableCUDA=OFF -DEnableLibSci=ON -DEnableMPI=ON \
-DEnableProfiling=ON \
-DCMAKE_CXX_FLAGS='-fopenmp -DTHREADED_BLAS' \
-DCMAKE_CXX_FLAGS_RELEASE='-O3 -g -DNDEBUG' \
-DGSL_PATH="$GSL_TASK_ROOT" \
-DFFTW3_PATH=/opt/cray/pe/fftw/3.3.10.11/x86_trento
cmake --build "$TASK_ROOT/profile-pt111-45/cpu-threaded-profile/build" \
--target jdftx -j8
The unprofiled CPU build used the same flags with EnableProfiling=OFF in
cpu-threaded-blas/build. Both builds completed. The longer timing campaign
used one MPI rank, 16 cores, one GCD, 128 GiB, and a four-hour allocation limit.
It set OMP_NUM_THREADS=16, OMP_DYNAMIC=FALSE, and OMP_MAX_ACTIVE_LEVELS=1,
with the original pseudopotential directory. The batch finished with 0:0.
Case |
Recorded result |
|---|---|
First 16-thread cold CPU run |
Timed out after nine SCF cycles at 2,400 s |
Longer cold CPU timing run |
3,047.022 s wall; 17 cycles; validated energy |
Three GPU timing runs in the follow-up |
Mean 117.819 s wall; 17 cycles each |
Separate full CPU function profile |
3,156.476 s wall; 17 cycles; validated energy |
Eight-thread restart profile |
Six cycles; 663.11 s internal; F = −4006.3488631327054 Ha |
Eight-thread cold-start control |
Timed out after 45 minutes and ten cycles |
The restart read converged GPU wavefunctions, fillings, and eigenvalues, so
its timing belongs to a restart calculation. PROFILER: lines report inclusive
elapsed times; profiled runs were kept separate from uninstrumented samples.
Completion checklist¶
Timing records identify the source revision, input, libraries, and resources.
Independent runs use fresh output directories and record convergence and normal termination.
Comparisons distinguish cold starts, restarts, and profiled runs, and use matching inputs, resources, and warmup protocols.
Historical comparisons across allocations are identified as such.
Further reading¶
See Episode 2: JDFTx with HIP on LUMI for the HIP build and regression tests, and Episode 1: Building JDFTx on HPC systems for the CPU and CUDA build records.