Episode 3: JDFTx HIP benchmarks and profiling on LUMI (draft)

Note

Draft — work in progress. The results and supporting evidence are still under review.

These GPU timings and CPU profiles come from the HIP validation draft. They use the original port from Episode 2: JDFTx with HIP on LUMI with the solver fix listed below.

Before you start

Prerequisites

  • The environment from Episode 2: JDFTx with HIP on LUMI and the development revision listed below.

  • Access to a LUMI AMD GPU allocation with the resources for the chosen case.

  • The original benchmark inputs, pseudopotentials, and timing wrappers.

The benchmark inputs, wrappers, raw logs, and profiling report are not bundled with the tutorial. Results below are taken from the draft; the calculations have not been rerun for this revision.

Recorded configuration

Component

Recorded version or setting

Validation date in source draft

8 October 2026

Source branch for recorded fix run

fix/hip-metal-surface-9573a3e5

Original HIP port

9573a3e57944f5d18f818c183bf42f4f7b9201a5

Solver cutoff fix

fa9be096adf8b9dbbe6ee2ee015a16c0749c991e

Historical optimized build used for comparison

994a542b

GPU target

MI250X GCD, gfx90a

Programming environment / ROCm

CPE 25.09 / ROCm 6.4.4

Compiler

CCE 20.0.0; ROCm Clang 19 for HIP

Standard Pt45 timing resources

One MPI rank, 16 CPU threads, one GCD

Prepare the workspace variables

From the workspace used in Episode 2, set:

export TASK_ROOT="$PWD"
export SOURCE_ROOT="$TASK_ROOT/source"
source "$TASK_ROOT/env.sh"

The commands use the original build/ directory. Its executable must include the solver fix below; Episode 2’s unmodified port is a different build. Prepare the benchmark directories and timing wrappers before running the examples.

Build used for these results

Solver fix

The fix changes the HIP solver cutoff from 32 to 96 in jdftx/core/matrixLinalg.cpp. Small matrices use CPU LAPACK; larger matrices remain eligible for hipSOLVER. The Hermitian eigensolver uses the GPU at N >= 96; other routines have their own inequalities around the same cutoff. CUDA keeps its cutoff of 32.

The cutoff is a matrix dimension, not a plane-wave energy cutoff. This branch uses a compile-time value and does not implement the later JDFTX_GPU_SOLVER_CUTOFF environment override.

This change avoided the charged-metalSurface allocation failure on ROCm 6.4.4; its underlying cause remains unresolved. The branch contains the original port and this fix, without the later launch-policy, batching, or FFT optimizations.

Recorded rebuild and regression tests

The validation used an incremental rebuild:

cd "$TASK_ROOT"
source ./env.sh
cmake --build build --target jdftx_gpu PseudopotentialLibrary -j8

The binaries were built before the patch was committed, so their banner still says 9573a3e5. According to the draft, the tested patch was committed unchanged. The GPU library SHA-256 was:

1d94ed2d25383781ba2832e63e8919a4c83a9bdd198bf94e193b6090394538a6

A newly configured build of the fix should carry the fix revision in its banner.

The regression run used the loop from Episode 2: JDFTx with HIP on LUMI, with BUILD_ROOT set to the original build/ and a fresh tests-fix/ directory. All 11 test scripts passed, including neutral and charged metalSurface. The build-and-test driver took 374.809 seconds.

The run included 45 numerical checks. Two optional graphene band checks were skipped because Octave was absent; the graphene energy and chemical-potential checks passed.

1. Larger GPU calculations

256-band metalSurface

The neutral metalSurface input used the original three-Pt-atom geometry, 48 electrons, ten reduced k-points, and elec-n-bands 256. From the prepared benchmark directory, the timing wrapper launched:

source "$TASK_ROOT/env.sh"
cd "$TASK_ROOT/benchmark-256"
export SRCDIR="$PWD"

"$TASK_ROOT/build/jdftx_gpu" -i neutral.in -d -o neutral.out

It passed on one GPU with eight threads: 68.7955 seconds wall time and 61.86 seconds internal duration.

45-atom Pt(111) slab

The five-layer, 3×3 lateral vacuum slab had 45 atoms, 720 electrons, 540 bands, and four reduced k-points. The input and Pt pseudopotential hashes matched the retained historical benchmark. Each calculation ran to SCF convergence.

The first interactive run used:

source "$TASK_ROOT/env.sh"
export PSEUDODIR="$TASK_ROOT/build/pseudopotentials"
cd "$TASK_ROOT/benchmark-pt111-45"
python3 time-run.py

It converged in 17 cycles: 163.2131 seconds wall time and 133.46 seconds internal duration. This was one fresh run with eight threads, without an excluded warmup.

A separate batch job used the historical timing protocol:

Resource or setting

Value

Nodes / MPI tasks

1 / 1

CPUs per task / OpenMP threads

16 / 16

GPUs

One MI250X GCD

Memory / wall-time limit

128 GiB / 20 minutes

Sampling

One excluded warmup, three fresh measured runs

Stack

CPE 25.09 / ROCm 6.4.4

The payload sourced env.sh, set OMP_NUM_THREADS=16 and PSEUDODIR, and ran the Python timing wrapper. It checked hashes, linked ROCm libraries, allocated resources, convergence, normal termination, and energy. Submission used the validation workspace’s broker, not a standalone sbatch script; its profile and payload are not provided here.

Measurement

Recorded result

Measured wall times

116.8959, 115.9475, 116.8030 s

Mean wall time

116.5488 s

Sample standard deviation

0.5228 s

Mean internal duration

113.2233 s

SCF cycles

17 in every run

Largest energy difference from historical pilot

1.2e-10 Hartree

Job exit status

0:0

The historical optimized 994a542b build averaged 100.4734 seconds with matching input, stack, resources, and warmup protocol. The cutoff-only build took 16.0% longer. Different allocations were used, so this was a historical comparison rather than a source A/B test within one allocation.

The first batch wrapper failed before launching JDFTx because Python 3.6 does not accept subprocess.check_output(text=True). The corrected wrapper used universal_newlines=True. The timings cover the whole GPU-enabled calculation. No kernel timings or runtime hipSOLVER API trace were collected.

2. Understand the HIP port

The manual port at 9573a3e5 shares kernels and numerical algorithms between CUDA and HIP. Episode 2: JDFTx with HIP on LUMI shows the API header and CMake excerpts.

GPU API boundary

jdftx/core/GpuApi.h retains CUDA names in the implementation and maps them to HIP when GPU_BACKEND_HIP is defined:

Existing interface

HIP implementation

CUDA runtime and memory operations

HIP runtime, such as cudaMalloc → hipMalloc

cuBLAS

hipBLAS

cuFFT

hipFFT

cuSOLVER, when enabled

hipSOLVER

The CUDA branch includes the original CUDA libraries. GPU utility headers include this shared header instead of CUDA headers directly. As a result, CUDA names remain in source and logs; a message mentioning a “cuda device” can describe an AMD device in the HIP build.

Build and device compilation

EnableHIP compiles the existing .cu files as HIP and links hip::host, roc::hipblas, and hip::hipfft. EnableHipSolver adds roc::hipsolver. The _gpu executable names and CUSOLVER_ENABLED guard stay shared. HIP defines GPU_ENABLED and GPU_BACKEND_HIP; CUDA uses GPU_BACKEND_CUDA.

The port requires CMake 3.21 or newer. CMake rejects CUDA and HIP together, a solver without its backend, and CUDA LibXC with HIP.

GPU_DEVICE_COMPILE covers both CUDA’s __CUDA_ARCH__ and HIP’s __HIP_DEVICE_COMPILE__. The radial-function and spline code uses that guard for device coefficients and shared-memory operations. Other changes route CUDA includes through GpuApi.h, adjust the function pointer passed to HIP’s kernel-attribute query, and add an explicit __global__ qualifier to the slab Coulomb kernel specialization.

The cutoff fix is a separate change to CPU/GPU solver dispatch. To inspect the two changes in the supplied source tree:

git -C "$SOURCE_ROOT" show 9573a3e5 -- \
  jdftx/CMakeLists.txt jdftx/core/GpuApi.h

git -C "$SOURCE_ROOT" show fa9be096 -- jdftx/core/matrixLinalg.cpp

3. CPU profiling follow-up

The first CPU executable failed during Pt45 LCAO initialization, while the GPU run converged. A CPU timing comparison therefore required a separate build.

The follow-up used separate CPU builds with threaded LibSci, without changing the source. On the recorded CPE stack, -fopenmp selected libsci_cray_mp.so.6, and -DTHREADED_BLAS told JDFTx to let BLAS own its parallelism. The draft notes that the CMake ThreadedBLAS option alone did not add that definition for LibSci in this source.

The profiling build used:

source "$TASK_ROOT/env.sh"
cmake -S "$SOURCE_ROOT/jdftx" \
  -B "$TASK_ROOT/profile-pt111-45/cpu-threaded-profile/build" \
  -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=cc -DCMAKE_CXX_COMPILER=CC \
  -DEnableHIP=OFF -DEnableCUDA=OFF -DEnableLibSci=ON -DEnableMPI=ON \
  -DEnableProfiling=ON \
  -DCMAKE_CXX_FLAGS='-fopenmp -DTHREADED_BLAS' \
  -DCMAKE_CXX_FLAGS_RELEASE='-O3 -g -DNDEBUG' \
  -DGSL_PATH="$GSL_TASK_ROOT" \
  -DFFTW3_PATH=/opt/cray/pe/fftw/3.3.10.11/x86_trento
cmake --build "$TASK_ROOT/profile-pt111-45/cpu-threaded-profile/build" \
  --target jdftx -j8

The unprofiled CPU build used the same flags with EnableProfiling=OFF in cpu-threaded-blas/build. Both builds completed. The longer timing campaign used one MPI rank, 16 cores, one GCD, 128 GiB, and a four-hour allocation limit. It set OMP_NUM_THREADS=16, OMP_DYNAMIC=FALSE, and OMP_MAX_ACTIVE_LEVELS=1, with the original pseudopotential directory. The batch finished with 0:0.

Case

Recorded result

First 16-thread cold CPU run

Timed out after nine SCF cycles at 2,400 s

Longer cold CPU timing run

3,047.022 s wall; 17 cycles; validated energy

Three GPU timing runs in the follow-up

Mean 117.819 s wall; 17 cycles each

Separate full CPU function profile

3,156.476 s wall; 17 cycles; validated energy

Eight-thread restart profile

Six cycles; 663.11 s internal; F = −4006.3488631327054 Ha

Eight-thread cold-start control

Timed out after 45 minutes and ten cycles

The restart read converged GPU wavefunctions, fillings, and eigenvalues, so its timing belongs to a restart calculation. PROFILER: lines report inclusive elapsed times; profiled runs were kept separate from uninstrumented samples.

Completion checklist

  • Timing records identify the source revision, input, libraries, and resources.

  • Independent runs use fresh output directories and record convergence and normal termination.

  • Comparisons distinguish cold starts, restarts, and profiled runs, and use matching inputs, resources, and warmup protocols.

  • Historical comparisons across allocations are identified as such.

Further reading

See Episode 2: JDFTx with HIP on LUMI for the HIP build and regression tests, and Episode 1: Building JDFTx on HPC systems for the CPU and CUDA build records.