Episode 2: JDFTx with HIP on LUMI

Build the original JDFTx HIP port at 9573a3e57 for LUMI AMD GPUs. The port reuses JDFTx’s GPU kernels through an API mapping header and a HIP build option. Episode 3: JDFTx HIP benchmarks and profiling on LUMI (draft) contains the later development results and profiling.

Before you start

Prerequisites

  • Access to a LUMI AMD GPU allocation and the recorded CPE/ROCm environment.

  • Familiarity with JDFTx builds, shell commands, and GPU calculations.

  • Git and access to the public ENCCS JDFTx repository.

The recipe uses the CPE/ROCm environment from an Apptainer validation workspace. Its compiler modules were already loaded; a container recipe is not supplied. The commands have not been rerun for the unmodified 9573a3e57 revision.

Build environment

Component

Version or setting

Source revision

9573a3e57944f5d18f818c183bf42f4f7b9201a5

GPU target

MI250X GCD, gfx90a

Programming environment

CPE 25.09

C/C++ compiler

CCE 20.0.0 through cc and CC

HIP compiler

ROCm Clang 19

ROCm

6.4.4

CMake

3.31.11

GSL

2.8, cpeAMD-25.09-sequential

CPU BLAS/LAPACK

Cray LibSci 25.09.0

CPU FFTW

Cray FFTW 3.3.10.11

MPI

Cray MPICH 9.0.1

GPU libraries

HIP runtime, hipBLAS, hipFFT, hipSOLVER and rocSOLVER

Regression-test resources

One MPI process, eight CPU threads, one GCD

How the HIP port reuses the GPU code

JDFTx already separates GPU operations from the electronic-structure code. The port keeps one set of kernels and numerical algorithms, adding the HIP API mappings and build settings at that boundary.

Map library calls in GpuApi.h

jdftx/core/GpuApi.h selects HIP headers and maps CUDA names to HIP. The following excerpt shows the runtime and library mappings:

#ifdef GPU_BACKEND_HIP

#include <hip/hip_runtime.h>
#include <hipblas/hipblas.h>
#include <hipfft/hipfft.h>
#ifdef CUSOLVER_ENABLED
#include <hipsolver/hipsolver.h>
#endif

//HIP runtime:
#define cudaDeviceProp hipDeviceProp_t
#define cudaError_t hipError_t
#define cudaSuccess hipSuccess
#define cudaMemcpyDeviceToDevice hipMemcpyDeviceToDevice
#define cudaMemcpyDeviceToHost hipMemcpyDeviceToHost
#define cudaMemcpyHostToDevice hipMemcpyHostToDevice
#define cudaDeviceSynchronize hipDeviceSynchronize
#define cudaFree hipFree
#define cudaFreeHost hipHostFree
#define cudaFuncGetAttributes(attr, func) hipFuncGetAttributes((attr), reinterpret_cast<const void*>(func))
#define cudaGetDevice hipGetDevice
#define cudaGetDeviceCount hipGetDeviceCount
#define cudaGetDeviceProperties hipGetDeviceProperties
#define cudaMalloc hipMalloc
#define cudaMallocHost hipHostMalloc
#define cudaMallocManaged hipMallocManaged
#define cudaMemcpy hipMemcpy
#define cudaMemset hipMemset
#define cudaSetDevice hipSetDevice
// ... other runtime mappings omitted ...

//hipBLAS:
#define cublasHandle_t hipblasHandle_t
#define cublasOperation_t hipblasOperation_t
#define cublasCreate hipblasCreate
#define cublasDaxpy_v2 hipblasDaxpy
#define cublasDdot_v2 hipblasDdot
#define cublasZgemm hipblasZgemm_v2
#define cublasZtrsm hipblasZtrsm_v2
#define CUBLAS_OP_N HIPBLAS_OP_N
#define CUBLAS_OP_T HIPBLAS_OP_T
// ... other BLAS mappings omitted ...

//hipFFT:
#define cufftHandle hipfftHandle
#define cufftDestroy hipfftDestroy
#define cufftExecD2Z hipfftExecD2Z
#define cufftExecZ2D hipfftExecZ2D
#define cufftExecZ2Z hipfftExecZ2Z
#define cufftPlan3d hipfftPlan3d
#define CUFFT_FORWARD HIPFFT_FORWARD
#define CUFFT_INVERSE HIPFFT_BACKWARD
// ... other FFT constants omitted ...

//hipSOLVER (optional):
#ifdef CUSOLVER_ENABLED
#define cusolverDnHandle_t hipsolverHandle_t
#define cusolverDnCreate hipsolverDnCreate
#define cusolverDnZgetrf hipsolverDnZgetrf
#define cusolverDnZgetrs hipsolverDnZgetrs
#define cusolverDnZheevd hipsolverDnZheevd
#define cusolverDnZheevd_bufferSize hipsolverDnZheevd_bufferSize
#define cusolverDnZpotrf hipsolverDnZpotrf
#define cusolverDnZpotrs hipsolverDnZpotrs
// ... other solver mappings omitted ...
#endif

#else //CUDA backend

#include <cuda_runtime.h>
#include <cublas_v2.h>
#include <cufft.h>
#ifdef CUSOLVER_ENABLED
#include <cusolverDn.h>
#endif

#endif

GPU utility headers include GpuApi.h instead of including CUDA headers directly. Existing calls keep their CUDA spelling, while the selected backend supplies the types, runtime, and libraries:

  • Runtime: device selection, allocation, memory copies, and synchronization.

  • BLAS: vector operations, matrix multiplication, and triangular solves.

  • FFT: transform plans and real/complex transforms.

  • Solver: optional matrix factorizations, linear solves, and Hermitian eigensolvers, selected by the existing CUSOLVER_ENABLED guard.

A few mappings adapt API details: cudaFuncGetAttributes casts the kernel pointer for HIP, several BLAS calls use different version suffixes, and CUFFT_INVERSE maps to HIPFFT_BACKWARD. The header covers the interfaces used by this JDFTx revision.

Compile the same kernels with HIP

In jdftx/CMakeLists.txt, the HIP branch changes the compiler language and linked libraries for the existing GPU sources. This excerpt starts inside elseif(EnableHIP), after its CMake-version check:

    enable_language(HIP)
    find_package(hip CONFIG REQUIRED)
    find_package(hipblas CONFIG REQUIRED)
    find_package(hipfft CONFIG REQUIRED)
    set_source_files_properties(${gpukernelsSources} PROPERTIES LANGUAGE HIP)
    set(GPU_LIBRARIES hip::host roc::hipblas hip::hipfft)
    set(GPU_RUNTIME_LIBRARY hip::host)

gpukernelsSources contains the existing .cu files. Setting their language to HIP lets the build reuse them, while GPU_BACKEND_HIP selects the API mappings above. CUDA and HIP retain the same _gpu executable names.

The port changed 13 files, including the new 124-line API header. Device-code changes included recognizing HIP’s compilation macro and adding __global__ to one kernel specialization.

Source excerpts are from the named commit’s GpuApi.h and CMakeLists.txt; the JDFTx code is licensed under GNU GPL v3 or later. Omissions are marked.

1. Prepare the source and workspace

Create a working directory and clone the ENCCS fork’s hip-support branch. Pin the exact commit so the recipe keeps using this version if the branch changes. Run these commands in the same shell:

mkdir jdftx-hip-lumi
cd jdftx-hip-lumi
export TASK_ROOT="$PWD"
export SOURCE_ROOT="$TASK_ROOT/source"
export BUILD_ROOT="$TASK_ROOT/build-hip-9573a3e5"

git clone --branch hip-support https://github.com/ENCCS/jdftx.git "$SOURCE_ROOT"
git -C "$SOURCE_ROOT" switch --detach 9573a3e57944f5d18f818c183bf42f4f7b9201a5

This development branch belongs to the ENCCS fork; the upstream JDFTx repository is shankar1729/jdftx.

2. Prepare the recorded environment

The validation workspace had these modules loaded:

Module

Purpose

PrgEnv-cray/8.6.0, cce/20.0.0, craype/2.7.35

Cray compiler wrappers

craype-x86-trento

CPU target

craype-accel-amd-gfx90a

GPU target

cray-mpich/9.0.1

MPI

cray-libsci/25.09.0

CPU BLAS/LAPACK

cray-fftw/3.3.10.11

CPU FFTW

It also included libfabric 1.22.0, craype-network-ofi, perftools-base 25.09.0, xpmem, and cray-dsmml. The draft did not reconstruct or purge that baseline; its task script explicitly loaded only cray-fftw.

Save the following as env.sh under TASK_ROOT. The paths identify the recorded LUMI installations:

module load cray-fftw

export PATH=/appl/lumi/SW/LUMI-25.09/common/EB/buildtools/25.09-CMake3/bin:$PATH
export ROCM_PATH=/appl/lumi/SW/LUMI-25.09/G/EB/rocm/6.4.4
export GSL_TASK_ROOT=/appl/lumi/SW/LUMI-25.09/G/EB/GSL/2.8-cpeAMD-25.09-sequential
export LD_LIBRARY_PATH="$GSL_TASK_ROOT/lib:$ROCM_PATH/lib:${LD_LIBRARY_PATH:-}"

export OMP_NUM_THREADS=8
export JDFTX_SUFFIX=_gpu

Source it and inspect the environment:

source "$TASK_ROOT/env.sh"
module list
cmake --version
"$ROCM_PATH/bin/hipcc" --version

In a new shell, set the workspace variables again and source env.sh. Check the module versions and paths if using another image or login environment.

3. Configure and compile

Configure the separate build directory selected in step 1:

set -euo pipefail

cmake -S "$SOURCE_ROOT/jdftx" -B "$BUILD_ROOT" \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER=cc \
  -DCMAKE_CXX_COMPILER=CC \
  -DCMAKE_HIP_COMPILER="$ROCM_PATH/llvm/bin/clang++" \
  -DCMAKE_HIP_ARCHITECTURES=gfx90a \
  -DCMAKE_PREFIX_PATH="$ROCM_PATH" \
  -DEnableHIP=ON \
  -DEnableHipSolver=ON \
  -DEnableCUDA=OFF \
  -DEnableCuSolver=OFF \
  -DEnableLibSci=ON \
  -DEnableMPI=ON \
  -DGSL_PATH="$GSL_TASK_ROOT" \
  -DFFTW3_PATH=/opt/cray/pe/fftw/3.3.10.11/x86_trento

cmake --build "$BUILD_ROOT" \
  --target jdftx_gpu PseudopotentialLibrary -j8

ldd "$BUILD_ROOT/jdftx_gpu"

C and C++ compilation use CrayClang 20; GPU compilation uses ROCm’s Clang 19. EnableLibSci=ON uses BLAS/LAPACK supplied by the Cray wrappers. The pseudopotential target supplies files used by the regression tests.

4. Run the regression tests

Run inside the allocated GPU environment. The upstream runner skips calculations whose output already ends in Done!; create a fresh test directory for each test run:

source "$TASK_ROOT/env.sh"
export OMP_NUM_THREADS=8
export JDFTX_SUFFIX=_gpu
export JDFTX_LAUNCH=""

TEST_ROOT=$(mktemp -d "$TASK_ROOT/tests-repro-XXXXXX")
ctest --test-dir "$BUILD_ROOT" -N

for name in openShell vibrations moleculeSolvation ionSolvation \
            latticeOpt metalBulk plusU spinOrbit graphene metalSurface checkpoint
do
  bash "$SOURCE_ROOT/jdftx/test/runTest.sh" \
    "$name" "$SOURCE_ROOT/jdftx/test" "$TEST_ROOT" "$BUILD_ROOT"
  cat "$TEST_ROOT/$name/summary"
done

Read each test’s summary and output for failures and skipped checks. Save the results with the source revision and module list. The passing results in Episode 3: JDFTx HIP benchmarks and profiling on LUMI (draft) belong to a later revision.

Completion checklist

  • The source tree is clean and pinned to 9573a3e57.

  • The build uses the recorded compiler and library environment and targets gfx90a.

  • Regression calculations run in a fresh directory; optional skipped checks are distinguished from passing numerical checks.

Further reading

See Episode 3: JDFTx HIP benchmarks and profiling on LUMI (draft) for the later results and profiling, or Episode 1: Building JDFTx on HPC systems for CPU and CUDA builds.