Episode 2: JDFTx with HIP on LUMI¶
Build the original JDFTx HIP port
at 9573a3e57 for LUMI AMD GPUs. The port
reuses JDFTx’s GPU kernels through an API mapping header and a HIP build option.
Episode 3: JDFTx HIP benchmarks and profiling on LUMI (draft) contains the later development results and profiling.
Before you start¶
Prerequisites
Access to a LUMI AMD GPU allocation and the recorded CPE/ROCm environment.
Familiarity with JDFTx builds, shell commands, and GPU calculations.
Git and access to the public ENCCS JDFTx repository.
The recipe uses the CPE/ROCm environment from an Apptainer validation
workspace. Its compiler modules were already loaded; a container recipe is
not supplied. The commands have not been rerun for the unmodified 9573a3e57
revision.
Build environment¶
Component |
Version or setting |
|---|---|
Source revision |
|
GPU target |
MI250X GCD, |
Programming environment |
CPE 25.09 |
C/C++ compiler |
CCE 20.0.0 through |
HIP compiler |
ROCm Clang 19 |
ROCm |
6.4.4 |
CMake |
3.31.11 |
GSL |
2.8, cpeAMD-25.09-sequential |
CPU BLAS/LAPACK |
Cray LibSci 25.09.0 |
CPU FFTW |
Cray FFTW 3.3.10.11 |
MPI |
Cray MPICH 9.0.1 |
GPU libraries |
HIP runtime, hipBLAS, hipFFT, hipSOLVER and rocSOLVER |
Regression-test resources |
One MPI process, eight CPU threads, one GCD |
How the HIP port reuses the GPU code¶
JDFTx already separates GPU operations from the electronic-structure code. The port keeps one set of kernels and numerical algorithms, adding the HIP API mappings and build settings at that boundary.
Map library calls in GpuApi.h¶
jdftx/core/GpuApi.h selects HIP headers and maps CUDA names to HIP. The
following excerpt shows the runtime and library mappings:
#ifdef GPU_BACKEND_HIP
#include <hip/hip_runtime.h>
#include <hipblas/hipblas.h>
#include <hipfft/hipfft.h>
#ifdef CUSOLVER_ENABLED
#include <hipsolver/hipsolver.h>
#endif
//HIP runtime:
#define cudaDeviceProp hipDeviceProp_t
#define cudaError_t hipError_t
#define cudaSuccess hipSuccess
#define cudaMemcpyDeviceToDevice hipMemcpyDeviceToDevice
#define cudaMemcpyDeviceToHost hipMemcpyDeviceToHost
#define cudaMemcpyHostToDevice hipMemcpyHostToDevice
#define cudaDeviceSynchronize hipDeviceSynchronize
#define cudaFree hipFree
#define cudaFreeHost hipHostFree
#define cudaFuncGetAttributes(attr, func) hipFuncGetAttributes((attr), reinterpret_cast<const void*>(func))
#define cudaGetDevice hipGetDevice
#define cudaGetDeviceCount hipGetDeviceCount
#define cudaGetDeviceProperties hipGetDeviceProperties
#define cudaMalloc hipMalloc
#define cudaMallocHost hipHostMalloc
#define cudaMallocManaged hipMallocManaged
#define cudaMemcpy hipMemcpy
#define cudaMemset hipMemset
#define cudaSetDevice hipSetDevice
// ... other runtime mappings omitted ...
//hipBLAS:
#define cublasHandle_t hipblasHandle_t
#define cublasOperation_t hipblasOperation_t
#define cublasCreate hipblasCreate
#define cublasDaxpy_v2 hipblasDaxpy
#define cublasDdot_v2 hipblasDdot
#define cublasZgemm hipblasZgemm_v2
#define cublasZtrsm hipblasZtrsm_v2
#define CUBLAS_OP_N HIPBLAS_OP_N
#define CUBLAS_OP_T HIPBLAS_OP_T
// ... other BLAS mappings omitted ...
//hipFFT:
#define cufftHandle hipfftHandle
#define cufftDestroy hipfftDestroy
#define cufftExecD2Z hipfftExecD2Z
#define cufftExecZ2D hipfftExecZ2D
#define cufftExecZ2Z hipfftExecZ2Z
#define cufftPlan3d hipfftPlan3d
#define CUFFT_FORWARD HIPFFT_FORWARD
#define CUFFT_INVERSE HIPFFT_BACKWARD
// ... other FFT constants omitted ...
//hipSOLVER (optional):
#ifdef CUSOLVER_ENABLED
#define cusolverDnHandle_t hipsolverHandle_t
#define cusolverDnCreate hipsolverDnCreate
#define cusolverDnZgetrf hipsolverDnZgetrf
#define cusolverDnZgetrs hipsolverDnZgetrs
#define cusolverDnZheevd hipsolverDnZheevd
#define cusolverDnZheevd_bufferSize hipsolverDnZheevd_bufferSize
#define cusolverDnZpotrf hipsolverDnZpotrf
#define cusolverDnZpotrs hipsolverDnZpotrs
// ... other solver mappings omitted ...
#endif
#else //CUDA backend
#include <cuda_runtime.h>
#include <cublas_v2.h>
#include <cufft.h>
#ifdef CUSOLVER_ENABLED
#include <cusolverDn.h>
#endif
#endif
GPU utility headers include GpuApi.h instead of including CUDA headers
directly. Existing calls keep their CUDA spelling, while the selected backend
supplies the types, runtime, and libraries:
Runtime: device selection, allocation, memory copies, and synchronization.
BLAS: vector operations, matrix multiplication, and triangular solves.
FFT: transform plans and real/complex transforms.
Solver: optional matrix factorizations, linear solves, and Hermitian eigensolvers, selected by the existing
CUSOLVER_ENABLEDguard.
A few mappings adapt API details: cudaFuncGetAttributes casts the kernel
pointer for HIP, several BLAS calls use different version suffixes, and
CUFFT_INVERSE maps to HIPFFT_BACKWARD. The header covers the interfaces used
by this JDFTx revision.
Compile the same kernels with HIP¶
In jdftx/CMakeLists.txt, the HIP branch changes the compiler language and
linked libraries for the existing GPU sources. This excerpt starts inside
elseif(EnableHIP), after its CMake-version check:
enable_language(HIP)
find_package(hip CONFIG REQUIRED)
find_package(hipblas CONFIG REQUIRED)
find_package(hipfft CONFIG REQUIRED)
set_source_files_properties(${gpukernelsSources} PROPERTIES LANGUAGE HIP)
set(GPU_LIBRARIES hip::host roc::hipblas hip::hipfft)
set(GPU_RUNTIME_LIBRARY hip::host)
gpukernelsSources contains the existing .cu files. Setting their language
to HIP lets the build reuse them, while GPU_BACKEND_HIP selects the API
mappings above. CUDA and HIP retain the same _gpu executable names.
The port changed 13 files, including the new 124-line API header. Device-code
changes included recognizing HIP’s compilation macro and adding __global__
to one kernel specialization.
Source excerpts are from the named commit’s GpuApi.h and CMakeLists.txt;
the JDFTx code is licensed under GNU GPL v3 or later. Omissions are marked.
1. Prepare the source and workspace¶
Create a working directory and clone the ENCCS fork’s hip-support branch.
Pin the exact commit so the recipe keeps using this version if the branch
changes. Run these commands in the same shell:
mkdir jdftx-hip-lumi
cd jdftx-hip-lumi
export TASK_ROOT="$PWD"
export SOURCE_ROOT="$TASK_ROOT/source"
export BUILD_ROOT="$TASK_ROOT/build-hip-9573a3e5"
git clone --branch hip-support https://github.com/ENCCS/jdftx.git "$SOURCE_ROOT"
git -C "$SOURCE_ROOT" switch --detach 9573a3e57944f5d18f818c183bf42f4f7b9201a5
This development branch belongs to the
ENCCS fork; the upstream
JDFTx repository is shankar1729/jdftx.
2. Prepare the recorded environment¶
The validation workspace had these modules loaded:
Module |
Purpose |
|---|---|
|
Cray compiler wrappers |
|
CPU target |
|
GPU target |
|
MPI |
|
CPU BLAS/LAPACK |
|
CPU FFTW |
It also included libfabric 1.22.0, craype-network-ofi, perftools-base 25.09.0,
xpmem, and cray-dsmml. The draft did not reconstruct or purge that
baseline; its task script explicitly loaded only cray-fftw.
Save the following as env.sh under TASK_ROOT. The paths identify the
recorded LUMI installations:
module load cray-fftw
export PATH=/appl/lumi/SW/LUMI-25.09/common/EB/buildtools/25.09-CMake3/bin:$PATH
export ROCM_PATH=/appl/lumi/SW/LUMI-25.09/G/EB/rocm/6.4.4
export GSL_TASK_ROOT=/appl/lumi/SW/LUMI-25.09/G/EB/GSL/2.8-cpeAMD-25.09-sequential
export LD_LIBRARY_PATH="$GSL_TASK_ROOT/lib:$ROCM_PATH/lib:${LD_LIBRARY_PATH:-}"
export OMP_NUM_THREADS=8
export JDFTX_SUFFIX=_gpu
Source it and inspect the environment:
source "$TASK_ROOT/env.sh"
module list
cmake --version
"$ROCM_PATH/bin/hipcc" --version
In a new shell, set the workspace variables again and source env.sh. Check
the module versions and paths if using another image or login environment.
3. Configure and compile¶
Configure the separate build directory selected in step 1:
set -euo pipefail
cmake -S "$SOURCE_ROOT/jdftx" -B "$BUILD_ROOT" \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_COMPILER=cc \
-DCMAKE_CXX_COMPILER=CC \
-DCMAKE_HIP_COMPILER="$ROCM_PATH/llvm/bin/clang++" \
-DCMAKE_HIP_ARCHITECTURES=gfx90a \
-DCMAKE_PREFIX_PATH="$ROCM_PATH" \
-DEnableHIP=ON \
-DEnableHipSolver=ON \
-DEnableCUDA=OFF \
-DEnableCuSolver=OFF \
-DEnableLibSci=ON \
-DEnableMPI=ON \
-DGSL_PATH="$GSL_TASK_ROOT" \
-DFFTW3_PATH=/opt/cray/pe/fftw/3.3.10.11/x86_trento
cmake --build "$BUILD_ROOT" \
--target jdftx_gpu PseudopotentialLibrary -j8
ldd "$BUILD_ROOT/jdftx_gpu"
C and C++ compilation use CrayClang 20; GPU compilation uses ROCm’s Clang 19.
EnableLibSci=ON uses BLAS/LAPACK supplied by the Cray wrappers. The
pseudopotential target supplies files used by the regression tests.
4. Run the regression tests¶
Run inside the allocated GPU environment. The upstream runner skips
calculations whose output already ends in Done!; create a fresh test
directory for each test run:
source "$TASK_ROOT/env.sh"
export OMP_NUM_THREADS=8
export JDFTX_SUFFIX=_gpu
export JDFTX_LAUNCH=""
TEST_ROOT=$(mktemp -d "$TASK_ROOT/tests-repro-XXXXXX")
ctest --test-dir "$BUILD_ROOT" -N
for name in openShell vibrations moleculeSolvation ionSolvation \
latticeOpt metalBulk plusU spinOrbit graphene metalSurface checkpoint
do
bash "$SOURCE_ROOT/jdftx/test/runTest.sh" \
"$name" "$SOURCE_ROOT/jdftx/test" "$TEST_ROOT" "$BUILD_ROOT"
cat "$TEST_ROOT/$name/summary"
done
Read each test’s summary and output for failures and skipped checks. Save
the results with the source revision and module list. The passing results in
Episode 3: JDFTx HIP benchmarks and profiling on LUMI (draft) belong to a later revision.
Completion checklist¶
The source tree is clean and pinned to
9573a3e57.The build uses the recorded compiler and library environment and targets
gfx90a.Regression calculations run in a fresh directory; optional skipped checks are distinguished from passing numerical checks.
Further reading¶
See Episode 3: JDFTx HIP benchmarks and profiling on LUMI (draft) for the later results and profiling, or Episode 1: Building JDFTx on HPC systems for CPU and CUDA builds.