Tutorial: Quantum ESPRESSO on HPC systems

Credit

This material is adapted with permission from https://gitlab.com/QEF/materials-for-max-coe-enccs-workshop-2022

Slides

Setup

• Download the pseudopotentials: data/materials-for-max-coe-enccs-workshop-2022-v0.9-pseudo.tar.gz

---

EXERCISE 0 - First serial run

Files needed:

ex0-run.slurm templateatom-pbe.in

#!/bin/bash
#SBATCH --job-name=USERjob
#SBATCH --nodes 1
#SBATCH --time=00:00:30
#SBATCH --partition=cpu
#SBATCH --reservation=maxcpu
#SBATCH --ntasks-per-socket=...
#SBATCH --ntasks-per-node=...
#SBATCH --hint=nomultithread
#SBATCH --cpus-per-task=...
#SBATCH --output=sysout.out
#SBATCH --error=syserr.err

export EXDIR=${PWD}/..

module purge
module load QuantumESPRESSO/7.1-foss-2022a

export INDIR=${EXDIR}/inputs
export ESPRESSO_PSEUDO=${EXDIR}/../pseudo

# number of threads (serial)
export OMP_NUM_THREADS=...
# execute PW
mpirun ...

We will only use pw.x for this hands-on.
The build-in module of QEv7.1 is loaded in the batch file:

$ module load QuantumESPRESSO/7.1-foss-2022a

Check that the module works by submitting a quick serial test.
Open the Slurm batch file, fill the dots with the right numbers/commands (dots at lines 7, 8, 10, 23, 25) and submit it:

$ sbatch ./ex0-run.slurm

To see the submission status of your job:

$ squeue -u YOUR_USERNAME

Solution

Solution

ex0-run.slurmatom-pbe.out

#!/bin/bash
#SBATCH --job-name=USERjob
#SBATCH --nodes 1
#SBATCH --time=00:00:30
#SBATCH --partition=cpu
#SBATCH --ntasks-per-socket=1
#SBATCH --ntasks-per-node=1
#SBATCH --hint=nomultithread
#SBATCH --cpus-per-task=1
#SBATCH --output=sysout.out
#SBATCH --error=syserr.err

export EXDIR=${PWD}/..

module purge
module load QuantumESPRESSO/7.1-foss-2022a

#input file directory
export INDIR=${EXDIR}/inputs
#pseudo file directory
export ESPRESSO_PSEUDO=${EXDIR}/../pseudo

# number of threads (serial)
export OMP_NUM_THREADS=1
# execute PW
mpirun -np 1 pw.x -in ${INDIR}/atom-pbe.in > atom-pbe.out

---

EXERCISE 1 - Parallelization with pools

Files needed:

ex1-pools.slurm templatepw.CuO.scf.in

#!/bin/bash
#SBATCH --job-name=USERjob
#SBATCH --nodes 1
#SBATCH --exclusive
#SBATCH --time=00:20:00
#SBATCH --partition=cpu
#SBATCH --reservation=maxcpu
#SBATCH --ntasks-per-socket=2
#SBATCH --ntasks-per-node=16
#SBATCH --hint=nomultithread
#SBATCH --cpus-per-task=1
#SBATCH --output=sysout.out
#SBATCH --error=syserr.err
# # SBATCH --mail-user=YOUR_EMAIL - if you want

module purge
module load QuantumESPRESSO/7.1-foss-2022a

export EXDIR=${PWD}/..
export INDIR=${EXDIR}/inputs
export ESPRESSO_PSEUDO=${EXDIR}/../pseudo
export OMP_NUM_THREADS=1

mpiopt="-mca pml ucx -mca btl ^uct,tcp,openib,vader --map-by socket:PE=1 --rank-by core --report-bindings "

for ip in ...
do
mpirun $mpiopt -np 16 pw.x -npool "$ip" -ndiag 1 -i ${INDIR}/pw.CuO.scf.in > pw_CuO_${ip}pools.out
done

Try to predict the best value of npool and check by performing a number of runs.

1. Open the batch file ex1-pools.slurm and customize the user-related SLURM options like job-name and mail-user (not essential at all);
   Replace the dots with a list of proper values for npool, e.g:

   for ip in 1 2 3 4 5 6    # not the necessarily the right values here!
   do
   

2. Submit the job file:

   $ sbatch ./ex1-pools.slurm
   

3. Look for total WALL time at the end of each output file:

           PWSCF        :   3m26.15s CPU   3m30.27s WALL
   

   and plot TIME(npool). Which is the best npool value? Why?

4. You can try with different numbers of MPI tasks.

Solution

Solution

pw_CuO.out:

     Program PWSCF v.7.1 starts on  9Nov2022 at 10:35: 0 

     This program is part of the open-source Quantum ESPRESSO suite
     for quantum simulation of materials; please cite
         "P. Giannozzi et al., J. Phys.:Condens. Matter 21 395502 (2009);
         "P. Giannozzi et al., J. Phys.:Condens. Matter 29 465901 (2017);
         "P. Giannozzi et al., J. Chem. Phys. 152 154105 (2020);
          URL http://www.quantum-espresso.org", 
     in publications or presentations arising from this work. More details at
     http://www.quantum-espresso.org/quote

     Parallel version (MPI & OpenMP), running on      16 processor cores
     Number of MPI processes:                16
     Threads/MPI process:                     1

     MPI processes distributed on     1 nodes
     998766 MiB available memory on the printing compute node when the environment starts

     Reading input from /ceph/hpc/home/eufabrizior/materials-for-max-coe-enccs-workshop-2022/Day-1/ex1-pools/../inputs/pw.CuO.scf.in
Warning: card  &CELL ignored
Warning: card     CELL_DYNAMICS = BFGS, ignored
Warning: card     PRESS = 0 ignored
Warning: card  / ignored

     Current dimensions of program PWSCF are:
     Max number of different atomic species (ntypx) = 10
     Max number of k-points (npk) =  40000
     Max angular momentum in pseudopotentials (lmaxx) =  4
     Found identity + (  0.2500  0.2500  0.0000) symmetry
     This is a supercell, fractional translations are disabled
     Message from routine setup:
     using ibrav=0 with symmetry is DISCOURAGED, use correct ibrav instead

     K-points division:     npool     =       8
     R & G space division:  proc/nbgrp/npool/nimage =       2
     Subspace diagonalization in iterative solution of the eigenvalue problem:
     a serial algorithm will be used


     Parallelization info
     --------------------
     sticks:   dense  smooth     PW     G-vecs:    dense   smooth      PW
     Min        3192    1273    355               241517    61104    9108
     Max        3193    1274    356               241518    61109    9111
     Sum        6385    2547    711               483035   122213   18219

     Using Slab Decomposition



     bravais-lattice index     =            0
     lattice parameter (alat)  =      17.6992  a.u.
     unit-cell volume          =    4367.8091 (a.u.)^3
     number of atoms/cell      =           64
     number of atomic types    =            2
     number of electrons       =       800.00
     number of Kohn-Sham states=          480
     kinetic-energy cutoff     =      35.0000  Ry
     charge density cutoff     =     350.0000  Ry
     scf convergence threshold =      1.0E-07
     mixing beta               =       0.5000
     number of iterations used =            8  plain     mixing
     Exchange-correlation= SLA  PW   PBE  PBE
                           (   1   4   3   4   0   0   0)

     celldm(1)=  17.699175  celldm(2)=   0.000000  celldm(3)=   0.000000
     celldm(4)=   0.000000  celldm(5)=   0.000000  celldm(6)=   0.000000

     crystal axes: (cart. coord. in units of alat)
               a(1) = (   1.000000   0.000000   0.000000 )  
               a(2) = (   0.000000   0.730365   0.000000 )  
               a(3) = (  -0.184506   0.000000   1.078610 )  

     reciprocal axes: (cart. coord. in units 2 pi/alat)
               b(1) = (  1.000000 -0.000000  0.171059 )  
               b(2) = (  0.000000  1.369178 -0.000000 )  
               b(3) = (  0.000000  0.000000  0.927119 )  


     PseudoPot. # 1 for O  read from file:
     /ceph/hpc/home/eufabrizior/materials-for-max-coe-enccs-workshop-2022/Day-1/ex1-pools/../../pseudo/o_pbe_v1.2.uspp.F.UPF
     MD5 check sum: 734c27235a0248c51dbae37a1fbe46ec
     Pseudo is Ultrasoft + core correction, Zval =  6.0
     Generated by new atomic code, or converted to UPF format
     Using radial grid of  737 points,  5 beta functions with: 
                l(1) =   0
                l(2) =   0
                l(3) =   1
                l(4) =   1
                l(5) =   2
     Q(r) pseudized with  8 coefficients,  rinner =    0.900   0.900   0.900
                                                       0.900   0.900

     PseudoPot. # 2 for Cu read from file:
     /ceph/hpc/home/eufabrizior/materials-for-max-coe-enccs-workshop-2022/Day-1/ex1-pools/../../pseudo/cu_pbe_v1.2.uspp.F.UPF
     MD5 check sum: 6e991ff952a84172a2a52a1c1e996048
     Pseudo is Ultrasoft + core correction, Zval = 19.0
     Generated by new atomic code, or converted to UPF format
     Using radial grid of  867 points,  6 beta functions with: 
                l(1) =   0
                l(2) =   0
                l(3) =   1
                l(4) =   1
                l(5) =   2
                l(6) =   2
     Q(r) pseudized with  8 coefficients,  rinner =    1.150   1.150   1.150
                                                       1.150   1.150

     atomic species   valence    mass     pseudopotential
        O              6.00     1.00000     O ( 1.00)
        Cu            19.00     1.00000     Cu( 1.00)

     Starting magnetic structure 
     atomic species   magnetization
        O            0.000
        Cu           0.500

      2 Sym. Ops., with inversion, found



   Cartesian axes

     site n.     atom                  positions (alat units)
         1           O   tau(   1) = (  -0.0230632   0.1522811   0.1348263  )
         2           O   tau(   2) = (  -0.1614426   0.5780840   0.9437839  )
         3           O   tau(   3) = (  -0.0691897   0.5780840   0.4044788  )
         4           O   tau(   4) = (   0.2269368   0.3348724   0.1348263  )
         5           O   tau(   5) = (   0.0885574   0.0303102   0.9437839  )
         6           O   tau(   6) = (   0.1808103   0.0303102   0.4044788  )
         7           O   tau(   7) = (  -0.1153162   0.1522811   0.6741314  )
         8           O   tau(   8) = (   0.1346838   0.3348724   0.6741314  )
         9           O   tau(   9) = (  -0.0230632   0.5174637   0.1348263  )
        10           O   tau(  10) = (  -0.1614426   0.2129014   0.9437839  )
        11           O   tau(  11) = (  -0.0691897   0.2129014   0.4044788  )
        12           O   tau(  12) = (   0.2269368   0.7000550   0.1348263  )
        13           O   tau(  13) = (   0.0885574   0.3954927   0.9437839  )
        14           O   tau(  14) = (   0.1808103   0.3954927   0.4044788  )
        15           O   tau(  15) = (  -0.1153162   0.5174637   0.6741314  )
        16           O   tau(  16) = (   0.1346838   0.7000550   0.6741314  )
        17           O   tau(  17) = (   0.4769368   0.1522811   0.1348263  )
        18           O   tau(  18) = (   0.3385574   0.5780840   0.9437839  )
        19           O   tau(  19) = (   0.4308103   0.5780840   0.4044788  )
        20           O   tau(  20) = (   0.7269368   0.3348724   0.1348263  )
        21           O   tau(  21) = (   0.5885574   0.0303102   0.9437839  )
        22           O   tau(  22) = (   0.6808103   0.0303102   0.4044788  )
        23           O   tau(  23) = (   0.3846838   0.1522811   0.6741314  )
        24           O   tau(  24) = (   0.6346838   0.3348724   0.6741314  )
        25           O   tau(  25) = (   0.4769368   0.5174637   0.1348263  )
        26           O   tau(  26) = (   0.3385574   0.2129014   0.9437839  )
        27           O   tau(  27) = (   0.4308103   0.2129014   0.4044788  )
        28           O   tau(  28) = (   0.7269368   0.7000550   0.1348263  )
        29           O   tau(  29) = (   0.5885574   0.3954927   0.9437839  )
        30           O   tau(  30) = (   0.6808103   0.3954927   0.4044788  )
        31           O   tau(  31) = (   0.3846838   0.5174637   0.6741314  )
        32           O   tau(  32) = (   0.6346838   0.7000550   0.6741314  )
        33           Cu  tau(  33) = (   0.1250000   0.0912956   0.0000000  )
        34           Cu  tau(  34) = (   0.8750000   0.6390695   0.0000000  )
        35           Cu  tau(  35) = (   0.8288735   0.0912956   0.2696526  )
        36           Cu  tau(  36) = (   0.0788735   0.6390695   0.2696526  )
        37           Cu  tau(  37) = (   0.3750000   0.2738869   0.0000000  )
        38           Cu  tau(  38) = (   0.0788735   0.2738869   0.2696526  )
        39           Cu  tau(  39) = (   0.3288735   0.0912956   0.2696526  )
        40           Cu  tau(  40) = (   0.0327471   0.0912956   0.5393051  )
        41           Cu  tau(  41) = (   0.7827471   0.6390695   0.5393051  )
        42           Cu  tau(  42) = (   0.7366206   0.0912956   0.8089577  )
        43           Cu  tau(  43) = (  -0.0133794   0.6390695   0.8089577  )
        44           Cu  tau(  44) = (   0.2827471   0.2738869   0.5393051  )
        45           Cu  tau(  45) = (  -0.0133794   0.2738869   0.8089577  )
        46           Cu  tau(  46) = (   0.2366206   0.0912956   0.8089577  )
        47           Cu  tau(  47) = (   0.1250000   0.4564782   0.0000000  )
        48           Cu  tau(  48) = (   0.8750000   0.2738869   0.0000000  )
        49           Cu  tau(  49) = (   0.8288735   0.4564782   0.2696526  )
        50           Cu  tau(  50) = (   0.3750000   0.6390695   0.0000000  )
        51           Cu  tau(  51) = (   0.3288735   0.4564782   0.2696526  )
        52           Cu  tau(  52) = (   0.0327471   0.4564782   0.5393051  )
        53           Cu  tau(  53) = (   0.7827471   0.2738869   0.5393051  )
        54           Cu  tau(  54) = (   0.7366206   0.4564782   0.8089577  )
        55           Cu  tau(  55) = (   0.2827471   0.6390695   0.5393051  )
        56           Cu  tau(  56) = (   0.2366206   0.4564782   0.8089577  )
        57           Cu  tau(  57) = (   0.6250000   0.0912956   0.0000000  )
        58           Cu  tau(  58) = (   0.5788735   0.6390695   0.2696526  )
        59           Cu  tau(  59) = (   0.5788735   0.2738869   0.2696526  )
        60           Cu  tau(  60) = (   0.5327471   0.0912956   0.5393051  )
        61           Cu  tau(  61) = (   0.4866206   0.6390695   0.8089577  )
        62           Cu  tau(  62) = (   0.4866206   0.2738869   0.8089577  )
        63           Cu  tau(  63) = (   0.6250000   0.4564782   0.0000000  )
        64           Cu  tau(  64) = (   0.5327471   0.4564782   0.5393051  )

     number of k points=     8  Methfessel-Paxton smearing, width (Ry)=  0.0100
                       cart. coord. in units 2pi/alat
        k(    1) = (   0.0000000   0.0000000   0.0000000), wk =   0.1250000
        k(    2) = (   0.0000000   0.0000000  -0.4635595), wk =   0.1250000
        k(    3) = (   0.0000000  -0.6845891   0.0000000), wk =   0.1250000
        k(    4) = (   0.0000000  -0.6845891  -0.4635595), wk =   0.1250000
        k(    5) = (  -0.5000000   0.0000000  -0.0855294), wk =   0.1250000
        k(    6) = (  -0.5000000   0.0000000  -0.5490889), wk =   0.1250000
        k(    7) = (  -0.5000000  -0.6845891  -0.0855294), wk =   0.1250000
        k(    8) = (  -0.5000000  -0.6845891  -0.5490889), wk =   0.1250000

     Dense  grid:   483035 G-vectors     FFT dimensions: ( 108,  80, 120)

     Smooth grid:   122213 G-vectors     FFT dimensions: (  72,  50,  75)

     Estimated max dynamical RAM per process >     906.88 MB

     Estimated total dynamical RAM >      12.56 GB

     Check: negative core charge=   -0.000002
     Generating pointlists ...
     new r_m :   0.0859 (alat units)  1.5202 (a.u.) for type    1
     new r_m :   0.0859 (alat units)  1.5202 (a.u.) for type    2

     Initial potential from superposition of free atoms

     starting charge     767.9999, renormalised to     800.0000
     Starting wfcs are  544 atomic wfcs

     total cpu time spent up to now is       24.8 secs

     Self-consistent Calculation

     iteration #  1     ecut=    35.00 Ry     beta= 0.50
     Davidson diagonalization with overlap
     ethr =  1.00E-02,  avg # of iterations =  3.6

     total cpu time spent up to now is       78.4 secs

     total energy              =  -13814.48627541 Ry
     estimated scf accuracy    <     256.67013473 Ry

     total magnetization       =   100.86 Bohr mag/cell
     absolute magnetization    =   107.40 Bohr mag/cell

     iteration #  2     ecut=    35.00 Ry     beta= 0.50
     Davidson diagonalization with overlap
     ethr =  1.00E-02,  avg # of iterations =  3.0

     total cpu time spent up to now is      134.3 secs

     total energy              =  -13919.03207246 Ry
     estimated scf accuracy    <     423.44018938 Ry

     total magnetization       =     7.91 Bohr mag/cell
     absolute magnetization    =    11.88 Bohr mag/cell

     iteration #  3     ecut=    35.00 Ry     beta= 0.50
     Davidson diagonalization with overlap
     ethr =  1.00E-02,  avg # of iterations =  2.0

     total cpu time spent up to now is      173.8 secs

     total energy              =  -13936.10197903 Ry
     estimated scf accuracy    <      79.23357079 Ry

     total magnetization       =    18.87 Bohr mag/cell
     absolute magnetization    =    25.51 Bohr mag/cell

     iteration #  4     ecut=    35.00 Ry     beta= 0.50
     Davidson diagonalization with overlap
     ethr =  9.90E-03,  avg # of iterations =  2.1

     total cpu time spent up to now is      210.1 secs

     total energy              =  -13951.84215466 Ry
     estimated scf accuracy    <      22.27988907 Ry

     total magnetization       =    32.26 Bohr mag/cell
     absolute magnetization    =    39.66 Bohr mag/cell

     End of self-consistent calculation

     convergence NOT achieved after   4 iterations: stopping

     Writing config to output data dir ./out/pwscf.save/

     init_run     :     20.24s CPU     20.86s WALL (       1 calls)
     electrons    :    181.97s CPU    185.32s WALL (       1 calls)

     Called by init_run:
     wfcinit      :      9.13s CPU      9.45s WALL (       1 calls)
     potinit      :      4.44s CPU      4.64s WALL (       1 calls)
     hinit0       :      5.63s CPU      5.66s WALL (       1 calls)

     Called by electrons:
     c_bands      :    161.64s CPU    163.85s WALL (       4 calls)
     sum_band     :     13.17s CPU     13.74s WALL (       4 calls)
     v_of_rho     :      6.38s CPU      6.89s WALL (       5 calls)
     newd         :      2.03s CPU      2.26s WALL (       5 calls)
     mix_rho      :      0.21s CPU      0.23s WALL (       4 calls)

     Called by c_bands:
     init_us_2    :      1.14s CPU      1.20s WALL (      18 calls)
     init_us_2:cp :      1.14s CPU      1.20s WALL (      18 calls)
     cegterg      :    127.17s CPU    129.24s WALL (       8 calls)

     Called by *egterg:
     cdiaghg      :     14.55s CPU     14.62s WALL (      30 calls)
     h_psi        :     51.73s CPU     52.81s WALL (      32 calls)
     s_psi        :     14.98s CPU     15.05s WALL (      32 calls)
     g_psi        :      0.17s CPU      0.17s WALL (      22 calls)

     Called by h_psi:
     h_psi:calbec :     15.18s CPU     15.28s WALL (      32 calls)
     vloc_psi     :     21.23s CPU     21.80s WALL (      32 calls)
     add_vuspsi   :     15.04s CPU     15.11s WALL (      32 calls)

     General routines
     calbec       :     20.32s CPU     20.45s WALL (      40 calls)
     fft          :      3.37s CPU      3.62s WALL (     123 calls)
     ffts         :      0.33s CPU      0.34s WALL (      18 calls)
     fftw         :     21.55s CPU     22.15s WALL (   26754 calls)
     interpolate  :      0.62s CPU      0.64s WALL (      10 calls)
     davcio       :      0.00s CPU      0.01s WALL (       8 calls)

     Parallel routines

     PWSCF        :   3m26.15s CPU   3m30.27s WALL


   This run was terminated on:  10:38:31   9Nov2022            

=------------------------------------------------------------------------------=
   JOB DONE.
=------------------------------------------------------------------------------=

---

EXERCISE 2 - Parallelization of the eigenvalue problem

Files needed:

ex2-diag.slurm templatepw.CuO.scf.in

#!/bin/bash
#SBATCH --job-name=USERjob
#SBATCH --nodes 1
#SBATCH --exclusive
#SBATCH --time=00:20:00
#SBATCH --partition=cpu
#SBATCH --reservation=maxcpu
#SBATCH --ntasks-per-socket=2
#SBATCH --ntasks-per-node=16
#SBATCH --hint=nomultithread
#SBATCH --cpus-per-task=1
#SBATCH --output=sysout.out
#SBATCH --error=syserr.err
# # SBATCH --mail-user=YOUR_EMAIL - if you want

module purge
module load QuantumESPRESSO/7.1-foss-2022a

export EXDIR=${PWD}/..
export INDIR=${EXDIR}/inputs
export ESPRESSO_PSEUDO=${EXDIR}/../pseudo
export OMP_NUM_THREADS=1

mpiopt="-mca pml ucx -mca btl ^uct,tcp,openib,vader --map-by socket:PE=1 --rank-by core --report-bindings "

for id in ....
do
mpirun $mpiopt -np 16 pw.x -npool 4 -ndiag "$id" -i ${INDIR}/pw.CuO.scf.in > pw_CuO_${id}diag.out
done

Play with the ndiag parameter by performing a number of runs and seeing variations in the WALL time (if any).
You can also change the fixed value of npools (the default value for this exercise is 4).

1. Replace the dots with a list of proper values for ndiag, e.g:

   for id in 1 2 3 4 5 6     # not necessarily the right values here!
   do
   

2. Submit the job file:

   $ sbatch ./ex2-diag.slurm
   

3. Check the total WALL time at the end of the output file and plot TIME(ndiag).
   Which is the best ndiag value (if any)? Why?

Solution

Solution

pw_CuO_4diag.out:

     Program PWSCF v.7.1 starts on  9Nov2022 at 12:22:39 

     This program is part of the open-source Quantum ESPRESSO suite
     for quantum simulation of materials; please cite
         "P. Giannozzi et al., J. Phys.:Condens. Matter 21 395502 (2009);
         "P. Giannozzi et al., J. Phys.:Condens. Matter 29 465901 (2017);
         "P. Giannozzi et al., J. Chem. Phys. 152 154105 (2020);
          URL http://www.quantum-espresso.org", 
     in publications or presentations arising from this work. More details at
     http://www.quantum-espresso.org/quote

     Parallel version (MPI & OpenMP), running on      16 processor cores
     Number of MPI processes:                16
     Threads/MPI process:                     1

     MPI processes distributed on     1 nodes
     990353 MiB available memory on the printing compute node when the environment starts

     Reading input from /ceph/hpc/home/eufabrizior/materials-for-max-coe-enccs-workshop-2022/Day-1/ex2-ndiag/../inputs/pw.CuO.scf.in
Warning: card  &CELL ignored
Warning: card     CELL_DYNAMICS = BFGS, ignored
Warning: card     PRESS = 0 ignored
Warning: card  / ignored

     Current dimensions of program PWSCF are:
     Max number of different atomic species (ntypx) = 10
     Max number of k-points (npk) =  40000
     Max angular momentum in pseudopotentials (lmaxx) =  4
     Found identity + (  0.2500  0.2500  0.0000) symmetry
     This is a supercell, fractional translations are disabled
     Message from routine setup:
     using ibrav=0 with symmetry is DISCOURAGED, use correct ibrav instead

     K-points division:     npool     =       4
     R & G space division:  proc/nbgrp/npool/nimage =       4
     Subspace diagonalization in iterative solution of the eigenvalue problem:
     one sub-group per band group will be used
     scalapack distributed-memory algorithm (size of sub-group:  2*  2 procs)


     Parallelization info
     --------------------
     sticks:   dense  smooth     PW     G-vecs:    dense   smooth      PW
     Min        1596     636    177               120758    30550    4553
     Max        1597     637    178               120760    30555    4556
     Sum        6385    2547    711               483035   122213   18219

     Using Slab Decomposition



     bravais-lattice index     =            0
     lattice parameter (alat)  =      17.6992  a.u.
     unit-cell volume          =    4367.8091 (a.u.)^3
     number of atoms/cell      =           64
     number of atomic types    =            2
     number of electrons       =       800.00
     number of Kohn-Sham states=          480
     kinetic-energy cutoff     =      35.0000  Ry
     charge density cutoff     =     350.0000  Ry
     scf convergence threshold =      1.0E-07
     mixing beta               =       0.5000
     number of iterations used =            8  plain     mixing
     Exchange-correlation= SLA  PW   PBE  PBE
                           (   1   4   3   4   0   0   0)

     celldm(1)=  17.699175  celldm(2)=   0.000000  celldm(3)=   0.000000
     celldm(4)=   0.000000  celldm(5)=   0.000000  celldm(6)=   0.000000

     crystal axes: (cart. coord. in units of alat)
               a(1) = (   1.000000   0.000000   0.000000 )  
               a(2) = (   0.000000   0.730365   0.000000 )  
               a(3) = (  -0.184506   0.000000   1.078610 )  

     reciprocal axes: (cart. coord. in units 2 pi/alat)
               b(1) = (  1.000000 -0.000000  0.171059 )  
               b(2) = (  0.000000  1.369178 -0.000000 )  
               b(3) = (  0.000000  0.000000  0.927119 )  


     PseudoPot. # 1 for O  read from file:
     /ceph/hpc/home/eufabrizior/materials-for-max-coe-enccs-workshop-2022/Day-1/ex2-ndiag/../../pseudo/o_pbe_v1.2.uspp.F.UPF
     MD5 check sum: 734c27235a0248c51dbae37a1fbe46ec
     Pseudo is Ultrasoft + core correction, Zval =  6.0
     Generated by new atomic code, or converted to UPF format
     Using radial grid of  737 points,  5 beta functions with: 
                l(1) =   0
                l(2) =   0
                l(3) =   1
                l(4) =   1
                l(5) =   2
     Q(r) pseudized with  8 coefficients,  rinner =    0.900   0.900   0.900
                                                       0.900   0.900

     PseudoPot. # 2 for Cu read from file:
     /ceph/hpc/home/eufabrizior/materials-for-max-coe-enccs-workshop-2022/Day-1/ex2-ndiag/../../pseudo/cu_pbe_v1.2.uspp.F.UPF
     MD5 check sum: 6e991ff952a84172a2a52a1c1e996048
     Pseudo is Ultrasoft + core correction, Zval = 19.0
     Generated by new atomic code, or converted to UPF format
     Using radial grid of  867 points,  6 beta functions with: 
                l(1) =   0
                l(2) =   0
                l(3) =   1
                l(4) =   1
                l(5) =   2
                l(6) =   2
     Q(r) pseudized with  8 coefficients,  rinner =    1.150   1.150   1.150
                                                       1.150   1.150

     atomic species   valence    mass     pseudopotential
        O              6.00     1.00000     O ( 1.00)
        Cu            19.00     1.00000     Cu( 1.00)

     Starting magnetic structure 
     atomic species   magnetization
        O            0.000
        Cu           0.500

      2 Sym. Ops., with inversion, found



   Cartesian axes

     site n.     atom                  positions (alat units)
         1           O   tau(   1) = (  -0.0230632   0.1522811   0.1348263  )
         2           O   tau(   2) = (  -0.1614426   0.5780840   0.9437839  )
         3           O   tau(   3) = (  -0.0691897   0.5780840   0.4044788  )
         4           O   tau(   4) = (   0.2269368   0.3348724   0.1348263  )
         5           O   tau(   5) = (   0.0885574   0.0303102   0.9437839  )
         6           O   tau(   6) = (   0.1808103   0.0303102   0.4044788  )
         7           O   tau(   7) = (  -0.1153162   0.1522811   0.6741314  )
         8           O   tau(   8) = (   0.1346838   0.3348724   0.6741314  )
         9           O   tau(   9) = (  -0.0230632   0.5174637   0.1348263  )
        10           O   tau(  10) = (  -0.1614426   0.2129014   0.9437839  )
        11           O   tau(  11) = (  -0.0691897   0.2129014   0.4044788  )
        12           O   tau(  12) = (   0.2269368   0.7000550   0.1348263  )
        13           O   tau(  13) = (   0.0885574   0.3954927   0.9437839  )
        14           O   tau(  14) = (   0.1808103   0.3954927   0.4044788  )
        15           O   tau(  15) = (  -0.1153162   0.5174637   0.6741314  )
        16           O   tau(  16) = (   0.1346838   0.7000550   0.6741314  )
        17           O   tau(  17) = (   0.4769368   0.1522811   0.1348263  )
        18           O   tau(  18) = (   0.3385574   0.5780840   0.9437839  )
        19           O   tau(  19) = (   0.4308103   0.5780840   0.4044788  )
        20           O   tau(  20) = (   0.7269368   0.3348724   0.1348263  )
        21           O   tau(  21) = (   0.5885574   0.0303102   0.9437839  )
        22           O   tau(  22) = (   0.6808103   0.0303102   0.4044788  )
        23           O   tau(  23) = (   0.3846838   0.1522811   0.6741314  )
        24           O   tau(  24) = (   0.6346838   0.3348724   0.6741314  )
        25           O   tau(  25) = (   0.4769368   0.5174637   0.1348263  )
        26           O   tau(  26) = (   0.3385574   0.2129014   0.9437839  )
        27           O   tau(  27) = (   0.4308103   0.2129014   0.4044788  )
        28           O   tau(  28) = (   0.7269368   0.7000550   0.1348263  )
        29           O   tau(  29) = (   0.5885574   0.3954927   0.9437839  )
        30           O   tau(  30) = (   0.6808103   0.3954927   0.4044788  )
        31           O   tau(  31) = (   0.3846838   0.5174637   0.6741314  )
        32           O   tau(  32) = (   0.6346838   0.7000550   0.6741314  )
        33           Cu  tau(  33) = (   0.1250000   0.0912956   0.0000000  )
        34           Cu  tau(  34) = (   0.8750000   0.6390695   0.0000000  )
        35           Cu  tau(  35) = (   0.8288735   0.0912956   0.2696526  )
        36           Cu  tau(  36) = (   0.0788735   0.6390695   0.2696526  )
        37           Cu  tau(  37) = (   0.3750000   0.2738869   0.0000000  )
        38           Cu  tau(  38) = (   0.0788735   0.2738869   0.2696526  )
        39           Cu  tau(  39) = (   0.3288735   0.0912956   0.2696526  )
        40           Cu  tau(  40) = (   0.0327471   0.0912956   0.5393051  )
        41           Cu  tau(  41) = (   0.7827471   0.6390695   0.5393051  )
        42           Cu  tau(  42) = (   0.7366206   0.0912956   0.8089577  )
        43           Cu  tau(  43) = (  -0.0133794   0.6390695   0.8089577  )
        44           Cu  tau(  44) = (   0.2827471   0.2738869   0.5393051  )
        45           Cu  tau(  45) = (  -0.0133794   0.2738869   0.8089577  )
        46           Cu  tau(  46) = (   0.2366206   0.0912956   0.8089577  )
        47           Cu  tau(  47) = (   0.1250000   0.4564782   0.0000000  )
        48           Cu  tau(  48) = (   0.8750000   0.2738869   0.0000000  )
        49           Cu  tau(  49) = (   0.8288735   0.4564782   0.2696526  )
        50           Cu  tau(  50) = (   0.3750000   0.6390695   0.0000000  )
        51           Cu  tau(  51) = (   0.3288735   0.4564782   0.2696526  )
        52           Cu  tau(  52) = (   0.0327471   0.4564782   0.5393051  )
        53           Cu  tau(  53) = (   0.7827471   0.2738869   0.5393051  )
        54           Cu  tau(  54) = (   0.7366206   0.4564782   0.8089577  )
        55           Cu  tau(  55) = (   0.2827471   0.6390695   0.5393051  )
        56           Cu  tau(  56) = (   0.2366206   0.4564782   0.8089577  )
        57           Cu  tau(  57) = (   0.6250000   0.0912956   0.0000000  )
        58           Cu  tau(  58) = (   0.5788735   0.6390695   0.2696526  )
        59           Cu  tau(  59) = (   0.5788735   0.2738869   0.2696526  )
        60           Cu  tau(  60) = (   0.5327471   0.0912956   0.5393051  )
        61           Cu  tau(  61) = (   0.4866206   0.6390695   0.8089577  )
        62           Cu  tau(  62) = (   0.4866206   0.2738869   0.8089577  )
        63           Cu  tau(  63) = (   0.6250000   0.4564782   0.0000000  )
        64           Cu  tau(  64) = (   0.5327471   0.4564782   0.5393051  )

     number of k points=     8  Methfessel-Paxton smearing, width (Ry)=  0.0100
                       cart. coord. in units 2pi/alat
        k(    1) = (   0.0000000   0.0000000   0.0000000), wk =   0.1250000
        k(    2) = (   0.0000000   0.0000000  -0.4635595), wk =   0.1250000
        k(    3) = (   0.0000000  -0.6845891   0.0000000), wk =   0.1250000
        k(    4) = (   0.0000000  -0.6845891  -0.4635595), wk =   0.1250000
        k(    5) = (  -0.5000000   0.0000000  -0.0855294), wk =   0.1250000
        k(    6) = (  -0.5000000   0.0000000  -0.5490889), wk =   0.1250000
        k(    7) = (  -0.5000000  -0.6845891  -0.0855294), wk =   0.1250000
        k(    8) = (  -0.5000000  -0.6845891  -0.5490889), wk =   0.1250000

     Dense  grid:   483035 G-vectors     FFT dimensions: ( 108,  80, 120)

     Smooth grid:   122213 G-vectors     FFT dimensions: (  72,  50,  75)

     Estimated max dynamical RAM per process >     503.96 MB

     Estimated total dynamical RAM >       7.18 GB

     Check: negative core charge=   -0.000002
     Generating pointlists ...
     new r_m :   0.0859 (alat units)  1.5202 (a.u.) for type    1
     new r_m :   0.0859 (alat units)  1.5202 (a.u.) for type    2

     Initial potential from superposition of free atoms

     starting charge     767.9999, renormalised to     800.0000
     Starting wfcs are  544 atomic wfcs

     total cpu time spent up to now is       18.3 secs

     Self-consistent Calculation

     iteration #  1     ecut=    35.00 Ry     beta= 0.50
     Davidson diagonalization with overlap
     ethr =  1.00E-02,  avg # of iterations =  3.6

     total cpu time spent up to now is       65.6 secs

     total energy              =  -13814.48629833 Ry
     estimated scf accuracy    <     256.67010713 Ry

     total magnetization       =   100.86 Bohr mag/cell
     absolute magnetization    =   107.40 Bohr mag/cell

     iteration #  2     ecut=    35.00 Ry     beta= 0.50
     Davidson diagonalization with overlap
     ethr =  1.00E-02,  avg # of iterations =  3.0

     total cpu time spent up to now is      119.8 secs

     total energy              =  -13919.03203722 Ry
     estimated scf accuracy    <     423.44026184 Ry

     total magnetization       =     7.91 Bohr mag/cell
     absolute magnetization    =    11.88 Bohr mag/cell

     iteration #  3     ecut=    35.00 Ry     beta= 0.50
     Davidson diagonalization with overlap
     ethr =  1.00E-02,  avg # of iterations =  2.0

     total cpu time spent up to now is      161.0 secs

     total energy              =  -13936.10189575 Ry
     estimated scf accuracy    <      79.23343546 Ry

     total magnetization       =    18.87 Bohr mag/cell
     absolute magnetization    =    25.51 Bohr mag/cell

     iteration #  4     ecut=    35.00 Ry     beta= 0.50
     Davidson diagonalization with overlap
     ethr =  9.90E-03,  avg # of iterations =  2.1

     total cpu time spent up to now is      198.1 secs

     total energy              =  -13951.84219563 Ry
     estimated scf accuracy    <      22.27972501 Ry

     total magnetization       =    32.26 Bohr mag/cell
     absolute magnetization    =    39.66 Bohr mag/cell

     End of self-consistent calculation

     convergence NOT achieved after   4 iterations: stopping

     Writing config to output data dir ./out/pwscf.save/

     init_run     :     17.57s CPU     18.14s WALL (       1 calls)
     electrons    :    175.59s CPU    179.83s WALL (       1 calls)

     Called by init_run:
     wfcinit      :     10.15s CPU     10.52s WALL (       1 calls)
     potinit      :      2.62s CPU      2.72s WALL (       1 calls)
     hinit0       :      4.38s CPU      4.42s WALL (       1 calls)

     Called by electrons:
     c_bands      :    159.51s CPU    162.88s WALL (       4 calls)
     sum_band     :     12.29s CPU     12.83s WALL (       4 calls)
     v_of_rho     :      2.89s CPU      3.09s WALL (       5 calls)
     newd         :      1.32s CPU      1.53s WALL (       5 calls)
     mix_rho      :      0.16s CPU      0.18s WALL (       4 calls)

     Called by c_bands:
     init_us_2    :      1.08s CPU      1.13s WALL (      36 calls)
     init_us_2:cp :      1.08s CPU      1.13s WALL (      36 calls)
     cegterg      :    149.95s CPU    153.27s WALL (      16 calls)

     Called by *egterg:
     cdiaghg      :     23.47s CPU     24.04s WALL (      58 calls)
     h_psi        :     57.38s CPU     58.93s WALL (      62 calls)
     s_psi        :     17.22s CPU     17.30s WALL (      62 calls)
     g_psi        :      0.21s CPU      0.21s WALL (      42 calls)

     Called by h_psi:
     h_psi:calbec :     17.76s CPU     17.91s WALL (      62 calls)
     vloc_psi     :     22.00s CPU     22.94s WALL (      62 calls)
     add_vuspsi   :     17.25s CPU     17.35s WALL (      62 calls)

     General routines
     calbec       :     23.67s CPU     23.88s WALL (      78 calls)
     fft          :      1.57s CPU      1.70s WALL (     123 calls)
     ffts         :      0.31s CPU      0.32s WALL (      18 calls)
     fftw         :     22.48s CPU     23.45s WALL (   53488 calls)
     interpolate  :      0.41s CPU      0.42s WALL (      10 calls)
     davcio       :      0.00s CPU      0.01s WALL (       8 calls)

     Parallel routines

     PWSCF        :   3m13.35s CPU   3m18.22s WALL


   This run was terminated on:  12:25:58   9Nov2022            

=------------------------------------------------------------------------------=
   JOB DONE.
=------------------------------------------------------------------------------=

---

EXERCISE 3 - MPI + OpenMP parallelization

Files needed:

ex3-omp.slurm

#!/bin/bash
#SBATCH --job-name=USERjob
#SBATCH --nodes 1
#SBATCH --exclusive
#SBATCH --time=00:20:00
#SBATCH --partition=cpu
#SBATCH --reservation=maxcpu
#SBATCH --ntasks-per-socket=2
#SBATCH --ntasks-per-node=...
#SBATCH --hint=nomultithread
#SBATCH --cpus-per-task=...
#SBATCH --output=sysout.out
#SBATCH --error=syserr.err
# # SBATCH --mail-user=YOUR_EMAIL - if you want

module purge
module load QuantumESPRESSO/7.1-foss-2022a

export EXDIR=${PWD}/..
export INDIR=${EXDIR}/inputs
export ESPRESSO_PSEUDO=${EXDIR}/../pseudo

mpiopt="-mca pml ucx -mca btl ^uct,tcp,openib,vader --map-by socket:PE=4 --rank-by core --report-bindings "

for nthr in ....
do
export OMP_NUM_THREADS="$nthr"
mpirun $mpiopt -np ... pw.x -npool ... -i ${INDIR}/pw.CuO.scf.in > pw_run16x${nthr}.out
done

Find out how to best exploit the available CPU resources, by playing with the MPI-related parameters (number of tasks, npools) together with the number of threads.
Use the batch file ex3-omp.slurm to submit your jobs (modify it at your convenience).
Hints:

0. Know the size of your node, e.g. the amount of cores at your disposal;

1. See how the time scaling of your jobs changes just by varying the number of tasks (keep just 1 thread each at first).
   Adapt the npool parameter at each run.

2. Now you can start to explore the OpenMP parallelization by varying the number of threads (avoid hyperthreading).

3. Do multiple WALL_TIME plots in function of the number of MPI tasks and OpenMP threads. Which is the best configuration for this exercise?

Solution

Solution

ex3-omp.slurmpw_run16x1.outpw_run16x2.outpw_run16x4.out

#!/bin/bash
#SBATCH --job-name=USERjob
#SBATCH --nodes 1
#SBATCH --exclusive
#SBATCH --time=00:20:00
#SBATCH --partition=cpu
#SBATCH --ntasks-per-socket=2
#SBATCH --ntasks-per-node=16
#SBATCH --hint=nomultithread
#SBATCH --cpus-per-task=4
#SBATCH --output=sysout.out
#SBATCH --error=syserr.err
# # SBATCH --mail-user=YOUR_EMAIL - if you want

module purge
module load QuantumESPRESSO/7.1-foss-2022a

export EXDIR=${PWD}/..
export INDIR=${EXDIR}/inputs
export ESPRESSO_PSEUDO=${EXDIR}/../pseudo

mpiopt="-mca pml ucx -mca btl ^uct,tcp,openib,vader --map-by socket:PE=4 --rank-by core --report-bindings "

for nthr in 1 2 4
do
export OMP_NUM_THREADS="$nthr"
mpirun $mpiopt -np 16 pw.x -npool 8 -i ${INDIR}/pw.CuO.scf.in > pw_run16x${nthr}.out
done

EXERCISE 4 - Hands on the GPUs

Files needed:

ex4-cpu.slurmex4-gpu.slurm

#!/bin/bash
#SBATCH --job-name=USERjob
#SBATCH --nodes 1
#SBATCH --exclusive
#SBATCH --time=00:10:00
#SBATCH --partition=cpu
#SBATCH --reservation=maxcpu
#SBATCH --ntasks-per-node=16
#SBATCH --cpus-per-task=8
#SBATCH --output=sysout.out
#SBATCH --error=syserr.err
# # SBATCH --mail-user=YOUR_EMAIL - if you want

# module purge
# module use ${WORK}/modules
# echo $WORK
# module load QuantumESPRESSO/DEV-NVHPC-21.2-FIXMAG
module purge
module load QuantumESPRESSO/7.1-foss-2022a

export EXDIR=${PWD}/..
export INDIR=${EXDIR}/inputs
export ESPRESSO_PSEUDO=${EXDIR}/../pseudo

export OMP_NUM_THREADS=4

mpiopt="-mca pml ucx -mca btl ^uct,tcp,openib,vader --map-by socket:PE=4 --rank-by socket --report-bindings "

mpirun $mpiopt -np 16 pw.x -i ${INDIR}/pw.CnSnI3.in > pw.CnSnI3.cpu.out

Test the power of the GPUs (roughly).

0. Know the size of your node. Look at: https://doc.vega.izum.si/general-spec/
   How many cores? How many GPUs?

1. Use the batch file ex4-gpu.slurm to submit a few MPI+GPU runs. You can also check what happens with more MPI tasks than GPUs.
   Dots at lines 7, 9, 10, 25, 27, 29.

2. Enable openMP threading. Do you see any improvement?

3. Consider your best CPU run. How many GPUs were necessary to match the performance?
   If you don’t have your optimized CPU batch file from exercise 3 you can use the one in the reference folder.

Solution

Solution

ex4-cpu.slurmex4-gpu.slurmpw.CnSnI3.cpu.outpw.CnSnI3_gpu.out

#!/bin/bash
#SBATCH --job-name=USERjob
#SBATCH --nodes 1
#SBATCH --exclusive
#SBATCH --time=00:10:00
#SBATCH --partition=cpu
#SBATCH --reservation=maxcpu
#SBATCH --ntasks-per-node=16
#SBATCH --cpus-per-task=8
#SBATCH --output=sysout.out
#SBATCH --error=syserr.err
# # SBATCH --mail-user=YOUR_EMAIL - if you want

# module purge
# module use ${WORK}/modules
# echo $WORK
# module load QuantumESPRESSO/DEV-NVHPC-21.2-FIXMAG
module purge
module load QuantumESPRESSO/7.1-foss-2022a

export EXDIR=${PWD}/..
export INDIR=${EXDIR}/inputs
export ESPRESSO_PSEUDO=${EXDIR}/../pseudo

export OMP_NUM_THREADS=4

mpiopt="-mca pml ucx -mca btl ^uct,tcp,openib,vader --map-by socket:PE=4 --rank-by socket --report-bindings "

mpirun $mpiopt -np 16 pw.x -i ${INDIR}/pw.CnSnI3.in > pw.CnSnI3.cpu.out