:-) GROMACS - gmx mdrun, 2021.3 (-: GROMACS is written by: Andrey Alekseenko Emile Apol Rossen Apostolov Paul Bauer Herman J.C. Berendsen Par Bjelkmar Christian Blau Viacheslav Bolnykh Kevin Boyd Aldert van Buuren Rudi van Drunen Anton Feenstra Gilles Gouaillardet Alan Gray Gerrit Groenhof Anca Hamuraru Vincent Hindriksen M. Eric Irrgang Aleksei Iupinov Christoph Junghans Joe Jordan Dimitrios Karkoulis Peter Kasson Jiri Kraus Carsten Kutzner Per Larsson Justin A. Lemkul Viveca Lindahl Magnus Lundborg Erik Marklund Pascal Merz Pieter Meulenhoff Teemu Murtola Szilard Pall Sander Pronk Roland Schulz Michael Shirts Alexey Shvetsov Alfons Sijbers Peter Tieleman Jon Vincent Teemu Virolainen Christian Wennberg Maarten Wolf Artem Zhmurov and the project leaders: Mark Abraham, Berk Hess, Erik Lindahl, and David van der Spoel Copyright (c) 1991-2000, University of Groningen, The Netherlands. Copyright (c) 2001-2019, The GROMACS development team at Uppsala University, Stockholm University and the Royal Institute of Technology, Sweden. check out http://www.gromacs.org for more information. GROMACS is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version. GROMACS: gmx mdrun, version 2021.3 Executable: /veracruz/projects/t/training/gromacs-2021.3/bin/gmx Data prefix: /veracruz/projects/t/training/gromacs-2021.3 Working dir: /veracruz/home/m/mabraham/git/gromacs-gpu-performance-master/content/exercises/using-pme Process ID: 174461 Command line: gmx mdrun -ntmpi 1 -noconfout -resetstep 10000 -g default GROMACS version: 2021.3 Verified release checksum is c5bf577cc74de0e05106b7b6426476abb7f6530be7b4a2c64f637d6a6eca8fcb Precision: mixed Memory model: 64 bit MPI library: thread_mpi OpenMP support: enabled (GMX_OPENMP_MAX_THREADS = 64) GPU support: CUDA SIMD instructions: AVX_512 FFT library: fftw-3.3.9-sse2-avx RDTSCP usage: enabled TNG support: enabled Hwloc support: disabled Tracing support: disabled C compiler: /apps/software/GCCcore/9.3.0/bin/cc GNU 9.3.0 C compiler flags: -mavx512f -mfma -Wno-missing-field-initializers -fexcess-precision=fast -funroll-all-loops -pthread -O3 -DNDEBUG C++ compiler: /apps/software/GCCcore/9.3.0/bin/c++ GNU 9.3.0 C++ compiler flags: -mavx512f -mfma -Wno-missing-field-initializers -fexcess-precision=fast -funroll-all-loops -pthread -fopenmp -O3 -DNDEBUG CUDA compiler: /apps/software/CUDAcore/11.0.2/bin/nvcc nvcc: NVIDIA (R) Cuda compiler driver;Copyright (c) 2005-2020 NVIDIA Corporation;Built on Thu_Jun_11_22:26:38_PDT_2020;Cuda compilation tools, release 11.0, V11.0.194;Build cuda_11.0_bu.TC445_37.28540450_0 CUDA compiler flags:-std=c++17;-gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_52,code=sm_52;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-Wno-deprecated-gpu-targets;-gencode;arch=compute_35,code=compute_35;-gencode;arch=compute_53,code=compute_53;-gencode;arch=compute_80,code=compute_80;-use_fast_math;;-mavx512f -mfma -Wno-missing-field-initializers -fexcess-precision=fast -funroll-all-loops -pthread -fopenmp -O3 -DNDEBUG CUDA driver: 11.10 CUDA runtime: 11.0 Running on 1 node with total 80 cores, 80 logical cores, 1 compatible GPU Hardware detected: CPU info: Vendor: Intel Brand: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz Family: 6 Model: 85 Stepping: 4 Features: aes apic avx avx2 avx512f avx512cd avx512bw avx512vl avx512secondFMA clfsh cmov cx8 cx16 f16c fma hle htt intel lahf mmx msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd rdtscp rtm sse2 sse3 sse4.1 sse4.2 ssse3 tdt x2apic Number of AVX-512 FMA units: 2 Hardware topology: Only logical processor count GPU info: Number of GPUs detected: 1 #0: NVIDIA Tesla V100-PCIE-16GB, compute cap.: 7.0, ECC: yes, stat: compatible ++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++ M. J. Abraham, T. Murtola, R. Schulz, S. Páll, J. C. Smith, B. Hess, E. Lindahl GROMACS: High performance molecular simulations through multi-level parallelism from laptops to supercomputers SoftwareX 1 (2015) pp. 19-25 -------- -------- --- Thank You --- -------- -------- ++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++ S. Páll, M. J. Abraham, C. Kutzner, B. Hess, E. Lindahl Tackling Exascale Software Challenges in Molecular Dynamics Simulations with GROMACS In S. Markidis & E. Laure (Eds.), Solving Software Challenges for Exascale 8759 (2015) pp. 3-27 -------- -------- --- Thank You --- -------- -------- ++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++ S. Pronk, S. Páll, R. Schulz, P. Larsson, P. Bjelkmar, R. Apostolov, M. R. Shirts, J. C. Smith, P. M. Kasson, D. van der Spoel, B. Hess, and E. Lindahl GROMACS 4.5: a high-throughput and highly parallel open source molecular simulation toolkit Bioinformatics 29 (2013) pp. 845-54 -------- -------- --- Thank You --- -------- -------- ++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++ B. Hess and C. Kutzner and D. van der Spoel and E. Lindahl GROMACS 4: Algorithms for highly efficient, load-balanced, and scalable molecular simulation J. Chem. Theory Comput. 4 (2008) pp. 435-447 -------- -------- --- Thank You --- -------- -------- ++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++ D. van der Spoel, E. Lindahl, B. Hess, G. Groenhof, A. E. Mark and H. J. C. Berendsen GROMACS: Fast, Flexible and Free J. Comp. Chem. 26 (2005) pp. 1701-1719 -------- -------- --- Thank You --- -------- -------- ++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++ E. Lindahl and B. Hess and D. van der Spoel GROMACS 3.0: A package for molecular simulation and trajectory analysis J. Mol. Mod. 7 (2001) pp. 306-317 -------- -------- --- Thank You --- -------- -------- ++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++ H. J. C. Berendsen, D. van der Spoel and R. van Drunen GROMACS: A message-passing parallel molecular dynamics implementation Comp. Phys. Comm. 91 (1995) pp. 43-56 -------- -------- --- Thank You --- -------- -------- ++++ PLEASE CITE THE DOI FOR THIS VERSION OF GROMACS ++++ https://doi.org/10.5281/zenodo.5053201 -------- -------- --- Thank You --- -------- -------- The number of OpenMP threads was set by environment variable OMP_NUM_THREADS to 20 Input Parameters: integrator = md tinit = 0 dt = 0.002 nsteps = 20000 init-step = 0 simulation-part = 1 mts = false comm-mode = Linear nstcomm = 100 bd-fric = 0 ld-seed = 1993 emtol = 10 emstep = 0.01 niter = 20 fcstep = 0 nstcgsteep = 1000 nbfgscorr = 10 rtpi = 0.05 nstxout = 0 nstvout = 0 nstfout = 0 nstlog = 0 nstcalcenergy = 10 nstenergy = 0 nstxout-compressed = 0 compressed-x-precision = 1000 cutoff-scheme = Verlet nstlist = 10 pbc = xyz periodic-molecules = false verlet-buffer-tolerance = 0.005 rlist = 1 coulombtype = PME coulomb-modifier = Potential-shift rcoulomb-switch = 0 rcoulomb = 1 epsilon-r = 1 epsilon-rf = 1 vdw-type = Cut-off vdw-modifier = Potential-shift rvdw-switch = 0 rvdw = 1 DispCorr = EnerPres table-extension = 1 fourierspacing = 0.12 fourier-nx = 96 fourier-ny = 96 fourier-nz = 128 pme-order = 4 ewald-rtol = 1e-05 ewald-rtol-lj = 0.001 lj-pme-comb-rule = Geometric ewald-geometry = 0 epsilon-surface = 0 tcoupl = V-rescale nsttcouple = 10 nh-chain-length = 0 print-nose-hoover-chain-variables = false pcoupl = No pcoupltype = Semiisotropic nstpcouple = -1 tau-p = 5 compressibility (3x3): compressibility[ 0]={ 0.00000e+00, 0.00000e+00, 0.00000e+00} compressibility[ 1]={ 0.00000e+00, 0.00000e+00, 0.00000e+00} compressibility[ 2]={ 0.00000e+00, 0.00000e+00, 0.00000e+00} ref-p (3x3): ref-p[ 0]={ 0.00000e+00, 0.00000e+00, 0.00000e+00} ref-p[ 1]={ 0.00000e+00, 0.00000e+00, 0.00000e+00} ref-p[ 2]={ 0.00000e+00, 0.00000e+00, 0.00000e+00} refcoord-scaling = COM posres-com (3): posres-com[0]= 0.00000e+00 posres-com[1]= 0.00000e+00 posres-com[2]= 0.00000e+00 posres-comB (3): posres-comB[0]= 0.00000e+00 posres-comB[1]= 0.00000e+00 posres-comB[2]= 0.00000e+00 QMMM = false qm-opts: ngQM = 0 constraint-algorithm = Lincs continuation = false Shake-SOR = false shake-tol = 0.0001 lincs-order = 4 lincs-iter = 1 lincs-warnangle = 45 nwall = 0 wall-type = 9-3 wall-r-linpot = -1 wall-atomtype[0] = -1 wall-atomtype[1] = -1 wall-density[0] = 0 wall-density[1] = 0 wall-ewald-zfac = 3 pull = false awh = false rotation = false interactiveMD = false disre = No disre-weighting = Conservative disre-mixed = false dr-fc = 1000 dr-tau = 0 nstdisreout = 100 orire-fc = 0 orire-tau = 0 nstorireout = 100 free-energy = no cos-acceleration = 0 deform (3x3): deform[ 0]={ 0.00000e+00, 0.00000e+00, 0.00000e+00} deform[ 1]={ 0.00000e+00, 0.00000e+00, 0.00000e+00} deform[ 2]={ 0.00000e+00, 0.00000e+00, 0.00000e+00} simulated-tempering = false swapcoords = no userint1 = 0 userint2 = 0 userint3 = 0 userint4 = 0 userreal1 = 0 userreal2 = 0 userreal3 = 0 userreal4 = 0 applied-forces: electric-field: x: E0 = 0 omega = 0 t0 = 0 sigma = 0 y: E0 = 0 omega = 0 t0 = 0 sigma = 0 z: E0 = 0 omega = 0 t0 = 0 sigma = 0 density-guided-simulation: active = false group = protein similarity-measure = inner-product atom-spreading-weight = unity force-constant = 1e+09 gaussian-transform-spreading-width = 0.2 gaussian-transform-spreading-range-in-multiples-of-width = 4 reference-density-filename = reference.mrc nst = 1 normalize-densities = true adaptive-force-scaling = false adaptive-force-scaling-time-constant = 4 grpopts: nrdf: 67964.4 49247.5 196258 ref-t: 310 310 310 tau-t: 0.1 0.1 0.1 annealing: No No No annealing-npoints: 0 0 0 acc: 0 0 0 nfreeze: N N N energygrp-flags[ 0]: 0 Changing nstlist from 10 to 100, rlist from 1 to 1.132 1 GPU selected for this run. Mapping of GPU IDs to the 2 GPU tasks in the 1 rank on this node: PP:0,PME:0 PP tasks will do (non-perturbed) short-ranged interactions on the GPU PP task will update and constrain coordinates on the CPU PME tasks will do all aspects on the GPU Using 1 MPI thread Non-default thread affinity set, disabling internal thread affinity Using 20 OpenMP threads The -resetstep functionality is deprecated, and may be removed in a future version. System total charge: 0.000 Will do PME sum in reciprocal space for electrostatic interactions. ++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++ U. Essmann, L. Perera, M. L. Berkowitz, T. Darden, H. Lee and L. G. Pedersen A smooth particle mesh Ewald method J. Chem. Phys. 103 (1995) pp. 8577-8592 -------- -------- --- Thank You --- -------- -------- Using a Gaussian width (1/beta) of 0.320163 nm for Ewald Potential shift: LJ r^-12: -1.000e+00 r^-6: -1.000e+00, Ewald -1.000e-05 Initialized non-bonded Ewald tables, spacing: 9.33e-04 size: 1073 Generated table with 1066 data points for 1-4 COUL. Tabscale = 500 points/nm Generated table with 1066 data points for 1-4 LJ6. Tabscale = 500 points/nm Generated table with 1066 data points for 1-4 LJ12. Tabscale = 500 points/nm Long Range LJ corr.: 6.7202e-04 Using GPU 8x8 nonbonded short-range kernels Using a dual 8x8 pair-list setup updated with dynamic, rolling pruning: outer list: updated every 100 steps, buffer 0.132 nm, rlist 1.132 nm inner list: updated every 14 steps, buffer 0.001 nm, rlist 1.001 nm At tolerance 0.005 kJ/mol/ps per atom, equivalent classical 1x1 list would be: outer list: updated every 100 steps, buffer 0.283 nm, rlist 1.283 nm inner list: updated every 14 steps, buffer 0.057 nm, rlist 1.057 nm Using Lorentz-Berthelot Lennard-Jones combination rule Removing pbc first time Initializing LINear Constraint Solver ++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++ B. Hess and H. Bekker and H. J. C. Berendsen and J. G. E. M. Fraaije LINCS: A Linear Constraint Solver for molecular simulations J. Comp. Chem. 18 (1997) pp. 1463-1472 -------- -------- --- Thank You --- -------- -------- The number of constraints is 13620 ++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++ S. Miyamoto and P. A. Kollman SETTLE: An Analytical Version of the SHAKE and RATTLE Algorithms for Rigid Water Models J. Comp. Chem. 13 (1992) pp. 952-962 -------- -------- --- Thank You --- -------- -------- The -noconfout functionality is deprecated, and may be removed in a future version. ++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++ G. Bussi, D. Donadio and M. Parrinello Canonical sampling through velocity rescaling J. Chem. Phys. 126 (2007) pp. 014101 -------- -------- --- Thank You --- -------- -------- There are: 141677 Atoms Constraining the starting coordinates (step 0) Constraining the coordinates at t0-dt (step 0) Center of mass motion removal mode is Linear We have the following groups for center of mass motion removal: 0: System RMS relative constraint deviation after constraining: 3.88e-06 Initial temperature: 279.104 K Started mdrun on rank 0 Thu Sep 9 10:24:52 2021 Step Time 0 0.00000 Energies (kJ/mol) Bond Angle Proper Dih. Ryckaert-Bell. Improper Dih. 8.51521e+02 8.91826e+04 8.15628e+04 1.56545e+04 1.81361e+03 Improper Dih. LJ-14 Coulomb-14 LJ (SR) Disper. corr. 3.30514e+03 4.75419e+04 3.40936e+05 7.67685e+04 -3.46877e+04 Coulomb (SR) Coul. recip. Potential Kinetic En. Total Energy -2.51729e+06 2.10383e+04 -1.87332e+06 3.63069e+05 -1.51025e+06 Conserved En. Temperature Pres. DC (bar) Pressure (bar) Constr. rmsd -1.51025e+06 2.78605e+02 -3.54583e+02 2.48651e+03 3.79109e-06 step 2700: timed with pme grid 96 96 128, coulomb cutoff 1.000: 458.7 M-cycles step 2900: timed with pme grid 84 84 112, coulomb cutoff 1.116: 492.1 M-cycles step 3100: timed with pme grid 72 72 96, coulomb cutoff 1.302: 568.8 M-cycles step 3300: timed with pme grid 80 80 96, coulomb cutoff 1.288: 556.4 M-cycles step 3500: timed with pme grid 80 80 100, coulomb cutoff 1.236: 534.3 M-cycles step 3700: timed with pme grid 80 80 104, coulomb cutoff 1.189: 547.9 M-cycles step 3900: timed with pme grid 80 80 108, coulomb cutoff 1.172: 507.0 M-cycles step 4100: timed with pme grid 84 84 108, coulomb cutoff 1.145: 503.3 M-cycles step 4300: timed with pme grid 84 84 112, coulomb cutoff 1.116: 490.7 M-cycles step 4500: timed with pme grid 96 96 112, coulomb cutoff 1.104: 486.9 M-cycles step 4700: timed with pme grid 96 96 120, coulomb cutoff 1.030: 473.1 M-cycles step 4900: timed with pme grid 96 96 128, coulomb cutoff 1.000: 457.2 M-cycles optimal pme grid 96 96 128, coulomb cutoff 1.000 step 10000: resetting all time and cycle counters Restarted time on rank 0 Thu Sep 9 10:25:23 2021 Step Time 20000 40.00000 Energies (kJ/mol) Bond Angle Proper Dih. Ryckaert-Bell. Improper Dih. 4.41870e+04 8.69871e+04 8.15106e+04 1.56428e+04 1.89968e+03 Improper Dih. LJ-14 Coulomb-14 LJ (SR) Disper. corr. 3.34053e+03 4.34528e+04 3.30520e+05 7.54139e+04 -3.46877e+04 Coulomb (SR) Coul. recip. Potential Kinetic En. Total Energy -2.51033e+06 2.08797e+04 -1.84119e+06 4.04224e+05 -1.43696e+06 Conserved En. Temperature Pres. DC (bar) Pressure (bar) Constr. rmsd -1.51014e+06 3.10186e+02 -3.54583e+02 6.45532e+01 3.84561e-06 Energy conservation over simulation part #1 of length 40 ns, time 0 to 40 ns Conserved energy drift: 1.91e-05 kJ/mol/ps per atom <====== ############### ==> <==== A V E R A G E S ====> <== ############### ======> Statistics over 20001 steps using 2001 frames Energies (kJ/mol) Bond Angle Proper Dih. Ryckaert-Bell. Improper Dih. 4.09641e+04 8.82327e+04 8.19726e+04 1.67079e+04 1.88455e+03 Improper Dih. LJ-14 Coulomb-14 LJ (SR) Disper. corr. 3.32914e+03 4.35621e+04 3.30459e+05 7.69735e+04 -3.46877e+04 Coulomb (SR) Coul. recip. Potential Kinetic En. Total Energy -2.51006e+06 2.02293e+04 -1.84043e+06 4.03865e+05 -1.43657e+06 Conserved En. Temperature Pres. DC (bar) Pressure (bar) Constr. rmsd -1.51020e+06 3.09910e+02 -3.54583e+02 1.19701e+02 0.00000e+00 Total Virial (kJ/mol) 1.28607e+05 -2.67250e+01 -1.15607e+02 -2.64590e+01 1.29194e+05 5.10137e+02 -1.14269e+02 5.10220e+02 1.28475e+05 Pressure (bar) 1.31009e+02 -4.78552e-01 1.45042e+00 -4.83984e-01 1.16603e+02 -1.03928e+01 1.42312e+00 -1.03945e+01 1.11492e+02 T-Protein T-DOPCT-Water_and_ions 3.09755e+02 3.09154e+02 3.10154e+02 M E G A - F L O P S A C C O U N T I N G NB=Group-cutoff nonbonded kernels NxN=N-by-N cluster Verlet kernels RF=Reaction-Field VdW=Van der Waals QSTab=quadratic-spline table W3=SPC/TIP3p W4=TIP4p (single or pairs) V&F=Potential and force V=Potential only F=Force only Computing: M-Number M-Flops % Flops ----------------------------------------------------------------------------- Pair Search distance check 1365.796448 12292.168 0.0 NxN QSTab Elec. + LJ [F] 1094735.825280 44884168.836 85.2 NxN QSTab Elec. + LJ [V&F] 121759.047616 7183783.809 13.6 1,4 nonbonded interactions 810.081000 72907.290 0.1 Shift-X 14.309377 85.856 0.0 Bonds 300.600057 17735.403 0.0 Angles 683.558349 114837.803 0.2 Propers 858.675859 196636.772 0.4 Impropers 70.207020 14603.060 0.0 RB-Dihedrals 66.886688 16521.012 0.0 Virial 141.863722 2553.547 0.0 Stop-CM 14.309377 143.094 0.0 Calc-Ekin 283.495677 7654.383 0.0 Lincs 136.213620 8172.817 0.0 Lincs-Mat 715.871580 2863.486 0.0 Constraint-V 1251.905178 11267.147 0.0 Constraint-Vir 111.669558 2680.069 0.0 Settle 326.492646 120802.279 0.2 ----------------------------------------------------------------------------- Total 52669708.832 100.0 ----------------------------------------------------------------------------- R E A L C Y C L E A N D T I M E A C C O U N T I N G On 1 MPI rank, each using 20 OpenMP threads Computing: Num Num Call Wall time Giga-Cycles Ranks Threads Count (s) total sum % ----------------------------------------------------------------------------- Neighbor search 1 20 101 0.861 41.234 4.5 Launch GPU ops. 1 20 10001 1.087 52.041 5.7 Force 1 20 10001 5.450 260.980 28.6 Wait PME GPU gather 1 20 10001 0.908 43.503 4.8 Reduce GPU PME F 1 20 10001 0.839 40.164 4.4 Wait GPU NB local 0.994 47.593 5.2 NB X/F buffer ops. 1 20 19901 2.777 132.958 14.6 Update 1 20 10001 1.169 56.003 6.1 Constraints 1 20 10001 1.670 79.959 8.8 Rest 3.324 159.171 17.4 ----------------------------------------------------------------------------- Total 19.079 913.607 100.0 ----------------------------------------------------------------------------- Core t (s) Wall t (s) (%) Time: 381.558 19.079 1999.9 (ns/day) (hour/ns) Performance: 90.582 0.265 Finished mdrun on rank 0 Thu Sep 9 10:25:42 2021