:-) GROMACS - gmx mdrun, 2021.3 (-: GROMACS is written by: Andrey Alekseenko Emile Apol Rossen Apostolov Paul Bauer Herman J.C. Berendsen Par Bjelkmar Christian Blau Viacheslav Bolnykh Kevin Boyd Aldert van Buuren Rudi van Drunen Anton Feenstra Gilles Gouaillardet Alan Gray Gerrit Groenhof Anca Hamuraru Vincent Hindriksen M. Eric Irrgang Aleksei Iupinov Christoph Junghans Joe Jordan Dimitrios Karkoulis Peter Kasson Jiri Kraus Carsten Kutzner Per Larsson Justin A. Lemkul Viveca Lindahl Magnus Lundborg Erik Marklund Pascal Merz Pieter Meulenhoff Teemu Murtola Szilard Pall Sander Pronk Roland Schulz Michael Shirts Alexey Shvetsov Alfons Sijbers Peter Tieleman Jon Vincent Teemu Virolainen Christian Wennberg Maarten Wolf Artem Zhmurov and the project leaders: Mark Abraham, Berk Hess, Erik Lindahl, and David van der Spoel Copyright (c) 1991-2000, University of Groningen, The Netherlands. Copyright (c) 2001-2019, The GROMACS development team at Uppsala University, Stockholm University and the Royal Institute of Technology, Sweden. check out http://www.gromacs.org for more information. GROMACS is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version. GROMACS: gmx mdrun, version 2021.3 Executable: /veracruz/projects/t/training/gromacs-2021.3/bin/gmx Data prefix: /veracruz/projects/t/training/gromacs-2021.3 Working dir: /veracruz/home/m/mabraham/git/gromacs-gpu-performance-master/content/exercises/performance-with-bonded Process ID: 105842 Command line: gmx mdrun -ntmpi 1 -noconfout -g default GROMACS version: 2021.3 Verified release checksum is c5bf577cc74de0e05106b7b6426476abb7f6530be7b4a2c64f637d6a6eca8fcb Precision: mixed Memory model: 64 bit MPI library: thread_mpi OpenMP support: enabled (GMX_OPENMP_MAX_THREADS = 64) GPU support: CUDA SIMD instructions: AVX_512 FFT library: fftw-3.3.9-sse2-avx RDTSCP usage: enabled TNG support: enabled Hwloc support: disabled Tracing support: disabled C compiler: /apps/software/GCCcore/9.3.0/bin/cc GNU 9.3.0 C compiler flags: -mavx512f -mfma -Wno-missing-field-initializers -fexcess-precision=fast -funroll-all-loops -pthread -O3 -DNDEBUG C++ compiler: /apps/software/GCCcore/9.3.0/bin/c++ GNU 9.3.0 C++ compiler flags: -mavx512f -mfma -Wno-missing-field-initializers -fexcess-precision=fast -funroll-all-loops -pthread -fopenmp -O3 -DNDEBUG CUDA compiler: /apps/software/CUDAcore/11.0.2/bin/nvcc nvcc: NVIDIA (R) Cuda compiler driver;Copyright (c) 2005-2020 NVIDIA Corporation;Built on Thu_Jun_11_22:26:38_PDT_2020;Cuda compilation tools, release 11.0, V11.0.194;Build cuda_11.0_bu.TC445_37.28540450_0 CUDA compiler flags:-std=c++17;-gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_52,code=sm_52;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-Wno-deprecated-gpu-targets;-gencode;arch=compute_35,code=compute_35;-gencode;arch=compute_53,code=compute_53;-gencode;arch=compute_80,code=compute_80;-use_fast_math;;-mavx512f -mfma -Wno-missing-field-initializers -fexcess-precision=fast -funroll-all-loops -pthread -fopenmp -O3 -DNDEBUG CUDA driver: 11.10 CUDA runtime: 11.0 Running on 1 node with total 80 cores, 80 logical cores, 1 compatible GPU Hardware detected: CPU info: Vendor: Intel Brand: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz Family: 6 Model: 85 Stepping: 4 Features: aes apic avx avx2 avx512f avx512cd avx512bw avx512vl avx512secondFMA clfsh cmov cx8 cx16 f16c fma hle htt intel lahf mmx msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd rdtscp rtm sse2 sse3 sse4.1 sse4.2 ssse3 tdt x2apic Number of AVX-512 FMA units: 2 Hardware topology: Only logical processor count GPU info: Number of GPUs detected: 1 #0: NVIDIA Tesla V100-PCIE-16GB, compute cap.: 7.0, ECC: yes, stat: compatible ++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++ M. J. Abraham, T. Murtola, R. Schulz, S. Páll, J. C. Smith, B. Hess, E. Lindahl GROMACS: High performance molecular simulations through multi-level parallelism from laptops to supercomputers SoftwareX 1 (2015) pp. 19-25 -------- -------- --- Thank You --- -------- -------- ++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++ S. Páll, M. J. Abraham, C. Kutzner, B. Hess, E. Lindahl Tackling Exascale Software Challenges in Molecular Dynamics Simulations with GROMACS In S. Markidis & E. Laure (Eds.), Solving Software Challenges for Exascale 8759 (2015) pp. 3-27 -------- -------- --- Thank You --- -------- -------- ++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++ S. Pronk, S. Páll, R. Schulz, P. Larsson, P. Bjelkmar, R. Apostolov, M. R. Shirts, J. C. Smith, P. M. Kasson, D. van der Spoel, B. Hess, and E. Lindahl GROMACS 4.5: a high-throughput and highly parallel open source molecular simulation toolkit Bioinformatics 29 (2013) pp. 845-54 -------- -------- --- Thank You --- -------- -------- ++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++ B. Hess and C. Kutzner and D. van der Spoel and E. Lindahl GROMACS 4: Algorithms for highly efficient, load-balanced, and scalable molecular simulation J. Chem. Theory Comput. 4 (2008) pp. 435-447 -------- -------- --- Thank You --- -------- -------- ++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++ D. van der Spoel, E. Lindahl, B. Hess, G. Groenhof, A. E. Mark and H. J. C. Berendsen GROMACS: Fast, Flexible and Free J. Comp. Chem. 26 (2005) pp. 1701-1719 -------- -------- --- Thank You --- -------- -------- ++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++ E. Lindahl and B. Hess and D. van der Spoel GROMACS 3.0: A package for molecular simulation and trajectory analysis J. Mol. Mod. 7 (2001) pp. 306-317 -------- -------- --- Thank You --- -------- -------- ++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++ H. J. C. Berendsen, D. van der Spoel and R. van Drunen GROMACS: A message-passing parallel molecular dynamics implementation Comp. Phys. Comm. 91 (1995) pp. 43-56 -------- -------- --- Thank You --- -------- -------- ++++ PLEASE CITE THE DOI FOR THIS VERSION OF GROMACS ++++ https://doi.org/10.5281/zenodo.5053201 -------- -------- --- Thank You --- -------- -------- The number of OpenMP threads was set by environment variable OMP_NUM_THREADS to 20 Input Parameters: integrator = md tinit = 0 dt = 0.002 nsteps = 50000 init-step = 0 simulation-part = 1 mts = false comm-mode = Linear nstcomm = 100 bd-fric = 0 ld-seed = -1994413355 emtol = 10 emstep = 0.01 niter = 20 fcstep = 0 nstcgsteep = 1000 nbfgscorr = 10 rtpi = 0.05 nstxout = 0 nstvout = 0 nstfout = 0 nstlog = 0 nstcalcenergy = 100 nstenergy = 500 nstxout-compressed = 0 compressed-x-precision = 1000 cutoff-scheme = Verlet nstlist = 10 pbc = xyz periodic-molecules = false verlet-buffer-tolerance = 0.005 rlist = 0.967 coulombtype = Reaction-Field coulomb-modifier = Potential-shift rcoulomb-switch = 0 rcoulomb = 0.9 epsilon-r = 1 epsilon-rf = inf vdw-type = Cut-off vdw-modifier = Potential-shift rvdw-switch = 0 rvdw = 0.9 DispCorr = No table-extension = 1 fourierspacing = 0.12 fourier-nx = 0 fourier-ny = 0 fourier-nz = 0 pme-order = 4 ewald-rtol = 1e-05 ewald-rtol-lj = 0.001 lj-pme-comb-rule = Geometric ewald-geometry = 0 epsilon-surface = 0 tcoupl = V-rescale nsttcouple = 10 nh-chain-length = 0 print-nose-hoover-chain-variables = false pcoupl = No pcoupltype = Isotropic nstpcouple = -1 tau-p = 1 compressibility (3x3): compressibility[ 0]={ 0.00000e+00, 0.00000e+00, 0.00000e+00} compressibility[ 1]={ 0.00000e+00, 0.00000e+00, 0.00000e+00} compressibility[ 2]={ 0.00000e+00, 0.00000e+00, 0.00000e+00} ref-p (3x3): ref-p[ 0]={ 0.00000e+00, 0.00000e+00, 0.00000e+00} ref-p[ 1]={ 0.00000e+00, 0.00000e+00, 0.00000e+00} ref-p[ 2]={ 0.00000e+00, 0.00000e+00, 0.00000e+00} refcoord-scaling = No posres-com (3): posres-com[0]= 0.00000e+00 posres-com[1]= 0.00000e+00 posres-com[2]= 0.00000e+00 posres-comB (3): posres-comB[0]= 0.00000e+00 posres-comB[1]= 0.00000e+00 posres-comB[2]= 0.00000e+00 QMMM = false qm-opts: ngQM = 0 constraint-algorithm = Lincs continuation = false Shake-SOR = false shake-tol = 0.0001 lincs-order = 4 lincs-iter = 1 lincs-warnangle = 30 nwall = 0 wall-type = 9-3 wall-r-linpot = -1 wall-atomtype[0] = -1 wall-atomtype[1] = -1 wall-density[0] = 0 wall-density[1] = 0 wall-ewald-zfac = 3 pull = false awh = false rotation = false interactiveMD = false disre = No disre-weighting = Conservative disre-mixed = false dr-fc = 1000 dr-tau = 0 nstdisreout = 100 orire-fc = 0 orire-tau = 0 nstorireout = 100 free-energy = no cos-acceleration = 0 deform (3x3): deform[ 0]={ 0.00000e+00, 0.00000e+00, 0.00000e+00} deform[ 1]={ 0.00000e+00, 0.00000e+00, 0.00000e+00} deform[ 2]={ 0.00000e+00, 0.00000e+00, 0.00000e+00} simulated-tempering = false swapcoords = no userint1 = 0 userint2 = 0 userint3 = 0 userint4 = 0 userreal1 = 0 userreal2 = 0 userreal3 = 0 userreal4 = 0 applied-forces: electric-field: x: E0 = 0 omega = 0 t0 = 0 sigma = 0 y: E0 = 0 omega = 0 t0 = 0 sigma = 0 z: E0 = 0 omega = 0 t0 = 0 sigma = 0 density-guided-simulation: active = false group = protein similarity-measure = inner-product atom-spreading-weight = unity force-constant = 1e+09 gaussian-transform-spreading-width = 0.2 gaussian-transform-spreading-range-in-multiples-of-width = 4 reference-density-filename = reference.mrc nst = 1 normalize-densities = true adaptive-force-scaling = false adaptive-force-scaling-time-constant = 4 grpopts: nrdf: 49110 ref-t: 300 tau-t: 0.1 annealing: No annealing-npoints: 0 acc: 0 0 0 nfreeze: N N N energygrp-flags[ 0]: 0 Changing nstlist from 10 to 25, rlist from 0.967 to 1.12 1 GPU selected for this run. Mapping of GPU IDs to the 1 GPU task in the 1 rank on this node: PP:0 PP tasks will do (non-perturbed) short-ranged interactions on the GPU PP task will update and constrain coordinates on the CPU Using 1 MPI thread Non-default thread affinity set, disabling internal thread affinity Using 20 OpenMP threads System total charge: 0.000 Reaction-Field: epsRF = 0, rc = 0.9, krf = 0.685871, crf = 1.66667, epsfac = 138.935 The electrostatics potential has its minimum at r = 0.9 Potential shift: LJ r^-12: -3.541e+00 r^-6: -1.882e+00 Generated table with 1060 data points for 1-4 COUL. Tabscale = 500 points/nm Generated table with 1060 data points for 1-4 LJ6. Tabscale = 500 points/nm Generated table with 1060 data points for 1-4 LJ12. Tabscale = 500 points/nm Using GPU 8x8 nonbonded short-range kernels Using a dual 8x8 pair-list setup updated with dynamic, rolling pruning: outer list: updated every 25 steps, buffer 0.220 nm, rlist 1.120 nm inner list: updated every 4 steps, buffer 0.022 nm, rlist 0.922 nm At tolerance 0.005 kJ/mol/ps per atom, equivalent classical 1x1 list would be: outer list: updated every 25 steps, buffer 0.279 nm, rlist 1.179 nm inner list: updated every 4 steps, buffer 0.034 nm, rlist 0.934 nm Using Lorentz-Berthelot Lennard-Jones combination rule Removing pbc first time Initializing LINear Constraint Solver ++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++ B. Hess and H. Bekker and H. J. C. Berendsen and J. G. E. M. Fraaije LINCS: A Linear Constraint Solver for molecular simulations J. Comp. Chem. 18 (1997) pp. 1463-1472 -------- -------- --- Thank You --- -------- -------- The number of constraints is 999 ++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++ S. Miyamoto and P. A. Kollman SETTLE: An Analytical Version of the SHAKE and RATTLE Algorithms for Rigid Water Models J. Comp. Chem. 13 (1992) pp. 952-962 -------- -------- --- Thank You --- -------- -------- The -noconfout functionality is deprecated, and may be removed in a future version. ++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++ G. Bussi, D. Donadio and M. Parrinello Canonical sampling through velocity rescaling J. Chem. Phys. 126 (2007) pp. 014101 -------- -------- --- Thank You --- -------- -------- There are: 24040 Atoms Constraining the starting coordinates (step 0) Constraining the coordinates at t0-dt (step 0) Center of mass motion removal mode is Linear We have the following groups for center of mass motion removal: 0: rest RMS relative constraint deviation after constraining: 2.66e-06 Initial temperature: 291.407 K Started mdrun on rank 0 Thu Sep 9 09:04:08 2021 Step Time 0 0.00000 Energies (kJ/mol) Bond Angle Proper Dih. Improper Dih. LJ-14 2.84633e+01 4.44083e+03 5.70370e+03 2.49923e+02 2.00482e+03 Coulomb-14 LJ (SR) Coulomb (SR) Potential Kinetic En. 1.68045e+04 4.16604e+04 -3.80700e+05 -3.09808e+05 6.00249e+04 Total Energy Conserved En. Temperature Pressure (bar) Constr. rmsd -2.49783e+05 -2.49783e+05 2.94007e+02 5.62203e+02 3.08739e-06 Step Time 50000 100.00000 Energies (kJ/mol) Bond Angle Proper Dih. Improper Dih. LJ-14 1.55262e+03 4.53208e+03 5.48804e+03 2.29797e+02 1.89325e+03 Coulomb-14 LJ (SR) Coulomb (SR) Potential Kinetic En. 1.60834e+04 4.13108e+04 -3.80939e+05 -3.09849e+05 6.15606e+04 Total Energy Conserved En. Temperature Pressure (bar) Constr. rmsd -2.48289e+05 -2.46855e+05 3.01529e+02 -2.23744e+02 2.69746e-06 Energy conservation over simulation part #1 of length 100 ns, time 0 to 100 ns Conserved energy drift: 1.22e-03 kJ/mol/ps per atom <====== ############### ==> <==== A V E R A G E S ====> <== ############### ======> Statistics over 50001 steps using 501 frames Energies (kJ/mol) Bond Angle Proper Dih. Improper Dih. LJ-14 1.60102e+03 4.43185e+03 5.56728e+03 2.57945e+02 1.82698e+03 Coulomb-14 LJ (SR) Coulomb (SR) Potential Kinetic En. 1.59863e+04 4.22947e+04 -3.82794e+05 -3.10827e+05 6.12745e+04 Total Energy Conserved En. Temperature Pressure (bar) Constr. rmsd -2.49553e+05 -2.48296e+05 3.00127e+02 -2.21685e+02 0.00000e+00 Total Virial (kJ/mol) 2.20947e+04 2.14691e+01 -5.20121e+01 2.14473e+01 2.20505e+04 4.72912e+01 -5.21329e+01 4.73765e+01 2.20808e+04 Pressure (bar) -2.27047e+02 -3.09583e+00 6.38328e+00 -3.09290e+00 -2.18545e+02 -9.28741e+00 6.39950e+00 -9.29886e+00 -2.19462e+02 M E G A - F L O P S A C C O U N T I N G NB=Group-cutoff nonbonded kernels NxN=N-by-N cluster Verlet kernels RF=Reaction-Field VdW=Van der Waals QSTab=quadratic-spline table W3=SPC/TIP3p W4=TIP4p (single or pairs) V&F=Potential and force V=Potential only F=Force only Computing: M-Number M-Flops % Flops ----------------------------------------------------------------------------- Pair Search distance check 4356.253104 39206.278 0.1 NxN RF Elec. + LJ [F] 1048041.420096 39825573.964 97.8 NxN RF Elec. + LJ [V&F] 10607.420352 572800.699 1.4 1,4 nonbonded interactions 267.055341 24034.981 0.1 Shift-X 48.104040 288.624 0.0 Bonds 52.701054 3109.362 0.0 Angles 185.203704 31114.222 0.1 Propers 279.105582 63915.178 0.2 Impropers 21.100422 4388.888 0.0 Virial 12.066585 217.199 0.0 Stop-CM 12.068080 120.681 0.0 Calc-Ekin 480.848080 12982.898 0.0 Lincs 49.952997 2997.180 0.0 Lincs-Mat 253.815228 1015.261 0.0 Constraint-V 1200.348012 10803.132 0.0 Constraint-Vir 11.526507 276.636 0.0 Settle 366.822008 135724.143 0.3 ----------------------------------------------------------------------------- Total 40728569.325 100.0 ----------------------------------------------------------------------------- R E A L C Y C L E A N D T I M E A C C O U N T I N G On 1 MPI rank, each using 20 OpenMP threads Computing: Num Num Call Wall time Giga-Cycles Ranks Threads Count (s) total sum % ----------------------------------------------------------------------------- Neighbor search 1 20 2001 3.034 145.276 18.6 Launch GPU ops. 1 20 50001 1.017 48.687 6.2 Force 1 20 50001 1.763 84.421 10.8 Wait GPU NB local 1 20 50001 5.997 287.192 36.8 NB X/F buffer ops. 1 20 98001 1.229 58.845 7.5 Update 1 20 50001 0.945 45.264 5.8 Constraints 1 20 50003 1.714 82.080 10.5 Rest 0.611 29.262 3.7 ----------------------------------------------------------------------------- Total 16.310 781.026 100.0 ----------------------------------------------------------------------------- NOTE: 19 % of the run time was spent in pair search, you might want to increase nstlist (this has no effect on accuracy) Core t (s) Wall t (s) (%) Time: 326.182 16.310 1999.9 (ns/day) (hour/ns) Performance: 529.746 0.045 Finished mdrun on rank 0 Thu Sep 9 09:04:24 2021