9
$\begingroup$

I have been using the HPCE on our campus for MD simulations using GROMACS. However, when I use the following script, after the job starts to run on the cluster, gets stuck at Step 0 in the log file.

#!/bin/bash
#PBS -e errorfile.err
#PBS -o logfile.log
#PBS -l walltime=24:00:00
#PBS -l select=4:ncpus=6
cd $PBS_O_WORKDIR
module load intel2020 gromacs2020.1
gmx mdrun -s npt.tpr -deffnm npt -v     

The same would run smoothly if I submit the job with high number of processors like 40 or 80. However, there is generally a long queue for such high usage. What could be the reason for the job to not proceed beyond step 0 even after hours for lower number of processors?

Example output files when I had terminated the system after 12min. The .err file:

    GROMACS:      gmx mdrun, version 2020.1
    Executable:   /lfs/sware/gromacs2020.1/bin/gmx
    Data prefix:  /lfs/sware/gromacs2020.1
    Working dir:  /lfs/usrhome/phd/ch18d408/poly/long/350/1/try
    Command line:
      gmx mdrun -s npt.tpr -deffnm npt -v
    
    Reading file npt.tpr, VERSION 2020.1 (single precision)
    Changing nstlist from 10 to 100, rlist from 1 to 1
    
    Using 6 MPI threads
    
    Non-default thread affinity set, disabling internal thread affinity
    
    Using 6 OpenMP threads per tMPI thread
    
    starting mdrun 'polymer'
    500000 steps,   1000.0 ps.
    
    step 0
    
    Received the TERM signal, stopping within 100 steps

And the .log file:

    GROMACS:      gmx mdrun, version 2020.1
Executable:   /lfs/sware/gromacs2020.1/bin/gmx
Data prefix:  /lfs/sware/gromacs2020.1
Working dir:  /lfs/usrhome/phd/ch18d408/poly/long/350/1/try
Process ID:   15343
Command line:
  gmx mdrun -s npt.tpr -deffnm npt -v

GROMACS version:    2020.1
Verified release checksum is 5cde61b9d46b24153ba84f499c996612640b965eff9a218f8f5e561f94ff4e43
Precision:          single
Memory model:       64 bit
MPI library:        thread_mpi
OpenMP support:     enabled (GMX_OPENMP_MAX_THREADS = 64)
GPU support:        CUDA
SIMD instructions:  AVX_512
FFT library:        Intel MKL
RDTSCP usage:       enabled
TNG support:        enabled
Hwloc support:      disabled
Tracing support:    disabled
C compiler:         /lfs/sware/intel2019/compilers_and_libraries_2019.0.117/linux/mpi/intel64/bin/mpiicc Intel 19.0.0.20180804
C compiler flags:   -xCORE-AVX512 -qopt-zmm-usage=high -mkl=sequential -std=gnu99 -ip -funroll-all-loops -alias-const -ansi-alias -no-prec-div -fimf-domain-exclusion=14 -qoverride-limits
C++ compiler:       /lfs/sware/intel2019/compilers_and_libraries_2019.0.117/linux/mpi/intel64/bin/mpiicpc Intel 19.0.0.20180804
C++ compiler flags: -xCORE-AVX512 -qopt-zmm-usage=high -mkl=sequential -ip -funroll-all-loops -alias-const -ansi-alias -no-prec-div -fimf-domain-exclusion=14 -qoverride-limits -qopenmp
CUDA compiler:      /lfs/sware/cuda-10.1/bin/nvcc nvcc: NVIDIA (R) Cuda compiler driver;Copyright (c) 2005-2019 NVIDIA Corporation;Built on Sun_Jul_28_19:07:16_PDT_2019;Cuda compilation tools, release 10.1, V10.1.243
CUDA compiler flags:-std=c++14;-gencode;arch=compute_30,code=sm_30;-gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_52,code=sm_52;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_35,code=compute_35;-gencode;arch=compute_50,code=compute_50;-gencode;arch=compute_52,code=compute_52;-gencode;arch=compute_60,code=compute_60;-gencode;arch=compute_61,code=compute_61;-gencode;arch=compute_70,code=compute_70;-gencode;arch=compute_75,code=compute_75;-use_fast_math;;-xCORE-AVX512 -qopt-zmm-usage=high -mkl=sequential -ip -funroll-all-loops -alias-const -ansi-alias -no-prec-div -fimf-domain-exclusion=14 -qoverride-limits -qopenmp
CUDA driver:        0.0
CUDA runtime:       N/A


Running on 1 node with total 40 cores, 40 logical cores (GPU detection deactivated)
Hardware detected:
  CPU info:
    Vendor: Intel
    Brand:  Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
    Family: 6   Model: 85   Stepping: 7
    Features: aes apic avx avx2 avx512f avx512cd avx512bw avx512vl clfsh cmov cx8 cx16 f16c fma hle htt intel lahf mmx msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd rdtscp rtm sse2 sse3 sse4.1 sse4.2 ssse3 tdt x2apic
    Number of AVX-512 FMA units: 2
  Hardware topology: Only logical processor count


++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++
M. J. Abraham, T. Murtola, R. Schulz, S. Páll, J. C. Smith, B. Hess, E.
Lindahl
GROMACS: High performance molecular simulations through multi-level
parallelism from laptops to supercomputers
SoftwareX 1 (2015) pp. 19-25
-------- -------- --- Thank You --- -------- --------


++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++
S. Páll, M. J. Abraham, C. Kutzner, B. Hess, E. Lindahl
Tackling Exascale Software Challenges in Molecular Dynamics Simulations with
GROMACS
In S. Markidis & E. Laure (Eds.), Solving Software Challenges for Exascale 8759 (2015) pp. 3-27
-------- -------- --- Thank You --- -------- --------


++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++
S. Pronk, S. Páll, R. Schulz, P. Larsson, P. Bjelkmar, R. Apostolov, M. R.
Shirts, J. C. Smith, P. M. Kasson, D. van der Spoel, B. Hess, and E. Lindahl
GROMACS 4.5: a high-throughput and highly parallel open source molecular
simulation toolkit
Bioinformatics 29 (2013) pp. 845-54
-------- -------- --- Thank You --- -------- --------


++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++
B. Hess and C. Kutzner and D. van der Spoel and E. Lindahl
GROMACS 4: Algorithms for highly efficient, load-balanced, and scalable
molecular simulation
J. Chem. Theory Comput. 4 (2008) pp. 435-447
-------- -------- --- Thank You --- -------- --------


++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++
D. van der Spoel, E. Lindahl, B. Hess, G. Groenhof, A. E. Mark and H. J. C.
Berendsen
GROMACS: Fast, Flexible and Free
J. Comp. Chem. 26 (2005) pp. 1701-1719
-------- -------- --- Thank You --- -------- --------


++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++
E. Lindahl and B. Hess and D. van der Spoel
GROMACS 3.0: A package for molecular simulation and trajectory analysis
J. Mol. Mod. 7 (2001) pp. 306-317
-------- -------- --- Thank You --- -------- --------


++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++
H. J. C. Berendsen, D. van der Spoel and R. van Drunen
GROMACS: A message-passing parallel molecular dynamics implementation
Comp. Phys. Comm. 91 (1995) pp. 43-56
-------- -------- --- Thank You --- -------- --------


++++ PLEASE CITE THE DOI FOR THIS VERSION OF GROMACS ++++
https://doi.org/10.5281/zenodo.3685919
-------- -------- --- Thank You --- -------- --------


The number of OpenMP threads was set by environment variable OMP_NUM_THREADS to 6

Input Parameters:
   integrator                     = md
   tinit                          = 0
   dt                             = 0.002
   nsteps                         = 500000
   init-step                      = 0
   simulation-part                = 1
   comm-mode                      = Linear
   nstcomm                        = 100
   bd-fric                        = 0
   ld-seed                        = -464515634
   emtol                          = 10
   emstep                         = 0.01
   niter                          = 20
   fcstep                         = 0
   nstcgsteep                     = 1000
   nbfgscorr                      = 10
   rtpi                           = 0.05
   nstxout                        = 0
   nstvout                        = 0
   nstfout                        = 0
   nstlog                         = 500
   nstcalcenergy                  = 100
   nstenergy                      = 500
   nstxout-compressed             = 500
   compressed-x-precision         = 1000
   cutoff-scheme                  = Verlet
   nstlist                        = 10
   pbc                            = xyz
   periodic-molecules             = false
   verlet-buffer-tolerance        = 0.005
   rlist                          = 1
   coulombtype                    = PME
   coulomb-modifier               = Potential-shift
   rcoulomb-switch                = 0
   rcoulomb                       = 1
   epsilon-r                      = 1
   epsilon-rf                     = inf
   vdw-type                       = Cut-off
   vdw-modifier                   = Potential-shift
   rvdw-switch                    = 0
   rvdw                           = 1
   DispCorr                       = EnerPres
   table-extension                = 1
   fourierspacing                 = 0.16
   fourier-nx                     = 128
   fourier-ny                     = 128
   fourier-nz                     = 128
   pme-order                      = 4
   ewald-rtol                     = 1e-05
   ewald-rtol-lj                  = 0.001
   lj-pme-comb-rule               = Geometric
   ewald-geometry                 = 0
   epsilon-surface                = 0
   tcoupl                         = V-rescale
   nsttcouple                     = 10
   nh-chain-length                = 0
   print-nose-hoover-chain-variables = false
   pcoupl                         = Berendsen
   pcoupltype                     = Isotropic
   nstpcouple                     = 10
   tau-p                          = 2
   compressibility (3x3):
      compressibility[    0]={ 4.50000e-05,  0.00000e+00,  0.00000e+00}
      compressibility[    1]={ 0.00000e+00,  4.50000e-05,  0.00000e+00}
      compressibility[    2]={ 0.00000e+00,  0.00000e+00,  4.50000e-05}
   ref-p (3x3):
      ref-p[    0]={ 3.50000e+02,  0.00000e+00,  0.00000e+00}
      ref-p[    1]={ 0.00000e+00,  3.50000e+02,  0.00000e+00}
      ref-p[    2]={ 0.00000e+00,  0.00000e+00,  3.50000e+02}
   refcoord-scaling               = COM
   posres-com (3):
      posres-com[0]=-4.31721e-03
      posres-com[1]= 2.26425e-01
      posres-com[2]= 9.22131e-01
   posres-comB (3):
      posres-comB[0]=-4.31721e-03
      posres-comB[1]= 2.26425e-01
      posres-comB[2]= 9.22131e-01
   QMMM                           = false
   QMconstraints                  = 0
   QMMMscheme                     = 0
   MMChargeScaleFactor            = 1
qm-opts:
   ngQM                           = 0
   constraint-algorithm           = Lincs
   continuation                   = false
   Shake-SOR                      = false
   shake-tol                      = 0.0001
   lincs-order                    = 4
   lincs-iter                     = 1
   lincs-warnangle                = 30
   nwall                          = 0
   wall-type                      = 9-3
   wall-r-linpot                  = -1
   wall-atomtype[0]               = -1
   wall-atomtype[1]               = -1
   wall-density[0]                = 0
   wall-density[1]                = 0
   wall-ewald-zfac                = 3
   pull                           = false
   awh                            = false
   rotation                       = false
   interactiveMD                  = false
   disre                          = No
   disre-weighting                = Conservative
   disre-mixed                    = false
   dr-fc                          = 1000
   dr-tau                         = 0
   nstdisreout                    = 100
   orire-fc                       = 0
   orire-tau                      = 0
   nstorireout                    = 100
   free-energy                    = no
   cos-acceleration               = 0
   deform (3x3):
      deform[    0]={ 0.00000e+00,  0.00000e+00,  0.00000e+00}
      deform[    1]={ 0.00000e+00,  0.00000e+00,  0.00000e+00}
      deform[    2]={ 0.00000e+00,  0.00000e+00,  0.00000e+00}
   simulated-tempering            = false
   swapcoords                     = no
   userint1                       = 0
   userint2                       = 0
   userint3                       = 0
   userint4                       = 0
   userreal1                      = 0
   userreal2                      = 0
   userreal3                      = 0
   userreal4                      = 0
   applied-forces:
     electric-field:
       x:
         E0                       = 0
         omega                    = 0
         t0                       = 0
         sigma                    = 0
       y:
         E0                       = 0
         omega                    = 0
         t0                       = 0
         sigma                    = 0
       z:
         E0                       = 0
         omega                    = 0
         t0                       = 0
         sigma                    = 0
     density-guided-simulation:
       active                     = false
       group                      = protein
       similarity-measure         = inner-product
       atom-spreading-weight      = unity
       force-constant             = 1e+09
       gaussian-transform-spreading-width = 0.2
       gaussian-transform-spreading-range-in-multiples-of-width = 4
       reference-density-filename = reference.mrc
       nst                        = 1
       normalize-densities        = true
       adaptive-force-scaling     = false
       adaptive-force-scaling-time-constant = 4
grpopts:
   nrdf:       31229
   ref-t:         298
   tau-t:         0.5
annealing:          No
annealing-npoints:           0
   acc:            0           0           0
   nfreeze:           N           N           N
   energygrp-flags[  0]: 0

Changing nstlist from 10 to 100, rlist from 1 to 1


Initializing Domain Decomposition on 6 ranks
Dynamic load balancing: locked
Minimum cell size due to atom displacement: 0.703 nm
Initial maximum distances in bonded interactions:
    two-body bonded interactions: 0.403 nm, LJ-14, atoms 30848 30853
  multi-body bonded interactions: 0.403 nm, Ryckaert-Bell., atoms 30853 30848
Minimum cell size due to bonded interactions: 0.443 nm
Maximum distance for 5 constraints, at 120 deg. angles, all-trans: 0.764 nm
Estimated maximum distance required for P-LINCS: 0.764 nm
This distance will limit the DD cell size, you can override this with -rcon
Scaling the initial minimum size with 1/0.8 (option -dds) = 1.25
Using 0 separate PME ranks, as there are too few total
 ranks for efficient splitting
Optimizing the DD grid for 6 cells with a minimum initial size of 0.956 nm
The maximum allowed number of cells is: X 20 Y 20 Z 20
Domain decomposition grid 6 x 1 x 1, separate PME ranks 0
PME domain decomposition: 6 x 1 x 1
Domain decomposition rank 0, coordinates 0 0 0

The initial number of communication pulses is: X 1
The initial domain decomposition cell size is: X 3.33 nm

The maximum allowed distance for atoms involved in interactions is:
                 non-bonded interactions           1.000 nm
(the following are initial values, they could change due to box deformation)
            two-body bonded interactions  (-rdd)   1.000 nm
          multi-body bonded interactions  (-rdd)   1.000 nm
              virtual site constructions  (-rcon)  3.333 nm
  atoms separated by up to 5 constraints  (-rcon)  3.333 nm

When dynamic load balancing gets turned on, these settings will change to:
The maximum number of communication pulses is: X 1
The minimum size for domain decomposition cells is 1.000 nm
The requested allowed shrink of DD cells (option -dds) is: 0.80
The allowed shrink of domain decomposition cells is: X 0.30
The maximum allowed distance for atoms involved in interactions is:
                 non-bonded interactions           1.000 nm
            two-body bonded interactions  (-rdd)   1.000 nm
          multi-body bonded interactions  (-rdd)   1.000 nm
              virtual site constructions  (-rcon)  1.000 nm
  atoms separated by up to 5 constraints  (-rcon)  1.000 nm

Using 6 MPI threads

Non-default thread affinity set, disabling internal thread affinity

Using 6 OpenMP threads per tMPI thread

System total charge: -0.000
Will do PME sum in reciprocal space for electrostatic interactions.

++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++
U. Essmann, L. Perera, M. L. Berkowitz, T. Darden, H. Lee and L. G. Pedersen 
A smooth particle mesh Ewald method
J. Chem. Phys. 103 (1995) pp. 8577-8592
-------- -------- --- Thank You --- -------- --------

Using a Gaussian width (1/beta) of 0.320163 nm for Ewald
Potential shift: LJ r^-12: -1.000e+00 r^-6: -1.000e+00, Ewald -1.000e-05
Initialized non-bonded Coulomb Ewald tables, spacing: 9.33e-04 size: 1073

Generated table with 1000 data points for 1-4 COUL.
Tabscale = 500 points/nm
Generated table with 1000 data points for 1-4 LJ6.
Tabscale = 500 points/nm
Generated table with 1000 data points for 1-4 LJ12.
Tabscale = 500 points/nm

Using SIMD 4x8 nonbonded short-range kernels

Using a 4x8 pair-list setup:
  updated every 100 steps, buffer 0.000 nm, rlist 1.000 nm
At tolerance 0.005 kJ/mol/ps per atom, equivalent classical 1x1 list would be:
  updated every 100 steps, buffer 0.000 nm, rlist 1.000 nm

Using Lorentz-Berthelot Lennard-Jones combination rule

Long Range LJ corr.: <C6> 5.0152e-04

Removing pbc first time

Initializing Parallel LINear Constraint Solver

++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++
B. Hess
P-LINCS: A Parallel Linear Constraint Solver for molecular simulation
J. Chem. Theory Comput. 4 (2008) pp. 116-122
-------- -------- --- Thank You --- -------- --------

The number of constraints is 6385
There are constraints between atoms in different decomposition domains,
will communicate selected coordinates each lincs iteration

Linking all bonded interactions to atoms
There are 18465 inter update-group virtual sites,
will an extra communication step for selected coordinates and forces

Intra-simulation communication will occur every 10 steps.

++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++
G. Bussi, D. Donadio and M. Parrinello
Canonical sampling through velocity rescaling
J. Chem. Phys. 126 (2007) pp. 014101
-------- -------- --- Thank You --- -------- --------

There are: 12539 Atoms
There are: 18465 VSites
Atom distribution over 6 domains: av 5167 stddev 199 min 4989 max 5507

Constraining the starting coordinates (step 0)

Constraining the coordinates at t0-dt (step 0)
Center of mass motion removal mode is Linear
We have the following groups for center of mass motion removal:
  0:  rest
RMS relative constraint deviation after constraining: 2.31e-05
Initial temperature: 300.037 K

Started mdrun on rank 0 Fri Oct  2 15:05:27 2020

           Step           Time
              0        0.00000

   Energies (kJ/mol)
          Angle Ryckaert-Bell.  Improper Dih.          LJ-14     Coulomb-14
    4.45652e+02   -4.13382e+02    2.39002e+00    5.76587e+02    3.22453e+02
        LJ (SR)  Disper. corr.   Coulomb (SR)   Coul. recip. Position Rest.

You can also download the .err and .log files here. The job would just get killed after the time allocated if I do not terminate.

$\endgroup$
  • 2
    $\begingroup$ Difficult to tell but in some cases GROMACS can silently segfault without terminating, so it's just left hanging. I have had problems with dynamic load balancing doing that so maybe try turning it off and see how it goes. $\endgroup$ – Godzilla Oct 1 at 11:38
  • $\begingroup$ can you post the pertinent part of the log file as well as any other output the cluster/gromacs gives you about the error? I use SLURM to run gromacs, usually it will also have something to say about an error. $\endgroup$ – Charlie Crown Oct 2 at 0:46
  • $\begingroup$ @Godzilla Thankyou. I tried with switching off the dlb and it worked only for one case. In the others the job atleast proceeded beyond 0th step but is way too slower than how it runs on a single node. $\endgroup$ – Kavya Mrudula Oct 2 at 9:48
  • $\begingroup$ @CharlieCrown I have added the log file and the .err file to the question. But I do not know what I am missing out in those files that could correct this problem. $\endgroup$ – Kavya Mrudula Oct 2 at 9:50
  • $\begingroup$ @KavyaMrudula Next time if the output files are no more than a few hundred, lines, please insert them into the question. We don't want to have to download them from an unfamiliar upload site. If the output files are longer than a few hundred lines, please upload them here: github.com/HPQC-LABS/Modeling_Matters in a folder titled according to the post number (you can see from this question's URL that it is post #2398). $\endgroup$ – Nike Dattani Oct 18 at 19:25
4
$\begingroup$

Turns out that it works with a simple addition of -ntomp and -np flags in the last line. Though I still do not know the answer to why it wouldn't run if not for them, when it can work for the higher number of processors like 40 or 80.

The edited script:

#!/bin/bash
#PBS -e errorfile.err
#PBS -o logfile.log
#PBS -l walltime=24:00:00
#PBS -l select=4:ncpus=6
cd $PBS_O_WORKDIR
module load intel2020 gromacs2020.1
gmx mdrun -ntomp 6 -ntmpi 4 -s npt.tpr -deffnm npt -v 
| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Did you try with "-ntomp 6 -ntmpi 6" to replicate the scenario in the log file above? It's possible this PBS script only reserves some fraction of the memory on the node, and that increasing the number of MPI threads too far exceeds that reservation. $\endgroup$ – Anyon Nov 13 at 15:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.