4
$\begingroup$

I have recently installed SIESTA v.4.1.5 in ubuntu 22.04 LTS without any error. But, it's taking a very large time for the same SCF calculations done in past.

Does this issue occur due to versions of libraries or compilers?

Please suggest some solution. I have attached arch.make file here.

# 
# Copyright (C) 1996-2016   The SIESTA group
#  This file is distributed under the terms of the
#  GNU General Public License: see COPYING in the top directory
#  or http://www.gnu.org/copyleft/gpl.txt.
# See Docs/Contributors.txt for a list of contributors.
#
.SUFFIXES:
.SUFFIXES: .f .F .o .a .f90 .F90

SIESTA_ARCH=x86_64-unknown-linux-gnu--Gfortran

FPP=
FPP_OUTPUT= 
FC=mpif90
CC=mpicc
RANLIB=echo

SYS=nag

SP_KIND=4
DP_KIND=8
KINDS=$(SP_KIND) $(DP_KIND)

# Add any other sensible compilation flags here
FFLAGS=-g -O2 -fallow-argument-mismatch

FPPFLAGS= -DMPI -DFC_HAVE_FLUSH -DFC_HAVE_ABORT -DCDF -DGRID_DP -DPHI_GRID_SP
LDFLAGS= -L/usr/lib/x86_64-linux-gnu/hdf5/openmpi -L//usr/lib/x86_64-linux-gnu/  

ARFLAGS_EXTRA=

FCFLAGS_fixed_f=
FCFLAGS_free_f90=
FPPFLAGS_fixed_F=
FPPFLAGS_free_F90=

INCFLAGS=  -I/usr/include -I.
BLAS_LIBS= -lblas
LAPACK_LIBS= -llapack
BLACS_LIBS=
SCALAPACK_LIBS=/usr/lib/x86_64-linux-gnu/libscalapack-openmpi.so



NETCDF_LIBS=-lnetcdff -lnetcdf
NETCDF_INTERFACE=libnetcdf_f90.a

COMP_LIBS += libncdf.a libfdict.a 
FPPFLAGS += -DNCDF -DNCDF_4 


#FPPFLAGS += -DSIESTA__ELPA -I/usr/local/include/elpa-2017.11.001/modules -I/usr/local/include/elpa-2017.11.001/elpa

LIBS= $(NETCDF_LIBS) $(SCALAPACK_LIBS) $(BLACS_LIBS) $(LAPACK_LIBS) $(BLAS_LIBS) -lhdf5_fortran -lhdf5 -lz
#LIBS += -L/usr/local/lib -lelpa
#SIESTA needs an F90 interface to MPI
#This will give you SIESTA's own implementation
#If your compiler vendor offers an alternative, you may change
#to it here.
MPI_INTERFACE=libmpi_f90.a
MPI_INCLUDE=.

#Dependency rules are created by autoconf according to whether
#discrete preprocessing is necessary or not.
.F.o:
    $(FC) -c $(FFLAGS) $(INCFLAGS) $(FPPFLAGS) $(FPPFLAGS_fixed_F)  $< 
.F90.o:
    $(FC) -c $(FFLAGS) $(INCFLAGS) $(FPPFLAGS) $(FPPFLAGS_free_F90) $< 
.f.o:
    $(FC) -c $(FFLAGS) $(INCFLAGS) $(FCFLAGS_fixed_f)  $<
.f90.o:
    $(FC) -c $(FFLAGS) $(INCFLAGS) $(FCFLAGS_free_f90)  $<
$\endgroup$
9
  • $\begingroup$ In the past, in the same computer with the same arch.make file? $\endgroup$
    – Camps
    Jul 2, 2022 at 15:10
  • $\begingroup$ yes, with the same computer and the same arch.make, $\endgroup$
    – sushil
    Jul 2, 2022 at 15:15
  • $\begingroup$ All the rest: library version, compiler version, MPI manager and version? $\endgroup$
    – Camps
    Jul 2, 2022 at 15:21
  • $\begingroup$ yes, those are changed due to new versions in ubuntu 22.04 compare to ubuntu 21.10 $\endgroup$
    – sushil
    Jul 2, 2022 at 15:23
  • 2
    $\begingroup$ Is the number of processes you are using equal to the number of cores in your machine? Are your BLAS and/or LAPACK threaded? What happens if you export OMP_NUM_THREADS=1 before you start SIESTA running? $\endgroup$
    – Ian Bush
    Jul 2, 2022 at 16:55

1 Answer 1

1
$\begingroup$

Turning my comment into an answer...

From what you have said it looks as though you are fully populating the cores on your machine with MPI processes, generally a good idea provided the problem you are trying to solve is big enough to exploit the parallelism so available. Now this is should be a good idea. Unfortunately many BLAS and LAPACK implementations assume that if you do not set the environment variable OMP_NUM_THREADS to a value then whatever operation is being done by that library should set the number of threads equal to the number of cores available in the machine. For an MPI code this is extremely irritating, as it can lead to massive overpopulation of the machine.

For instance let us say you have a quad-core machine, such as the laptop I am now on, and so logically start a job using 4 MPI processes. The problem is that when your job enter a BLAS or LAPACK routine each of those 4 processes will spawn 4 threads, so your code is now trying to run 4x4=16 threads of execution despite you only have 4 cores available! Nett result, slooooooooooow running of the code as you have observed. To avoid this use

export OMP_NUM_THREADS=1

to limit the number of threads per MPI process to 1, and so avoiding the overpopulation described above.

Just to take the argument a little further another sensible set up would be to run 2 MPI processes and set OMP_NUM_THREADS to 2, thus giving you 2 processes each with 2 threads, and so running on the 4 cores available.

Most BLAS and LAPACK implementations nowadays are threaded, and this is undoubtedly a good thing in general. However most (all?) implementations have the default behaviour of assuming that each call should grab all resources available if OMP_NUM_THREADS is not set - something that annoys me no end, it should just use 1 thread in such circumstances IMO. Thus I recommend you always set OMP_NUM_THREADS before you run a program that uses BLAS or LAPACK, and if it is purely MPI parallelism you require you should set it to 1.

$\endgroup$
3
  • $\begingroup$ Does this issue occur due to the new versions of LAPACK and BLAS? Because earlier, I had not encountered such issue. $\endgroup$
    – sushil
    Jul 4, 2022 at 15:23
  • $\begingroup$ That would be guess, yes $\endgroup$
    – Ian Bush
    Jul 4, 2022 at 15:48
  • $\begingroup$ Once again thank you @IanBush $\endgroup$
    – sushil
    Jul 7, 2022 at 7:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .