1
$\begingroup$

I have been recently trying to optimize a particularly large system Mo2C in QE using the pw.x executable and encountering the following MPI errors:

[compute-0-0:285011] *** An error occurred in MPI_Comm_free
[compute-0-0:285011] *** reported by process [140734245568513,140642999074824]
[compute-0-0:285011] *** on communicator MPI_COMM_WORLD
[compute-0-0:285011] *** MPI_ERR_COMM: invalid communicator
[compute-0-0:285011] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[compute-0-0:285011] ***    and potentially your MPI job)
[compute-0-0.local:284985] PMIX ERROR: UNREACHABLE in file server/pmix_server.c at line 2198
[compute-0-0.local:284985] PMIX ERROR: UNREACHABLE in file server/pmix_server.c at line 2198
[compute-0-0.local:284985] PMIX ERROR: UNREACHABLE in file server/pmix_server.c at line 2198
[compute-0-0.local:284985] PMIX ERROR: UNREACHABLE in file server/pmix_server.c at line 2198
[compute-0-0.local:284985] 9 more processes have sent help message help-mpi-errors.txt / mpi_errors_are_fatal
[compute-0-0.local:284985] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

The input file for the job is:

&CONTROL
    calculation   = "relax"
    forc_conv_thr =  1.00000e-03
    nstep         = 100
    pseudo_dir    = "/home/user42/pseudopot"
    tprnfor       = .TRUE.
    tstress       = .TRUE.
/

&SYSTEM
    a                         =  1.44246e+01
    c                         =  3.26030e+01
    degauss                   =  1.00000e-02
    ecutrho                   =  7.20000e+02
    ecutwfc                   =  8.00000e+01
    ibrav                     = 4
    nat                       = 75
    nspin                     = 2
    ntyp                      = 2
    vdw_corr                  ="grimme-d2"
    occupations               = "smearing"
    smearing                  = "gaussian"
    starting_magnetization(1) =  2.00000e-01
    starting_magnetization(2) =  0.00000e+00
/

&ELECTRONS
    conv_thr         =  1.00000e-06
    electron_maxstep = 200
    mixing_beta      =  4.00000e-01
    startingpot      = "atomic"
    startingwfc      = "atomic+random"
/

&IONS
    ion_dynamics = "bfgs"
/

&CELL
/

K_POINTS {automatic}
 2  2  1  0 0 0

ATOMIC_SPECIES
Mo   95.94000  Mo.pbe-spn-kjpaw_psl.1.0.0.UPF
C    12.01070  C.pbe-n-kjpaw_psl.1.0.0.UPF

ATOMIC_POSITIONS {angstrom}
Mo   -0.000000   1.665610  17.691990
Mo    2.884921   1.665610  17.691990
Mo    5.769843   1.665610  17.691990
Mo    8.654764   1.665610  17.691990
Mo   11.539685   1.665610  17.691990
Mo   -1.442461   4.164025  17.691990
Mo    1.442461   4.164025  17.691990
Mo    4.327382   4.164025  17.691990
Mo    7.212303   4.164025  17.691990
Mo   10.097225   4.164025  17.691990
Mo   -2.884921   6.662441  17.691990
Mo   -0.000000   6.662441  17.691990
Mo    2.884921   6.662441  17.691990
Mo    5.769843   6.662441  17.691990
Mo    8.654764   6.662441  17.691990
Mo   -4.327382   9.160856  17.691990
Mo   -1.442461   9.160856  17.691990
Mo    1.442461   9.160856  17.691990
Mo    4.327382   9.160856  17.691990
Mo    7.212303   9.160856  17.691990
Mo   -5.769843  11.659271  17.691990
Mo   -2.884921  11.659271  17.691990
Mo   -0.000000  11.659271  17.691990
Mo    2.884921  11.659271  17.691990
Mo    5.769843  11.659271  17.691990
C    0.000000   0.000000  16.301511  
C    2.884921   0.000000  16.301511  
C    5.769843   0.000000  16.301511  
C    8.654764   0.000000  16.301511  
C   11.539685   0.000000  16.301511  
C   -1.442461   2.498415  16.301511  
C    1.442461   2.498415  16.301511  
C    4.327382   2.498415  16.301511  
C    7.212303   2.498415  16.301511  
C   10.097225   2.498415  16.301511  
C   -2.884921   4.996830  16.301511  
C   -0.000000   4.996830  16.301511  
C    2.884921   4.996830  16.301511  
C    5.769843   4.996830  16.301511  
C    8.654764   4.996830  16.301511  
C   -4.327382   7.495246  16.301511  
C   -1.442461   7.495246  16.301511  
C    1.442461   7.495246  16.301511  
C    4.327382   7.495246  16.301511  
C    7.212303   7.495246  16.301511  
C   -5.769843   9.993661  16.301511  
C   -2.884921   9.993661  16.301511  
C   -0.000000   9.993661  16.301511  
C    2.884921   9.993661  16.301511  
C    5.769843   9.993661  16.301511  
Mo   -0.000000   1.665610  14.911032  
Mo    2.884921   1.665610  14.911032  
Mo    5.769843   1.665610  14.911032  
Mo    8.654764   1.665610  14.911032  
Mo   11.539685   1.665610  14.911032  
Mo   -1.442461   4.164025  14.911032  
Mo    1.442461   4.164025  14.911032  
Mo    4.327382   4.164025  14.911032  
Mo    7.212303   4.164025  14.911032  
Mo   10.097225   4.164025  14.911032  
Mo   -2.884921   6.662441  14.911032  
Mo   -0.000000   6.662441  14.911032  
Mo    2.884921   6.662441  14.911032  
Mo    5.769843   6.662441  14.911032  
Mo    8.654764   6.662441  14.911032  
Mo   -4.327382   9.160856  14.911032  
Mo   -1.442461   9.160856  14.911032  
Mo    1.442461   9.160856  14.911032  
Mo    4.327382   9.160856  14.911032  
Mo    7.212303   9.160856  14.911032  
Mo   -5.769843  11.659271  14.911032  
Mo   -2.884921  11.659271  14.911032  
Mo   -0.000000  11.659271  14.911032  
Mo    2.884921  11.659271  14.911032  
Mo    5.769843  11.659271  14.911032  

The job is being run on a cluster system where parallel QE is installed. Other optimization jobs with smaller systems are running perfectly fine, but this or larger systems are encountering this similar error.

Can someone suggest some workaround or reasoning behind this?

$\endgroup$
5
  • 3
    $\begingroup$ Unfortunately it is all but impossible to tell what has gone wrong with what is provided - there might be a clue in the output just above the ones you show here, but I doubt it. Ultimately "An error occurred in MPI_Comm_free" is probably caused by a bug somewhere and code diving will be needed. As you have a repeatable case that displays the problem I'd refer this problem to the QE developers and see what they have to say. Make sure you say exactly how the code was compiled, what libraries you are using (especially MPI), how many cores you are using and any other relevant info. $\endgroup$
    – Ian Bush
    Commented May 31, 2023 at 9:55
  • $\begingroup$ Try adding the option -ndiag 1 when running pw.x. It disables parallel diagonalization that raised similar problems in versions <= 7.1 (see gitlab.com/QEF/q-e/-/blob/develop/Doc/release-notes#L30) $\endgroup$
    – Pie86
    Commented Jun 1, 2023 at 8:10
  • $\begingroup$ @IanBush Thank you for the comment, I will make sure to report it if the problem still persists post applying the suggestion given in the other comment. $\endgroup$
    – Avrana
    Commented Jun 1, 2023 at 10:25
  • $\begingroup$ @Pie86 thank you, I will try with this setting. $\endgroup$
    – Avrana
    Commented Jun 1, 2023 at 10:25
  • $\begingroup$ @Pie86 Your suggestion worked perfectly. Thankyou, it is much appreciated. However, I could not exactly understand the reasoning behind it. If you want to post your comment as an answer, I will be happy to accept it. $\endgroup$
    – Avrana
    Commented Jun 6, 2023 at 4:19

1 Answer 1

2
$\begingroup$

The comment by Pie86:

"Try adding the option -ndiag 1 when running pw.x. It disables parallel diagonalization that raised similar problems in versions <= 7.1 (see gitlab.com/QEF/q-e/-/blob/develop/Doc/release-notes#L30)"

OP:

"@Pie86 Your suggestion worked perfectly. Thankyou, it is much appreciated. However, I could not exactly understand the reasoning behind it. If you want to post your comment as an answer, I will be happy to accept it."

I'll be happy to delete this Community Wiki if Pie86 or someone else comments saying that they would be willing to write a full answer, but until then I'm removing this from the queue of unanswered questions.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .