Is this "internal error" reproducible when compiling OpenMolcas on other Intel compilers?

Question

The procedure I've been following to build OpenMolcas with an Intel Fortran compiler in the Compute Canada environment is as follows:

module load python
pip install pyparsing
mkdir build_OpenMolcas
git clone https://gitlab.com/Molcas/OpenMolcas.git
cd OpenMolcas
git submodule update --init External/lapack
cd ../build_OpenMolcas
module load intel
which ifort

The result is:

/cvmfs/restricted.computecanada.ca/easybuild/software/2017/Core/ifort/2016.4.258/compilers_and_libraries_2016.4.258/linux/bin/intel64/ifort

So then I use that path:

cmake ../OpenMolcas -D CMAKE_Fortran_COMPILER=/cvmfs/restricted.computecanada.ca/easybuild/software/2020/Core/intel/2020.1.217/compilers_and_libraries_2020.1.217/linux/bin/intel64/ifort
make -j12 # USE -j2 IF IT'S ON A PERSONAL COMPUTER !!!!

This works perfectly (reaches [100%] after about 15 minutes of real/wall time). However if I then compile with BIGOT=ON (in the same folder, I just run the following two commands immediately after the previously run make command finishes):

cmake ../OpenMolcas -D BIGOT=ON -D CMAKE_Fortran_COMPILER=/cvmfs/restricted.computecanada.ca/easybuild/software/2020/Core/intel/2020.1.217/compilers_and_libraries_2020.1.217/linux/bin/intel64/ifort
make -j12 # USE -j2 IF IT'S ON A PERSONAL COMPUTER !!!!

I get vague errors which did not tell me what to fix, other than to contact Intel support (which does not have an email address, nor a phone number, at least for people in the North American continent). For example:

/home/nike/OpenMolcas_HPQC_2022_12_30/src/ldf_ri_util/ldf_sortcoefficients.f(108): internal error: Please visit 'http://www.intel.com/software/products/support' for assistance.
         Call LDF_SortCanonical(AB,l_Scr,Work(ip_Scr),
^
[ Aborting due to internal error. ]
compilation aborted for /home/nike/OpenMolcas_HPQC_2022_12_30/src/ldf_ri_util/ldf_sortcoefficients.f (code 1)
make[2]: *** [CMakeFiles/ldf_ri_util/CMakeFiles/ldf_ri_util_obj.dir/build.make:1623: CMakeFiles/ldf_ri_util/CMakeFiles/ldf_ri_util_obj.dir/ldf_sortcoefficients.f.o] Error 1
make[2]: *** Waiting for unfinished jobs....

and:

/home/nike/OpenMolcas_HPQC_2022_12_30/src/integral_util/k2loop.f(383): internal error: Please visit 'http://www.intel.com/software/products/support' for assistance.
                     Call NewPK(CoorM(1,1),CoorM(1,2),
^
[ Aborting due to internal error. ]
compilation aborted for /home/nike/OpenMolcas_HPQC_2022_12_30/src/integral_util/k2loop.f (code 1)
make[2]: *** [CMakeFiles/integral_util/CMakeFiles/integral_util_obj.dir/build.make:1714: CMakeFiles/integral_util/CMakeFiles/integral_util_obj.dir/k2loop.f.o] Error 1
make[2]: *** Waiting for unfinished jobs....

and:

/home/nike/OpenMolcas_HPQC_2022_12_30/src/aniso_util/transHam.f(124): internal error: Please visit 'http://www.intel.com/software/products/support' for assistance.
          Call prMom('transHam:: magnetic moment, coordinate '//
^
[ Aborting due to internal error. ]
compilation aborted for /home/nike/OpenMolcas_HPQC_2022_12_30/src/aniso_util/transHam.f (code 1)
make[2]: *** [CMakeFiles/aniso_util/CMakeFiles/aniso_util_obj.dir/build.make:713: CMakeFiles/aniso_util/CMakeFiles/aniso_util_obj.dir/transHam.f.o] Error 1
make[2]: *** Waiting for unfinished jobs....

I encountered the same problem with other Intel compiler versions, and all Intel compilers that I have available are:

intel/2020.1.217     intel/2021.2.0     intel/2022.1.0 

Do others get the same error when compiling with BIGOT=ON and Intel Fortran compilers (following exactly the procedure above, except perhaps the module python and module intel parts which I had to run ion Compute Canada machines to make sure that Intel compilers and Python were available in the environment, and don't use -j12 if it's on a personal computer, I recommend -j2 which takes about 20 minutes on a 2016 laptop)?

I would like to know if this error occurs for other people (outside of the Compute Canada environment, for example).

Answer 1

I successfully compiled (using BIGOT=ON) without any issue.

My configuration:

Python 3.9.1
Compilers: Intel® oneAPI HPC Toolkit v2023.0 (Fortran compiler: version 2021.8.0 Build 20221119_000000)
CPU: Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
OS: openSUSE Leap 15.2

Answer 2

After two full work days of struggling with this, I've solved the problem!

On the Compute Canada environment (Cedar specifically) and with the settings described in my question, the only three error messages that I was given, were the ones provided in the initial post. These were compeltely unhelpful and referred to very stable parts of the code that I never modified. These errors were occurring in my own fork (I didn't check the upstream Master branch because I was working with my own fork as a developer, and since these error messages referred to parts of the software that had nothing to do with the parts that I changed, I didn't think my instructions above for compiling the upstream Master would result in any behavior different from what I observed in my fork). Because the upstream Master compiled perfectly in the environment described in the answer by Camps, I switched to that environment and got the following (much more helpful!) error when building my fork with make -j4:

[ 84%] Building Fortran object CMakeFiles/localisation/CMakeFiles/localisation_obj.dir/get_etwo_act.F90.o
/home/nike/OpenMolcas/src/level/level.f(394): error #6634: The shape matching rules of actual arguments and dummy arguments have been violated.   [MMLR]
     2  MMLR,CMM,NCMM,IVSR,IDSTT,RHOAB)
--------^
/home/nike/OpenMolcas/src/level/level.f(525): error #6634: The shape matching rules of actual arguments and dummy arguments have been violated.   [MMLR]
     2  PARM,MMLR,CMM,NCMM,IVSR,IDSTT,RHOAB)
-------------^

This was followed by a bunch of compiler progress:

[ 84%] Building Fortran object CMakeFiles/loprop/CMakeFiles/loprop_obj.dir/dynamic_properties.F90.o
compilation aborted for /home/nike/OpenMolcas/src/level/level.f (code 1)
make[2]: *** [CMakeFiles/level/CMakeFiles/level_obj.dir/build.make:80: CMakeFiles/level/CMakeFiles/level_obj.dir/level.f.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:8087: CMakeFiles/level/CMakeFiles/level_obj.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 84%] Building Fortran object CMakeFiles/localisation/CMakeFiles/localisation_obj.dir/get_nat_lorb.F90.o
[ 84%] Building Fortran object CMakeFiles/loprop/CMakeFiles/loprop_obj.dir/fluctuating.F90.o
[ 84%] Building Fortran object CMakeFiles/localisation/CMakeFiles/localisation_obj.dir/primo_localisation.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/mck_interface.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/mckinley_global.F90.o
[ 84%] Building Fortran object CMakeFiles/localisation/CMakeFiles/localisation_obj.dir/rdvec_localisation.F90.o
[ 84%] Building Fortran object CMakeFiles/loprop/CMakeFiles/loprop_obj.dir/make_fluctuating_charges.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/acore.F90.o
[ 84%] Building Fortran object CMakeFiles/localisation/CMakeFiles/localisation_obj.dir/wrvec_localisation.F90.o
[ 84%] Building Fortran object CMakeFiles/loprop/CMakeFiles/loprop_obj.dir/move_polar.F90.o
[ 84%] Building Fortran object CMakeFiles/loprop/CMakeFiles/loprop_obj.dir/polar.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/cmbn2dc.F90.o
[ 84%] Building Fortran object CMakeFiles/localisation/CMakeFiles/localisation_obj.dir/bitmap_localisation_atom.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/cmbnacb1.F90.o
[ 84%] Building Fortran object CMakeFiles/loprop/CMakeFiles/loprop_obj.dir/print_local.F90.o
[ 84%] Building Fortran object CMakeFiles/loprop/CMakeFiles/loprop_obj.dir/print_mpprop.F90.o
[ 84%] Building Fortran object CMakeFiles/localisation/CMakeFiles/localisation_obj.dir/bitmap_localisation.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/cmbnacb2.F90.o
[ 84%] Building Fortran object CMakeFiles/localisation/CMakeFiles/localisation_obj.dir/chk_input.F90.o
[ 84%] Building Fortran object CMakeFiles/loprop/CMakeFiles/loprop_obj.dir/readin_polar.F90.o
[ 84%] Building Fortran object CMakeFiles/localisation/CMakeFiles/localisation_obj.dir/domain_localisation.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/cmbneldot.F90.o
[ 84%] Building Fortran object CMakeFiles/localisation/CMakeFiles/localisation_obj.dir/genbmp_localisation.F90.o
[ 84%] Built target loprop_obj
[ 84%] Building Fortran object CMakeFiles/localisation/CMakeFiles/localisation_obj.dir/get_cnos.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/cmbnel.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/cmbns1_mck.F90.o
[ 84%] Building Fortran object CMakeFiles/localisation/CMakeFiles/localisation_obj.dir/getinfo_localisation_0.F90.o
[ 84%] Building Fortran object CMakeFiles/localisation/CMakeFiles/localisation_obj.dir/localisation.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/cmbns2a.F90.o
[ 84%] Building Fortran object CMakeFiles/localisation/CMakeFiles/localisation_obj.dir/localise_iterative.F90.o
[ 84%] Building Fortran object CMakeFiles/localisation/CMakeFiles/localisation_obj.dir/localise_noniterative.F90.o
[ 84%] Building Fortran object CMakeFiles/localisation/CMakeFiles/localisation_obj.dir/loc_nat_orb.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/cmbns2b.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/cmbns2.F90.o
[ 84%] Building Fortran object CMakeFiles/localisation/CMakeFiles/localisation_obj.dir/pao_analysis.F90.o
[ 84%] Building Fortran object CMakeFiles/localisation/CMakeFiles/localisation_obj.dir/readinp_localisation.F90.o
[ 84%] Building Fortran object CMakeFiles/localisation/CMakeFiles/localisation_obj.dir/testloc.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/cmbnt1_mck.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/cmbnt2.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/cnthlf_mck.F90.o
[ 84%] Built target localisation_obj
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/cntrct_mck.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/coreb.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/crsph_mck.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/ctldns.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/ctrlmo.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/dan.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/dede_mck.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/din.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/drvel1.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/drvetc.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/drvh2.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/drvk2_mck.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/drvn1_mck.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/elgrddot.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/elgrd.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/elmem.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/hssprt.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/k2loop_mck.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/knegrd_mck.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/knehss.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/knemem_mck.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/knemmh.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/ltocore.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/ltosph.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/lu2lu.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/m1grd_mck.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/m1kernel.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/m1mm1.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/m1mmh.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/makemo.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/mckinley_banner.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/mkfck.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/moacc.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/mult_sro.F90.o
[ 84%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/na2mem.F90.o
[ 85%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/nagrd_mck.F90.o
[ 85%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/namem_mck.F90.o
[ 85%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/nammh.F90.o
[ 85%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/nonatwo.F90.o
[ 85%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/ovrgrd_mck.F90.o
[ 85%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/ovrhss.F90.o
[ 85%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/ovrmem_mck.F90.o
[ 85%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/ovrmmh.F90.o
[ 85%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/pckint_mck.F90.o
[ 85%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/pckmo2.F90.o
[ 85%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/pickmo.F90.o
[ 85%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/prjgrd_mck.F90.o
[ 85%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/prjhss.F90.o
[ 85%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/prjmm1.F90.o
[ 85%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/prjmmh.F90.o
[ 85%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/psoao0_h.F90.o
[ 85%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/request_mclr_run.F90.o
[ 85%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/rtocore.F90.o
[ 85%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/rtosph.F90.o
[ 85%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/schint_mck.F90.o
[ 85%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/screen_mck.F90.o
[ 85%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/smadna.F90.o
[ 85%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/sort_mck.F90.o
[ 85%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/srogrd_mck.F90.o

and then an "internal error" message like the ones I described in the intial post (again in a totally unrelated part of the software, and even in a place unrelated to the three places in which this error was occuring before):

[ 85%] Building Fortran object CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/srohss.F90.o
/home/nike/OpenMolcas/src/mckinley/srogrd_mck.F90(174): internal error: Please visit 'http://www.intel.com/software/products/support' for assistance.
        call SmAdNa(Array(ipFin),nt,rFinal,mop,loper,JndGrd,iuvwx,Indx,idcar,One,tr)
^
[ Aborting due to internal error. ]
compilation aborted for /home/nike/OpenMolcas/src/mckinley/srogrd_mck.F90 (code 1)
make[2]: *** [CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/build.make:1263: CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/srogrd_mck.F90.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/Makefile2:8792: CMakeFiles/mckinley/CMakeFiles/mckinley_obj.dir/all] Error 2
make: *** [Makefile:158: all] Error 2

This means that the integral error message can basically happen anywhere in the code, due to a totally unrelated warning (or error in the case of the MOLCAS BIGOT=ON option which uses -Wall to turn all warnings into errors) elsewhere. For some reason the warning/error that triggered these "internal error" messges, didn't show up in the Compute Canada environment, but thankfully did in the environment described by Camps.

By doing literally nothing else but changing MMLR to MMLR(3) in both suggested locations of level.f and rerunning make -j4, the compilation completed to 100% with no warnings/errors (the "internal errors" in other parts of the software were "fixed" by fixing this unrelated file).

Why did I change MMLR to MMLR(3)?

It's strange, because right next to MMLR were PARM and CMM which should have sizes 4 and 3 respectively, and the compiler never complained about those (neither before changing MMLR to MMLR(3), nor after).

This thread on Intel Communities shows that NASTRAN a finite-element code developed by several companies for NASA starting in the 1960s, had the same error, and that the size of the array should be specified in the CALL statement (i.e. the "actual" argument) even if it's not specified in the subroutine's first line (i.e. the "dummy" arguments).

Before looking at the above Intel Community thread, I was unsuccessful in getting a solution from this StackOverflow thread which had the same shape matching rules error message, but I'm putting the link here in case it's useful to others. I had also opened tabs with this, this, this, this, this, this, this and this page, but that first Intel Community thread came up first, either by luck or because of Google watching precisely what I've been doing on my computer in the last two days, so I didn't need the other pages (but again I'm providing them here in case they're helpful to others with similar errors in the future).