View source for Converting GEOS code from SLES 11 to SLES 12

{{rightTOC}}

This page will try and detail the changes needed to move a code base from SLES 11 to SLES 12. 

If you have questions, please [mailto:siteam@gmao.gsfc.nasa.gov contact the SI Team].

= Generic SLES 12 Information = 

'''NOTE''': As of May 18, 2020, NCCS has moved to SLES 12 by default upon login. This means that when you do <tt>ssh discover</tt> you will now be on a SLES 12 head node and in the SLES 12 environment (SLURM 19, etc.). In order to get to SLES 11, you must then execute <tt>ssh discover-sles11</tt> from a head node. The SLES 11 nodes are not available directly through the Bastion service, so if you are using it, you won't be able to ssh directly to discover-sles11 from outside discover.

== NCCS SLES 12 FAQ ==

NCCS has created a [https://www.nccs.nasa.gov/nccs-users/instructional/using-discover/SLES12_faq Frequently Asked Questions page] to help answer some generic questions about the SLES 12 transition.

== Missing tools (xxdiff, ImageMagick, etc.) ==

You might notice that things like <tt>xxdiff</tt> and <tt>magick</tt> don't exist in your path anymore. On SLES 12, NCCS has moved many non-system-required utilities to LMod module files. So to get xxdiff, you should:

 module load xxdiff

and similar for ImageMagick, tkcvs, and more. If a utility is not available, contacting NCCS Support is the first step.

== Complaints on login about missing modules ==

You might see issues on logging into SLES 12 that modules cannot be found. The reason for this is often that you are loading modules in your <tt>.bashrc</tt> or <tt>.tcshrc</tt> files. A workaround while we are still in mixed SLES 11/SLES 12 mode is to do something like this for bash:

 <nowiki>if [[ -e /etc/os-release ]]
then
   module load <modules-available-on-sles12>
else
   module load <modules-available-on-sles11>
fi</nowiki>

and similar for tcsh:

 <nowiki>if ( -e /etc/os-release) then
   module load <modules-available-on-sles12>
else
   module load <modules-available-on-sles11>
endif</nowiki>

Here we are using the fact that <tt>/etc/os-release</tt> only exists on the SLES 12 systems as a proxy.

=== GEOSenv ===

This also applies to <tt>module use</tt> statements. If, for convenience, you have been loading <tt>GEOSenv</tt> at startup in interactive shells, something like:

 <nowiki>if [[ -e /etc/os-release ]]
then
   module use -a /discover/swdev/gmao_SIteam/modulefiles-SLES12
else
   module use -a /discover/swdev/gmao_SIteam/modulefiles-SLES11
fi
module load GEOSenv</nowiki>

or in tcsh:

 <nowiki>if (-e /etc/os-release) then
   module use -a /discover/swdev/gmao_SIteam/modulefiles-SLES12
else
   module use -a /discover/swdev/gmao_SIteam/modulefiles-SLES11
endif
module load GEOSenv</nowiki>

would work, as there is a <tt>GEOSenv</tt> in both the SLES 11 and SLES 12 SI Team modulefiles. (Of course, any other modules must be protected.)

If you don't do this, you'll get errors like this on SLES 12:

 <nowiki>Lmod has detected the following error: The following module(s) are unknown: "other/git" 

Please check the spelling or version number. Also try "module spider ..."

It is also possible your cache file is out-of-date; it may help to try:
  $ module --ignore-cache load "other/git"

Also make sure that all modulefiles written in TCL start with the string #%Module

Executing this command requires loading "other/git" which failed while processing the following
module(s): 

    Module fullname  Module Filename
    ---------------  ---------------
    GEOSenv          /discover/swdev/gmao_SIteam/modulefiles-SLES11/GEOSenv</nowiki>

== Missing shared libraries ==

Eventually you might see something like:

 rs_numtiles.x: error while loading shared libraries: libssl.so.0.9.8: cannot open shared object file: No such file or directory

What this ''usually'' means is that you are trying to run an executable built on SLES 11 on SLES 12. Not always, but in many cases.

== SSH is complaining ==

Once you move to SLES 12 you might see this when trying to use <tt>ssh</tt>:

 <nowiki>$ ssh HOST
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:<long list of letters>.
Please contact your system administrator.
Add correct host key in /home/USER/.ssh/known_hosts to get rid of this message.
Offending RSA key in /home/USER/.ssh/known_hosts:63
You can use following command to remove the offending key:
ssh-keygen -R HOST -f /home/USER/.ssh/known_hosts
ECDSA host key for HOST has changed and you have requested strict checking.
Host key verification failed.</nowiki>

This is due to upgrades related to OpenSSH on SLES 12. There are a few actions you can take. 

One, you can use the command it suggests:

 ssh-keygen -R HOST -f /home/USER/.ssh/known_hosts

to remove the offending fingerprint.

A more comprehensive change would be to remove/move <tt>~/.ssh/known_hosts</tt> and then re-establish it by ssh'ing into all the servers you often go to:

 ssh discover
 ssh discover-sles11
 ssh pfe
 ssh cvsacl
 ssh train
 ssh polar
 ssh ...

This will add new fingerprints for every server. '''BUT''' note that you will now see these errors/warnings if you ssh from <tt>discover-sles11</tt> as it will see the issue now.

= GEOS Specific SLES 12 =

== Building code on SLES 12 ==

NCCS has requested that, if at all possible, GEOS parallel builds be done on compute nodes rather than head nodes. If you use <tt>parallel_build.csh</tt> this is done by default for you. However, if you use the more manual CMake then Make/Ninja build process for GEOS, you should run the make step on a compute node. 

NCCS has noticed performance degradations on head nodes as people do builds like <tt>make -j12</tt> as (at the moment) there are not as many head nodes on SLES 12 during this transition time (some are still SLES 11).

== <tt>g5_modules</tt> ==

The first challenge is trying to get a <tt>g5_modules</tt> file that works with your tag. The first thing to do is look at the version of Baselibs being used. Note that if you want to take advantage of both the Haswell and Skylake nodes on SLES 12, you should use Intel MPI as your MPI stack.

NOTE: You should/must build with MPT on the Haswell compute nodes in SLES 12. If you try to build any code with MPT on a Skylake node it will fail due to missing libraries.


=== Baselibs 4 ===

If your Baselibs is based on version 4 (say, 4.0.6), the best one to try is:

 /gpfsm/dhome/mathomp4/GitG5Modules/SLES12/4.0.11/g5_modules.intel1805.impi1910

If you need MPT, you can use:

 /gpfsm/dhome/mathomp4/GitG5Modules/SLES12/4.0.11/g5_modules.intel1805.mpt217

NOTE: It may not be an exact version match but it is essentially equivalent (has the same version of ESMF), with newer versions of some libraries that needed updating for newer OSs and Intel 18+.

=== Baselibs 5 ===

Many of the tags with GEOS use Baselibs 5.1.x, so 5.1.8 is a good substitute. You can use:

 /gpfsm/dhome/mathomp4/GitG5Modules/SLES12/5.1.8-Github/g5_modules.intel1805.impi1910

or for MPT:

  /gpfsm/dhome/mathomp4/GitG5Modules/SLES12/5.1.8-Github/g5_modules.intel1805.mpt217

For 5.2.x tags, use:

 /gpfsm/dhome/mathomp4/GitG5Modules/SLES12/5.2.8-Github/g5_modules.intel1805.impi1910

or for MPT:

 /gpfsm/dhome/mathomp4/GitG5Modules/SLES12/5.2.8-Github/g5_modules.intel1805.mpt217

=== Baselibs 6 ===

If you using Baselibs 6.0.x, the files to use are:

  /gpfsm/dhome/mathomp4/GitG5Modules/SLES12/6.0.4/g5_modules.intel1805.impi1910

  /gpfsm/dhome/mathomp4/GitG5Modules/SLES12/6.0.4/g5_modules.intel1805.mpt217

== <tt>src/Config</tt> ==

'''NOTE:''' Most of the fundamental issues with moving to SLES 12 are due to the files in <tt>src/Config</tt>. The reason is that older tags of GEOS don't handle Intel 18+ and Intel MPI 19+ well due to differences in flags and library names. Without updating files in here, the whole build will fall apart because it doesn't know that Intel 18+ existed!

=== Heracles ===

For a Heracles-based tag, you can start with updating these files to <tt>bw_Heracles-5_4_p3-SLES12</tt>:
 
 ESMA_arch.mk
 fdp

=== Icarus ===

==== Old-style make system ====

For an older Icarus-based tag (one that doesn't have <tt>ifort.mk</tt> and <tt>mpi.mk</tt>, try updating these files to <tt>Icarus-3_2_p9-SLES12</tt>:

 ESMA_arch.mk
 fdp

==== New-style make system ====

For a new style make system with files like <tt>ifort.mk</tt> and <tt>mpi.mk</tt> you will need some changes due to changes in compiler and MPI stacks.

===== mpi.mk =====

For this file, the Intel MPI section should look like:

  ifdef I_MPI_ROOT
      FC := mpiifort
      INC_MPI := $(I_MPI_ROOT)/include64
      LIB_MPI := -L$(I_MPI_ROOT)/lib64 -lmpifort -lmpi # Intel MPI
      LIB_MPI_OMP := -L$(I_MPI_ROOT)/lib64 -lmpifort -lmpi # Intel MPI
  else

=== Jason ===

A good comparison for Jason tags would be to compare against <tt>Jason-3_6_p1</tt> when it comes to Config and other make issues.

== f2py ==

One large challenge will be f2py based files. Any f2py build that depends on <tt>$(LIB_SDF)</tt> will need updating due to GEOSpyD (the Python stack on SLES12). The fix can be demonstrated with <tt>GFIO_.so</tt>. It was originally built as:

 GFIO_.$(F2PYEXT): GFIO_py.F90 r4_install
        $(F2PY) -c -m GFIO_ $(M). $(M)$(INC_SDF) \
                 GFIO_py.F90 r4/libGMAO_gfio_r4.a $(LIB_SDF) $(LIB_SYS) \
                 only: gfioopen gfiocreate gfiodiminquire gfioinquire\
                       gfiogetvar gfiogetvart gfioputvar gfiogetbegdatetime\
                       gfiointerpxy gfiointerpnn gfiocoordnn gfioclose :</nowiki>

Notice how we pass in <tt>$(LIB_SDF)</tt> to f2py? The fix for this is to define a new <tt>$(XLIBS)</tt> and add that:

 '''XLIBS ='''
 '''ifeq ($(wildcard /etc/os-release/.*),)'''
    '''XLIBS = -L/usr/lib64 -lssl -lcrypto'''
 '''endif'''
 
 GFIO_.$(F2PYEXT): GFIO_py.F90 r4_install
        $(F2PY) -c -m GFIO_ $(M). $(M)$(INC_SDF) \
                 GFIO_py.F90 r4/libGMAO_gfio_r4.a $(LIB_SDF) $(LIB_SYS) '''$(XLIBS)'''\
                 only: gfioopen gfiocreate gfiodiminquire gfioinquire\
                       gfiogetvar gfiogetvart gfioputvar gfiogetbegdatetime\
                       gfiointerpxy gfiointerpnn gfiocoordnn gfioclose : </nowiki>

Here we use the fact that the file <tt>/etc/os-release</tt> doesn't exist on SLES 11.

NOTE that this does not present itself at compile time, but rather as a run-time error a la:

 ImportError: ..../Linux/lib/Python/GFIO_.so: undefined symbol: SSLeay

A (partial?) list of f2py builds that need this fix are:

 src/GMAO_Shared/GMAO_ods/GNUmakefile
 src/GMAO_Shared/Chem_Base/GNUmakefile
 src/GMAO_Shared/GMAO_gfio/GNUmakefile

== Hardcoded <tt>-openmp</tt> in make ==

=== Build time error ===

Another common theme is the inclusion of a hardcoded <tt>-openmp</tt> flag in <tt>GNUmakefile</tt>. The reason is that Intel deprecated and then by Intel 18, '''removed''' the <tt>-openmp</tt> flag and changed it to <tt>-qopenmp</tt>.

Examples can be seen in <tt>GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/Shared/Raster/src/GNUmakefile</tt>:

 RASTER_OMPFLAG =
 ifeq ($(ESMA_FC), ifort)
 #       RASTER_OMPFLAG = -openmp
 endif
 OPENMP_FLAG = -openmp

As said above, this flag was changed by Intel to be <tt>-qopenmp</tt> but a better change is to use the generic OMPFLAG alias:

 RASTER_OMPFLAG =
 ifeq ($(ESMA_FC), ifort)
 #       RASTER_OMPFLAG = '''$(OMPFLAG)'''
 endif
 OPENMP_FLAG = '''$(OMPFLAG)'''

=== Link-time error ===

If, when you build, your executable has an issue saying things like <tt>kmp...</tt> not found, this can often be an issue with either trying to link with <tt>-openmp</tt> or not linking to it at all. An example is <tt>Applications/NCEP_Etc/NCEP_enkf/GNUmakefile_in</tt> in <tt>GEOSadas-5_17_0p5B</tt>:

 USER_LDFLAGS = -openmp

which should become:

 USER_LDFLAGS = '''$(OMPFLAG)'''

== Issues with passing <tt>SetServices</tt> in <tt>ESMF_GridCompSetServices</tt> ==

You might occasionally get an error with a call to <tt>ESMF_GridCompSetServices</tt>. For example, building <tt>GEOSadas-5_17_0p5B</tt>, you will encounter:

 geos_pertmod.F90(328): error #7061: The characteristics of dummy argument 1 of the associated actual procedure differ from the characteristics of dummy argument 1 of the dummy procedure.   [AGCMPERT_SETSERVICES]
    call ESMF_GridCompSetServices ( pertmod_gc, agcmPert_SetServices, rc=ier)
 -----------------------------------------------^
 compilation aborted for geos_pertmod.F90 (code 1)

The issue is that Intel 18+ is much stricter with Fortran, and requires the that procedure being passed to <tt>ESMF_GridCompSetServices</tt> have the exact signature required by ESMF for a SetServices. This interface is:

     interface
       subroutine userRoutine(gridcomp, rc)
         use ESMF_CompMod
         implicit none
         type(ESMF_GridComp)        :: gridcomp ! must not be optional
         integer, intent(out)       :: rc       ! must not be optional
       end subroutine
     end interface

So, if we look at the SetServices in <tt>GEOS_AgcmPertGridComp.F90</tt>, we see:

 ! !IROUTINE: SetServices -- Sets ESMF services for this component
 
 ! !INTERFACE:
 
     subroutine SetServices ( GC, RC )
 
 ! !ARGUMENTS:
 
     type(ESMF_GridComp), intent(INOUT) :: GC  ! gridded component
     integer            , intent(  out) :: RC  ! return code

Notice that the <tt>GC</tt> is <tt>intent(INOUT)</tt>, but ESMF's interface does not have this. The solution is to remove the intent:

 ! !IROUTINE: SetServices -- Sets ESMF services for this component
 
 ! !INTERFACE:
 
     subroutine SetServices ( GC, RC )
 
 ! !ARGUMENTS:
 
     type(ESMF_GridComp)                :: GC  ! gridded component
     integer            , intent(  out) :: RC  ! return code

This can occur with other Gridded Components where RC is option, or has the wrong intent.

== Internal Compiler Error with <code>ADA_Module.f90</code> ==

Occasionally, when you try to build <tt>ADA_Module.f90</tt> with Intel 18+, you will get an Internal Compiler Error (ICE) and the build will crash. This is a bug in Intel and can be fixed by changing the optimization level of this file to anything but <code>-O2</code>. However, doing that can (and most likely will) change answers if this file is used. 

One possible workaround that has worked is to change your <code>TMPDIR</code> when building. Why? No idea. But it seems to help. Moreover, it seems to be more successful if the TMPDIR is in nobackup. Why? No idea! If building by hand, do:

 mkdir $NOBACKUP/tmpdir_for_build
 setenv TMPDIR $NOBACKUP/tmpdir_for_build

or

 export TMPDIR=$NOBACKUP/tmpdir_for_build

If using <code>parallel_build.csh</code>, use <code>parallel_build.csh -tmpdir $NOBACKUP/tmpdir_for_build</code>.

Note that depending on shell you might need pass your full nobackup path to <code>parallel_build.csh</code> instead of <code>$NOBACKUP</code>

=== Changing compilation level of <tt>ADA_Module.f90</tt> ===

If all this doesn't work, and you ''know'' you don't depend on the results of <tt>ADA_Module.f90</tt>, this snippet of code in the <tt>GNUmakefile</tt>:

 ADA_Module.o : ADA_Module.f90
 <tab>$(FC) $(subst O2,O3,$(FCFLAGS_local)) -c $<

will change the <code>-O2</code> to be <code>-O3</code>. It's been shown that <code>-O0</code>, and <code>-O1</code> also seem to work. Only <code>-O2</code> seems to trigger the ICE. 

NOTE: That <tt><tab></tt> in the above snippet means that must be a tab character. One of the "fun" vagaries of GNU Make.

== No rule to make target 'it' ==

This one is due to the differences in the base GNU include files. Namely, at some point larger C-block comments were added to standard include files. There are various ways to solve this.

=== .P90 file ===

If the error happens with a .P90 file, then <tt>src/Config/ESMA_base.mk</tt> needs to be changed such that:

 .P90.o
        @sed -e "/\!.*'/s/'//g" $< | $(CPP) -C '''-ansi''' -DANSI_CPP $(FPPFLAGS) > $*___.s90

becomes:

 .P90.o
        @sed -e "/\!.*'/s/'//g" $< | $(CPP) -C '''-nostdinc -std=c99''' -DANSI_CPP $(FPPFLAGS) > $*___.s90

In this case, the <tt>-nostdinc</tt> solved the 'it' issue. It also turns out cpp removed the <tt>-ansi</tt> flag, so we substitute <tt>-std=c99</tt>.

=== Other files ===

If this happens with another Fortran file, usually that means that directory is doing its own preprocessing. For example, <tt>GMAO_gems</tt> can encounter this because it has:

 $(SRCS): %.f90: src/%.f90
 	@echo Preprocessing $@ from $<
 	-@$(FPP) -P -D_PARALLEL -C $< >$@

 communication_primitives.f90: src/communication_primitives.mpi.f90
 	@echo Preprocessing $@ from $<
 	-@$(FPP) -P -D_PARALLEL -C $< >$@

This needs to change to:

 $(SRCS): %.f90: src/%.f90
 	@echo Preprocessing $@ from $<
 	-@$(FPP) -P -D_PARALLEL '''-nostdinc''' -C $< >$@

 communication_primitives.f90: src/communication_primitives.mpi.f90
 	@echo Preprocessing $@ from $<
 	-@$(FPP) -P -D_PARALLEL '''-nostdinc''' -C $< >$@

== Double continuation characters ==

Some codes have doubled continuation characters which lead to:

 odas/odas_decorrelation.F90(354): warning #5152: Invalid use of '&'. Not followed by comment or end of line
                chi2 = chi0, angle1 = odas_grid.angles(i, j, 1), angle2 = v.angle, scale1 = odas_grid.scales(i, j, 1), scale2 = & &
 ---------------------------------------------------------------------------------------------------------------------------------^

As the error says, remove the second one. Note that this simple error can throw a lot of additional errors due to the compiler incorrectly parsing files after this.

== Undefined references to MPI, PMPI, etc ==

If you encounter messages like this:

 ld: /discover/swdev/mathomp4/Baselibs/ESMA-Baselibs-4.0.11-SLES12/x86_64-unknown-linux-gnu/ifort_18.0.5.274-mpt_2.17/Linux/lib/libesmf.so: 
 undefined reference to `MPI::Op::Free()'

when linking an executable, it usually means that the link step is missing <tt>$(LIB_MPI)</tt>. Add it to the link flags (in this case *after* $(LIB_ESMF)).

== ld: failed to convert GOTPCREL relocation; relink with --no-relax ==

The S2S V2 code on transition to SLES 12 encountered this error:

 ld: failed to convert GOTPCREL relocation; relink with --no-relax

A possible solution (still under testing) is to do exactly what it says and add <tt>-Wl,--no-relax</tt> to <tt>USER_LDFLAGS</tt>

== Why so many Intel MPI (I_MPI) warnings?! ==

In running older tags with a newer Intel MPI, you might see a lot of startup messages like:

 <nowiki>[0] MPI startup(): I_MPI_DAPL_UD_ACK_SEND_POOL_SIZE variable has been removed from the product, its value is ignored
[0] MPI startup(): I_MPI_DAPL_UD_ACK_RECV_POOL_SIZE variable has been removed from the product, its value is ignored
[0] MPI startup(): I_MPI_DAPL_CHECK_MAX_RDMA_SIZE variable has been removed from the product, its value is ignored
[0] MPI startup(): I_MPI_DAPL_UD_RECV_BUFFER_NUM variable has been removed from the product, its value is ignored
[0] MPI startup(): I_MPI_DAPL_UD_SEND_BUFFER_NUM variable has been removed from the product, its value is ignored
[0] MPI startup(): I_MPI_DAPL_UD_REQ_EVD_SIZE variable has been removed from the product, its value is ignored
[0] MPI startup(): I_MPI_FABRICS_LIST environment variable is not supported.</nowiki>

This is because many older tags of the GEOS ADAS and GCM had Intel MPI environment variables set that no longer apply on SLES 12. In most cases, the recommendations are to comment out/remove any <tt>I_MPI_*</tt> flags as Intel MPI has removed *many* of these. Please try your jobs without any of these set.

Some exceptions are <tt>I_MPI_DEBUG</tt> which is still valid and the <tt>I_MPI_ADJUST</tt> family (see below). However, if things are still not working after commenting out, NCCS Support might need to be contacted.

=== Model crashes in an AllReduce ===

One thing we've found with Intel MPI on SLES 12 is sometimes the model will crash in an AllReduce call. If so, please try setting:

 setenv I_MPI_ADJUST_ALLREDUCE 12

The default value of this seems to have issues.

== My code seems to crash in Dynamics Startup ==

One user in trying to run a version of the MERRA-2 AGCM on SLES 12 had a crash around the Dynamics layer:

 <nowiki>  NOT using buffer I/O for file: fvcore_internal_rst
    Character Resource Parameter LAYOUT: fvcore_layout.rc
    Integer*4 Resource Parameter COLDSTART: 0
    Integer*4 Resource Parameter AGCM_IM:           180
    Integer*4 Resource Parameter RUN_DT:           450
 Number of Sponge Layers :   9
 Dynamics NSPLIT:   7
 Dynamics KSPLIT:   1
 Dynamics MSPLIT:   0
 Dynamics QSPLIT:   0

 ===================================================================================
 =   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
 =   RANK 0 PID 19837 RUNNING AT borgt005
 =   KILLED BY SIGNAL: 11 (Segmentation fault)
 ===================================================================================
</nowiki>

The solution was to make sure you have an <tt>input.nml</tt> file in your <tt>scratch</tt> directory. Most GEOS models of recent-ish vintage will have a line like:

 cd $SCRDIR

which means the script is now in the <tt>scratch</tt> directory. After this line add a:

 touch input.nml

and that will create the file if not there.

== My ensembles are crashing because of mpiexec_mpt ==

In many tags, if you use a ensemble, there are some hard coded <tt>mpiexec_mpt</tt> references that need to be changed for Intel MPI. An example is in:

 src/Applications/NCEP_Etc/NCEP_enkf/scripts/gmao/etc/AtmEnsConfig.csh

If you grep for <tt>mpiexec_mpt</tt> you'll see many references:

 <nowiki>AtmEnsConfig.csh
47:setenv MPIRUN_AOD  "mpiexec_mpt "
58:#setenv MPIRUN_ATMENKFAERO "mpiexec_mpt -np 384 $AENKFAEROX"
59:setenv MPIRUN_ATMENKFAERO "mpiexec_mpt -np 96 $AENKFAEROX"
67:setenv MPIRUN_ENSIAU  "mpiexec_mpt -np $ENSIAU_NCPUS $IAUX"
78:setenv MPIRUN_ATMENKF "mpiexec_mpt -np $AENKF_NCPUS $AENKFX"
90:setenv MPIRUN_ENSGCM  "mpiexec_mpt -np $ENSGCM_NCPUS GEOSgcm.x"   # mpiexec_mpt does not work in this context
100:setenv MPIRUN_ENSGCMADJ "mpiexec_mpt -np $ENSGCMADJ_NCPUS GEOSgcmPert.x"   # esma_mpirun does not work in this context
101:setenv ADMRUN_OPT_BEGIN "mpiexec_mpt -np $ENSGCMADJ_NCPUS GEOSgcmPert.x"
116:setenv MPIRUN_ENSANA  "mpiexec_mpt -np $ENSGSI_NCPUS GSIsa.x"     # esma_mpirun does not work in this context
129:setenv AENSTAT_MPIRUN "mpiexec_mpt -np $AENSTAT_NCPUS mp_stats.x"</nowiki>

An easy solution is to convert all the <tt>mpiexec_mpt</tt> to <tt>mpirun</tt> which is the command for Intel MPI. (NOTE: if you ever decide to run this with MPT, make sure you convert back to <tt>mpiexec_mpt</tt>. HPE MPT does have an <tt>mpirun</tt> command but it is '''WEIRD'''. Contact Matt if you want to hear a screed about it.)

== All my Perl scripts are failing! ==

As an overall comparison, a good tag to <tt>cvs diff</tt> against is '''<tt>GEOSadas-5_25_1_p2</tt>'''. You are looking for changes like those described below.

=== Shell ===

Many/All of the Perl scripts in the ADAS will fail due their dependence on an older version of <tt>Shell.pm</tt> that doesn't exist on SLES 12. Indeed, <tt>Shell.pm</tt> isn't even a correct call on SLES 12!

Now for many/most scripts, you don't even need a <tt>use Shell</tt> command. For example, if your file has:

 use Shell qw(rm);

or:

 use Shell qw(cat cut wc); # shell commands

and you search the script and there are no <tt>rm();</tt>, <tt>cat();</tt>, <tt>cut();</tt>, or <tt>wc</tt> calls, remove the line! Often it is cruft from years of accrual.

But, the main takeaway is, in the end, '''any''' Perl script with <tt>use Shell</tt> '''will need to be changed''' and <tt>use Shell</tt> must be removed.  In one case (see below), you can change to use <tt>use CPAN::Shell</tt>, which is the Shell module on SLES 12, but your goal, again, is to ''remove'' <tt>use Shell</tt>.

==== rm (the most common) ====

This is by far the most common change needed.  Change all:

 rm();

to:

 unlink();

==== cp ====

If you need <tt>cp();</tt> do not use:

 use Shell qw(cp);

use:

 use File::Copy "cp";

==== mv ====

For <tt>mv();</tt> use:

 use File::Copy "mv";

==== rcp rsh... ====

Change:

 use Shell qw(cat rcp rsh scp ssh);   # make C-shell commands available

to:

 use CPAN::Shell qw(cat rcp rsh scp ssh);   # make C-shell commands available

Note this doesn't work for some of the <tt>Shell</tt> commands above because, say, <tt>rm();</tt> was '''removed''' from <tt>Shell.pm</tt>.

=== timelocal.pl ===

Replace:
 require('timelocal.pl');
with:
 use Time::Local;

=== foreach loop issues ===

Another difference between the Perl on SLES 11 and in SLES 12 is in its handling of some foreach loops in GEOS. Examples can be found in <tt>fvsetup</tt> in some tags. For example:

 foreach $dir qw(ana diag daotovs etc obs prog rs run recycle fcst asens anasa) {

will fail with something like:

 syntax error at ./fvsetup line 4557, near "$dir qw(ana diag daotovs etc obs prog rs run recycle fcst asens anasa)"

The solution is that you have to surround the <tt>qw()</tt> with parentheses itself:

 foreach $dir (qw(ana diag daotovs etc obs prog rs run recycle fcst asens anasa)) {
              *                                                                 *
Essentially all your <tt>foreach</tt> calls should have parentheses around the array you are looping over.

A GEOSadas user found that these files:

 idcheck.pl
 fvsetup
 gen_silo_arc.pl
 monthly.yyyymm.pl.tmpl

had foreach issues.

== ImportError: No module named cross_validation ==

You might encounter this on SLES 12:

     from sklearn.cross_validation import cross_val_score
 ImportError: No module named cross_validation

This is due to the newer version of Python on SLES 12. A solution is to do:

 # sklearn changed where cross_val_score exists
 try:
     from sklearn.model_selection  import cross_val_score
 except ImportError:
     from sklearn.cross_validation import cross_val_score

== Use of <tt>#PBS</tt> pragmas in run scripts ==

With SLES 12, NCCS has removed support for using <tt>#PBS</tt> pragmas in SLURM <tt>sbatch</tt> scripts. You will need to convert them to the appropriate <tt>#SBATCH</tt> pragmas. You can find various pages about this on the web, such as [https://hpc.nih.gov/docs/pbs2slurm.html this one].

[[Category:SI Team]]