Using ESMPy on Discover

Load Modules

Base Modules

Only a couple Compiler+MPI combinations have been tested. These examples are based on the modules used in the GEOSagcm-bridge-DEVEL git repo:

comp/intel-17.0.0.098
mpi/impi-17.0.0.098
lib/mkl-17.0.1.132
other/comp/gcc-5.3-sp3
other/SSSO_Ana-PyD/SApd_4.1.1_py2.7_gcc-5.3-sp3

And you point to the Baselibs in:

/discover/swdev/mathomp4/Baselibs/TmpBaselibs/GMAO-Baselibs-5_0_2-ESMF-7_1_0_beta_snapshot_24/x86_64-unknown-linux-gnu/ifort_17.0.0.098-intelmpi_17.0.0.098

Extra Modules for mpi4py and ESMPy

Next, load mpi4py and ESMPy:

$ module use -a /home/mathomp4/modulefiles
$ module load python/mpi4py/2.0.0/ifort_17.0.0.098-intelmpi_17.0.0.098 python/ESMPy/7.1.0b24/ifort_17.0.0.098-intelmpi_17.0.0.098

This should set up pretty much everything. A simple test to make sure is run:

$ python -c 'import ESMF; import mpi4py'

If that doesn't crash, signs are good.

Running Examples

Copy Examples

To run the examples that ESMF provides, copy them from:

$ cp -r /discover/swdev/mathomp4/Baselibs/TmpBaselibs/GMAO-Baselibs-5_0_2-ESMF-7_1_0_beta_snapshot_24/src/esmf/src/addon/ESMPy/examples .

Download Test NC4 Files

Run on a head node

$ python examples/run_examples_dryrun.py

This command downloads some data files used in their examples. As it needs internet access, this doesn't work on a compute node.

Run Examples

On a compute node, run "Hello World"

$ mpirun -np 6 python examples/hello_world.py
srun.slurm: cluster configuration lacks support for cpu binding
Hello ESMPy World from PET (processor) 4!
Hello ESMPy World from PET (processor) 2!
Hello ESMPy World from PET (processor) 3!
Hello ESMPy World from PET (processor) 0!
Hello ESMPy World from PET (processor) 1!
Hello ESMPy World from PET (processor) 5!

If you want to run all their examples:

$ python examples/run_examples.py --parallel

To run one individually:

$ mpirun -np 6 python examples/ungridded_dimension_regrid.py
srun.slurm: cluster configuration lacks support for cpu binding
ESMPy Ungridded Field Dimensions Example
  interpolation mean relative error = 0.000768815903364
  mass conservation relative error  = 1.49257625157e-16

$ mpirun -np 6 python examples/grid_mesh_regrid.py
srun.slurm: cluster configuration lacks support for cpu binding
ESMPy Grid Mesh Regridding Example
  interpolation mean relative error = 0.00235869859211

$ mpirun -np 6 python ./examples/cubed_sphere_to_mesh_regrid.py
srun.slurm: cluster configuration lacks support for cpu binding
ESMPy cubed sphere Grid Mesh Regridding Example
  interpolation mean relative error = 0.00302911738799
  interpolation max relative (pointwise) error = 0.0101182527126

Launching MPI

ESMPy says they have a couple ways to launch mpi as noted in the API documentation. So far, only the mpirun methods works. I cannot figure out how to run the mpi_spawn_regrid.py example. If anyone does know, please inform me and edit this section, but it might just be "You can't on a cluster" or "You can't with Intel MPI".