Using ESMPy on Discover

Load Modules

Only a couple Compiler+MPI combinations have been tested. These examples are based on the modules used in the GEOSagcm-bridge-DEVEL git repo:

comp/intel-17.0.0.098
mpi/impi-17.0.0.098
lib/mkl-17.0.1.132
other/comp/gcc-5.3-sp3
other/SSSO_Ana-PyD/SApd_4.1.1_py2.7_gcc-5.3-sp3

And you point to the Baselibs in:

/discover/swdev/mathomp4/Baselibs/TmpBaselibs/GMAO-Baselibs-5_0_2-ESMF-7_1_0_beta_snapshot_17/x86_64-unknown-linux-gnu/ifort_17.0.0.098-intelmpi_17.0.0.098

(Equivalently, you can source g5_modules in the git DEVEL repo.)

Next, load mpi4py and ESMPy:

$ module use -a /home/mathomp4/modulefiles
$ module load python/mpi4py/2.0.0/ifort_17.0.0.098-intelmpi_17.0.0.098 python/ESMPy/7.1.0b17/ifort_17.0.0.098-intelmpi_17.0.0.098

This should set up pretty much everything. A simple test to make sure is run:

$ python -c 'import ESMF; import mpi4py'

If that doesn't crash, signs are good.

Running Examples

Copy Examples

To run the examples that ESMF provides, copy them from:

$ cp -r /discover/swdev/mathomp4/Baselibs/TmpBaselibs/GMAO-Baselibs-5_0_2-ESMF-7_1_0_beta_snapshot_17/src/esmf/src/addon/ESMPy/examples .

Download Test NC4 Files

Run on a head node

$ python examples/run_examples_dryrun.py

This command downloads some data files used in their examples. As it needs internet access, this doesn't work on a compute node.

Run Examples

On a compute node, run "Hello World"

$ mpirun -np 6 python examples/hello_world.py
srun.slurm: cluster configuration lacks support for cpu binding
Hello ESMPy World from PET (processor) 4!
Hello ESMPy World from PET (processor) 2!
Hello ESMPy World from PET (processor) 3!
Hello ESMPy World from PET (processor) 0!
Hello ESMPy World from PET (processor) 1!
Hello ESMPy World from PET (processor) 5!

If you want to run all their examples:

$ python examples/run_examples.py --parallel

To run one individually:

$ mpirun -np 6 python examples/ungridded_dimension_regrid.py
srun.slurm: cluster configuration lacks support for cpu binding
ESMPy Ungridded Field Dimensions Example
  interpolation mean relative error = 0.000768815903378
  mass conservation relative error  = 0.0

$ mpirun -np 6 python examples/grid_mesh_regrid.py
srun.slurm: cluster configuration lacks support for cpu binding
ESMPy Grid Mesh Regridding Example
  interpolation mean relative error = 0.00235869859211

Launching MPI

ESMPy says they have a couple ways to launch mpi as noted in the API documentation, but I like the mpirun way...because it seems to work. I cannot figure out how to run the mpi_spawn_regrid.py example. If anyone does know, please inform me, but it might just be "You can't on a cluster" or "You can't with Intel MPI".