wrfhydropy.parallel_teams_run

wrfhydropy.parallel_teams_run(arg_dict)[source]

Parallelizable function to run simulations across nodes. On the master node, python runs multiprocessing. Each separate process is a “team” of simulations to run. Multiprocessing makes MPI calls with a specific syntax to run the MPI executable on specfific (potentially other) nodes. This provides 2 levels of parallelism.

This function is called (in parallel) once for each team by multiprocessing. Each team runs its set of simulations sequentially but each simulation it runs is parallel via MPI. (In the case of ensemble-cycles each team runs an ensemble but the ensemble runs its members sequentially.)

Input:
arg_dict:
arg_dict == {
‘obj_name’string, either “member” or “cast” (or some other

object), matches the object name used in the team_dict below (first argument)

‘compose_dir’: <pathlib.Path absolute path to the cycle top

level/compse, dir where the individual cycle dirs are found>,

‘team_dict’<dict: the information needed for the team, see

below>

}

where:
team_dict == {
object_name: key/name is either ‘members’ or ‘casts’, the value
is a <list: groups of simulations to run either

simulation objects or their run_dirs>

‘nodes’<list: the nodes previously parsed from something

like$PBS_NODE_FILE>,

‘exe_cmd’<string: the MPI-specific model invokation

command>,

‘env’<dict: containging the environment in which to run

the cmds, may be None or ‘None’>

}

The ‘exe_cmd’ is a form of invocation for the distribution of MPI to be used. For openmpi, for example for OpenMPI, this is

exe_cmd: ‘mpirun –host {nodelist} -np {nproc} {cmd}’

The variables in brackets are expanded by internal variables. The ‘exe_cmd’ command substitutes the wrfhydropy of ‘wrf_hydro.exe’ convention for {cmd}. The {nproc} argument is the length of the list passed in the nodes argument, and the {nodellist} are the comma separated arguments in that list.

The “entry_cmd” and “exit_cmd” ARE TAKEN FROM THE JOB object.

  1. can be semicolon-separated commands

  2. where these are run depends on MPI. OpenMPI, for example, handles

these on the same processor set as the model runs.

Notes

Currently this is working/tested with openmpi and intel mpi. MPT requires MPI_SHEPERD env variable and it’s performance is not

satisfactory so far.