End-to-end overview of wrfhydropy: Simulation evaluation
The goal of this example is to show both the WRF-Hydro model API (application programming interface) and the ability of wrfhydropy to interact with model inputs and outputs: end-to-end capabilities all within python. The vision of of wrfhydropy is to work seamlessly in python across the input, setup, run, and analysis phases of the model life cycle to support reproducible science without stifling the user. This example aims to illustrate this.
Model inputs receive little attention here but can easily be manipulated using the xarray library which provides the general strategy for working with model inputs and outputs in wrfhydropy (sometimes in conjunction with pandas). The evaluation and plotting of the output at the end of the example provide some flavor of using xarray. Particularly for users new to python, it is worth noting that the xarray approach is attractive because it provides a path to data export from
python at every step. This allows users to choose points of departure from the confines of wrfhydropy.
The main focus of this example is the API, in particular the core objects which build a Simulation object: Model, Domain, Job, and optionally Scheduler. Building the Simulation object is key to understanding how to build the more complicated objects which contain it, to be covered separately: Ensembles, Cycles, and EnsembleCycles. The two main differences in wrfhydropy as compared to a shell-centered approach are the way it treats the concepts of “namelists” and “jobs”. These are covered at
length.
Note: This example provides zero scientific content.
0. Overview of this overview
At a high level, this example covers the following:
Preliminaries
Python libraries
Local model repository clone:
wrf_hydro_nwm_publicLocal croton_NY domain downloaded from the cloud
Namelists and configurations in wrfhydropy
The mind is bent for existing users of WRF-Hydro
Model-side namelists
Domain-side namelist “patches”
Compile options
Configuration
Domain object
Model object and compile
Job object
Further brain injuries for existing WRF-Hydro users
Scheduler object
Run modes: interactive, scheduled, and team (not shown here)
Simulation object: interactive run
add Domain, Model, Jobs
compose the simulation to disk
interactive run
Simulation object: scheduled run
add Domain, Model, Jobs, Scheduler
Simulations on disk and in memory
collecting output
opening output
Engineering test
Evaluation syntax: goodness of fit
bulk
group_by=feature_id
export stats to netcdf file
Plot
Total capitulation to matplotlib
1. Preliminaries
Steps to gather the required inputs and setup where the example will be run.
Python libraries
First define the python libraries we will use in this example. The wrfhydropy library should have previously been installed, see installing for help, and its dependencies cover what is used here.
[1]:
import os
import pandas as pd
import pathlib
import pickle
from pprint import pprint
import shlex
import subprocess
import sys
import wrfhydropy
/glade/work/jamesmcc/python_envs/368_/lib/python3.6/site-packages/dask/config.py:131: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
data = yaml.load(f.read()) or {}
Model repository
A clone of the wrf_hydro_nwm_public needs to be available locally.
[2]:
model_repo = pathlib.Path(
'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public')
If this does not exist somewhere on your machine, then
cd /path/to/desired_location
git clone https://github.com/NCAR/wrf_hydro_nwm_public.git
would result in the previous code block being:
model_repo = pathlib.Path(
'/path/to/desired_location/wrf_hydro_nwm_public')
The branch of the repository which is checked out will be the one used by wrfhydropy. If in doubt, git checkout master inside /path/to/desired_location/wrf_hydro_nwm_public/ will suffice for this example.
Next, we will establish an experiment directory and create it. This directory will contain where wrfhydropy writes this example to disk:
[3]:
experiment_dir = pathlib.Path(
'/glade/scratch/jamesmcc/wrfhydropy_end-to-end_example')
if not experiment_dir.exists():
os.mkdir(str(experiment_dir))
os.chdir(str(experiment_dir))
Domain retrieval from the cloud
This section is not important to understanding wrfhydropy, it’s just grabbing the model domain from the cloud. Only the first line is necessary to understand, it tels where the domain files are located. We use code in the wrf_hydro_nwm_public repository to provide us the croton_NY test domain. When a local copy of it does not previously exist, we will download the croton_NY domain in to a subdirectory of the experiment directory, the domain_dir.
[4]:
domain_dir = experiment_dir / 'domain'
if not domain_dir.exists():
sys.path.append(str(model_repo / 'tests/local/utils'))
from gdrive_download import download_file_from_google_drive
file_id = '1xFYB--zm9f8bFHESzgP5X5i7sZryQzJe'
file_target = 'gdrive_testcase.tar.gz'
download_file_from_google_drive(
file_id,
str(experiment_dir.joinpath(file_target)))
untar_cmd = 'tar -xf ' + file_target + '; mv example_case domain'
subprocess.run(
untar_cmd,
shell=True,
cwd=str(experiment_dir))
2. Namelists and configurations in wrfhydropy
Now that we have the model repository and a model domain locally, we can get started with wrfhydropy. We go straight to the main structure or concept that wrfhydropy introduces and which users must understand: namelists. The wrfhydropy package introduces some complexity to the concept of the namelists for the purposes of being able to easily swap domains while maintaining a certain “model configuration”. Or vice-versa, one may also want to swap configurations while maintining a
particular domain. The namelist approach in wrfhydropy supports this flexibility while also offering a mechanism for collecting sets of related namelists into a smaller set of files than would otherwise be needed.
For example, there is a configuration known as the “Analysis and Assimilation” configuration of the National Water Model. For the purposes of a single execution of the model, a configuration is defined by: 1) compile-time options, 2) run-time options, in Fortran these are namelists (hydro.namelist and namelist.hrldas), and 3) input files (such as domain files defined at run-time or files implicit in the code such as tables). This begs the question: What is the “Analysis and Assimilation”
configuration for a different domain? The parsing of namelist files by wrfhydropy provides an answer to the question: how do we maintain a “configuraton” on a different domain (and vice-versa).
Let us be clear that, in terms of the operational execution of the NWM, the state of the model is maintained by a set of process beyond running the model once with the 3 things comprising a configuraton above. Sepcifically, the model state is given in the analysis and forecast modes by “cycling” the model with ever changing forcing and observation files. Here we are not referring to the cycling workflow of the operational model. This is addressed in later examples. Here we confine our attention to serial (in time) execution of the model with fixed inputs, what we will term a “simulation”.
To tackle the notion of configuration across different domains, wrfhydropy separates the standard WRF-Hydro namelists in to two parts: model-side and domain-side namelists. Because WRF-Hydro begins with 2 namelist files, the result is 4 namelist files, as illustrated in the table below.
WRF-Hydro namelist file |
= |
wrfhydropy model-side example file |
wrfhydropy domain-side example file |
config name |
||
|---|---|---|---|---|---|---|
namelist.hrldas |
= |
hrldas_namelists.json |
hrldas_namelist_patches.json |
‘some_config’ |
||
hydro.namelist |
= |
hrldas_namelists.json |
hydro_namelist_patches.json |
‘some_config’ |
The user presents wrfhydropy with the files in two right-hand columns and it creates the files in the left-hand columns, which is what the model requires at run time. The files in the left-hand column have specific names, which are hard-wired in the WRF-Hydro code. The JSON files supplied to wrfhydropy do not need specific names, the names here are the conventions we have used in wrf_hydro_nwm_public (model-side) and in the croton_NY domain (domain-side) which has been
downloaded. The names of these files are presented to wrfhydropy at its run time.
Moreover, the files supplied to wrfhydropy potentially contain multiple named configurations in either or both of the respective model- and domain- sides of the namelists. To actually generate the namelist files needed by the model, wrfhydropy also needs a configuration name to pull the specific, desired namelists from the JSON files before combining the two sides of the namelists to present to the model. In this way, the JSON namelist files provide a way of saving multiple namelists
(more importantly, diffs from a single base namelist in each) which may be a relevant collection for a given experiment or some other purpose. The collection of configurations in the specific JSON files used in this example are intended for the purposes of wrf_hydro_nwm_public model testing (they are not identical to the NWM operational configurations) and are used here for convenience rather than a specific purpose.
Model-side namelists
As mentioned, for testing purposes we have created model-side namelists in wrf_hydro_nwm_public. We can examine those in our clone of the repository. Notice the names of all the available configurations in the file, starting with “base”, from which the remaining configurations are differences:
[5]:
hrldas_model_side_file = model_repo / 'src/hrldas_namelists.json'
with open(hrldas_model_side_file, 'r') as file:
print(file.read())
{
"base": {
"noahlsm_offline": {
"btr_option": 1,
"canopy_stomatal_resistance_option": 1,
"dynamic_veg_option": 4,
"forcing_timestep": "NULL_specified_in_domain.json",
"frozen_soil_option": 1,
"glacier_option": 2,
"hrldas_setup_file": "NULL_specified_in_domain.json",
"indir": "NULL_specified_in_domain.json",
"khour": 144,
"noah_timestep": 3600,
"nsoil": 4,
"outdir": "./",
"output_timestep": 3600,
"pcp_partition_option": 1,
"radiative_transfer_option": 3,
"restart_filename_requested": "NULL_specified_in_domain.json",
"restart_frequency_hours": 24,
"rst_bi_in": 0,
"rst_bi_out": 0,
"runoff_option": 3,
"snow_albedo_option": 1,
"soil_thick_input": [0.1, 0.3, 0.6, 1.0],
"spatial_filename": "NULL_specified_in_domain.json",
"split_output_count": 1,
"start_day": "NULL_specified_in_domain.json",
"start_hour": "NULL_specified_in_domain.json",
"start_min": "NULL_specified_in_domain.json",
"start_month": "NULL_specified_in_domain.json",
"start_year": "NULL_specified_in_domain.json",
"supercooled_water_option": 1,
"surface_drag_option": 1,
"surface_resistance_option": 4,
"tbot_option": 2,
"temp_time_scheme_option": 3,
"zlvl": 10.0
},
"wrf_hydro_offline": {
"forc_typ": "NULL_specified_in_domain.json"
}
},
"nwm_ana": {
"noahlsm_offline": {},
"wrf_hydro_offline": {}
},
"nwm_long_range" : {
"noahlsm_offline": {},
"wrf_hydro_offline": {}
},
"nwm_hi_ana" : {
"noahlsm_offline": {
"snow_albedo_option": 2
},
"wrf_hydro_offline": {}
},
"nwm_channel-only" : {
"noahlsm_offline": {},
"wrf_hydro_offline": {}
},
"nwm_output_ana" : {
"noahlsm_offline": {
"khour": 1,
"restart_frequency_hours": 1
},
"wrf_hydro_offline": {}
},
"nwm_output_long_range" : {
"noahlsm_offline": {
"khour": 1,
"restart_frequency_hours": 1
},
"wrf_hydro_offline": {}
},
"gridded": {
"noahlsm_offline": {},
"wrf_hydro_offline": {}
},
"gridded_no_lakes": {
"noahlsm_offline": {},
"wrf_hydro_offline": {}
},
"reach": {
"noahlsm_offline": {},
"wrf_hydro_offline": {}
}
}
This is a JSON file with hierarchical data structures similar to a python dictionary. The “base” configuration sets the basis for the other named configuratons which follow, which only specify differences from this base. You can see that the “base” config is identical to both the nwm_ana and “nwm_long_range” configurations (among others). Differences from “base” are succinctly summarized in the named configurations.
The next things to notice are lines like: "forcing_timestep": "NULL_specified_in_domain.json". This value of the field indicates that it is considered a domain-side option and its value is to come from the domain-side file (and the specified configuration for that file). Before looking at the domain-side namelists, let’s take a look a the other model-side namelist corresponding to hydro.namelist. It is treated exactly as for the namelist.hrldas above.
[6]:
hydro_model_side_file = model_repo / 'src/hydro_namelists.json'
with open(hydro_model_side_file, 'r') as file:
print(file.read())
{
"base": {
"hydro_nlist": {
"aggfactrt": 4,
"channel_option": 2,
"chanobs_domain": 1,
"chanrtswcrt": 1,
"chrtout_domain": 1,
"chrtout_grid": 1,
"compound_channel": true,
"dtrt_ch": 300,
"dtrt_ter": 10,
"dxrt": 250,
"frxst_pts_out": 0,
"geo_finegrid_flnm": "NULL_specified_in_domain.json",
"geo_static_flnm": "NULL_specified_in_domain.json",
"gw_restart": 1,
"gwbaseswcrt": 4,
"gwbuckparm_file": "NULL_specified_in_domain.json",
"hydrotbl_f": "NULL_specified_in_domain.json",
"igrid": 1,
"io_config_outputs": 0,
"io_form_outputs": 4,
"land_spatial_meta_flnm": "NULL_specified_in_domain.json",
"lsmout_domain": 1,
"nsoil": 4,
"order_to_write": 1,
"out_dt": 60,
"outlake": 1,
"output_channelbucket_influx": 0,
"output_gw": 1,
"ovrtswcrt": 1,
"reservoir_persistence_usgs": false,
"restart_file": "NULL_specified_in_domain.json",
"route_lake_f": "NULL_specified_in_domain.json",
"route_link_f": "NULL_specified_in_domain.json",
"rst_bi_in": 0,
"rst_bi_out": 0,
"rst_dt": 1440,
"rst_typ": 1,
"rstrt_swc": 0,
"rt_option": 1,
"rtout_domain": 1,
"split_output_count": 1,
"subrtswcrt": 1,
"sys_cpl": 1,
"t0outputflag": 1,
"udmap_file": "NULL_specified_in_domain.json",
"udmp_opt": 1,
"zsoil8": [-0.1, -0.4, -1.0, -2.0]
},
"nudging_nlist": {
"biaswindowbeforet0": false,
"invdisttimeweightbias": true,
"maxagepairsbiaspersist": 3,
"minnumpairsbiaspersist": 1,
"nlastobs": 480,
"noconstinterfbias": false,
"nudginglastobsfile": "NULL_specified_in_domain.json",
"nudgingparamfile": "NULL_specified_in_domain.json",
"persistbias": true,
"readtimesliceparallel": true,
"temporalpersistence": true,
"timeslicepath": "NULL_specified_in_domain.json"
}
},
"nwm_ana": {
"hydro_nlist": {},
"nudging_nlist": {}
},
"nwm_long_range": {
"hydro_nlist": {
"aggfactrt": 1,
"dxrt": 1000,
"ovrtswcrt": 0,
"rst_typ": 0,
"rtout_domain": 0,
"subrtswcrt": 0
},
"nudging_nlist": {
"maxagepairsbiaspersist": 24,
"persistbias": false
}
},
"nwm_hi_ana": {
"hydro_nlist": {
"compound_channel": false,
"out_dt": 60,
"aggfactrt": 10,
"dxrt": 100
},
"nudging_nlist": {}
},
"nwm_channel-only": {
"hydro_nlist": {
"lsmout_domain": 0,
"output_gw": 0,
"output_channelbucket_influx": 2
},
"nudging_nlist": {}
},
"nwm_channel-only_forecasts": {
"hydro_nlist": {
"lsmout_domain": 0,
"output_gw": 0,
"output_channelbucket_influx": 2,
"reservoir_persistence_usgs": false
},
"nudging_nlist": {}
},
"nwm_output_ana": {
"hydro_nlist": {
"chanobs_domain": 0,
"chrtout_grid": 0,
"io_config_outputs": 1,
"io_form_outputs": 2,
"lsmout_domain": 0,
"output_channelbucket_influx": 2,
"output_gw": 0,
"rst_dt": 60
},
"nudging_nlist": {}
},
"nwm_output_long_range": {
"hydro_nlist": {
"aggfactrt": 1,
"chanobs_domain": 0,
"chrtout_grid": 0,
"dxrt": 1000,
"io_config_outputs": 4,
"io_form_outputs": 2,
"lsmout_domain": 0,
"out_dt": 60,
"output_gw": 0,
"ovrtswcrt": 0,
"rst_dt": 60,
"rst_typ": 0,
"rtout_domain": 0,
"subrtswcrt": 0
},
"nudging_nlist": {
"maxagepairsbiaspersist": 24,
"persistbias": false
}
},
"gridded": {
"hydro_nlist": {
"compound_channel": false,
"gwbaseswcrt": 1,
"gwbasmskfil": "NULL_specified_in_domain.json",
"channel_option": 3,
"dtrt_ch": 10,
"rstrt_swc": 1,
"udmp_opt": 0
},
"nudging_nlist": {
"biaswindowbeforet0": "",
"invdisttimeweightbias": "",
"maxagepairsbiaspersist": "",
"minnumpairsbiaspersist": "",
"nlastobs": "",
"noconstinterfbias": "",
"nudginglastobsfile": "",
"nudgingparamfile": "",
"persistbias": "",
"readtimesliceparallel": "",
"temporalpersistence": "",
"timeslicepath": ""
}
},
"gridded_no_lakes": {
"hydro_nlist": {
"compound_channel": false,
"gwbaseswcrt": 1,
"gwbasmskfil": "NULL_specified_in_domain.json",
"channel_option": 3,
"dtrt_ch": 10,
"rstrt_swc": 1,
"udmp_opt": 0
},
"nudging_nlist": {
"biaswindowbeforet0": "",
"invdisttimeweightbias": "",
"maxagepairsbiaspersist": "",
"minnumpairsbiaspersist": "",
"nlastobs": "",
"noconstinterfbias": "",
"nudginglastobsfile": "",
"nudgingparamfile": "",
"persistbias": "",
"readtimesliceparallel": "",
"temporalpersistence": "",
"timeslicepath": ""
}
},
"reach": {
"hydro_nlist": {
"compound_channel": false,
"gwbaseswcrt": 1,
"gwbasmskfil": "NULL_specified_in_domain.json",
"dtrt_ch": 10,
"outlake": 0,
"rstrt_swc": 0,
"t0outputflag": 0,
"udmp_opt": 0
},
"nudging_nlist": {
"biaswindowbeforet0": "",
"invdisttimeweightbias": "",
"maxagepairsbiaspersist": "",
"minnumpairsbiaspersist": "",
"nlastobs": "",
"noconstinterfbias": "",
"nudginglastobsfile": "",
"nudgingparamfile": "",
"persistbias": "",
"readtimesliceparallel": "",
"temporalpersistence": "",
"timeslicepath": ""
}
}
}
Again, the JSON file provides a “base” namelist and the following configuratons only specifiy differences. And again, domain-side values are indicated by “NULL_specified_in_domain.json”.
A final point is that model-side JSON namelists can be any similarly structured JSON file. There is nothing particularly special about these namelists which are located in the model repository. They exist specifically to support automated model testing. As will be pointed out below, the user can tell wrfhydropy the location of the JSON namelist files to use (though it looks for these specific files in the model repository by default). Generally, an experiment will create it’s own model-side
and domain-side JSON namelists. Only two such model-side JSON namelist files are necessary while 2 domain-side namelists are necessary for each domain.
Domain-side namelists
Just as for model-side namelists, the domain side namelists provide a “base” configuration and
[7]:
hrldas_domain_side_file = domain_dir / 'hrldas_namelist_patches.json'
with open(hrldas_domain_side_file, 'r') as file:
print(file.read())
{
"base": {
"noahlsm_offline": {
"hrldas_setup_file": "./NWM/DOMAIN/wrfinput_d01.nc",
"khour": 144,
"indir": "./FORCING",
"forcing_timestep": 3600,
"spatial_filename": "./NWM/DOMAIN/soil_properties.nc",
"restart_filename_requested": "./NWM/RESTART/RESTART.2011082600_DOMAIN1",
"start_year": 2011,
"start_month": 8,
"start_day": 26,
"start_hour": 0,
"start_min": 0
},
"wrf_hydro_offline": {
"forc_typ": 1
}
},
"nwm_ana": {
"noahlsm_offline": {},
"wrf_hydro_offline": {}
},
"nwm_short_range" : {
"noahlsm_offline": {},
"wrf_hydro_offline": {}
},
"nwm_medium_range" : {
"noahlsm_offline": {},
"wrf_hydro_offline": {}
},
"nwm_long_range" : {
"noahlsm_offline": {
"hrldas_setup_file": "./NWM/DOMAIN_LR/wrfinput_d01.nc",
"indir": "./FORCING",
"forcing_timestep": 3600,
"spatial_filename": "./NWM/DOMAIN_LR/soil_properties.nc",
"restart_filename_requested": "./NWM/RESTART_LR/RESTART.2011082600_DOMAIN1"
},
"wrf_hydro_offline": {}
},
"nwm_channel-only" : {
"noahlsm_offline": {
"indir": "./FORCING_channel-only"
},
"wrf_hydro_offline": {
"forc_typ": 10
}
},
"nwm_output_ana": {
"noahlsm_offline": {},
"wrf_hydro_offline": {}
},
"nwm_output_long_range" : {
"noahlsm_offline": {
"hrldas_setup_file": "./NWM/DOMAIN_LR/wrfinput_d01.nc",
"indir": "./FORCING",
"forcing_timestep": 3600,
"spatial_filename": "./NWM/DOMAIN_LR/soil_properties.nc",
"restart_filename_requested": "./NWM/RESTART_LR/RESTART.2011082600_DOMAIN1"
},
"wrf_hydro_offline": {}
},
"gridded" : {
"noahlsm_offline": {
"hrldas_setup_file": "./Gridded/DOMAIN/wrfinput_d01.nc",
"spatial_filename": "./Gridded/DOMAIN/soil_properties.nc",
"restart_filename_requested": "./Gridded/RESTART/RESTART.2011082600_DOMAIN1"
},
"wrf_hydro_offline": {}
},
"gridded_no_lakes": {
"noahlsm_offline": {
"hrldas_setup_file": "./Gridded_no_lakes/DOMAIN/wrfinput_d01.nc",
"spatial_filename": "./Gridded_no_lakes/DOMAIN/soil_properties.nc",
"restart_filename_requested": "./Gridded_no_lakes/RESTART/RESTART.2011082600_DOMAIN1"
},
"wrf_hydro_offline": {}
},
"reach": {
"noahlsm_offline": {
"hrldas_setup_file": "./Reach/DOMAIN/wrfinput_d01.nc",
"spatial_filename": "./Reach/DOMAIN/soil_properties.nc",
"restart_filename_requested": "./Reach/RESTART/RESTART.2011082600_DOMAIN1"
},
"wrf_hydro_offline": {}
}
}
We do not show the hydro_namelist_patches.json, it is very similar and can be examined by the user.
To summarize the namelist handling of wrfhydropy: while wrfhydropy breaks the native WRF-Hydro namelists into parts, the files we use to build the native namelists may contain many named configurations which can be specified as deviations from a base config in that file. The model-side namelists contain the full set of WRF-Hydro options and where an option is considered domain-side that option’s value is set to NULL_specified_in_domain.json. The domain-side namelist contains only
the options relevant to the domain, for the base and each named configuration.
As mentioned, each “domain” or set of domain files will require it’s own domain-side namelits which are specific to the files in that domain. If multiple domains are generated using the same filenames and file structure conventions then these JSON namelists can be identical across the multiple domains and only one of each domain-side JSON file is required.
3. Compile Options
The definition of a configuration above included compile-time options in addition to the run-time options and input files covered by the namelists above. Similarly to the namelists, the compile options for named-configurations’ are strored in JSON files and retrieved by name by wrfhydropy. However, because compile options relate only to the model, all compile time options are necessarily model-side. There is only one compile options file. The compile options JSON file also does NOT follow
the convention of using a “base” namelist and diffs. The compile options JSON file just uses configuration names as this level of complexity has never seemed warranted in this case.
Here is contents of the compile options file that comes with the model repository:
[8]:
compile_options_file = model_repo / 'src/compile_options.json'
with open(compile_options_file, 'r') as file:
print(file.read())
{
"nwm": {
"WRF_HYDRO": 1,
"HYDRO_D": 0,
"SPATIAL_SOIL": 1,
"WRF_HYDRO_RAPID": 0,
"WRFIO_NCD_LARGE_FILE_SUPPORT": 1,
"NCEP_WCOSS": 0,
"WRF_HYDRO_NUDGING": 1
},
"gridded": {
"WRF_HYDRO": 1,
"HYDRO_D": 0,
"SPATIAL_SOIL": 1,
"WRF_HYDRO_RAPID": 0,
"WRFIO_NCD_LARGE_FILE_SUPPORT": 1,
"NCEP_WCOSS": 0,
"WRF_HYDRO_NUDGING": 0
},
"gridded_no_lakes": {
"WRF_HYDRO": 1,
"HYDRO_D": 0,
"SPATIAL_SOIL": 1,
"WRF_HYDRO_RAPID": 0,
"WRFIO_NCD_LARGE_FILE_SUPPORT": 1,
"NCEP_WCOSS": 0,
"WRF_HYDRO_NUDGING": 0
},
"reach": {
"WRF_HYDRO": 1,
"HYDRO_D": 0,
"SPATIAL_SOIL": 1,
"WRF_HYDRO_RAPID": 0,
"WRFIO_NCD_LARGE_FILE_SUPPORT": 1,
"NCEP_WCOSS": 0,
"WRF_HYDRO_NUDGING": 0
}
}
As with the namelist JSON files, the compile options JSON file can be any file and its path is specified in the initialization of wrfhydropy Model objects. The init method also provides a means for overiding compile options.
4. Configuration
We will use the named configuration nwm_ana which was seen in the above namelists.
[9]:
config = 'nwm_ana'
The following detail may not be obvious until we get to Simulation.add(), but note that wrfhydropy requires the same configuration name be used for Model and Domain objects when they are combined into a Simulation object. The flexibility of allowing different named configurations for Model and Domain presents risks that we haven’t deemed acceptable.
5. Domain object
The domain object contains all the domain-side information necessary to run the model. To instantiate or initalize (__init__) a Domain object, help is available on-line (in python) and is the same information presented at our (readthedocs page)[https://wrfhydropy.readthedocs.io/en/latest/]. This shows the arguments availble to the __init__ method, with some description:
[10]:
help(wrfhydropy.Domain)
Help on class Domain in module wrfhydropy.core.domain:
class Domain(builtins.object)
| Class for a WRF-Hydro domain, which constitutes all domain-specific files needed for a
| setup.
|
| Methods defined here:
|
| __init__(self, domain_top_dir:str, domain_config:str, compatible_version:str=None, hydro_namelist_patch_file:str='hydro_namelist_patches.json', hrldas_namelist_patch_file:str='hrldas_namelist_patches.json')
| Instantiate a Domain object
| Args:
| domain_top_dir: Parent directory containing all domain directories and files.
| domain_config: The domain configuration to use, options are 'NWM',
| 'Gridded', or 'Reach'
| compatible_version: String indicating the compatible model version, required if no
| .version file included in domain_top_dir.
| hydro_namelist_patch_file: Filename of json file containing namelist patches for
| hydro namelist
| hrldas_namelist_patch_file: Filename of json file containing namelist patches for
| hrldas namelist
|
| copy_files(self, dest_dir:str, symlink:bool=True)
| Copy domain files to a new directory
| Args:
| dir: The destination directory for domain files
| symlink: Symlink domain files instead of copy
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables (if defined)
|
| __weakref__
| list of weak references to the object (if defined)
To create an object containing the domain-side, we’ll use just the required arguments. The implicit paths to the JSON namelist patch (domain-side) files is at the top level of the domain and the names are the defaults. We could give different paths for both of these files. The
[11]:
domain = wrfhydropy.Domain(
domain_top_dir=domain_dir,
domain_config=config)
domain
[11]:
<wrfhydropy.core.domain.Domain at 0x2b4cb6d15208>
This last line simply indicates that no useful __repr__ (representation method) has been implemented for the domain object. One can examine wrfhydropy objects via the “dunder dict” mechanism. This will be illustrated for the Model object.
6. Model object
The environment used to build and run the model is not managed by wrfhydropy directly (however relevant parts of it are saved into the Model object, as shown below). Here we examine the modules used on cheyenne to compile the model.
[12]:
print(subprocess.run('module list', shell=True, stderr=subprocess.PIPE).stderr.decode('utf-8'))
Currently Loaded Modules:
1) intel/18.0.5 3) impi/2018.4.274 5) ncarcompilers/0.5.0
2) netcdf/4.7.3 4) ncarenv/1.3 6) nccmp/1.8.2.1
The above shows that we are using the intel compiler for this example. You may need to change this to your machine.
[13]:
model = wrfhydropy.Model(
model_repo / 'src',
compiler='ifort',
model_config=config)
model
[13]:
<wrfhydropy.core.model.Model at 0x2b4cb6db6198>
What is in a model object? Let’s take a look at the disctionary representation of the object (the “dunder dict”) BEFORE we compile:
[14]:
pprint(model.__dict__)
{'compile_dir': None,
'compile_log': None,
'compile_options': {'HYDRO_D': 0,
'NCEP_WCOSS': 0,
'SPATIAL_SOIL': 1,
'WRFIO_NCD_LARGE_FILE_SUPPORT': 1,
'WRF_HYDRO': 1,
'WRF_HYDRO_NUDGING': 1,
'WRF_HYDRO_RAPID': 0},
'compile_options_config_file': 'compile_options.json',
'compiler': 'ifort',
'configure_log': None,
'git_hash': '5913c15572a90556f26702dd6ff1d16c7ffb63df',
'hrldas_namelist_config_file': 'hrldas_namelists.json',
'hrldas_namelists': {'noahlsm_offline': {'btr_option': 1,
'canopy_stomatal_resistance_option': 1,
'dynamic_veg_option': 4,
'forcing_timestep': 'NULL_specified_in_domain.json',
'frozen_soil_option': 1,
'glacier_option': 2,
'hrldas_setup_file': 'NULL_specified_in_domain.json',
'indir': 'NULL_specified_in_domain.json',
'khour': 144,
'noah_timestep': 3600,
'nsoil': 4,
'outdir': './',
'output_timestep': 3600,
'pcp_partition_option': 1,
'radiative_transfer_option': 3,
'restart_filename_requested': 'NULL_specified_in_domain.json',
'restart_frequency_hours': 24,
'rst_bi_in': 0,
'rst_bi_out': 0,
'runoff_option': 3,
'snow_albedo_option': 1,
'soil_thick_input': [0.1,
0.3,
0.6,
1.0],
'spatial_filename': 'NULL_specified_in_domain.json',
'split_output_count': 1,
'start_day': 'NULL_specified_in_domain.json',
'start_hour': 'NULL_specified_in_domain.json',
'start_min': 'NULL_specified_in_domain.json',
'start_month': 'NULL_specified_in_domain.json',
'start_year': 'NULL_specified_in_domain.json',
'supercooled_water_option': 1,
'surface_drag_option': 1,
'surface_resistance_option': 4,
'tbot_option': 2,
'temp_time_scheme_option': 3,
'zlvl': 10.0},
'wrf_hydro_offline': {'forc_typ': 'NULL_specified_in_domain.json'}},
'hydro_namelist_config_file': 'hydro_namelists.json',
'hydro_namelists': {'hydro_nlist': {'aggfactrt': 4,
'channel_option': 2,
'chanobs_domain': 1,
'chanrtswcrt': 1,
'chrtout_domain': 1,
'chrtout_grid': 1,
'compound_channel': True,
'dtrt_ch': 300,
'dtrt_ter': 10,
'dxrt': 250,
'frxst_pts_out': 0,
'geo_finegrid_flnm': 'NULL_specified_in_domain.json',
'geo_static_flnm': 'NULL_specified_in_domain.json',
'gw_restart': 1,
'gwbaseswcrt': 4,
'gwbuckparm_file': 'NULL_specified_in_domain.json',
'hydrotbl_f': 'NULL_specified_in_domain.json',
'igrid': 1,
'io_config_outputs': 0,
'io_form_outputs': 4,
'land_spatial_meta_flnm': 'NULL_specified_in_domain.json',
'lsmout_domain': 1,
'nsoil': 4,
'order_to_write': 1,
'out_dt': 60,
'outlake': 1,
'output_channelbucket_influx': 0,
'output_gw': 1,
'ovrtswcrt': 1,
'reservoir_persistence_usgs': False,
'restart_file': 'NULL_specified_in_domain.json',
'route_lake_f': 'NULL_specified_in_domain.json',
'route_link_f': 'NULL_specified_in_domain.json',
'rst_bi_in': 0,
'rst_bi_out': 0,
'rst_dt': 1440,
'rst_typ': 1,
'rstrt_swc': 0,
'rt_option': 1,
'rtout_domain': 1,
'split_output_count': 1,
'subrtswcrt': 1,
'sys_cpl': 1,
't0outputflag': 1,
'udmap_file': 'NULL_specified_in_domain.json',
'udmp_opt': 1,
'zsoil8': [-0.1, -0.4, -1.0, -2.0]},
'nudging_nlist': {'biaswindowbeforet0': False,
'invdisttimeweightbias': True,
'maxagepairsbiaspersist': 3,
'minnumpairsbiaspersist': 1,
'nlastobs': 480,
'noconstinterfbias': False,
'nudginglastobsfile': 'NULL_specified_in_domain.json',
'nudgingparamfile': 'NULL_specified_in_domain.json',
'persistbias': True,
'readtimesliceparallel': True,
'temporalpersistence': True,
'timeslicepath': 'NULL_specified_in_domain.json'}},
'model_config': 'nwm_ana',
'object_id': None,
'pre_compile_cmd': None,
'source_dir': PosixPath('/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS'),
'table_files': [],
'version': 'v5.1.0-beta2\n',
'wrf_hydro_exe': None}
There is a lot of information contained in this Model object. Even more information is stored there after the model is compiled.
When the model is compiled, a compile_dir is required. It is VERY IMPORTANT to note that the model is actually compiled as per normal in the model_repo_path/src. The compile_dir is used to save the results of that (normal WRF-Hydro) compile along with a pickle (disk representation/serialization) of the wrfhydropy Model object. After compile, the Model object in memory and the copy of it stored to disk are identical. If we’ve already compiled the model and the
compile_dir exists, we can avoid recompiling the model.
[15]:
compile_dir = experiment_dir / 'compile'
if not compile_dir.exists():
model.compile(compile_dir)
else:
model = pickle.load(compile_dir.joinpath('WrfHydroModel.pkl').open('rb'))
# Caveat here, the config used for compiling is baked into the restored model object. A change/fix to this is easy to implement...
/glade/work/jamesmcc/python_envs/368_/lib/python3.6/site-packages/wrfhydropy-0.0.18-py3.6.egg/wrfhydropy/core/model.py:193: UserWarning: /glade/scratch/jamesmcc/wrfhydropy_end-to-end_example/compile directory does not exist, creating
warnings.warn(str(self.compile_dir.absolute()) + ' directory does not exist, creating')
Model successfully compiled into /glade/scratch/jamesmcc/wrfhydropy_end-to-end_example/compile
If one wants to recompile, it’s necessary to delete the compile_dir.
Note: Be careful with the unpickling above. Especially note that the configuration is baked into the unpickled model object and currently requires manual intervention to be changed. A model.restore() method or the like (init from file) could reduce pitfalls here and will likely be added to the API.
Let’s examine what’s in the compile directory:
[16]:
sorted(compile_dir.glob('*'))
[16]:
[PosixPath('/glade/scratch/jamesmcc/wrfhydropy_end-to-end_example/compile/.uid'),
PosixPath('/glade/scratch/jamesmcc/wrfhydropy_end-to-end_example/compile/WrfHydroModel.pkl'),
PosixPath('/glade/scratch/jamesmcc/wrfhydropy_end-to-end_example/compile/wrf_hydro.exe')]
The model executable, wrf_hydro.exe, is stored along with the serialization of the Model object. The .uid file stores the object id in an attempt to reduce mangling of the executable relative to the serialized model object.
Let’s look at the Model object now, AFTER compiling:
[17]:
pprint(model.__dict__)
{'compile_dir': PosixPath('/glade/scratch/jamesmcc/wrfhydropy_end-to-end_example/compile'),
'compile_log': CompletedProcess(args=['/bin/bash', '-c', './configure ifort; ./compile_offline_NoahMP.sh /glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/compile_options.sh'], returncode=0, stdout=b'Configured: ifort\nconfigure: Sourcing /glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/compile_options.sh for the compile options.\nmake[1]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/CPL/NoahMP_cpl\'\nrm -f *.o *.mod *.stb *~ Noah_hrldas_beta\nmake[1]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/CPL/NoahMP_cpl\'\n(make -f Makefile.comm clean)\nmake[1]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS\'\n(cd IO; make -f Makefile clean)\nmake[2]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/IO\'\nrm -f *.o *.mod *.stb *~ *.f\nmake[2]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/IO\'\n(cd OrchestratorLayer; make -f Makefile clean)\nmake[2]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/OrchestratorLayer\'\nrm -f *.o *.mod *.stb *~ *.f\nmake[2]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/OrchestratorLayer\'\n(cd utils ; make -f Makefile clean)\nmake[2]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/utils\'\nrm -f *.o *.mod *.stb *~\nmake[2]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/utils\'\nmake -C Routing/Overland clean\nmake[2]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Routing/Overland\'\nrm -f *.o \nrm -f *.mod\nrm -f overland_tests\nmake[2]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Routing/Overland\'\nmake -C Routing/Subsurface clean\nmake[2]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Routing/Subsurface\'\nrm -f *.o\nrm -f *.mod\nrm -f subsurface_tests\nmake[2]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Routing/Subsurface\'\nmake -C Routing/Reservoirs clean\nmake[2]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Routing/Reservoirs\'\nrm -f *.o\nrm -f *.mod\nrm -f reservoir_tests\nmake -C Level_Pool clean\nmake[3]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Routing/Reservoirs/Level_Pool\'\nrm -f *.o\nrm -f *.mod\nmake[3]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Routing/Reservoirs/Level_Pool\'\nmake -C Persistence_Level_Pool_Hybrid clean\nmake[3]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Routing/Reservoirs/Persistence_Level_Pool_Hybrid\'\nrm -f *.o\nrm -f *.mod\nmake[3]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Routing/Reservoirs/Persistence_Level_Pool_Hybrid\'\nmake -C RFC_Forecasts clean\nmake[3]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Routing/Reservoirs/RFC_Forecasts\'\nrm -f *.o\nrm -f *.mod\nmake[3]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Routing/Reservoirs/RFC_Forecasts\'\nmake[2]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Routing/Reservoirs\'\n(cd Data_Rec; make -f Makefile clean)\nmake[2]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Data_Rec\'\nrm -f *.o *.mod *.stb *~\nmake[2]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Data_Rec\'\n(cd HYDRO_drv; make -f Makefile clean)\nmake[2]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/HYDRO_drv\'\nrm -f *.o *.mod *.stb *~\nmake[2]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/HYDRO_drv\'\n(cd MPP; make -f Makefile clean)\nmake[2]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/MPP\'\nrm -f *.o *.mod *.stb *~\nmake[2]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/MPP\'\nmake -C Debug_Utilities/ clean\nmake[2]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Debug_Utilities\'\nrm -f *.o *.mod *.stb *~\nmake[2]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Debug_Utilities\'\n(cd Routing; make -f Makefile clean)\nmake[2]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Routing\'\nrm -f *.o *.mod *.stb *~\nmake[2]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Routing\'\n(rm -f lib/*.a */*.mod */*.o CPL/*/*.o CPL/*/*.mod)\nmake[1]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS\'\nmake[1]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Land_models/NoahMP\'\n(cd Utility_routines;\t\tmake clean)\nmake[2]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Land_models/NoahMP/Utility_routines\'\nrm -f *.o *.mod *.stb *~\nmake[2]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Land_models/NoahMP/Utility_routines\'\n(cd phys;\t\t\tmake clean)\nmake[2]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Land_models/NoahMP/phys\'\nrm -f *.o *.mod *.stb *~\nmake[2]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Land_models/NoahMP/phys\'\n(cd data_structures;\t\tmake clean)\nmake[2]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Land_models/NoahMP/data_structures\'\nrm -f *.o *.mod *.stb *~ *.f\nmake[2]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Land_models/NoahMP/data_structures\'\n(cd IO_code;\t\t\tmake clean)\nmake[2]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Land_models/NoahMP/IO_code\'\nrm -f *.o *.mod *.stb *~\nmake[2]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Land_models/NoahMP/IO_code\'\n(cd run;\t\t\tmake clean)\nmake[2]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Land_models/NoahMP/run\'\nrm -f *~ hrldas.exe\nmake[2]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Land_models/NoahMP/run\'\nmake[1]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Land_models/NoahMP\'\nif [ 0 -eq 1 ]; then \\\n\t(cd Rapid_routing; make -f makefile.cpl clean); \\\nfi\n(rm -f */*.mod */*.o lib/*.a Run/wrf_hydro.exe)\n(rm -f Run/wrf_hydro.exe )\n(make -f Makefile.comm BASIC)\nmake[1]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS\'\nmake -C MPP\nmake[2]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/MPP\'\n\nrm -f CPL_WRF.o CPL_WRF.mod CPL_WRF.stb *~\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include -o CPL_WRF.o -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod CPL_WRF.F\nmpif90 -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -c CPL_WRF.F\nar -r ../lib/libHYDRO.a CPL_WRF.o\n\nrm -f mpp_land.o mpp_land.mod mpp_land.stb *~\nmpif90 -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -c mpp_land.F\nar -r ../lib/libHYDRO.a mpp_land.o\n\nrm -f module_mpp_ReachLS.o module_mpp_ReachLS.mod module_mpp_ReachLS.stb *~\nmpif90 -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -c module_mpp_ReachLS.F\nar -r ../lib/libHYDRO.a module_mpp_ReachLS.o\n\nrm -f module_mpp_GWBUCKET.o module_mpp_GWBUCKET.mod module_mpp_GWBUCKET.stb *~\nmpif90 -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -c module_mpp_GWBUCKET.F\nar -r ../lib/libHYDRO.a module_mpp_GWBUCKET.o\nmake[2]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/MPP\'\nmake -C IO\nmake[2]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/IO\'\n\nmpif90 -o netcdf_layer.o -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include -I../mod netcdf_layer.f90\n\nar -r ../lib/libHYDRO.a netcdf_layer.o\ncp *.mod ../mod\nmake[2]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/IO\'\nmake -C utils\nmake[2]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/utils\'\nUtils Makefile:\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -o module_version.o -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -DNWM_VERSION=\\"none\\" -DWRF_HYDRO_VERSION=\\"v5.1.0-beta2\\" -I./ -I ../../MPP -I ../MPP -I ../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_version.F\n\nar -r ../lib/libHYDRO.a module_version.o\ncp *.mod ../mod\nUtils Makefile:\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -o module_hydro_stop.o -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -DNWM_VERSION=\\"none\\" -DWRF_HYDRO_VERSION=\\"v5.1.0-beta2\\" -I./ -I ../../MPP -I ../MPP -I ../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_hydro_stop.F\n\nar -r ../lib/libHYDRO.a module_hydro_stop.o\ncp *.mod ../mod\nmake[2]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/utils\'\nmake -C OrchestratorLayer\nmake[2]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/OrchestratorLayer\'\nOrchestrator Makefile:\nmpif90 -fpp -o io_manager.o -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include io_manager.f90\n\nar -r ../lib/libHYDRO.a io_manager.o\ncp *.mod ../mod\nOrchestrator Makefile:\nmpif90 -fpp -o config.o -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include config.f90\n\nar -r ../lib/libHYDRO.a config.o\ncp *.mod ../mod\nOrchestrator Makefile:\nmpif90 -fpp -o orchestrator.o -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include orchestrator.f90\n\nar -r ../lib/libHYDRO.a orchestrator.o\ncp *.mod ../mod\nmake[2]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/OrchestratorLayer\'\nmake -C Routing/Overland\nmake[2]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Routing/Overland\'\n#Build each sub module then build the module that depends on all sub modules\t\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_overland_control.F\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_overland_streams_and_lakes.F\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_overland_routing_properties.F\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_overland_mass_balance.F\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_overland.F\nar -r ../../lib/libHYDRO.a module_overland_control.o\nar -r ../../lib/libHYDRO.a module_overland_streams_and_lakes.o\nar -r ../../lib/libHYDRO.a module_overland_routing_properties.o\nar -r ../../lib/libHYDRO.a module_overland_mass_balance.o\nar -r ../../lib/libHYDRO.a module_overland.o\ncp *.mod ../../mod\nmake[2]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Routing/Overland\'\nmake -C Routing/Subsurface\nmake[2]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Routing/Subsurface\'\n#Build each sub module then build the module that depends on all sub modules\nmpif90 -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I ../../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_subsurface_grid_transform.F\nmpif90 -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I ../../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_subsurface_properties.F\nmpif90 -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I ../../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_subsurface_state.F\nmpif90 -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I ../../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_subsurface.F\nmpif90 -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I ../../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_subsurface_static_data.F\nmpif90 -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I ../../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_subsurface_output.F\nmpif90 -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I ../../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_subsurface_input.F\nar -r ../../lib/libHYDRO.a module_subsurface_grid_transform.o\nar -r ../../lib/libHYDRO.a module_subsurface_properties.o\nar -r ../../lib/libHYDRO.a module_subsurface_state.o\nar -r ../../lib/libHYDRO.a module_subsurface.o\nar -r ../../lib/libHYDRO.a module_subsurface_static_data.o\nar -r ../../lib/libHYDRO.a module_subsurface_output.o\nar -r ../../lib/libHYDRO.a module_subsurface_input.o\ncp *.mod ../../mod\nmake[2]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Routing/Subsurface\'\nmake -C Routing/Reservoirs\nmake[2]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Routing/Reservoirs\'\n#Build each sub module then build the module that depends on all sub modules\nmpif90 -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I ../../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_reservoir_utilities.F\nmpif90 -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I ../../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_reservoir.F\nmpif90 -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I ../../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_reservoir_read_timeslice_data.F\nmpif90 -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I ../../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_reservoir_read_rfc_time_series_data.F\nar -r ../../lib/libHYDRO.a module_reservoir_utilities.o\nar -r ../../lib/libHYDRO.a module_reservoir.o\nar -r ../../lib/libHYDRO.a module_reservoir_read_timeslice_data.o\nar -r ../../lib/libHYDRO.a module_reservoir_read_rfc_time_series_data.o\ncp *.mod ../../mod\n#Build the modules\nmake -C Level_Pool\nmake[3]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Routing/Reservoirs/Level_Pool\'\n#Build each sub module then build the module that depends on all sub modules\nmpif90 -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I ../../../MPP -I ../../../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_levelpool_properties.F\nmpif90 -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I ../../../MPP -I ../../../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_levelpool_state.F\nmpif90 -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I ../../../MPP -I ../../../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_levelpool.F\nmpif90 -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I ../../../MPP -I ../../../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_levelpool_tests.F\nar -r ../../../lib/libHYDRO.a module_levelpool_properties.o\nar -r ../../../lib/libHYDRO.a module_levelpool_state.o\nar -r ../../../lib/libHYDRO.a module_levelpool.o\nar -r ../../../lib/libHYDRO.a module_levelpool_tests.o\ncp *.mod ../../../mod\nmake[3]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Routing/Reservoirs/Level_Pool\'\nmake -C Persistence_Level_Pool_Hybrid\nmake[3]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Routing/Reservoirs/Persistence_Level_Pool_Hybrid\'\n#Build each sub module then build the module that depends on all sub modules\nmpif90 -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I ../../../MPP -I ../../../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_persistence_levelpool_hybrid_properties.F\nmpif90 -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I ../../../MPP -I ../../../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_persistence_levelpool_hybrid_state.F\nmpif90 -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I ../../../MPP -I ../../../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_persistence_levelpool_hybrid.F\nmpif90 -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I ../../../MPP -I ../../../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_persistence_levelpool_hybrid_tests.F\nar -r ../../../lib/libHYDRO.a module_persistence_levelpool_hybrid_properties.o\nar -r ../../../lib/libHYDRO.a module_persistence_levelpool_hybrid_state.o\nar -r ../../../lib/libHYDRO.a module_persistence_levelpool_hybrid.o\nar -r ../../../lib/libHYDRO.a module_persistence_levelpool_hybrid_tests.o\ncp *.mod ../../../mod\nmake[3]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Routing/Reservoirs/Persistence_Level_Pool_Hybrid\'\nmake -C RFC_Forecasts\nmake[3]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Routing/Reservoirs/RFC_Forecasts\'\n#Build each sub module then build the module that depends on all sub modules\nmpif90 -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I ../../../MPP -I ../../../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_rfc_forecasts_properties.F\nmpif90 -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I ../../../MPP -I ../../../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_rfc_forecasts_state.F\nmpif90 -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I ../../../MPP -I ../../../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_rfc_forecasts.F\nmpif90 -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I ../../../MPP -I ../../../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_rfc_forecasts_tests.F\nar -r ../../../lib/libHYDRO.a module_rfc_forecasts_properties.o\nar -r ../../../lib/libHYDRO.a module_rfc_forecasts_state.o\nar -r ../../../lib/libHYDRO.a module_rfc_forecasts.o\nar -r ../../../lib/libHYDRO.a module_rfc_forecasts_tests.o\ncp *.mod ../../../mod\nmake[3]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Routing/Reservoirs/RFC_Forecasts\'\nmake[2]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Routing/Reservoirs\'\nmake -C Data_Rec\nmake[2]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Data_Rec\'\n\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include -o module_namelist.o -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I../mod module_namelist.F\n\nar -r ../lib/libHYDRO.a module_namelist.o\ncp *.mod ../mod\n\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include -o module_RT_data.o -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I../mod module_RT_data.F\n\nar -r ../lib/libHYDRO.a module_RT_data.o\ncp *.mod ../mod\n\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include -o module_gw_gw2d_data.o -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I../mod module_gw_gw2d_data.F\n\nar -r ../lib/libHYDRO.a module_gw_gw2d_data.o\ncp *.mod ../mod\nmake[2]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Data_Rec\'\nmake -C Debug_Utilities\nmake[2]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Debug_Utilities\'\n\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include -o debug_dump_variable.o -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I../mod debug_dump_variable.F\n\nar -r ../lib/libHYDRO.a debug_dump_variable.o\ncp *.mod ../mod\nmake[2]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Debug_Utilities\'\nmake -C Routing\nmake[2]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Routing\'\nRouting Makefile:\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -o module_date_utilities_rt.o -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_date_utilities_rt.F\n\nar -r ../lib/libHYDRO.a module_date_utilities_rt.o\ncp *.mod ../mod\nRouting Makefile:\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -o module_UDMAP.o -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_UDMAP.F\n\nar -r ../lib/libHYDRO.a module_UDMAP.o\ncp *.mod ../mod\nRouting Makefile:\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -o module_HYDRO_utils.o -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_HYDRO_utils.F\n\nar -r ../lib/libHYDRO.a module_HYDRO_utils.o\ncp *.mod ../mod\nRouting Makefile:\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -o module_noah_chan_param_init_rt.o -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_noah_chan_param_init_rt.F\n\nar -r ../lib/libHYDRO.a module_noah_chan_param_init_rt.o\ncp *.mod ../mod\nRouting Makefile:\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -o module_GW_baseflow.o -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_GW_baseflow.F\n\nar -r ../lib/libHYDRO.a module_GW_baseflow.o\ncp *.mod ../mod\nRouting Makefile:\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -o ../nudging/module_date_utils_nudging.o -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include ../nudging/module_date_utils_nudging.F\n\nar -r ../lib/libHYDRO.a ../nudging/module_date_utils_nudging.o\ncp *.mod ../mod\nRouting Makefile:\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -o ../nudging/module_nudging_io.o -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include ../nudging/module_nudging_io.F\n\nar -r ../lib/libHYDRO.a ../nudging/module_nudging_io.o\ncp *.mod ../mod\nRouting Makefile:\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -o module_HYDRO_io.o -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_HYDRO_io.F\n\nar -r ../lib/libHYDRO.a module_HYDRO_io.o\ncp *.mod ../mod\nRouting Makefile:\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -o module_gw_gw2d.o -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_gw_gw2d.F\n\nar -r ../lib/libHYDRO.a module_gw_gw2d.o\ncp *.mod ../mod\nRouting Makefile:\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -o module_RT.o -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_RT.F\n\nar -r ../lib/libHYDRO.a module_RT.o\ncp *.mod ../mod\nRouting Makefile:\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -o Noah_distr_routing.o -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include Noah_distr_routing.F\n\nar -r ../lib/libHYDRO.a Noah_distr_routing.o\ncp *.mod ../mod\nRouting Makefile:\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -o Noah_distr_routing_overland.o -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include Noah_distr_routing_overland.F\n\nar -r ../lib/libHYDRO.a Noah_distr_routing_overland.o\ncp *.mod ../mod\nRouting Makefile:\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -o Noah_distr_routing_subsurface.o -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include Noah_distr_routing_subsurface.F\n\nar -r ../lib/libHYDRO.a Noah_distr_routing_subsurface.o\ncp *.mod ../mod\nRouting Makefile:\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -o ../nudging/module_nudging_utils.o -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include ../nudging/module_nudging_utils.F\n\nar -r ../lib/libHYDRO.a ../nudging/module_nudging_utils.o\ncp *.mod ../mod\nRouting Makefile:\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -o ../nudging/module_stream_nudging.o -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include ../nudging/module_stream_nudging.F\n\nar -r ../lib/libHYDRO.a ../nudging/module_stream_nudging.o\ncp *.mod ../mod\nRouting Makefile:\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -o module_channel_routing.o -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_channel_routing.F\n\nar -r ../lib/libHYDRO.a module_channel_routing.o\ncp *.mod ../mod\nRouting Makefile:\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -o module_lsm_forcing.o -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_lsm_forcing.F\n\nar -r ../lib/libHYDRO.a module_lsm_forcing.o\ncp *.mod ../mod\nRouting Makefile:\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -o module_NWM_io_dict.o -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_NWM_io_dict.F\n\nar -r ../lib/libHYDRO.a module_NWM_io_dict.o\ncp *.mod ../mod\nRouting Makefile:\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -o module_NWM_io.o -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_NWM_io.F\n\nar -r ../lib/libHYDRO.a module_NWM_io.o\ncp *.mod ../mod\nRouting Makefile:\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -o module_reservoir_routing.o -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_reservoir_routing.F\n\nar -r ../lib/libHYDRO.a module_reservoir_routing.o\ncp *.mod ../mod\nmake[2]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Routing\'\nmake -C HYDRO_drv\nmake[2]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/HYDRO_drv\'\n\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -o module_HYDRO_drv.o -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include -I../mod module_HYDRO_drv.F\n\nar -r ../lib/libHYDRO.a module_HYDRO_drv.o\ncp *.mod ../mod\nmake[2]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/HYDRO_drv\'\nmake[1]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS\'\nmake[1]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/CPL/NoahMP_cpl\'\n\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -o module_hrldas_HYDRO.o -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_hrldas_HYDRO.F\n\nar -r ../../lib/libHYDRO.a module_hrldas_HYDRO.o\ncp *.mod ../../mod\n\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -o hrldas_drv_HYDRO.o -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include hrldas_drv_HYDRO.F\n\nar -r ../../lib/libHYDRO.a hrldas_drv_HYDRO.o\ncp *.mod ../../mod\nmake[1]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/CPL/NoahMP_cpl\'\nif [ 0 -eq 1 ]; then \\\n\t(cd lib;rm -f librapid.a); \\\nfi\nif [ 0 -eq 1 ]; then \\\n\t(cd Rapid_routing; make -f makefile.cpl rapid); \\\nfi\nmake[1]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Land_models/NoahMP\'\n(cd Utility_routines;\t\tmake)\nmake[2]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Land_models/NoahMP/Utility_routines\'\n\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -D_HRLDAS_OFFLINE_ -o module_date_utilities.o -c -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian module_date_utilities.F\n\n\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -D_HRLDAS_OFFLINE_ -o module_model_constants.o -c -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian module_model_constants.F\n\n\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -D_HRLDAS_OFFLINE_ -o module_wrf_utilities.o -c -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian module_wrf_utilities.F\n\n\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -D_HRLDAS_OFFLINE_ -o kwm_string_utilities.o -c -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian kwm_string_utilities.F\n\nmake[2]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Land_models/NoahMP/Utility_routines\'\n(cd phys;\t\t\tmake)\nmake[2]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Land_models/NoahMP/phys\'\n\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -D_HRLDAS_OFFLINE_ -o module_sf_noahmplsm.o -c -I../Utility_routines -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian module_sf_noahmplsm.F\n\n\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -D_HRLDAS_OFFLINE_ -o module_sf_noahmp_glacier.o -c -I../Utility_routines -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian module_sf_noahmp_glacier.F\n\n\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -D_HRLDAS_OFFLINE_ -o module_sf_noahmp_groundwater.o -c -I../Utility_routines -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian module_sf_noahmp_groundwater.F\n\n\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -D_HRLDAS_OFFLINE_ -o module_sf_noahmpdrv.o -c -I../Utility_routines -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian module_sf_noahmpdrv.F\n\nmake[2]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Land_models/NoahMP/phys\'\n(cd data_structures;\t\tmake)\nmake[2]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Land_models/NoahMP/data_structures\'\nData structures Makefile:\nmpif90 -o state.o -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include -I../../../OrchestratorLayer state.f90\n\nar -r ../../../lib/libHYDRO.a state.o\ncp *.mod ../../../mod\nData structures Makefile:\nmpif90 -o forcing.o -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include -I../../../OrchestratorLayer forcing.f90\n\nar -r ../../../lib/libHYDRO.a forcing.o\ncp *.mod ../../../mod\nData structures Makefile:\nmpif90 -o parameters.o -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include -I../../../OrchestratorLayer parameters.f90\n\nar -r ../../../lib/libHYDRO.a parameters.o\ncp *.mod ../../../mod\nData structures Makefile:\nmpif90 -o geometry.o -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include -I../../../OrchestratorLayer geometry.f90\n\nar -r ../../../lib/libHYDRO.a geometry.o\ncp *.mod ../../../mod\nmake[2]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Land_models/NoahMP/data_structures\'\n(cd IO_code;\t\t\tmake)\nmake[2]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Land_models/NoahMP/IO_code\'\n\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -D_HRLDAS_OFFLINE_ -o module_hrldas_netcdf_io.o -c -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I ../MPP -I ../../../mod -I../Utility_routines -I./ -I ../../MPP -I ../MPP -I ../mod -I /glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_hrldas_netcdf_io.F\n\n\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -D_HRLDAS_OFFLINE_ -o module_NoahMP_hrldas_driver.o -c -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I./ -I ../../MPP -I ../MPP -I ../mod -I. \\\n-I../phys -I../Utility_routines -I../../../mod -I /glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include module_NoahMP_hrldas_driver.F\n\n\nmpif90 -fpp -DMPP_LAND -I ../Data_Rec -DWRF_HYDRO -DSPATIAL_SOIL -DWRF_HYDRO_NUDGING -D_HRLDAS_OFFLINE_ -o main_hrldas_driver.o -c -O2 -g -w -c -ftz -align all -fno-alias -fp-model precise -FR -convert big_endian -I ../MPP -I. \\\n-I../phys -I../Utility_routines -I../../../mod -I../../../MPP -I../data_structures -I /glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//include main_hrldas_driver.F\n\nmake[2]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Land_models/NoahMP/IO_code\'\n(cd run;\t\t\tmake)\nmake[2]: Entering directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Land_models/NoahMP/run\'\n\necho " -Wl,-rpath, -L -lhdf5 -lz"\n -Wl,-rpath, -L -lhdf5 -lz\nmpif90 -o hrldas.exe -I../IO_code -I../phys ../IO_code/main_hrldas_driver.o ../IO_code/module_hrldas_netcdf_io.o ../phys/module_sf_noahmpdrv.o ../phys/module_sf_noahmplsm.o ../phys/module_sf_noahmp_glacier.o ../phys/module_sf_noahmp_groundwater.o ../Utility_routines/module_wrf_utilities.o ../Utility_routines/module_model_constants.o ../Utility_routines/module_date_utilities.o ../Utility_routines/kwm_string_utilities.o ../IO_code/module_NoahMP_hrldas_driver.o -L../../../lib -lHYDRO -L/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5//lib -lnetcdff -lnetcdf \n\nmake[2]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Land_models/NoahMP/run\'\nmake[1]: Leaving directory \'/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Land_models/NoahMP\'\n\n*****************************************************************\nMake was successful\n\n*****************************************************************\nThe environment variables used in the compile:\nHYDRO_D=0\nNCEP_WCOSS=0\nNETCDF=/glade/u/apps/ch/opt/netcdf/4.7.3/intel/18.0.5/\nSPATIAL_SOIL=1\nWRF_HYDRO=1\nWRF_HYDRO_NUDGING=1\nWRF_HYDRO_RAPID=0\n', stderr=b'ar: creating ../lib/libHYDRO.a\n'),
'compile_options': {'HYDRO_D': 0,
'NCEP_WCOSS': 0,
'SPATIAL_SOIL': 1,
'WRFIO_NCD_LARGE_FILE_SUPPORT': 1,
'WRF_HYDRO': 1,
'WRF_HYDRO_NUDGING': 1,
'WRF_HYDRO_RAPID': 0},
'compile_options_config_file': 'compile_options.json',
'compiler': 'ifort',
'configure_log': None,
'git_hash': '5913c15572a90556f26702dd6ff1d16c7ffb63df',
'hrldas_namelist_config_file': 'hrldas_namelists.json',
'hrldas_namelists': {'noahlsm_offline': {'btr_option': 1,
'canopy_stomatal_resistance_option': 1,
'dynamic_veg_option': 4,
'forcing_timestep': 'NULL_specified_in_domain.json',
'frozen_soil_option': 1,
'glacier_option': 2,
'hrldas_setup_file': 'NULL_specified_in_domain.json',
'indir': 'NULL_specified_in_domain.json',
'khour': 144,
'noah_timestep': 3600,
'nsoil': 4,
'outdir': './',
'output_timestep': 3600,
'pcp_partition_option': 1,
'radiative_transfer_option': 3,
'restart_filename_requested': 'NULL_specified_in_domain.json',
'restart_frequency_hours': 24,
'rst_bi_in': 0,
'rst_bi_out': 0,
'runoff_option': 3,
'snow_albedo_option': 1,
'soil_thick_input': [0.1,
0.3,
0.6,
1.0],
'spatial_filename': 'NULL_specified_in_domain.json',
'split_output_count': 1,
'start_day': 'NULL_specified_in_domain.json',
'start_hour': 'NULL_specified_in_domain.json',
'start_min': 'NULL_specified_in_domain.json',
'start_month': 'NULL_specified_in_domain.json',
'start_year': 'NULL_specified_in_domain.json',
'supercooled_water_option': 1,
'surface_drag_option': 1,
'surface_resistance_option': 4,
'tbot_option': 2,
'temp_time_scheme_option': 3,
'zlvl': 10.0},
'wrf_hydro_offline': {'forc_typ': 'NULL_specified_in_domain.json'}},
'hydro_namelist_config_file': 'hydro_namelists.json',
'hydro_namelists': {'hydro_nlist': {'aggfactrt': 4,
'channel_option': 2,
'chanobs_domain': 1,
'chanrtswcrt': 1,
'chrtout_domain': 1,
'chrtout_grid': 1,
'compound_channel': True,
'dtrt_ch': 300,
'dtrt_ter': 10,
'dxrt': 250,
'frxst_pts_out': 0,
'geo_finegrid_flnm': 'NULL_specified_in_domain.json',
'geo_static_flnm': 'NULL_specified_in_domain.json',
'gw_restart': 1,
'gwbaseswcrt': 4,
'gwbuckparm_file': 'NULL_specified_in_domain.json',
'hydrotbl_f': 'NULL_specified_in_domain.json',
'igrid': 1,
'io_config_outputs': 0,
'io_form_outputs': 4,
'land_spatial_meta_flnm': 'NULL_specified_in_domain.json',
'lsmout_domain': 1,
'nsoil': 4,
'order_to_write': 1,
'out_dt': 60,
'outlake': 1,
'output_channelbucket_influx': 0,
'output_gw': 1,
'ovrtswcrt': 1,
'reservoir_persistence_usgs': False,
'restart_file': 'NULL_specified_in_domain.json',
'route_lake_f': 'NULL_specified_in_domain.json',
'route_link_f': 'NULL_specified_in_domain.json',
'rst_bi_in': 0,
'rst_bi_out': 0,
'rst_dt': 1440,
'rst_typ': 1,
'rstrt_swc': 0,
'rt_option': 1,
'rtout_domain': 1,
'split_output_count': 1,
'subrtswcrt': 1,
'sys_cpl': 1,
't0outputflag': 1,
'udmap_file': 'NULL_specified_in_domain.json',
'udmp_opt': 1,
'zsoil8': [-0.1, -0.4, -1.0, -2.0]},
'nudging_nlist': {'biaswindowbeforet0': False,
'invdisttimeweightbias': True,
'maxagepairsbiaspersist': 3,
'minnumpairsbiaspersist': 1,
'nlastobs': 480,
'noconstinterfbias': False,
'nudginglastobsfile': 'NULL_specified_in_domain.json',
'nudgingparamfile': 'NULL_specified_in_domain.json',
'persistbias': True,
'readtimesliceparallel': True,
'temporalpersistence': True,
'timeslicepath': 'NULL_specified_in_domain.json'}},
'model_config': 'nwm_ana',
'modules': b'\nCurrently Loaded Modules:\n 1) intel/18.0.5 3) impi/2018.'
b'4.274 5) ncarcompilers/0.5.0\n 2) netcdf/4.7.3 4) ncarenv/1.'
b'3 6) nccmp/1.8.2.1\n\n \n\n',
'object_id': 'c0bae457-6714-49f5-97c3-8317983ee791',
'pre_compile_cmd': None,
'source_dir': PosixPath('/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS'),
'table_files': [PosixPath('/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Run/GENPARM.TBL'),
PosixPath('/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Run/MPTABLE.TBL'),
PosixPath('/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Run/CHANPARM.TBL'),
PosixPath('/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Run/HYDRO.TBL'),
PosixPath('/glade/u/home/jamesmcc/WRF_Hydro/wrf_hydro_nwm_public/trunk/NDHMS/Run/SOILPARM.TBL')],
'version': 'v5.1.0-beta2\n',
'wrf_hydro_exe': PosixPath('/glade/scratch/jamesmcc/wrfhydropy_end-to-end_example/compile/wrf_hydro.exe')}
The above is a treasure trove of information about the “model” when problems arise. This information is key for doing reproducible science.
For some applications it is worth noting the table_files entry of the Model object. While there is no argument to the Model initialization specifying alternative table files, the ones supplied by the compile process can be altered after compile.
7. Job object
The concept of a job is formalized in wrfhydropy by the Job object and this can have surprising consequences compared to the relatively ad hoc notion used when running the model otherwise. The job has 2 main functions:
provide a time focused intervention into the configuration, including both the model- and domain- sides of the namelists. Without this we’d have to have a configuration for each period of time we want to run the model. As an additional convenience, restart and output file frequencies may also be controlled by the job.
specify how the model is to be executed by the system. This can vary depending on the flavor of MPI used to compile and execute the model and the number of processors to use.
Additional functionality:
execute user-defined commands immediately prior to and immediately after the model execution.
The arguments to instatiation of a Job object reveal its scope:
[18]:
help(wrfhydropy.Job)
Help on class Job in module wrfhydropy.core.job:
class Job(builtins.object)
| A Job represents run-time specific information for a given WRF-Hydro run. A Simulation
| consists of one or more jobs. For example, adding multiple Jobs can be used to split a
| Simulation into multiple runs to limit the wall-clock duration of each individual run.
|
| Methods defined here:
|
| __init__(self, job_id:str, model_start_time:Union[str, datetime.datetime]=None, model_end_time:Union[str, datetime.datetime]=None, restart_freq_hr:Union[int, dict]=None, output_freq_hr:Union[int, dict]=None, restart:bool=True, restart_file_time:Union[str, datetime.datetime, dict]=None, restart_dir:Union[str, pathlib.Path, dict]=None, exe_cmd:str=None, entry_cmd:str=None, exit_cmd:str=None)
| Instatiate a Job object.
| Args:
| job_id: A string identify the job
| model_start_time: The model start time to use for the WRF-Hydro model run. Can be
| a pandas.to_datetime compatible string or a pandas datetime object.
| model_end_time: The model end time to use for the WRF-Hydro model run. Can be
| a pandas.to_datetime compatible string or a pandas datetime object.
| restart_freq_hr: Restart write frequency, hours. Either an int or a dict. If int: Output
| write frequency, hours. If dict, must be of the form {'hydro': int, 'hrldas': int}
| which sets them independently. Non-positive values (those <=0) set the restart
| frequency for both models to -99999, which gives restarts at start of each month.
| output_freq_hr: Either an int or a dict. If int: Output write frequency, hours. If dict,
| must be of the form {'hydro': int, 'hrldas': int} which sets them independently.
| restart: Job is starting from a restart file. Use False for a cold start.
| restart_dir: The path in which to look for the restart files.
| restart_file_time: The time on the restart file, if not the same as the model_start_time.
| Eithera string (e.g. '2000-01-01 00') or a datetime object (datetime or pandas) or a dict
| the form {'hydro': date1, 'hrldas': date2} where dates are either strings or datetime
| objects.
| exe_cmd: The system-specific command to execute WRF-Hydro, for example 'mpirun -np
| 36 ./wrf_hydro.exe'. Can be left as None if jobs is added to a scheduler or if a
| scheduler is used in a simulation.
| entry_cmd: A command to run prior to executing WRF-Hydro, such as loading modules or
| libraries.
| exit_cmd: A command to run after completion of the job.
|
| clone(self, N) -> list
| Clone a job object N-times using deepcopy.
| Args:
| N: The number of time to clone the Job
| Returns:
| A list of Job objects
|
| pickle(self, path:str)
| Pickle job object to specified file path
| Args:
| path: The file path for pickle
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables (if defined)
|
| __weakref__
| list of weak references to the object (if defined)
|
| hrldas_namelist
|
| hrldas_times
|
| hydro_namelist
|
| hydro_times
|
| job_dir
| Path: Path to the run directory
|
| model_end_time
| datetime: The model time at the end of the execution.
|
| model_start_time
| datetime: The model time at the start of the execution.
A common example is a long model run which must be broken up into several jobs because of walltime limitations of schedulers. With wrfhydropy you can skip the non-reproducble interventions into the namelists in your run directory. Instead you can create and add multiple Jobs to your simulation before you run it. These jobs will run sequentially and, if you a Scheduler object, can be submitted to the queue as dependent jobs. This is shown below. Also shown below is how the provenance
and artifacts related to the individual jobs are be stored in subdirectories of the simulation run directory with distinct job names. This means that all jobs added to a Simulation need different job_ids.
For now we create 2 jobs which use the same execution command. The second job starts where the first job ends and looks for its restarts in the run directory.
[19]:
exe_cmd = 'mpirun -np 1 ./wrf_hydro.exe'
job_24hr = wrfhydropy.Job(
job_id='24hr',
exe_cmd = exe_cmd,
model_start_time = '2011-08-26',
model_end_time='2011-08-27',
restart_freq_hr=24,
output_freq_hr=1)
job_145hr = wrfhydropy.Job(
job_id='144hr',
exe_cmd = exe_cmd,
model_start_time = '2011-08-27',
model_end_time='2011-09-02',
restart_freq_hr=24,
output_freq_hr=1,
restart_dir='.')
8. Scheduler object
In this example, two run modes of wrfhydropy will be demonstrated: interactive and scheduled runs. A third run mode will be presented in a later example: teams runs. This latter mode is mean for executing lots of small jobs (each not exceeding a single node) simulataneously over multiple nodes (such as ensembles or forecasts).
Here we show the PBSCheyenne scheduler. PBS is notoriously different on different machines, so this object is tied to a specific machine. SLURM schedulers have been developed and used for wrfhydropy but need tested before they become part of the master branch. Please inquire if you need a SLRUM scheduler.
The scheduler takes common information and is described in its help.
[20]:
scheduler = wrfhydropy.schedulers.PBSCheyenne(
account='NRAL0017',
nproc=2,
nnodes=1,
ppn=1,
email_who='bob@dum.com',
email_when='abe',
queue='share',
walltime='00:10:00')
9. Simulation object: interactive run
We are finally ready to run the model! It is easy to assemble all the pieces into a Simulation object with its add() method. This is object-oriented compositon in action: the Simulation “has” the objects we add to it.
In this case we will not add the Scheduler which means we will have an interactive run.
[21]:
simulation = wrfhydropy.Simulation()
simulation.add(model)
simulation.add(domain)
simulation.add(job_24hr)
simulation.add(job_145hr)
At this point, our simulation object is constructed in only memory. While domain files exist on disk and we saw that the compiled model lives in a compile directory on disk, their combination in a Simulation object is currently something existing only in the imagination of python. Now we must get this Simulation on to disk so that the model execution can take place (the model does not run inside python). This is done by the Simulations compose() method. (This is perhaps a
confusing name for the method that writes the Simulation to disk since the add() method used immediately prior is also a kind of composition, in the object oriented programming sense). The compose() method requires being in an empty directory where the contents of the Simulation object are to be written to file.
[22]:
sim_dir = experiment_dir / 'simulation_interactive'
os.mkdir(sim_dir)
os.chdir(sim_dir)
simulation.compose()
Composing simulation into directory:'/glade/scratch/jamesmcc/wrfhydropy_end-to-end_example/simulation_interactive'
Getting domain files...
Making job directories...
Validating job input files
24hr
144hr
Model already compiled, copying files...
Simulation successfully composed
Note well: Before running the simulation, it is IMPERATIVE to look at the “composed”/on-disk run directory to verify its accuracy, there is no substitute for this human intervention to ensure correctness.
[23]:
ls -a
./ job_24hr/
../ .model_end_time
CHANPARM.TBL* MPTABLE.TBL*
FORCING@ nudgingLastObs.2011-08-26_00:00:00.nc@
GENPARM.TBL* NWM/
HYDRO_RST.2011-08-26_00:00_DOMAIN1@ RESTART.2011082600_DOMAIN1@
HYDRO.TBL* SOILPARM.TBL*
job_144hr/ wrf_hydro.exe@
First note the .model_end_time file.
[24]:
cat .model_end_time
2011-08-26 00:00:00
This indicates the valid time of the state of the model. There is no .model_start_time file, indicating that the model has not been run. We will revisit these files once we run the model.
Some users will find it shocking that there are no namelist files in the run directory. This is because the namelists are considered part of each job and therefore these are stashed in the individual job directories.
[25]:
ls job_24hr/
hydro.namelist namelist.hrldas
Without further ado,
[26]:
sim_exit_code = simulation.run()
Running job 24hr:
Wall start time: 2020-03-16 22:14:36
Model start time: 2011-08-26 00:00
Model end time: 2011-08-27 00:00
Running job 144hr:
Wall start time: 2020-03-16 22:14:45
Model start time: 2011-08-27 00:00
Model end time: 2011-09-02 00:00
[27]:
print(sim_exit_code)
0
Both jobs were successfully run, else there would be an error message and a non-zero exit code.
[28]:
cat .model_start_time; echo ''; cat .model_end_time; echo ''; ls -l --time-style=full-iso .model*
2011-08-27 00:00:00
2011-09-02 00:00:00
-rw-r--r-- 1 jamesmcc ncar 19 2020-03-16 22:15:36.917176000 -0600 .model_end_time
-rw-r--r-- 1 jamesmcc ncar 19 2020-03-16 22:14:45.215263000 -0600 .model_start_time
These files show us when the model last started and when the model last ended. If the model failed, there would not be a .model_end_time. The file times also show us how long the execution took. These are helpful diagnostic files produced by wrfhydropy which can aid the developer.
It is also worth looking again at a job directory.
[29]:
ls job_24hr/
24hr.stderr diag_hydro.00000 namelist.hrldas
24hr.stdout hydro.namelist WrfHydroJob_postrun.pkl
We find the provenance (namelists) and the artifacts (stdout, stderr, diag_hydro files) of the job preserved in the job directory.
10. Simulation object: scheduled run
We repeat the Simulation, except now we add a Scheduler. We can not copy the already run Simulation object because it has already been composed and run. But we can build a new, identical Simulation from the same components: object reuse in action!
[30]:
simulation_sched = wrfhydropy.Simulation()
simulation_sched.add(model)
simulation_sched.add(domain)
simulation_sched.add(job_24hr)
simulation_sched.add(job_145hr)
simulation_sched.add(scheduler)
As before, we need a empty directory into which we compose() the Simulation.
[31]:
sim_dir = experiment_dir / 'simulation_scheduled'
os.mkdir(sim_dir)
os.chdir(sim_dir)
simulation_sched.compose()
sched_exit_code = simulation_sched.run()
Composing simulation into directory:'/glade/scratch/jamesmcc/wrfhydropy_end-to-end_example/simulation_scheduled'
Getting domain files...
Making job directories...
Validating job input files
24hr
144hr
Model already compiled, copying files...
Simulation successfully composed
qsub_str: /bin/bash -c 'job_24hr=`qsub -h job_24hr/job_24hr.pbs`;job_144hr=`qsub -W depend=afterok:${job_24hr} job_144hr/job_144hr.pbs`;qrls ${job_24hr};'
Check the job queue:
[32]:
print(subprocess.run('qstat -u$USER', shell=True, stderr=subprocess.PIPE, stdout=subprocess.PIPE).stdout.decode('utf-8'))
chadmin1.ib0.cheyenne.ucar.edu:
Req'd Req'd Elap
Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----
1272477.chadmin jamesmcc shareex jupyterlab 23795 1 1 -- 06:00 R 00:05
1272530.chadmin jamesmcc shareex 144hr -- 1 1 -- 00:10 H --
1272529.chadmin jamesmcc shareex 24hr 31254 1 1 -- 00:10 R 00:00
1247120.chadmin jamesmcc regular target_ana 38568 3 108 -- 12:00 R 03:39
1247122.chadmin jamesmcc regular target_ana -- 3 108 -- 12:00 H --
1247126.chadmin jamesmcc regular target_ana -- 3 108 -- 12:00 H --
1247127.chadmin jamesmcc regular target_ana -- 3 108 -- 12:00 H --
1247129.chadmin jamesmcc regular target_ana -- 3 108 -- 12:00 H --
1247135.chadmin jamesmcc regular target_ana -- 3 108 -- 12:00 H --
1247138.chadmin jamesmcc regular target_ana -- 3 108 -- 12:00 H --
1247142.chadmin jamesmcc regular target_ana -- 3 108 -- 12:00 H --
1247148.chadmin jamesmcc regular target_ana -- 3 108 -- 12:00 H --
1247154.chadmin jamesmcc regular target_ana -- 3 108 -- 12:00 H --
1247157.chadmin jamesmcc regular target_ana -- 3 108 -- 12:00 H --
1247164.chadmin jamesmcc regular target_ana -- 3 108 -- 12:00 H --
1247171.chadmin jamesmcc regular target_ana -- 3 108 -- 12:00 H --
1247178.chadmin jamesmcc regular target_ana -- 3 108 -- 12:00 H --
1247181.chadmin jamesmcc regular target_ana -- 3 108 -- 12:00 H --
1247188.chadmin jamesmcc regular target_ana -- 3 108 -- 12:00 H --
1247193.chadmin jamesmcc regular target_ana -- 3 108 -- 12:00 H --
1247207.chadmin jamesmcc regular target_ana -- 3 108 -- 12:00 H --
1247215.chadmin jamesmcc regular target_ana -- 3 108 -- 12:00 H --
The 144hr job is held and is dependent on the running 24hr job. (Please ignore my other jobs present in the queue, if any).
11. Simulations on disk and in memory: collecting and opening output
As in the compose step, wrfhydropy users have to be aware of the gap between disk and what’s in memory in python. While the Simulations were run on disk, their output has yet to be seen by python.
[33]:
simulation_sched.output
[33]:
This simulation currently has no output. Run simulation.collect().
The message gives a hint as to what to do.
[34]:
simulation_sched.collect()
simulation_sched.output
[34]:
channel_rt: 168 files
channel_rt_grid: 0 files
chanobs: 168 files
lakeout: 168 files
gwout: 168 files
rtout: 168 files
ldasout: 168 files
restart_hydro: 8 files
restart_lsm: 8 files
restart_nudging: 8 files
The __repr__ shows how many files of each output type were found on disk. One can see the list of files by
[35]:
simulation_sched.output.chanobs[0:5] # just look at the first 5
[35]:
[PosixPath('/glade/scratch/jamesmcc/wrfhydropy_end-to-end_example/simulation_scheduled/201108260100.CHANOBS_DOMAIN1'),
PosixPath('/glade/scratch/jamesmcc/wrfhydropy_end-to-end_example/simulation_scheduled/201108260200.CHANOBS_DOMAIN1'),
PosixPath('/glade/scratch/jamesmcc/wrfhydropy_end-to-end_example/simulation_scheduled/201108260300.CHANOBS_DOMAIN1'),
PosixPath('/glade/scratch/jamesmcc/wrfhydropy_end-to-end_example/simulation_scheduled/201108260400.CHANOBS_DOMAIN1'),
PosixPath('/glade/scratch/jamesmcc/wrfhydropy_end-to-end_example/simulation_scheduled/201108260500.CHANOBS_DOMAIN1')]
The list of files can be converted to an xarray dataset:
[36]:
simulation_sched.output.open('chanobs')
n_files 168
Note that this may be an intensive or impossible operation for lots of files or for files with lots of coordinate data. The collection of output data using xarray will be treated in a future example, including cases of large datasets. In our example, the output list now reflects that the “chanobs” files have been brought in to an xarray Dataset.
[37]:
simulation_sched.output
[37]:
channel_rt: 168 files
channel_rt_grid: 0 files
chanobs:
<xarray.Dataset>
Dimensions: (feature_id: 4, time: 168)
Coordinates:
latitude (feature_id) float32 41.470795 41.473614 41.449814 41.40192
longitude (feature_id) float32 -73.76059 -73.69085 -73.73565 -73.68741
* feature_id (feature_id) int32 6226948 6226964 6227008 6227150
* time (time) datetime64[ns] 2011-08-26T01:00:00 ... 2011-09-02
Data variables:
crs (time) |S1 b'' b'' b'' b'' b'' b'' ... b'' b'' b'' b'' b'' b''
order (time, feature_id) int32 3 2 4 4 3 2 4 4 3 ... 4 3 2 4 4 3 2 4 4
elevation (time, feature_id) float32 180.48 183.15 ... 164.32 147.61
streamflow (time, feature_id) float32 0.16395454 0.0637129 ... 1.0080798
Attributes:
featureType: timeSeries
proj4: +proj=lcc +units=m +a=6370000.0 +b=6370000.0 +lat_1=3...
station_dimension: feature_id
Conventions: CF-1.6
lakeout: 168 files
gwout: 168 files
rtout: 168 files
ldasout: 168 files
restart_hydro: 8 files
restart_lsm: 8 files
restart_nudging: 8 files
[38]:
simulation_sched.output.chanobs
[38]:
<xarray.Dataset>
Dimensions: (feature_id: 4, time: 168)
Coordinates:
latitude (feature_id) float32 41.470795 41.473614 41.449814 41.40192
longitude (feature_id) float32 -73.76059 -73.69085 -73.73565 -73.68741
* feature_id (feature_id) int32 6226948 6226964 6227008 6227150
* time (time) datetime64[ns] 2011-08-26T01:00:00 ... 2011-09-02
Data variables:
crs (time) |S1 b'' b'' b'' b'' b'' b'' ... b'' b'' b'' b'' b'' b''
order (time, feature_id) int32 3 2 4 4 3 2 4 4 3 ... 4 3 2 4 4 3 2 4 4
elevation (time, feature_id) float32 180.48 183.15 ... 164.32 147.61
streamflow (time, feature_id) float32 0.16395454 0.0637129 ... 1.0080798
Attributes:
featureType: timeSeries
proj4: +proj=lcc +units=m +a=6370000.0 +b=6370000.0 +lat_1=3...
station_dimension: feature_id
Conventions: CF-1.612. Engineering test
To demonstrate working with simulation data within wrfhydropy using an xarray approach, let’s just compare the “chanobs” output from the two runs, interactive and scheduled. Since these were composed (in an OO sense) from the same parts, we expect identical output.
[39]:
simulation.collect()
simulation.output.open('chanobs')
simulation.output.chanobs.equals(simulation_sched.output.chanobs)
n_files 168
[39]:
True
13. Evaluation syntax: goodness of fit
A further demonstration of working with model output within wrfhydropy and while using an xarray approach is to calculate the goodness of fit stats. Since we know the results are identical, we expect perfect statistics in the evaulation.
The syntax here is that the collected model output files, an xarray.Dataset:simulation.output.chanobs, has a variable, an xarray.DataArray:streamflow. This data array has the attribute eval which has the method obs() which accepts an xarray.DataArray of observations.
[40]:
chanobs_eval = simulation.output.chanobs.streamflow.eval.obs(simulation_sched.output.chanobs.streamflow)
The above is simply “syntactic sugar” for the wrfhydropy.Evaluation object so that the evaluation can be chained to model output.
[41]:
help(wrfhydropy.Evaluation)
Help on class Evaluation in module wrfhydropy.core.evaluation:
class Evaluation(builtins.object)
| A dataset consisting of a modeled and observed dataframe.
| This class provides methods for calculating staistics.
|
| Methods defined here:
|
| __init__(self, observed:Union[pandas.core.frame.DataFrame, xarray.core.dataarray.DataArray], modeled:Union[pandas.core.frame.DataFrame, xarray.core.dataarray.DataArray], join_on:Union[list, str]=None, join_how:str='inner')
| Instantiate analysis class by joining modeled and observed datasets.
| Args:
| observed: Dataframe containing observed data
| modeled: Dataframe containing modelled data
| join_on: Optional, string or list of columns names to join datasets.
| Default is ['feature_id','time']
| join_how: Optional, how to perform teh dataframe join. Default is
| 'inner'. Options
| are 'inner','left','right'.
|
| brier(self, threshold:float, mod_col:str='modeled', obs_col:str='observed', time_col:str='time', weights=None)
| Calculate Brier score using the properscoring package.
| See :py:fun:`threshold_brier_score() <threshold_brier_score>`
| in :py:mod:`properscoring`.
| Grouping is not necessary because BRIER returns a value per forecast.
| Grouping would happen when computing BRIERS.
| The Eval object generally wants one observation per modeled data point,
| that is overkill for this function but we handle it in a consistent manner
| with the rest of Evaluation.
| Args:
| mod_col: Column name of modelled data
| obs_col: Column name of observed data.
| Returns:
| BRIER for each ensemble forecast against the observations.
|
| contingency(self, threshold:Union[float, str], time_window:str=None, mod_col:str='modeled', obs_col:str='observed', group_by:Union[list, str]=None, inf_as_na:bool=True, decimals:int=2)
| Calculate contingency statistics
|
| Accuracy (acc, fraction correct) = (hits + correct_negatives)/total
| Bias score (bias, frequency bias) = (hits + false_alarms)/(hits +
| misses)
| Probability of detection (pod, hit rate) = hits/(hits + misses)
| False alarm ratio (far) = false_alarms/(hits + false_alarms)
| Probability of false detection (pofd, false alarm rate) =
| false_alarms/(false_alarms + correct_negatives)
| Success ratio (sr) = hits/(hits + false_alarms)
| Critical success index (csi, threat score) = hits/(hits + misses +
| false_alarms)
| Gilbert skill score (gss, Equitable threat score) =
| (hits - hits_random)/(hits + misses +
| false_alarms - hits_random) where hits_random = ((hits + misses)*(hits +
| false_alarms))/total
| Hanssen and Kuipers discriminant (hk, Peirce's skill score) = (hits/(
| hits+misses)) - (false_alarms/(false_alarms + correct_negatives))
| Odds ratio (or) = (POD/(1-POD))/(POFD/(1-POFD))
| Odds ratio skill score (orss, Yule's Q) = ((hits * correct_negatives) -
| (misses * false_alarms))/((hits * correct_negatives) +
| (misses * false_alarms)
|
| Args:
| threshold: The threshold value for contingency stats or column name
| in self.data containing threshold value. The first value of the
| column will be used as the threshold value.
| TODO JLM: I Do NOT love an entire column where a single value is
| used. I guess this allows different thresholds for
| different groups within the data.frame.
|
| time_window: Calculate contingency statistics over a moving
| time window of specified width in seconds ('s'), hours ('h'),
| or days('d').
| mod_col: Column name of modelled data
| obs_col: Column name of observed data
| group_by: Column names to group by prior to calculating statistics
| inf_as_na: convert inf values to na?
| decimals: round stats to specified decimal places
| Returns:
| Pandas dataframe containing contingency statistics
|
| crps(self, mod_col:str='modeled', obs_col:str='observed', member_col:str='member', valid_time_col:str='valid_time', lead_time_col:str='lead_time', gage_col:str='gage', weights=None)
| Calculate CRPS (continuous ranked probability score) using the properscoring package.
| See :py:fun:`crps_ensemble() <crps_ensemble>`
| in :py:mod:`properscoring`.
|
| Grouping is not necessary because CRPS returns a value per forecast.
| Grouping would happen when computing CRPSS.
|
| The Eval object generally wants one observation per modeled data point,
| that is overkill for this function (since the ensemble takes one observations)
| but we handle it in a consistent manner with the rest of Evaluation.
|
| This function is setup to identify the ensemble dimension in the following way:
| 1. if "member_col" is present in the columns, then this is the ensemble dimension,
| which is a standard ensemble forecast way
| 2. else, the "valid_time" dimension is used. This is the time-lagged ensembles way.
| 3. NOT DONE: one could consider time-lagged ensembles of ensemble forecasts.
|
| Args:
| mod_col: str = 'modeled': Column name of modelled data
| obs_col: str = 'observed': Column name of observed data.
| member_col: str = 'member': Column name giving the members. If the column is present,
| evaluation is performed across the member dimension for each combination of
| other columns. If member is not present the valid_time lead_time and gage cols
| are used to calculate CRPS across lead-time for each valid_time, gage combination.
| This later option is the "timelagged" ensemble verification.
| valid_time_col: str = 'valid_time': I
| lead_time_col: str = 'lead_time',
| gage_col: str = 'gage',
|
| Returns:
| CRPS for each ensemble forecast against the observations.
|
| event(self, threshold:Union[float, str], mod_col:str='modeled', obs_col:str='observed', group_by:Union[list, str]=None, decimals:int=2)
| TODO: HUH? this is the same description as gof but returns a contingency table?
| Calculate goodness of fit statistics using the spotpy package.
| See :py:fun:`calculate_all_functions() <calculate_all_functions>`
| in :py:mod:`spotpy`.
| Args:
| mod_col: Column name of modelled data
| obs_col: Column name of observed data
| group_by: Column names to group by prior to calculating statistics
| decimals: round stats to specified decimal places
| threshold: Threshold value for high flow event or
| column name containing threshold value.
| dataframe
| Returns:
| Pandas dataframe containing contingency table
|
| gof(self, mod_col:str='modeled', obs_col:str='observed', group_by:Union[list, str]=None, inf_as_na:bool=True, decimals:int=2)
| Calculate goodness of fit statistics using the spotpy package.
| See :py:fun:`calculate_all_functions() <calculate_all_functions>`
| in :py:mod:`spotpy`.
| Args:
| mod_col: Column name of modelled data
| obs_col: Column name of observed data
| group_by: Column names to group by prior to calculating statistics
| inf_as_na: convert inf values to na?
| decimals: round stats to specified decimal places
| Returns:
| Pandas dataframe containing contingency table
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables (if defined)
|
| __weakref__
| list of weak references to the object (if defined)
Note there are other methods on the evaluation object. We call gof():
[42]:
chanobs_eval.gof()
[42]:
| value | |
|---|---|
| statistic | |
| agreementindex | 1.00 |
| bias | -0.00 |
| correlationcoefficient | 1.00 |
| covariance | 118.84 |
| decomposed_mse | 0.00 |
| kge | 1.00 |
| log_p | -0.92 |
| lognashsutcliffe | 1.00 |
| mae | 0.00 |
| mse | 0.00 |
| nashsutcliffe | 1.00 |
| pbias | 0.00 |
| rmse | 0.00 |
| rrmse | 0.00 |
| rsquared | 1.00 |
| rsr | 0.00 |
| volume_error | 0.00 |
| mean_obs | 7.29 |
| median_obs | 0.05 |
| std_obs | 10.91 |
| sample_size | 672.00 |
The above evaluation is across all space (gages) and time. Typically, a by-gage analysis is desired:
[43]:
import pandas as pd
gof_gages = chanobs_eval.gof(group_by='feature_id')
with pd.option_context('display.max_rows', None, 'display.max_columns', None):
print(gof_gages)
value
feature_id statistic
6226948 agreementindex 1.00
bias -0.00
correlationcoefficient 1.00
covariance 33.06
decomposed_mse 0.00
kge 1.00
log_p -0.92
lognashsutcliffe 1.00
mae 0.00
mse 0.00
nashsutcliffe 1.00
pbias 0.00
rmse 0.00
rrmse 0.00
rsquared 1.00
rsr 0.00
volume_error 0.00
mean_obs 4.67
median_obs 0.14
std_obs 5.77
sample_size 168.00
6226964 agreementindex 1.00
bias -0.00
correlationcoefficient 1.00
covariance 3.21
decomposed_mse 0.00
kge 1.00
log_p -0.92
lognashsutcliffe 1.00
mae 0.00
mse 0.00
nashsutcliffe 1.00
pbias 0.00
rmse 0.00
rrmse 0.00
rsquared 1.00
rsr 0.00
volume_error 0.00
mean_obs 1.12
median_obs 0.05
std_obs 1.80
sample_size 168.00
6227008 agreementindex 1.00
bias -0.00
correlationcoefficient 1.00
covariance 108.42
decomposed_mse 0.00
kge 1.00
log_p -0.92
lognashsutcliffe 1.00
mae 0.00
mse 0.00
nashsutcliffe 1.00
pbias 0.00
rmse 0.00
rrmse 0.00
rsquared 1.00
rsr 0.00
volume_error 0.00
mean_obs 8.45
median_obs 0.33
std_obs 10.44
sample_size 168.00
6227150 agreementindex 1.00
bias -0.00
correlationcoefficient 1.00
covariance 226.27
decomposed_mse 0.00
kge 1.00
log_p -0.92
lognashsutcliffe 1.00
mae 0.00
mse 0.00
nashsutcliffe 1.00
pbias 0.00
rmse 0.00
rrmse 0.00
rsquared 1.00
rsr 0.00
volume_error 0.00
mean_obs 14.92
median_obs 1.01
std_obs 15.09
sample_size 168.00
As mentioned, one goal of wrfhydropy is to provide an off-ramp from python to other languages. We can illustrate that here. Note that gof_gages is a pandas.dataframe. We can convert that easily to xarray and then export to NetCDF format.
[44]:
gof_gages_ds = gof_gages.to_xarray()
gof_gages_ds
[44]:
<xarray.Dataset>
Dimensions: (feature_id: 4, statistic: 21)
Coordinates:
* feature_id (feature_id) int64 6226948 6226964 6227008 6227150
* statistic (statistic) object 'agreementindex' 'bias' ... 'sample_size'
Data variables:
value (feature_id, statistic) float64 1.0 -0.0 1.0 ... 15.09 168.0[45]:
ret = gof_gages_ds.to_netcdf('chanobs_gof_gages.nc')
[46]:
%%bash
ncdump -h chanobs_gof_gages.nc
netcdf chanobs_gof_gages {
dimensions:
feature_id = 4 ;
statistic = 21 ;
variables:
int64 feature_id(feature_id) ;
string statistic(statistic) ;
double value(feature_id, statistic) ;
value:_FillValue = NaN ;
}
Then the data could be opened elsewhere, or appended to other data, etc. In R, the rwrfhydro::xr_datatable function is useful for reading xarray/NetCDF datasets, particularly when they involve a time dimension.
14. Plot
By no means is this an endorsement of matplotlib, but it is the most “canned” plotting option that works on xarray.DataArrays.
[48]:
simulation.output.chanobs.streamflow.plot(x='time', hue='feature_id', aspect=2, size=8)
[48]:
[<matplotlib.lines.Line2D at 0x2b4cb691f3c8>,
<matplotlib.lines.Line2D at 0x2b4cb69eb278>,
<matplotlib.lines.Line2D at 0x2b4cb69eb1d0>,
<matplotlib.lines.Line2D at 0x2b4cb69eb390>]
[ ]:
# Join in routelink gages and hue=gages plot?
Another option is to plot in R reading netcdf files in using rwrfhydro::xr_datatable and ggplot2.
[ ]:
# plot the domain?