Nesting

A multi-scale nesting system is available since r1763. It currently undergoes major testing and revisions. A detailed description will follow after the official release.

Job preparation

The following example provide a step-by-step tutorial on how to set up a nesting system with one parent (root) domain and two child domains.

1. Prepare parameter files for all domains, e.g. for run with topography and separate topography data for each domain:

   testrun_p3d (root/parent domain)
   testrun_02_p3d (child domain 1)
   testrun_03_p3d (child domain 2)
   testrun_topo
   testrun_02_topo
   testrun_03_topo
   

Note that numbering of the child domains must start with "02" as "01" is internally reserved for the root domain.

2. The nesting system is activated by specifying a &nestpar NAMELIST in the _p3d file of the root domain, e.g.:

    &nestpar domain_layouts = 'root domain',      1,  -1,  24,    0.0,    0.0,
                              'child domain 1',   2,   1,  24,  100.0,  100.0,
                              'child domain 2',   3,   2,  24,  200.0,  200.0,
             nesting_datatransfer_mode = 'mixed',
             nesting_mode = 'two-way',
   /
   

For details on the parameters, see nestpar.

3. Add file connection statements for all child domains to the .mrun.config file,

   #----------------------------------------------------------------------------
   # List of input-files for nested model runs
   #----------------------------------------------------------------------------
   PARIN_02                  in:job        nest#      $base_data/$fname/INPUT   _02_p3d
   PARIN_03                  in:job        nest#      $base_data/$fname/INPUT   _03_p3d
   TOPOGRAPHY_DATA_02        in:locopt     nest#      $base_data/$fname/INPUT   _02_topo
   TOPOGRAPHY_DATA_03        in:locopt     nest#      $base_data/$fname/INPUT   _03_topo
   #----------------------------------------------------------------------------
   # List of output-files for nested domains
   #----------------------------------------------------------------------------
   RUN_CONTROL_02     out:loc:tr   nest#     $base_data/$fname/MONITORING  _2_rc
   RUN_CONTROL_02     out:loc:tra  nestf     $base_data/$fname/MONITORING  _2_rc
   HEADER_02          out:loc:tr   nest#     $base_data/$fname/MONITORING  _2_header
   HEADER_02          out:loc:tra  nestf     $base_data/$fname/MONITORING  _2_header
   CPU_MEASURES_02    out:loc:tr   nest#     $base_data/$fname/MONITORING  _2_cpu
   CPU_MEASURES_02    out:loc:tra  nestf     $base_data/$fname/MONITORING  _2_cpu
   BINOUT_02          out:loc:flpe restart   $base_data/$fname/RESTART     _2_d3d
   #
   RUN_CONTROL_03     out:loc:tr   nest#     $base_data/$fname/MONITORING  _3_rc
   RUN_CONTROL_03     out:loc:tra  nestf     $base_data/$fname/MONITORING  _3_rc
   HEADER_03          out:loc:tr   nest#     $base_data/$fname/MONITORING  _3_header
   HEADER_03          out:loc:tra  nestf     $base_data/$fname/MONITORING  _3_header
   CPU_MEASURES_03    out:loc:tr   nest#     $base_data/$fname/MONITORING  _3_cpu
   CPU_MEASURES_03    out:loc:tra  nestf     $base_data/$fname/MONITORING  _3_cpu
   BINOUT_03          out:loc:flpe restart   $base_data/$fname/RESTART     _3_d3d
   #
   DATA_1D_PR_NETCDF_02    out:loc:tr pro#:prof   $base_data/$fname/OUTPUT _2_pr  nc
   DATA_1D_SP_NETCDF_02    out:loc:tr spo#:spof   $base_data/$fname/OUTPUT _2_sp  nc
   DATA_1D_TS_NETCDF_02    out:loc:tr tso#:tsof   $base_data/$fname/OUTPUT _2_ts  nc
   DATA_1D_PTS_NETCDF_02   out:loc:tr ptso#:ptsof $base_data/$fname/OUTPUT _2_pts nc
   DATA_2D_XY_NETCDF_02    out:loc:tr xyo#:xyof   $base_data/$fname/OUTPUT _2_xy  nc
   DATA_2D_XY_AV_NETCDF_02 out:loc:tr xyo#:xyof   $base_data/$fname/OUTPUT _2_xy_av  nc
   DATA_2D_XZ_NETCDF_02    out:loc:tr xzo#:xzof   $base_data/$fname/OUTPUT _2_xz  nc
   DATA_2D_XZ_AV_NETCDF_02 out:loc:tr xzo#:xzof   $base_data/$fname/OUTPUT _2_xz_av  nc
   DATA_2D_YZ_NETCDF_02    out:loc:tr yzo#:yzof   $base_data/$fname/OUTPUT _2_yz  nc
   DATA_2D_YZ_AV_NETCDF_02 out:loc:tr yzo#:yzof   $base_data/$fname/OUTPUT _2_yz_av  nc
   DATA_3D_NETCDF_02       out:loc:tr 3do#:3dof   $base_data/$fname/OUTPUT _2_3d  nc
   DATA_3D_AV_NETCDF_02    out:loc:tr 3do#:3dof   $base_data/$fname/OUTPUT _2_3d_av nc
   #
   DATA_1D_PR_NETCDF_03    out:loc:tr pro:prof   $base_data/$fname/OUTPUT _3_pr  nc
   DATA_1D_SP_NETCDF_03    out:loc:tr spo#:spof   $base_data/$fname/OUTPUT _3_sp  nc
   DATA_1D_TS_NETCDF_03    out:loc:tr tso#:tsof   $base_data/$fname/OUTPUT _3_ts  nc
   DATA_1D_PTS_NETCDF_03   out:loc:tr ptso#:ptsof $base_data/$fname/OUTPUT _3_pts nc
   DATA_2D_XY_NETCDF_03    out:loc:tr xyo#:xyof   $base_data/$fname/OUTPUT _3_xy  nc
   DATA_2D_XY_AV_NETCDF_03 out:loc:tr xyo#:xyof   $base_data/$fname/OUTPUT _3_xy_av  nc
   DATA_2D_XZ_NETCDF_03    out:loc:tr xzo#:xzof   $base_data/$fname/OUTPUT _3_xz  nc
   DATA_2D_XZ_AV_NETCDF_03 out:loc:tr xzo#:xzof   $base_data/$fname/OUTPUT _3_xz_av  nc
   DATA_2D_YZ_NETCDF_03    out:loc:tr yzo#:yzof   $base_data/$fname/OUTPUT _3_yz  nc
   DATA_2D_YZ_AV_NETCDF_03 out:loc:tr yzo#:yzof   $base_data/$fname/OUTPUT _3_yz_av  nc
   DATA_3D_NETCDF_03       out:loc:tr 3do#:3dof   $base_data/$fname/OUTPUT _3_3d  nc
   DATA_3D_AV_NETCDF_03    out:loc:tr 3do#:3dof   $base_data/$fname/OUTPUT _3_3d_av nc
   

Attention: To get NetCDF output from the children requires to set the d3par-parameter netcdf_data_format = 5 or 6 (parallel I/O support). Otherwise, the NetCDF files are created but will be empty (filled with zero data only).

4. Execute mrun and specify the nesting input files, i.e.,

   mrun -d testrun -r "d3# nest# ts# pr# ... tso# pro# ..." -X "72" ...
   

Make sure that the total number of PEs (here 72) the sum of the PEs allocated for the individual domains (3 x 24).