MOSAIK

Timestamp:

Feb 6, 2020 10:07:56 AM (4 years ago)

Author:

suehring

Comment:

--

project/results

@@ -58 +58 @@
 || Nested simulation - initial run || 03/01/2020 ||  || Lustre filesystem issues seem to be solved for now. Initialization of the child has been changed. Child is now initialized via dynamic driver rather than via the coupler. This way all atmosphere grid points are initialized appropriately. The nested simulation is at t=30min. '''First estimate of duration''': in 12 h real time on 6720 cores we will simulate about 1 h. With 30 hrs simulation time (00:00:00 UTC - 06:00:00 UTC, next day), we will need about 30 restarts. Since the machine in Berlin starts to fill up now with other users, we are only be able to do 1 simulation at a day (optimistic scenario), so this will take at least one month. ||
 || Nested simulation - restart run || 09/01/2020 ||  || Simulation crashes in reading the restart data for one PE in the child.  ||
-|| Nested simulation - restart run || 29/01/2020 ||  || After recurrent maintenance-related breaks on HLRN, restart simulation started again. Simulation alternately crashes either with a HDF 5 error in the parent or in reading the restart data. In the parent this happens while reading the Netcdf input data. At most of the ranks there is no problem with the NetCDF input, however, at some ranks the NF90_INQUIRE and NF90_INQUIRE_VARIABLE produces NetCDF error codes. In the child, the error is reproducible, even if the initial simulation is run again the problem occurs. This happens only at specific ranks. We will downscale the simulation to debug this more efficiently. '''(Un)fortunately these problems do not occur any more after HLRN runs more stable, so that the reason for these crashes cannot be traced back. '''||
+|| Nested simulation - restart run || 29/01/2020 ||  || After recurrent maintenance-related breaks on HLRN, restart simulation started again. Simulation alternately crashes either with a HDF 5 error in the parent or in reading the restart data. In the parent this happens while reading the Netcdf input data. At most of the ranks there is no problem with the NetCDF input, however, at some ranks the NF90_INQUIRE and NF90_INQUIRE_VARIABLE produces NetCDF error codes. In the child, the error is reproducible, even if the initial simulation is run again the problem occurs. This happens only at specific ranks. We will downscale the simulation to debug this more efficiently. '''(Un)fortunately these problems do not occur any more after HLRN runs more stable, so that the reason for these crashes cannot be traced back. ''' ||
 || Nested simulation || 06/02/2020 ||  || After several fixes on HLRN side, I started the whole simulation with debug prints again. Initial simulation did not show any problems. The following restart run also run fine, no problem with NF90_INQUIRE as well as with empty binary files. The second restart run is queued now. We are at t ~ 2940 s. ||