MOSAIK

Timestamp:

Jan 29, 2020 9:21:52 PM (5 years ago)

Author:

suehring

Comment:

--

project/results

@@ -57 +57 @@
 || Nested simulation || 02/01/2020 ||  || Simulations repeatedly hang / crash. The Lustre system in Berlin is still not full setup so that simulations repeatedly  hang / crash due to slow filesystem. ||
 || Nested simulation - initial run || 03/01/2020 ||  || Lustre filesystem issues seem to be solved for now. Initialization of the child has been changed. Child is now initialized via dynamic driver rather than via the coupler. This way all atmosphere grid points are initialized appropriately. The nested simulation is at t=30min. '''First estimate of duration''': in 12 h real time on 6720 cores we will simulate about 1 h. With 30 hrs simulation time (00:00:00 UTC - 06:00:00 UTC, next day), we will need about 30 restarts. Since the machine in Berlin starts to fill up now with other users, we are only be able to do 1 simulation at a day (optimistic scenario), so this will take at least one month. ||
-|| Nested simulation - restart run || 09/01/2020 ||  || Simulation crashes in reading the restart data. Bugtracing is on its way. (Error not reproducible, might possibly be connected to file-system issues on HLRN.) ||
-|| Nested simulation - restart run || 27/01/2020 ||  || After recurrent maintenance-related breaks on HLRN, restart simulation started again, restart data was read in, but simulation crashed with a HDF 5 error while reading the Netcdf input data. ||
+|| Nested simulation - restart run || 09/01/2020 ||  || Simulation crashes in reading the restart data for one PE in the child.  ||
+|| Nested simulation - restart run || 29/01/2020 ||  || After recurrent maintenance-related breaks on HLRN, restart simulation started again. Simulation alternately crashes either with a HDF 5 error in the parent or in reading the restart data. In the parent this happens while reading the Netcdf input data. At most of the ranks there is no problem with the NetCDF input, however, at some ranks the NF90_INQUIRE and NF90_INQUIRE_VARIABLE produces NetCDF error codes. In the child, the error is reproducible, even if the initial simulation is run again the problem occurs. This happens only at specific ranks. We will downscale the simulation to debug this more efficiently. ||
 
 ----