65 | 65 | || Nested simulation || 25/03/2020 || Simulation was running until exactly 05:00UTC. Crashed by floating overflow in the child domain. || Last restart time was at 04:55 UTC, flow fields, surface data look reasonable. Restarting from last restart step using traceback option and print statements revealed an floating overflow in output of averaged 3D variable 'theta' at grid point (k,j,i) = (97,117,968), which is far away from any building. Think this is also related to a restart problem where faulty data is read for pt_av. Proceeding without averaged data output worked. || |
66 | 66 | || Nested simulation || 01/04/2020 || Simulation has reached 06:05 UTC. || At the moment we are out of computing time. The IOP has been started, i.e. measurements are output. However, it turned out that the unstructured output of the virtual measurements consumes far too much CPU time at the moment. With smaller number of processes in test simulations this did not become obvious, however, with large number of processes the probability that IO processes interfere with each other becomes higher so that the slowdown of IO becomes more pronounced. First we need to accelerate the output before we can proceed. Moreover, with further debugging the reason of restart failures could be most probably narrowed down to file-system issues rather than palm-internal problems (sending trouble ticket to the computing center). || |