Changes between Version 10 and Version 11 of doc/app/runs

Timestamp:

Jul 4, 2017 9:20:48 AM (8 years ago)

Author:

raasch

Comment:

--

doc/app/runs

@@ -43 +43 @@
 = Handling of large binary restart- or output-files =
 
-In case of very large files, the copy of data from and to PALM's temporary working directory may need a long time. The CPU cores requested for the job run idle during that time and may consume significant amount of the job time without doing anything. The time required for copying can be spared by using a file link instead of copying the data.
+In case of very large files, the copy of data from and to '''mrun's''' temporary working directory may need a long time. The CPU cores requested for the job run idle during that time and may consume significant amount of the job time without doing anything. The time required for copying can be spared by using a file link instead of copying the data.
 {{{
    cp large_local_file  large_permanent file                                 # may take long time
    ln existing_large_local_TARGET_file  LINK_NAME_to_large_local_file        # is done immediately, i.e. requires almost no time
 }}}
-You can tell '''mrun''' to use {{{ln}}} instead of {{{cp}}} by giving the file attribute {{{ln}}} in the respective file connection statement:
+You can tell '''mrun''' to use {{{ln}}} instead of {{{cp}}} by giving the file attribute {{{ln}}} in the respective file connection statement, e.g.:
 {{{
-bla
+BININ   in:loc:lnpe  d3f       $base_data/$fname/RESTART  _d3d
+BINOUT  out:loc:lnpe restart   $base_data/$fname/RESTART  _d3d
 }}}
-However, performing a link requires that the link to a TARGET file with the name LINK_NAME must be located on the same physical file system as the TARGET file ... 
+However, performing a link requires that the link to a TARGET file with the name LINK_NAME must be located on the same physical file system as the TARGET file. If TARGET file and LINK_NAME are on different file systems, the TARGET file will be copied instead (and the advantage of using the {{{ln}}} command is lost).
+
+Most computing centers provide a file systems for fast I/O and this should be used as '''mrun's''' temporary working directory, which can be set in the configuration file by environment variable {{{tmp_user_catalog}}}. Since the LINK_NAME should be on the same file system, the user should provide a directory on that file system for storing the large files. Respective settings in the configuration file could be (example for Cray-XC40 at HLRN):
+{{{
+#
+# folder in which mrun's temporary working catalog is created (will be deleted after end of job)
+%tmp_user_catalog    /gfs2/work/<replace by username>     lccrayh parallel
+#
+# folder in which large binary files shall be stored
+%tmp_data_catalog    /gfs2/work/<replace by username>     lccrayh parallel
+#
+# file connection statements for restart files
+BININ   in:loc:lnpe  d3f       $tmp_data_catalog/$fname/RESTART  _d3d
+BINOUT  out:loc:lnpe restart   $tmp_data_catalog/$fname/RESTART  _d3d
+}}}
+Such fast file systems are generally not allowed to store files for a longer time, so the user has to take care for archiving himself.
+
+'''Attention:'''\\
+The {{{ln}}} file attribute and the above described method for storing large binary files has been introduced with revision number r2262. {{{mrun}}} does not create empty files/directories for restart files in the folder given with the file connection statement any more, as it was done with feature "fl" before.