Home

Context Navigation

palmrun

Timestamp:: Nov 20, 2018 3:46:29 PM (6 years ago)
Author:: scharf
Comment:: --

Legend:

: Unmodified
: Added
: Removed
: Modified

doc/app/palmrun

-                      v43
+                      v44
 == [#=batch Batch mode] ==
+{{{#!comment
+Running PALM in batch mode requires additional manual settings in the configuration file, which will be explained ([wiki:doc/palm_config here]).
+}}}
+Large simulation set-ups usually cannot be run interactively, since the large amount of required resources (memory as well as cpu-time) are only provided through batch environments. {{{palmrun}}} supports two different ways to run PALM in batch mode. In both cases it creates a batch job, i.e. a file containing directives for a queuing-system plus commands to run PALM, which is then either submitted to your local computer or to a remote computer. Running PALM in batch mode requires you to manually modify and extend your configuration file {{{.palm.config....}}}, and that a batch system (e.g. PBS or ...) is installed on the respective computer.
+Large simulation set-ups usually cannot be run interactively, since the large amount of required resources (memory as well as cpu-time) are only provided through batch environments. {{{palmrun}}} supports two different ways to run PALM in batch mode. In both cases it creates a batch job, i.e. a file containing directives for a queuing-system plus commands to run PALM, which is then either submitted to your local computer or to a remote computer. Running PALM in batch mode requires you to manually modify and extend your [wiki:doc/palm_confi configuration file], and that a batch system (e.g. PBS, Slurm, ...) is installed on the respective computer.
 === [#=batch_local Running PALM in batch on a local computer] ===
 …
    palmrun  ....-b -c <configuration identifier>  -t <cputime>  -X <total number of cores>  -T <MPI tasks per node>  -q <queue>
 }}}
+where
+ * {{{<configuration identifier>}}} is the configuration file containing your batch mode settings
+ * {{{<cputime>}}} is the maximum CPU time (wall clock time) in seconds requested by the batch job
+ * {{{<total number of cores>}}} is the total number of CPU cores (not CPUs!) that shall be used for your run
+ * {{{<MPI tasks per node>}}} is the number of MPI tasks to be started on one node of the computer system. Typically, {{{<MPI tasks per node>}}} is chosen as the total number of CPU cores available on one node, e.g. if a node has two CPUs with 12 cores each, then {{{<MPI tasks per node> = 24}}}.
+ * {{{<queue>>> is the name of the batch job queue that you like to use. See your batch system documentation about available queues and keep in mind that usually each queue has special limits for requested resources.
+ * {{{<memory per core>}}} is the memory in MByte requested by each core
+The first option {{{-b}}} is required to tell {{{palmrun}}} to create a batch job running on the local computer.
+**Note:** The first option {{{-b}}} is required to tell {{{palmrun}}} to create a batch job running on the local computer!
 Before entering the above command, you need to add information to your configuration file. You may edit an existing file (.e.g. {{{.palm.config.default}}}) or create a new one (e.g. by copying the default file to e.g. {{{.palm.config.batch}}} and then editing the new file). In general, you can not use the same configuration file for running interactive jobs and batch jobs as well since different settings are required. Let's assume here that you have created a new file {{{.palm.config.batch}}}. Edit this file and add those batch directives required by your batch system.  Keep in mind that there is a wide variety of batch systems and that many computer centers introduce their own special settings for these systems. Please read the documentation of your respective batch system carefully in order to figure out the required settings for your system (e.g. to run an MPI job on multiple cores). The following lines give a minimum example for the portable batch system (PBS).
 …
 ||-M   ||" "  ||Makefile to compile the PALM code and utility programs. By default, the name of the makefile is {{{Makefile}}}, and it is expected to be in the folder that is given by variable {{{source_path}}} in the configuration file.  ||
 ||-O   ||1    ||OpenMP threads to be started per MPI task. Environment variable {{{OMP_NUM_THREADS}}} will be set to this value  ||
 ||-q   ||none ||name of the job queue to which batch jobs will be submitted  ||
+||-q   ||none ||name of the job queue to which batch jobs will be submitted. See your batch system documentation about available queues and keep in mind that usually each queue has special limits for requested resources. ||
 ||-r   ||test ||The name of the run given by {{{-r}}} tells {{{palmrun}}} to use the NAMELIST file {{{<run_identifier>_p3d}}} from {{{JOBS/<run_identifier>/INPUT}}}. It also determines folders and names of output files generated by PALM using informations from the default file configuration file {{{..../trunk/SCRIPTS/.palm.iofiles}}}. Chapter [wiki:doc/app/palm_iofiles PALM iofiles] explains the format of this file and how you can modify or extend it. ||
 ||-s   ||" "  ||List of subroutines (Fortran file names) from the SVN repository (under {{{.../trunk/SOURCES}}}) that shall be compiled for this run. Compiled files will be exclusively used for the run and not be put in the MAKE_DEPOSITORY. In case of {{{-s LM}}}, all files in the repository that have been modified by the used will be compiled.  ||
 ||-t   ||" "  ||cpu time request for batch jobs in seconds  ||
 ||-T   ||" "  ||MPI tasks per node request for batch jobs  ||
+||-t   ||" "  ||maximum CPU time (wall clock time) in seconds requested by the batch job  ||
+||-T   ||" "  ||number of MPI tasks to be started on one node of the computer system. Typically, {{{<MPI tasks per node>}}} is chosen as the total number of CPU cores available on one node, e.g. if a node has two CPUs with 12 cores each, then {{{<MPI tasks per node> = 24}}}. ||
 ||-v   ||---  ||Suppresses parts of {{{palmrun}}}'s terminal output and prevents {{{palmrun}}} queries  ||
 ||-V   ||---  ||Use existing {{{SOURCES_FOR_RUN_...}}} folder. Prevents {{{palmrun}}} from creating a new {{{SOURCES_FOR_RUN_...}}} folder. Use this option if you do not want the user interface files to be compiled again.  ||
 …
 ||-W   ||" "  ||Name (id) of a previous job. Can be used as variable {{{ {{previous_job}} }}} as part of job directives in the configuration file, in order to prevent the job to start before the specified previous job has been finished. The job name must be the one that have been given by the batch system.  ||
 ||-x   ||---  ||Causes {{{palmrun}}} to output excessive debug information for both interactive sessions as well as batch jobs.  ||
 ||-X   ||1    ||Total number of cores to be used for the run.  The argument should not be larger than the maximum number of cores available on your computer (except in case of hyperthreading), because that would usually slow down the performance significantly.||
+||-X   ||1    ||Total number of cores (not CPUs!) to be used for the run.  The argument should not be larger than the maximum number of cores available on your computer (except in case of hyperthreading), because that would usually slow down the performance significantly.||
 ||-y   ||---  ||Use file appendix {{{_O}}} for local PALM-I/O files in case of uncoupled ocean runs, e.g. if the run is a precursor run and files shall later be used for coupled atmosphere-ocean runs. ||
 ||-Y   ||" "  ||In case of a coupled atmosphere-ocean run, the parameter tells PALM how many cores shall be assigned to the atmosphere- and ocean-model, respectively. For example, in case of {{{-X 64  -Y "16 48""}}} 16 cores are assigned to the atmosphere model, and 48 cores to the ocean model.  ||