| 99 | 99 | ||linker_options ||Compiler options to be used to link the PALM executable. Typically, these are paths to libraries used by PALM, e.g. NetCDF, FFTW, MPI, etc. You may repeat the options that you have given with {{{compiler_options}}} here. See your local system documentation / software manuals for required path settings. Requirements differ from system to system and also depend on the respective libraries that you are using. See [wiki:doc/app/recommended_compiler_options] for specific path settings that we, the PALM group, are using on our computers. Be aware, that these settings probably will not work on your computer system. ||no default value || |
| 100 | 100 | ||'''hostfile''' ||'''Name of the hostfile that is used by MPI to determine the nodes on which the MPI processes are started.'''\\\\ {{{palmrun}}} automatically generates the hostfile if you set {{{auto}}}. All MPI processes will then be started on the node on which {{{palmrun}}} is executed. The real name of the hostfile will then be set to {{{hostfile}}} (instead of {{{auto}}}) and, depending on your local MPI implementation, you may have to give this name in the {{{execute_command}}}. MPI implementations on large computer centers often do not require to explicitly specify a hostfile (in such a case you can remove this line from the configuration file), or the batch systems provides a hostfile which name you may access via environment variables (e.g. {{{$PBS_NODEFILE}}}) and which needs to be given in the {{{execute_command}}}. Please see your local system / batch system documentation about the hostfile policy on your system. ||no default value || |
| 124 | | Strings in double curly brackets are interpreted as variables and are replaced by {{{palmrun}}} based on settings via specific {{{palmrun}}} options or settings in the environment variable section of the configurations file. From the given batch directives, {{{palmrun}}} generates a batch script (file), also called batch job, which is then submitted to the batch system. If you like to check this batch script, then run {{{palmrun}}} with additional option {{{-F}}}, which will write the batch script to file {{{jobfile.#####}}} in your current working directory, where {{{#####}}} is a 5-digit random number (which is part of the so-called job-id). A batch job will not be submitted. |
| | 124 | Strings in double curly brackets are interpreted as variables and are replaced by {{{palmrun}}} based on settings via specific {{{palmrun}}} options or settings in the environment variable section of the configurations file. From the given batch directives, {{{palmrun}}} generates a batch script (file), also called batch job, which is then submitted to the batch system using the submit command that has been defined by {{{submit_command}}} (see environment variable section above). If you like to check the generated batch script, then run {{{palmrun}}} with additional option {{{-F}}}, which will write the batch script to file {{{jobfile.#####}}} in your current working directory, where {{{#####}}} is a 5-digit random number (which is part of the so-called job-id). A batch job will not be submitted. |
| 130 | | ||job_id ||job name under which you can find the job in the respective job queue ||generated from {{{palmrun}}} option {{{-d}}} plus a 5-digit random number, separated by a dot, e.g. {{{palmrun -d abcde ...}}} may generate {{{abcde.12345}}} || |
| 131 | | ||cpu_hours \\ cpu_minutes \\ cpu_seconds \\ cputime ||cpu time requested by the job split in hours, minutes and seconds. {{{cputime}}} is the requested time in seconds. ||calculated from {{{palmrun}}} option {{{-t}}}, e.g. in the above example, {{{palmrun -t 3662 ...}}} will generate {{{1:1:2}}} || |
| 132 | | ||timestring |
| 133 | | ||nodes ||number of compute nodes requested by the job ||calculated from {{{palmrun}}} options {{{-X}}} (total number of cores to be used), {{{-T}}} (number of MPI tasks to be started on each node), and {{{-O}}} (number of OpenMP threads to be started by each MPI task). {{{nodes}}} is calculated as {{{cores / ( tasks_per_node * threads_per_task)}}}. {{{threads_per_task}}} is one in pure MPI applications. If {{{tasks_per_node * threads_per_task}}} is not an integral divisor of the total number of cores, less tasks/threads will run on the last node. || |
| 134 | | ||tasks_per_node ||. ||. || |
| 135 | | ||threads_per_task ||. ||. || |
| 136 | | ||cores ||. ||. || |
| 137 | | ||mpi_tasks ||. ||cores / threads_per_task || |
| 138 | | ||job_protocol_file ||. ||. || |
| 139 | | ||memory ||requested memory in MByte ||as given by {{{palmrun}}} option {{{-m}}} or as set in the configuration file via {{{%memory}}}. Option overwrites the setting in the configuration file. || |
| 140 | | ||queue ||. ||. || |
| | 130 | ||job_id ||Job name under which you can find the job in the respective job queue ||generated from {{{palmrun}}} option {{{-d}}} plus a 5-digit random number, separated by a dot, e.g. {{{palmrun -d abcde ...}}} may generate {{{abcde.12345}}} || |
| | 131 | ||cpu_hours \\ cpu_minutes \\ cpu_seconds \\ cputime ||cpu time requested by the job split in hours, minutes and seconds. {{{cputime}}} is the requested time in seconds. ||Calculated from {{{palmrun}}} option {{{-t}}}, e.g. in the above example, {{{palmrun -t 3662 ...}}} will generate {{{1:1:2}}} || |
| | 132 | ||timestring ||Requested CPU time in format hh:mm:ss ||calculated from {{{palmrun}}} option {{{-t}}} || |
| | 133 | ||tasks_per_node ||Number of MPI tasks to be started on each requested node ||as given by {{{palmrun}}} option {{{-T}}} || |
| | 134 | ||threads_per_task ||Number of OpenMP threads to be started by each MPI task ||as given by {{{palmrun}}} option {{{-O}}} || |
| | 135 | ||cores ||Total number of cores requested by the job ||as given by {{{palmrun}}} option {{{-X}}} || |
| | 136 | ||nodes ||Number of compute nodes requested by the job ||calculated from {{{palmrun}}} options {{{-X}}} (total number of cores to be used), {{{-T}}} (number of MPI tasks to be started on each node), and {{{-O}}} (number of OpenMP threads to be started by each MPI task). {{{nodes}}} is calculated as {{{cores / ( tasks_per_node * threads_per_task)}}}. {{{threads_per_task}}} is one in pure MPI applications. If {{{tasks_per_node * threads_per_task}}} is not an integral divisor of the total number of cores, less tasks/threads will run on the last node. || |
| | 137 | ||mpi_tasks ||Total number of MPI tasks to be started ||calculated as {{{cores / threads_per_task}}} || |
| | 138 | ||job_protocol_file ||Name of the file (including path) to which the job protocol is written ||generated from {{{palmrun}}} options {{{-h}}} and {{{-d}}} and the path set by environment variable {{{local_jobcatalog}}}. As an example, if {{{local_jobcatalog = /home/user/job_queue}}}, the call of {{{palmrun -d testrun -h mycluster ....}}} gives a job protocol file {{{home/user/job_queue/mycluster_testrun}}}. || |
| | 139 | ||memory ||Requested memory in MByte ||as given by {{{palmrun}}} option {{{-m}}} or as set in the configuration file via {{{%memory}}}. Option overwrites the setting in the configuration file. || |
| | 140 | ||queue ||Batch queue to which the job is submitted ||as given by {{{palmrun}}} option {{{-q}}}. If the option is omitted, a default queue defined by variable {{{default_queue}}} is used. || |
| | 149 | |
| | 150 | === additional directives for batch jobs on remote hosts=== |
| | 151 | |
| | 152 | If {{{palmrun}}} is used in remote batch mode, i.e. the batch job is submitted from your local computer to a remote computer, additional batch job directives are required to guarantee that the job protocol file is sent back to your local computer after the batch job has finished on the remote system. Since the job protocol file is often only available after the job has finished, a small additional job is started at the end of the batch job, which only purpose is to transfer the job protocol from the remote to the local system. Batch directives for this job are given in the configuration file too. Add the string {{{BDT:}}} at the beginning of each directive. As for the main job directives (that start with {{{BD:}}}), we can only give a general example here, which is again for an OpenPBS based batch system. |
| | 153 | {{{ |
| | 154 | BDT:#!/bin/bash |
| | 155 | BDT:#PBS -A {{project_account}} |
| | 156 | BDT:#PBS -N job_protocol_transfer |
| | 157 | BDT:#PBS -l walltime=00:30:00 |
| | 158 | BDT:#PBS -l nodes=1:ppn=1 |
| | 159 | BDT:#PBS -o {{job_transfer_protocol_file}} |
| | 160 | BDT:#PBS -j oe |
| | 161 | BDT:#PBS -q dataq |
| | 162 | }}} |
| | 163 | Keep in mind to request only few resources because this job just carries out a file transfer via scp. Computing centers often offer a special queue for these kind of jobs ({{{dataq}}} in the above example). The variable {{{job_transfer_protocol_file}}} is determined by {{{palmrun}}}. In case that you did not receive the job protocol, you may look into the protocol file of this transfer job. You can/should find this file under the name {{{last_job_transfer_protocol}}} on the remote host in the directory defined by {{{remote_jobcatalog}}}. A new job overwrites the transfer protocol of a previous job. |
| | 164 | |
| | 165 | |
| | 166 | |
