= '''This page is under construction! ''' = \\\\\\ = The PALM configuration file Running PALM with the {{{palmrun}}} script or compiling PALM with {{{palmbuild}}} requires a configuration file in the working directory from where the scripts are called. The configuration file contains information about * the compiler and compiler options to be used * paths to required libraries (e.g. MPI, netCDF, or fftw) * preprocessor directives to switch on/off special features of PALM * paths to be used for storing PALM's input and output files * paths where user-interface routines and job protocol files can be found * the execute command to be used for starting the PALM executable * settings of other UNIX environment variable * batch job directives * unix commands to be executed before / after the PALM code is started, or that should be carried out in case of errors that appeared during the run If PALM has been successfully installed by the automatic installer, the installer creates a configuration file named {{{.palm.config.default}}} in the working directory of the user (which by default is assumed to be {{{$HOME/palm/current_version}}}). This file should contain all relevant information to run PALM on the respective computer. The installer may fail for various reasons, e.g. because of mismatches in the user's system configuration. Problems may appear if the user has installed more than one FORTRAN compiler, and/or if the netCDF libraries have been generated with a different compiler than is used for compiling PALM. Furthermore, the automatic installer cannot be used on many larger computer systems where software (compilers, libraries) is managed via the {{{module}}} environment, or if you like to run batch jobs an a remote host. In all these cases, the user has to setup the configuration file manually. == How to generate a configuration file manually As a first step, copy the default template of the configuration file (which is part of the PALM download) to your working directory: {{{ cd $HOME/palm/current_version cp trunk/SCRIPTS/.palm.config.default . }}} === Configuration file format The configuration file is an ASCII file which may be opened with any editor. The format of the file is simple. Lines need to start either with {{{#}}}, {{{%}}}, {{{IC:}}}, {{{OC:}}}, {{{EC:}}}, {{{BD:}}}, or {{{BDT:}}}. Empty line are allowed too. * lines starting with a {{{#}}} in the first column are comment lines {{{ # this is a comment line }}} * lines starting with a {{{%}}} in the first column are defining unix environment variables used in {{{palmrun}}} and {{{palmbuild}}}. {{{ %var value of var }}} means that a variable named {{{var}}} with value {{{value of var}}} is created. There must be at least one blank between the variable name and its value. The value may contain an arbitrary number of blanks. The value can contain already defined variables: {{{ %default_folder /work/abcd %subfolder1 $default_folder/efgh }}} which means that {{{subfolder1}}} has the value {{{/work/abcd/efgh}}}. You can also use any variables that are already defined within the {{{palmrun}}}/{{{palmbuild}}} scripts. The most important one is {{{jobname}}}, which value is given with {{{palmrun}}} option {{{-d}}}, and which defines the so-called ''jobname''. This variable is mentioned here because it is used in the default configuration files for naming I/O files and to better sort I/O files from different PALM runs in an organized folder structure. Beside the value replacement using the UNIX shell syntax (i.e. {{{$abcd}}} gives the value of variable {{{abcd}}}), another way to insert values of environment variables is to write them in double curly brackets, i.e. {{{ {{abcd}} }}} will be replaced by the value of {{{abce}}}. This way of variable replacement is required for some special variables and in the job directives. * lines starting with {{{IC:}}} define any unix commands that are executed by {{{palmrun}}} just before the PALM code is started. A typical input command might be {{{ IC:ulimit -s unlimited }}} to increase the system's stack size (if this is not unlimited by default), which is required for PALM runs with larger memory demands. * lines starting with {{{OC:}}} define unix commands that are executed by {{{palmrun}}} just after the PALM code has stopped. For example, you may inform yourself about termination of the program by sending an email: {{{ OC:echo "PALM simulation $jobname has finished" | mailx username@email-address }}} * lines starting with {{{EC:}}} define unix commands that shall be executed in case that the PALM code or the {{{palmrun}}} script terminated because of any kind of error. You can restrict execution of error commands to specific kinds of error: {{{ EC:[[ ]] && error-command }}} * lines starting with {{{BD:}}} define directives that are required for batch jobs, i.e. if PALM shall be run in batch mode. Explanations for batch directives are given further below. * lines starting with {{{BDT:}}} define directives for an additional batch job that is required in case of running PALM in batch mode on a remote host. This additional job transfers the job protocol file of the main PALM job back to the local host. === environment variables and their meaning The [source:palm/trunk/SCRIPTS/.palm.config.default default template] contains settings for the minimum set of variables that always need to be defined, plus suggestions for other variables that you may need to uncomment, in case that you like/need to use them. The following table lists all relevant variables and their meaning. Default values are either those which are set in the scripts {{{palmrun}}} and {{{palmbuild}}} or which are suggested in the template file. ||='''Variable name''' =||='''meaning''' =||='''default value''' =|| |----------- ||base_directory ||Working directory from where {{{palmrun}}} or {{{palmbuild}}} is called. This folder must contain the configuration file {{{.palm.config....}}}. ||$HOME/palm/current_version || ||base_data ||Directory where the PALM I/O-files are stored. This variable is used in the file configuration file (see [source:palm/trunk/SCRIPTS/.palm.iofiles .palm.iofiles], and the [wiki:doc/app/palm_iofiles description] of this file). Since this variable is also used to determine file locations on remote hosts, it must use the {{{~}}} instead of {{{$HOME}}}. ||~/palm/current_version/JOBS || ||source_path ||Path to PALM's FORTRAN source files. This is the place where the automatic installer has put the download, or which has been defined in the user's {{{svn checkout}}} command. ||$HOME/palm/current_version/trunk/SOURCE || ||user_source_path ||Path to the [wiki:doc/app/userint user interface routines]. The variable {{{jobname}}} that is used in the default path is replaced by the argument given with {{{palmrun}}}-option {{{-d}}}. ||$base_directory/JOBS/$jobname/USER_CODE || ||fast_io_catalog ||Path to a file system with fast discs (if available). This folder is used so store the temporary catalog generated by {{{palmrun}}} during each run. It should also be used to store large I/O files (e.g. restart data or 3D-output) in order to reduce I/O time. This variable is used in the default {{{.palm.iofiles}}} for the restart data files. The folder must be accessible from all compute nodes, i.e. it must reside in a global file system. WARNING: {{{/tmp}}} will only work on single node systems! In case of batch jobs on remote hosts, the variable refers to a folder on the remote host. The variable has no default value and must be set by the user. ||no default value || ||local_jobcatalog ||Folder on the local host to store the batch job protocols. In case of batch jobs running on remote hosts, the job protocol will be created on the {{{remote_jobcatalog}}} and then be transferred via scp to the {{{local_jobcatalog}}}. ||$HOME/job_queue || ||remote_jobcatalog ||In case of batch jobs running on remote hosts, the job protocol will be put in this folder, and then automatically transferred via scp to the {{{local_jobcatalog}}}. The transfer is done by a separate small batch job, which directives are defined by the {{{BDT:}}} lines. The variable has no default value and must be set by the user. Absolute paths need to be given. Using {{{$HOME}}} is not allowed / does not work. ||no default value || ||local_ip ||IP-address of your local computer / the computer on which you call the {{{palmrun}}}/{{{palmbuild}}} command. You may use {{{127.0.0.0}}} if you are running PALM in interactive mode or in batch mode on your local computer. The address is only used to identify where to send the output data in case of batch jobs on a remote host. ||no default value || # your local UNIX username %local_username # ip address of the remote host #%remote_ip # username on the remote host #%remote_username # ssh-key to be used for ssh/scp calls to the remote host #%ssh_key ~/.ssh/id_rsa # name of login-node on the remote machine #%remote_loginnode # default queue to be used if palmrun-option -q is omitted #%defaultqueue # command to submit batch jobs #%submit_command /opt/moab/default/bin/msub -E # compilername to generate MPI executables %compiler_name mpif90 # compilername to generate non-MPI executables running on one core %compiler_name_ser ifort # preprocessor directives to be used for compiling the PALM code %cpp_options -cpp -D__parallel -DMPI_REAL=MPI_DOUBLE_PRECISION -DMPI_2REAL=MPI_2DOUBLE_PRECISION -D__fftw -D__netcdf # used for parallel compilation %make_options -j 4 # options to be used to compile PALM %compiler_options -openmp -fpe0 -O3 -xHost -fp-model source -ftz -fno-alias -no-prec-div -no-prec-sqrt -ip -nbs -I /muksoft/packages/fftw/3.3.4/include -L/muksoft/packages/fftw/3.3.4/lib64 -lfftw3 -I /muksoft/packages/netcdf/4_intel/include -L/muksoft/packages/netcdf/4_intel/lib -lnetcdf -lnetcdff # options to be used to link the PALM executable %linker_options -openmp -fpe0 -O3 -xHost -fp-model source -ftz -fno-alias -no-prec-div -no-prec-sqrt -ip -nbs -I /muksoft/packages/fftw/3.3.4/include -L/muksoft/packages/fftw/3.3.4/lib64 -lfftw3 -I /muksoft/packages/netcdf/4_intel/include -L/muksoft/packages/netcdf/4_intel/lib -lnetcdf -lnetcdff # name of hostfile to be used %hostfile auto # command to start the PALM executable %execute_command mpiexec -machinefile hostfile -n {{mpi_tasks}} palm # memory request per core #%memory 2300 # module commands to load required libraries #%module_commands module load fftw cray-hdf5-parallel cray-netcdf-hdf5parallel # special commands to be carried out at login and start of batch jobs on the remote host #%login_init_cmd module switch craype-ivybridge craype-haswell === batch job directives