| 13 | | * The old configuration file {{{.mrun.config}}} has been split into two files {{{.palm.config.<configuration_identifier>}}} and {{{.palm.iofiles}}}, where {{{<configuration_identifier>}}} is an arbitrary string that you can define. "Configuration" means a setting for a specific computer with a specific compiler, compiler options, libraries, etc. If you like to run {{{palm}}} with different configurations, e.g. one with debug options switched on, and one with high optimization, you need to create separate configuration files for each configuration, e.g. {{{.palm.config.optimized}}} and {{{.palm.config.debug}}}. This replaces the old block structure in {{{.mrun.config}}}. The configuration file to be used is defined by {{{palmrun}}}- or {{{palmbuild}}}-option {{{-h}}}., e.g. {{{palmrun ... -h optimized}}} will use {{{.palm.config.optimized}}} \\\\ You will need only one file {{{.palm.iofiles}}} which contains the file connection statements to be used for all configurations. |
| | 13 | * The old configuration file {{{.mrun.config}}} has been split into two files {{{.palm.config.<configuration_identifier>}}} and {{{.palm.iofiles}}}, where {{{<configuration_identifier>}}} is an arbitrary string that you can define. "Configuration" means a setting for a specific computer with a specific compiler, compiler options, libraries, etc. If you like to run {{{palm}}} with different configurations, e.g. one with debug options switched on, and one with high optimization, you need to create separate configuration files for each configuration, e.g. {{{.palm.config.optimized}}} and {{{.palm.config.debug}}}. This replaces the old block structure in {{{.mrun.config}}}. The configuration file to be used is defined by {{{palmrun}}}- or {{{palmbuild}}}-option {{{-h}}}., e.g. {{{palmrun ... -h optimized}}} will use {{{.palm.config.optimized}}} \\\\ You will need only one file {{{.palm.iofiles}}} which contains the file connection statements to be used for all configurations. \\\\ The utility program {{{interpret_config}}} has been removed. The configuration files are now directly interpreted by the shellscripts. |
| | 49 | |
| | 50 | * For running PALM on a remote host in batch, additional settings are required in the configuration file. The following is an example for using the Cray-XC40 of HLRN as a remote host: |
| | 51 | {{{ |
| | 52 | #column 1 column 2 |
| | 53 | #name of variable value of variable (~ must not be used) |
| | 54 | #---------------------------------------------------------------------------- |
| | 55 | %base_data ~/palm/current_version/JOBS |
| | 56 | %base_directory $HOME/palm/current_version |
| | 57 | %source_path $HOME/palm/current_version/trunk/SOURCE |
| | 58 | %user_source_path $base_directory/JOBS/$fname/USER_CODE |
| | 59 | %fast_io_catalog /gfs2/work/niksiraa |
| | 60 | %local_jobcatalog /home/raasch/job_queue |
| | 61 | %remote_jobcatalog /home/h/niksiraa/job_queue |
| | 62 | # |
| | 63 | %local_ip 130.75.105.103 |
| | 64 | %local_username raasch |
| | 65 | %remote_ip 130.75.4.1 |
| | 66 | %remote_username niksiraa |
| | 67 | %remote_loginnode hlogin1 |
| | 68 | %ssh_key id_rsa_hlrn |
| | 69 | %defaultqueue mpp2testq |
| | 70 | %submit_command /opt/moab/default/bin/msub -E |
| | 71 | # |
| | 72 | %compiler_name ftn |
| | 73 | %compiler_name_ser ftn |
| | 74 | %cpp_options -e Z -DMPI_REAL=MPI_DOUBLE_PRECISION -DMPI_2REAL=MPI_2DOUBLE_PRECISION -D__parallel -D__netcdf -D__netcdf4 -D__netcdf4_parallel -D__fftw |
| | 75 | %make_options -j 4 |
| | 76 | %compiler_options -em -O3 -hnoomp -hnoacc -hfp3 -hdynamic |
| | 77 | %linker_options -em -O3 -hnoomp -hnoacc -hfp3 -hdynamic -dynamic |
| | 78 | %execute_command aprun -n {{MPI_TASKS}} -N {{TASKS_PER_NODE}} palm |
| | 79 | %memory 2300 |
| | 80 | %module_commands module load fftw cray-hdf5-parallel cray-netcdf-hdf5parallel |
| | 81 | %login_init_cmd module switch craype-ivybridge craype-haswell |
| | 82 | # |
| | 83 | # BATCH-directives to be used for batch jobs. If $-characters are required, hide them with \\\ |
| | 84 | BD:#!/bin/bash |
| | 85 | BD:#PBS -A {{PROJECT_ACCOUNT}} |
| | 86 | BD:#PBS -N {{JOB_ID}} |
| | 87 | BD:#PBS -l walltime={{CPU_HOURS}}:{{CPU_MINUTES}}:{{CPU_SECONDS}} |
| | 88 | BD:#PBS -l nodes={{NODES}}:ppn={{TASKS_PER_NODE}} |
| | 89 | BD:#PBS -o {{JOBFILE}} |
| | 90 | BD:#PBS -j oe |
| | 91 | BD:#PBS -q {{QUEUE}} |
| | 92 | # |
| | 93 | # BATCH-directives for batch jobs used to send back the jobfile from a remote to a local host |
| | 94 | BDT:#!/bin/bash |
| | 95 | BDT:#PBS -A {{PROJECT_ACCOUNT}} |
| | 96 | BDT:#PBS -N job_protocol_transfer |
| | 97 | BDT:#PBS -l walltime=00:30:00 |
| | 98 | BDT:#PBS -l nodes=1:ppn=1 |
| | 99 | BDT:#PBS -o {{JOB_TRANSFER_PROTOCOL_FILE}} |
| | 100 | BDT:#PBS -j oe |
| | 101 | BDT:#PBS -q dataq |
| | 102 | # |
| | 103 | #---------------------------------------------------------------------------- |
| | 104 | # INPUT-commands, executed before running PALM - lines must start with "IC:" |
| | 105 | #---------------------------------------------------------------------------- |
| | 106 | IC:export ATP_ENABLED=1 |
| | 107 | IC:export MPICH_GNI_BTE_MULTI_CHANNEL=disabled |
| | 108 | IC:ulimit -s unlimited |
| | 109 | }}} |
| | 110 | * Some additional settings are required here: \\\\ |
| | 111 | - {{{fast_io_catalog}}} is the one to be used on the remote host. |
| | 112 | - IP-addresses and user names have to be given for the local AND the remote host. Usually, the remote host IP-address is the one for the login-node. |
| | 113 | - {{{remote_loginnode}}}: on many of the large computer systems, the compute nodes do not allow for {{{ssh}}}- or {{{scp}}}-commands in order to transfer data to the local host or to start restart jobs. If {{{remote_loginnode}}} is set, {{{palmrun}}} tries to start these commands via the login-node. '''Attention:''' In most cases, the systems to not accept an IP-address. You have to give the mnemonic name of the login-node. |
| | 114 | - {{{ssh_key}}}: here you can give the filename of a special ssh-key for using ssh / scp without password. The key must be in folder {{{~/.ssh}}}. This is a special setting for the HLRN-system and should not be required on other systems. |
| | 115 | - {{{default_queue}}}: if you do not set the queue via {{{palmrun}}}-option {{{-q}}}, this queue will be taken as the default queue. Other than {{{mrun}}}, {{{palmrun}}} does not check for valid queue names any more. |
| | 116 | - To |
