= The PALM file connection file PALM expects all input files to be in the temporary working directory from where it is executed. Also, it writes all output data to this directory. The file connnection file tells {{{palmrun}}}, where the user has stored his input files, and where the user likes to have the output files copied after a PALM run has finished. You can find the default file connnection file that is used by {{{palmrun}}} under [source:palm/trunk/SCRIPTS/.palm.iofiles]. In general, the file connection file does not need to be touched by the user at all. \\ == Format of the file connection file The file is given in ASCII format and it generally contains one line per input/output file, which is called ''file connection statement''. A file connection statement is separated into a maximum of six columns. The first five columns are obligatory. The meaning of the columns are explained based on the following example, which shows an excerpt of the default file: {{{ #--------------------------------------------------------------------------------------------- # List of input-files #--------------------------------------------------------------------------------------------- PARIN in:tr d3# $base_data/$run_identifier/INPUT _p3d* PARIN in:tr d3r $base_data/$run_identifier/INPUT _p3dr* TOPOGRAPHY_DATA inopt:tr d3#:d3r $base_data/$run_identifier/INPUT _topo* DATA_1D_PR_NETCDF inopt prr $base_data/$run_identifier/OUTPUT _pr* nc # #--------------------------------------------------------------------------------------------- # List of output-files #--------------------------------------------------------------------------------------------- BINOUT* out:lnpe restart $fast_io_catalog/$run_identifier/RESTART _d3d DATA_3D_NETCDF* out:tr * $base_data/$run_identifier/OUTPUT _3d nc }}} The '''first column''' gives the local filename in PALM's temporary working directory. PALM expects all filenames in capital letters. The '''second column''' gives the file attributes, which need to be separated by a colon (":"). Two attributes are allowed. The '''first attribute''' says, whether the file is an input ({{{in}}}) or an output file ({{{out}}}). {{{palmrun}}} immediately terminates, if it cannot find the user's input file, because it assumes that PALM cannot run without files with attribute {{{in}}}. If you do not want {{{palmrun}}} to terminate, you may give the attribute {{{inopt}}}, which says that this file is optional. The '''second attribute''' can be * {{{tr}}} \\ In case of running jobs on remote hosts, transfer the file between the remote and local host via {{{scp}}}. \\\\Input files will be transferred from the user's folder on the local host to palmrun's temporary working directory on the remote host, before PALM is executed. This transfer is actually done before the batch job to the remote host is submitted. Files will be stored on the remote host in folder {{{$fast_io_catalog/SOURCES_FOR_RUN_$run_identifier}}}. As soon as the batch job executes, files are copied from the SOURCES_FOR_RUN-folder to the temporary working directory. \\\\Output files will be transferred from the temporary working directory on the remote host to the user's folder on the local host, after PALM has finished execution. * {{{tra}}} \\ Can only be used for ASCII output files. Same as {{{tr}}}, but the local file from the remote host will be appended, if a respective file already exists in the user's folder on the local host. In such a case, no new file cycle will be created. * {{{pe}}} \\ The filename is interpreted as a directory name, and it is assumed that the respective folder contains multiple files, one from each MPI process. Filenames are expected as {{{_######}}}, where {{{######}}} is the six-digit MPI-id number. If you are running PALM on 4 cores (4 MPI processes), files will be {{{_000000}}}, {{{_000001}}}, {{{_000002}}}, and {{{_000003}}}. {{{palmrun}}} will copy the complete folders. * {{{lnpe}}} \\ Same as {{{pe}}}, but {{{palmrun}}} tries to move the files using {{{ln}}} instead of {{{cp}}}, which reduces the time to copy the file drastically, but only if both folders (the local temporary folder and the user's folder) reside on the same file system. This attribute is given for restart files in the default file connection file, because they may have huge sizes. * {{{trpe}}} \\ Same as {{{pe}}}, but the folders are transferred between the local and the remote host via {{{scp}}}. The '''third column''' gives the so-called activation strings. Two or more strings need to be separated by colons (":"). Activation strings are those given in the {{{palmrun}}} option {{{-a}}}. The respective file connection statement is only processed (i.e. the file is copied), if at least one of its activation strings is set with option {{{-a}}}. In case of the above example, {{{-a "d3#"}}} causes {{{palmrun}}} to provide the input files {{{PARIN}}} and {{{TOPOGRAPHY_DATA}}}. Option {{{-a "d3# restart"}}} additional copies/saves the output file {{{BINOUT}}}. Note that activation strings need to be separated by blanks when given with the option {{{-a}}} and that they need to be embraced by {{{"}}}, even if only one string is given (otherwise, a {{{#}}} would cause the shell to interpret the remaining part of the command as a comment). For output files, a wildcard may be given in the third column. Respective local files will then always be copied, in case they exist, independently from the activation strings given with option {{{-a}}}. No warning will be issued if no file exists. Wildcards in the third column are not allowed for input files. The path to the user's permanent file is given in the '''fourth column'''. It may contain environment variables that have been defined in the file connection file. {{{$run_identifier}}} will be replaced with the argument of {{{palmrun}}} option {{{-r}}}. The '''fifth column''' gives the appendix of the user's permanent file. The '''sixth column''' is optional and may only contain the string {{{nc}}}. This string has to be given if the user's permanent file is a NetCDF file with suffix {{{.nc}}}. \\ == Determination of full file paths The full path for input and output files is based on columns 4-6 and the argument of {{{palmrun}}} option {{{-r}}}: {{{ /<-r option>[.] }}} As an example, the file connection statement {{{ DATA_3D_NETCDF* out:tr * $base_data/$run_identifier/OUTPUT _3d nc }}} plus entry {{{ %base_data ~/palm/current_version/JOBS }}} in the file connection file, plus the call {{{palmrun -r example_test ...}}} will create the output file {{{ ~/palm/current_version/JOBS/example_test/OUTPUT/example_test_3d.nc }}} \\ == Using wildcards in filenames In case of nested runs, masked 3d-output and wind turbine model output, PALM expects or creates multiple input/output files with additional suffixes. As an example, a nested run with a parent and one child requires two parameter files for steering the simulation, where the second file is expected to have the suffix {{{_N02}}}. Each additional nest would require another parameter file with suffixes {{{_N03}}}, {{{_N04}}}, etc. In order to simplify the file connection file, only one file connection statement {{{ PARIN in:tr d3# $base_data/$run_identifier/INPUT _p3d* }}} is required for this, where the wildcard in {{{_p3d*}}} causes {{{palmrun}}} to copy files with these suffixes too. Assuming an INPUT folder with files {{{ example_test_p3d example_test_p3d_N02 example_test_p3d_N03 }}} {{{palmrun}}} will copy them to local files {{{ PARIN PARIN_N02 PARIN_N03 }}} The same holds for output files. Here the wildcard must be given as part of the local filename (first column). The example given below assumes one nest (root domain plus one child domain), two output masks for the root domain, and three masks for the child. Assuming a file connection statement for an output file {{{ DATA_MASK_NETCDF* out:tr * $base_data/$run_identifier/OUTPUT _masked nc }}} and following local files that have been created by PALM {{{ DATA_MASK_NETCDF_M01 DATA_MASK_NETCDF_M02 DATA_MASK_NETCDF_N02_M01 DATA_MASK_NETCDF_N02_M02 DATA_MASK_NETCDF_N02_M03 }}} {{{palmrun}}} will copy these local temporary files to permanent files {{{ ....../OUTPUT/example_test_masked_M01.nc ....../OUTPUT/example_test_masked_M02.nc ....../OUTPUT/example_test_masked_N02_M01.nc ....../OUTPUT/example_test_masked_N02_M02.nc ....../OUTPUT/example_test_masked_N02_M03.nc }}} Instead of using wildcards in filenames, you may use separate file connection statements, one for each possible suffix. \\ == How to modify the file connection file If you like to modify the file connection file, e.g. because your user-interface requires or creates additional input/output files, you need to add additional file connection statements to the file connection file. In such a case, just copy the default file to your working directory, e.g. {{{ cd ~/palm/current_version cp trunk/SCRIPTS/.palm.iofiles . }}} and modify the copied file. If existing, {{{palmrun}}} will automatically take the file connection file from the working directory.