PALM   a parallelized LES model

technical/numerical documentation
Current fixed release: 3.7a
Last change of this document $Id: technical_documentation.html 532 2010-04-21 13:33:38Z heinze $

Table of contents

1.0  History of changes

The following table documents all changes to the PALM repository since 2009. Older changes can be found here.

Entries in the fifth column specify the type of the change: change  of existing code (C), new code (N) or bugfix (B). More detailed information about the changes may sometimes be found in the header comment lines of the respective routines.
svn revision
last release
affected routines
Large scale vertical motion (subsidence/ascent) can be added as additional source to the prognostic equation for humidity/passive scalar
character length for dopr_unit enlarged

Adjustment of the NCL scripts to allow for output of png files
modules, prognostic_equations

.ncl.config.default, cross_sections.ncl, profiles.ncl, timeseries.ncl, spectra.ncl

Bugfix: unit of hyp changed to dbar
Bugfix: treatment of special character * in NCL variables containing lists corrected
List of variable names have to specified using blank separated lists and single quotes
Adjustment of the NCL scripts and palmplot to allow for special characters in NetCDF variable names
palmplot, .ncl.config.default, cross_sections.ncl, profiles.ncl, timeseries.ncl, spectra.ncl
Bugfix: "/" in netcdf profile variable names replaced by ":"
NetCDF4 support for particle data. Total number of particles is allowed to increase during the run, when netcdf_data_format > 2 is used.
advec_particles, check_open, netcdf

Special characters like * and " are now allowed for NetCDF variable names. Replacement of these characters has been removed (routine clean_netcdf_varname removed)

Simple installation method has been documented (see chapter 5.0). Respective scripts and configuration files have been updated.
palm_simple_install, palm_simple_run

opt=1 is forced for PGI-compiler in routine buoyancy by compiler-directive. Otherwise, program aborts with SIGFPE.
NCL scripts can be run with the shell script palmplot.
palmplot, .ncl.config.default, cross_sections.ncl, profiles.ncl
bugfix for rev 504: remode_md was given a wrong value
hack for better usage of mrun on Kyoto computing center machine (lckyoto)
combine_plot_fields is now directly called within mrun and must not be given by an output command in the configuration file any more. Please remove the respective output command (OC:combine_plot_fields.x) from your configuration file!

Utility programs (combine_plot_fields, interpret_config) are created by mbuild separately for each block in the configuration file. Depending on option -K, mrun is using the utility programs for the respective block.

Utility programs are also compiled in parallel, depending on variable %mopts in the configuration file..

Default configuration files updated.
mbuild, mrun, UTIL/Makefile, SCRIPTS/.mrun.config.*
Particle data for restart runs are only written if environment variable write_binary = .T., i.e. particle data are now handled in the same way as the other restart data.
Output in NetCDF4-format.

NetCDF4 allows for parallel I/O. Also, variables are not limited in size any more (i.e. arrays can be larger than 2GB). The new d3par-parameter netcdf_data_format.can be used to select among the different NetCDF file formats (1: classic format, 2: 64-bit-offset format (this is now the default), 3: NetCDF4/HDF5 format, 4: NetCDF4 format with NF90_CLASSIC_MODEL bit set). Setting netcdf_data_format > 2 only affects 2d- and 3d-output, i.e. there is no parallel output for profiles, time series, spectra, and particles. Parallel output of particle data may realized in the future.

In case of parallel output, all PEs write into one file, which means that postprocessing the data with combine_plot_fields (merging data from single files - one per PE - into one file) is not neccessary. In this case, flag files are automatically set, telling combine_plot_fields that nothing is to be done.

Using NetCDF4 format requires a NetCDF4/HDF library. Respective settings for -I, -L, and -l compiler/loader options for the SGI-ICE at HLRN can be found in the new example configuration file ..../trunk/SCRIPTS/.mrun.config.imuk_ice2_netcdf4

So far, NetCDF parallel I/O has only been tested a little. Further performance optimizations may be required, especially in case of large data sets (e.g. data chunking, or lustre file system settings with lfs setstripe). Please report bugs and performance problems immediately!

combine_plot_fields outputs cpu-time needed for file merging.

Modules to be loaded for compilation (mbuild) or job execution (mrun) can/should now be given in the configuration file (.mrun.config) using variable modules. Example:

   %modules     ifort/11.0.069:netcdf       lcsgih parallel

This method replaces the (undocumented) mpilib-variable.

All fixed settings of modules in the scripts mbuild, mrun, and subjob have been removed! Please set the modules variable appropriately and remove the mpilib variable in your configuration file.

Compilation time can be reduced be using parallel make. The number of allowed simultaneous compiler calls (threads) is set in the configuration file using variable mopt, e.g.

   %mopts  -j:4   lcsgih parallel

allows a maximum of 4 routines to be compiled simultaneously (provided that the makefile does not show any dependencies between these routines).
check_open, check_parameters, close_file, data_output_2d, data_output_3d, header, modules, netcdf, parin


mbuild, mrun, subjob

d3par-parameters netcdf_64bit and netcdf_64bit_3d have been removed. Use the new d3par-variable netcdf_data_format = 2 for choosing the NetCDF 64bit-offset format (this is the default from now on). The offset-format can not be set independently for the 3d-output-data any more.

d3par-parameters netcdf_format_mask, netcdf_format_mask_av, and variables nc_format_mask, format_parallel_io removed. They are replaced by the new parameter netcdf_data_format. This change concerns the masked data output, which is still not documented.
check_open, close_file,
data_output_mask, header, init_masks, modules, parin

Bugfix in trunk/UTIL/Makefile: forgot to compile for interpret_config.

Bugfix: timeseries data have to be collected by PE0 (user_statistics).

ngp_3d, ngp_3d_inner changed to 64 bit. Otherwise, on large grids > 2*10**9 grid points some global statistics (e.g. total kinetic energy) have wrong values.
init_3d_model, modules
Typo in file headers removed ("current" instead of "actual").
compare_palm_logs is additionally compiled with mbuild -u . (Makefile in trunk/UTIL)

make options (mopts) to be set by configuration file implemented. (mrun, mbuild)

humidity = .T. can be used for runs with topography. wall_humidityflux and
wall_scalarflux are the corresponding new inipar parameters.
(check_parameters, init_3d_model, parin)

Large scale vertical motion (subsidence/ascent) can be added as additional source to the prognostic equation for potential temperature. See new parameters large_scale_subsidence, ws_vertical_gradient, and ws_vertical_gradient_level. (check_parameters, header, Makefile, modules, parin, prognostic_equations, read_var_list, subsidence, write_var_list)

Scripts have been updated for Cray-XT5 machine (lcxt5m) and Hitachi-machine (lckyoto).

A simple method for installing and running palm (with limited features) has been added. (Makefile, new scripts palm_simple_install, palm_simple_run)

Masked data output has been implemented but it is not documented in the user manual so far. This feature can already use NetCDF4 file format. (check_open, check_parameters, close_file, modules, netcdf, parin, time_integration, user_parin, new: data_output_mask, init_masks, user_data_output_mask)
check_open, check parameters, close_file, header, init_3d_model, Makefile, mbuild, modules, mrun, netcdf, parin, prognostic_equations, read_var_list, time_integration, user_parin, write_var_list

new: data_output_mask,  init_masks, palm_simple_install, palm_simple_run, subsidence, user_data_output_mask

2d-decomposition is default for Cray-XT machines. (init_pegrid)

var_ts is replaced by dots_max. (modules, init_3d_model)

Every cloud droplet has now an own weighting factor and can be deleted due to collisions. Condensation and collision of cloud droplets are adjusted accordingly. (advec_particles)

Collision efficiency for large cloud droplets has changed according to table of Rogers and Yau. (collision_efficiency)
advec_particles, collision_efficiency, init_3d_model, init_pegrid, modules

Bugfix for generating serial jobs. (subjob)

Bugfix: index problem concerning gradient_level indices removed. (header)

Dimension of array stat in cascade change to prevent type problems with mpi2 libraries. (poisfft_hybrid)

Loop was split to make runs reproducible when using ifort compiler. (disturb_field)

Bugfix: exchange of ghost points for prho included. (time_integration)

Bugfix in calculation of time-averaged surface heatfluxes. (sum_up_3d_data)

Bugfix in calculation of precipitation_rate. (calc_precipitation)

Bugfix: initial data assignments to some dvrp arrays changed due to error messages from gfortran compiler. (modules)

Bugfix in calculation of cloud droplet velocity. (advec_particles)

Bugfix in transfer of particles at south/left edge. (advec_particles)

Bugfix in calculation of collision efficiency. (collision_efficiency)
advec_particles, calc_precipitation, collision_efficiency, disturb_field, header, modules, poisfft_hybrid, subjob, sum_up_3d_data, time_integration

Most important changes: atmosphere-ocean coupling has been improved and steering of dvr-software is more user-friendly now. Beside that, the release contains a lot of optimizations, changes, and bugfixes which have accumulated over the last months.

The atmosphere-ocean coupling allows to use independent precursor runs in order to account for different spin-up times. The time when coupling has to be started is given by new inipar parameter coupling_start_time. The precursor ocean run has to be started using new mrun option "-y" in order to add the appendix "_O" to all output files. (check_for_restart, check_parameters, data_output_2d, data_output_3d, data_output_profiles, data_output_ptseries, data_output_spectra, data_output_tseries, header, init_coupling, modules, mrun, parin, read_var_list, surface_coupler, time_integration, write_var_list)

Clipping of dvrp output implemented. Default colourtable for particles implemented, particle attributes (color, dvrp_size) can be set with new parameters particle_color, particle_dvrpsize, color_interval, dvrpsize_interval. Slicer attributes (dvrp) are set with new routine set_slicer_attributes_dvrp and are controlled with existing parameters slicer_range_limits. (init_dvrp, data_output_dvrp, modules, set_slicer_attributes_dvrpuser_data_output_dvrp)

Polygon reduction for topography and ground plate isosurface. Reduction level for buildings can be chosen with parameter cluster_size. (init_dvrp)

Variables bc_lr / bc_ns  in most subroutines replaced by LOGICAL variables bc_lr_cyc,
bc_ns_cyc for speed optimization. This gives a significant speedup of more than 10%. (check_parameters, diffusion_u, diffusion_v, diffusion_w, modules)

Scripts have been adapted for machine lck (Yonsei Univ.). (mrun, mbuild, subjob)

Additional timestep criterion in case of simulations with plant canopy. (timestep)

Check for illegal entries in section_xy|xz|yz that exceed nz+1|ny+1|nx+1. (check_parameters)

External pressure gradient can be used as driving force. (check_parameters, header, init_3d_model, modules, parin, prognostic_equations, read_var_list, write_var_list)

New topography case 'single_street_canyon'. (header, init_grid, modules, parin, read_var_list, user_check_parameters, user_header, user_init_grid, write_var_list)

Option to predefine a target bulk velocity for conserve_volume_flow. (check_parameters, header, init_3d_model, modules, parin, read_var_list, write_var_list)

Option for user defined 2D data output in xy cross sections at z=nzb+1. (data_output_2d, user_data_output_2d)

xy cross section output of surface heatfluxes (sensible and latent). (average_3d_data, check_parameters, data_output_2d, modules, read_3d_binary, sum_up_3d_data, write_3d_binary)
average_3d_data, check_for_restart, check_parameters, data_output_2d, data_output_3d, data_output_dvrp, data_output_profiles, data_output_ptseries, data_output_spectra, data_output_tseries, diffusion_u, diffusion_v, diffusion_w, init_coupling, init_dvrp, init_grid, init_3d_model, header, mbuild, modules, mrun, package_parin, parin, prognostic_equations, read_3d_binary, read_var_list, subjob, sum_up_3d_data, surface_coupler, timestep, time_integration, user_check_parameters, user_data_output_2d, user_data_output_dvrp, user_header, user_init_grid, write_3d_binary, write_var_list

new: set_particle_attributes, set_slicer_attributes_dvrp

Output of messages replaced by message handling routine. (advec_particles, advec_s_bc, buoyancy, calc_spectra, check_for_restart, check_open, coriolis, cpu_log, data_output_2d, data_output_3d, data_output_dvrp, data_output_profiles, data_output_spectra, fft_xy, flow_statistics, headerinit_1d_model, init_3d_model, init_dvrp, init_grid, init_particles, init_pegrid, netcdf, parin, plant_canopy_model, poisfft_hybrid, poismg, read_3d_binary, read_var_list, surface_coupler, temperton_fft, timestep, user_actions, user_data_output_dvrp, user_dvrp_coltab, user_init_grid, user_init_plant_canopy, user_parin, user_read_restart_data, user_spectra)

Output of NetCDF messages with aid of message handling routine. (check_open, close_file, data_output_2d, data_output_3d, data_output_profiles, data_output_ptseries, data_output_spectradata_output_tseries, netcdf)

__lcmuk changed to __lc to avoid problems with Intel compiler on sgi-ice. (poisfft)

For extended NetCDF files, the updated title attribute includes an update of time_average_text where appropriate. (netcdf)

In case of restart runs without extension, initial profiles are not written to NetCDF-file anymore. So far, time axis always started at t=0. (data_output_profiles, modules, read_var_list, write_var_list)

initializing_actions='read_data_for_recycling' renamed to 'cyclic_fill'. A cyclic fill can now
be used independent of turbulent_inflow. (check_parameters, header, init_3d_model)

2 NetCDF error numbers changed. (data_output_3d)

A link to the webspage appendix_a.html is printed for further information about possible reasons of the error that appeared. (message)

Temperature gradient criterion for estimating the boundary layer height replaced by the gradient criterion of Sullivan et al. (1998). (flow_statistics)

NetCDF unit attribute in timeseries output in case of statistic regions added. (netcdf)

Maximum number of tails is calculated from maximum number of particles and skip_particles_for_tail. (init_particles)

Value of vertical_particle_advection is allowed to differ for each particle group. (advec_particles, header, modules)

First constant in array den also defined as type double. (eqn_state_seawater)

Parameter dvrp_psize moved from particles_par to dvrp_graphics_par. (package_parin)

topography_grid_convention moved from userpar to inipar (check_parameters, header, parin, read_var_list, user_check_parameters, user_header, user_init_grid, user_parin, write_var_list)

Default value of grid_matching changed to 'strict'. (modules)

Adjustments for runs on lcxt4 (necessary due to a software update on CRAY) and for coupled runs on ibmy. (mrun, subjob)
advec_particles, advec_s_bc, buoyancy, calc_spectra, check_for_restart, check_open, check_parameters, close_file, coriolis, cpu_log, data_output_2d, data_output_3d, data_output_dvrp, data_output_profiles, data_output_ptseries, data_output_spectra, data_output_tseries, eqn_state_seawater, fft_xy, flow_statistics, header, init_1d_model, init_3d_model, init_dvrp, init_grid, init_particles, init_pegrid, message, mrun, netcdf, package_parin, parin, plant_canopy_model, poisfft, poisfft_hybrid, poismg, read_3d_binary, read_var_list, subjob, surface_coupler, temperton_fft, timestep, user_actions,user_check_parameters, user_data_output_dvrp, user_dvrp_coltab, user_header, user_init_grid, user_init_plant_canopy, user_parin, user_read_restart_data, user_spectra, write_var_list

Bugfix: Initial hydrostatic pressure profile in case of ocean runs is now calculated in 5 iteration steps. (init_ocean)

Bugfix: wrong sign in buoyancy production of ocean part in case of not using the reference density (only in 3D routine production_e). (production_e)

Bugfix: output of averaged 2d/3d quantities requires that an avaraging interval has been set, respective error message is included. (check_parameters)

Bugfix: Output on unit 14 only if requested by write_binary. (user_last_actions)

Bugfix to avoid zero division by km_neutral. (production_e)

Bugfix for extended NetCDF files: In order to avoid 'data mode' errors if updated attributes are larger than their original size, NF90_PUT_ATT is called in 'define mode' enclosed by NF90_REDEF and NF90_ENDDEF calls. This implies a possible performance loss; an alternative strategy would be to ensure equal attribute size in a job chain. (netcdf)

Bugfix: correction of initial volume flow for non-flat topography. (init_3d_model)

Bugfix: zero initialization of arrays within buildings for 'cyclic_fill'. (init_3d_model)

Bugfix: to_be_resorted => s_av for time-averaged scalars. (data_output_2d, data_output_3d)

Bugfix: avoid that ngp_2dh_s_inner becomes zero. (init_3_model)

Typographical error: unit of wpt in dots_unit. (modules)

Bugfix: error in check, if particles moved further than one subdomain length. This check must not be applied for newly released particles. (advec_particles)

Bugfix: several tail counters are initialized, particle_tail_coordinates is only written to file if its third index is > 0, arrays for tails are allocated with a minimum size of 10 tails if there is no tail initially. (advec_particles, init_particles)

Bugfix: pressure included for profile output. (check_parameters)

Bugfix: Type of count and count_rate changed to default INTEGER on NEC machines. (cpu_log)

Bugfix: output of particle time series only if particle advection is switched on. (time_integration)

Bugfix: qsws was calculated in case of constant heatflux = .FALSE.. (prandtl_fluxes)

Bugfix: averaging along z is not allowed for 2d quantities (e.g. u* and z0). (data_output_2d)

Typographical errors. (netcdf)

If the inversion height calculated by the prerun is zero, inflow_damping_height must be explicitly specified. (init_3d_model)

Small bugfix concerning 3d 64bit netcdf output format. (header)

Bugfix: dt_fixed removed from the restart file, because otherwise, no change from a fixed to a variable timestep would be possible in restart runs. (read_var_list, write_var_list)

Bugfix: initial setting of time_coupling in coupled restart runs. (time_integration)
advec_particles, check_parameters, cpu_log, data_output_2d, data_output_3d, header, init_3d_model, init_particles, init_ocean, modules, netcdf, prandtl_fluxes, production_e, read_var_list, time_integration, user_last_actions, write_var_list
This release contains further additions, changes, and bugfixes for the SGI-ICE system, as well as important changes like the splitting of the user-interface into single files, a complete revision of the pre-compiling mechanism allowing to use different make-depositories simultaneously, and a revision of the reading-mechanism from restart files, which has no more restrictions in case that the previous and current run have different domain/subdomain sizes. A new handling of all kind of output messages is introduced with this release and will be applied to all existing messages within the next releases.

New features are:

Restart runs on SGI-ICE are working. (mrun)
2d-decomposition is default on SGI-ICE systems. (init_pegrid)

Ocean-atmosphere coupling realized with MPI-1. mrun adjusted for this case (-Y option). Check that PALM is called with mrun-option "-K parallel" for coupling. Adjustments in mrun, mbuild, and subjob for lcxt4.

DVRP arguments changed to single precision, mode pathlines added.

User can add additional routines in files user_additional_routines.

User can check user parameters and deduce further quantities in user_check_parameters
check_for_restart, check_parameters, data_output_dvrp, init_dvrp, init_pegrid, local_stop, modules, package_parin, palm, surface_coupler, timestep

Makefile, mbuild, mrun, subjob

init_coupling, user_additional_routines, user_check_parameters

User interface has been split into single files.

Type of variables count and count_rate changed to INTEGER(8) in order to avoid "out of range" problems, which result in measured negative time intervals. (cpu_log, local_tremain, local_tremain_ini)

Reading mechanism from restart files completely revised. The subdomain/total domain size is now allowed to vary arbitrarily between the current and previous run. (read_var_list, read_3d_binary, user_read_restart_data)

Precompilation mechanism (mbuild) completely revised: the source-tarfile remains in the source directory (trunk/SOURCE). One depository per block (given in the config-file) is created. Always all files from the source directory are copied to the respective depository. No additional file checks are done any more (mrun version 2.1). The depository name used contains the conditions given by mrun-option "-K". (mbuild, mrun)

Output messages (including required stop of execution) can now be handled with the new subroutine message. All output messages will be replaced by this routine step by step within the next revisions.

Neumann boundary condition at k=nzb is explicitly set for better reading, although this has been already done in boundary_conds. (advec_s_bc)

Origin of the xy-coordinate system shifted from the center of the first grid cell (indices i=0, j=0) to the south-left corner of this cell. (netcdf) Topography definition according to new user parameter topography_grid_convention. (init_grid, modules, user_header, user_init_grid, user_parin)
advec_s_bc, check_parameters, cpu_log, init_grid, local_stop, local_tremain, local_tremain_ini, modules, netcdf, read_3d_binary, read_var_list, user_read_restart_data

Makefile, mbuild, mrun


message, user_3d_data_averaging, user_actions, user_advec_particles, user_check_data_output, user_check_data_output_pr, user_data_output_2d, user_data_output_3d, user_data_output_dvrp, user_define_netcdf_grid, user_dvrp_coltab, user_header, user_init_3d_model, user_init, user_init_grid, user_init_particles, user_init_plant_canopy, user_last_actions, user_parin, user_parin, user_particle_attributes, user_read_restart_data, user_spectra, user_statistics

Bugfix: array d is reallocated in case that multigrid is used. (calc_spectra)

Bugfixes for nonparallel execution. (check_for_restart, cpu_statistics, inflow_turbulence, timestep)

Size of pf3d adjusted to the required output size (1 gridpoint less, along all three dimensions), because output of a subset of the data (pf3d(nxa:nxe...)) in the NF90_PUT_VAR statement caused segmentation fault with the INTEL compiler. (combine_plot_fields)

Bugfix: error in zu index in case of section_xy=-1. (header)

Inconsistency removed: as the thermal stratification is not taken into account for the evaluation of the wall fluxes at vertical walls, the eddy viscosity km must not be used for the evaluation of the velocity gradients dudy, dwdy, dvdx and dwdx. (production_e)

Bugfix in calculating k index in case of oceans runs. (sort_particles)

Bugfix: no output of particle concentration and radius unless particles have been started. (data_output_2d)

Bugfix: reading of spectrum_x|y from restart files ignored if total numbers of grid points do not match. (read_3d_binary)

Bugfix: abort in case that absolute temperature is below zero. (init_cloud_physics)
advec_particles, calc_spectra, check_for_restart, cpu_statistics, data_output_2d, header, inflow_turbulence, init_cloud_physics, production_e, read_3d_binary, timestep



2.0  How to change the source code and how to document modifications

Currently (Sep 23rd, 2009), only selected users are allowed to commit changes to the repository. Rules given below are preliminary, until we have switched to the "trac"-System.

  1. Change source code in your current working copy. Document changes in the respective file headers unter "Current revisions:". Additionally, document all changes in a local file (e.g. ~/palm/current_version/CURRENT_MODIFICATIONS), including the files that have been changed. Classify changes in three cases, new (N), changed (C), and bugfix (B). Try to be as short and as precise as possible in describing the changes!

  2. Test the changed version, unless it works sufficiently.

  3. Commit the changes to the repository

       svn commit -m "message string" trunk

    where "message string" is a short, meaningful summary of the changes.

    Alternatively, in case that you have made quite a lot of changes, you can also give the contents of your file CURRENT_MODIFICATIONS as the log message:

       svn commit -F CURRENT_MODIFICATIONS trunk

  4. Enter the changes into the technical documentation (under .../trunk/DOC/tec/technical_documentation.html).

  5. Move the change comments in the file headers from "Current revisions:" to "Former revisions:". Set the current svn id in front of that.


    before move:

    ! Current revisions:
    ! -----------------
    ! new dummy argument abcd
    ! Former revisions:
    ! -----------------
    ! §Id: init_3d_model.f90 425 2010-02-06 13:37:48Z raasch §
    ! 407 2009-12-01 15:01:15Z maronga
    ! var_ts is replaced by dots_max

    after move:

    ! Current revisions:
    ! -----------------
    ! Former revisions:
    ! -----------------
    ! §Id: init_3d_model.f90 425 2010-02-06 13:37:48Z raasch §

    ! 425 2010-02-06 13:37:48Z raasch
    new dummy argument abcd
    ! 407 2009-12-01 15:01:15Z maronga
    ! var_ts is replaced by dots_max

    Note that the "§" in this example should read "$", but "$" cannot be used here, because otherwise svn would replace these entries after every change to this technical documentation.

  6. Commit the changes in the technical documentation and the file headers:

       svn commit -m "last commit documented" trunk

3.0  Description of selected parts of the model source code

This section gives links to existing descriptions about special parts of the PALM code, e.g. special numerical methods that are used (including mathematical background). Documents are stored in single files with different formats. Some are in German (sorry for the inconvenience).