PALM technical/numerical documentation

PALM a parallelized LES model

technical/numerical documentation


Current model version:	3.7 (rev 392)
Last change of this document:	$Id: technical_documentation.html 392 2009-09-24 10:39:14Z raasch $

Table of contents

1.0

2.0 How to change the model source code

3.0 Description of selected parts of the model source code

1.0 History of changes

The following table documents all changes to the model since July, 7th 1998. Entries in the fourth column specify the type of the change: change of existing code (C), new code (N) or bugfix (B). More detailed information about the changes may be found in the header comment lines of the respective routines.

date author model
version type description affected routines

06/07/98 SR 1.0 C Änderung von Default-Werten (initializing_actions, end_time, fcl_factor, use_prior_plot1d_parameters), zusätzliche Überprüfung von Parametern auf Zulässigkeit (prandtl_layer, psolver, nx, ny, nz, fcl_factor), Höhe der Schnittebenen der Horizontalschnitte wird auf Plots exakt ausgegeben (uv- bzw. w-Gitter). modules, check_parameters, plot_2d

" SR 1.0 N Neues Modul test_variables wird in (fast) allen Programmteilen benutzt (USE). Wichtig: dieses Modul steht in der Datei module_test.f90. Diese Datei muß in der mrun- und add_library - Konfigurationsdatei bei der Variablen module_files aufgeführt werden. (fast) alle
Neu: module_test

16/07/98 SR 1.0 N Anpassung an die T3E-Rechner in Juelich (cpp - Direktiven) advec_s_bc, check_open, check_parameters, cpu_zeitmessung, init_pegrid, local_getenv, local_system, local_tremain, local_tremain_ini, poisfft

24/07/98 MS 1.0 C Anpassung des Druckloesealgorithmusses an Vorgehensweise von Schmidt, Schumann und Volkert (DFVLR-Mitteilung 84-15, S. 20-26) poisfft

27/07/98 MS 1.0 C Berechnung von l in maketri geschieht nun ausserhalb der k-Schleife, da unabhaengig von k, dadurch Rechenzeitersparnis poisfft

05/08/98 SR 1.0 C Entfernung der Sonderbehandlung der Advektionsterme (Piascek-Williams) bei k=1. Terme werden nun überall gleich berechnet. Evtl. leichte Erhöhung der Varianzen in Oberflächennähe. advec_u_pw, advec_v_pw, advec_w_pw

" SR 1.0 C Mischungsweganpassung (s. adjust_mixing_length) erfolgt nur noch bei k=1. Unrealistische sekundäre Maxima in den Km-Profilen oberhalb der Prandtl-Schicht werden damit vermieden. Leichte Änderungen in den oberflächennahen Turbulenzgrößen sind die Folge. diffusivities

1.0

Falsche Berechnung der benötigten CPU-Zeit pro Sekunde simulierter Zeit bei Fortsetzungsläufen korrigiert.

Berechnung von cross_uxmin(1) und cross_uxmax(1) aus den Werten von ug und vg entfernt. Lieferte falsche Wertebereiche für Plots von Vertikalprofilen, wenn im ersten Koordinatenkreuz nicht die Windgeschwindigkeiten gezeichnet wurden.

header, check_parameters

" SR 1.0 N Das erlaubte Maximum und Minimum der Rif-Zahl ist nun mittels der Initialparameter rif_max und rif_min einstellbar (Werte waren bisher fest auf -2.0, +1.0 eingestellt). check_parameters, header, init_1d_model, modules, parin, prandtl_fluxes, read_var_list, write_var_list

23/09/98

1.0

Initialisierung von Feldern im Fall von initializing_action = "set_constant_profiles" und "set_1d-model_profiles" korrigiert.

2D-Plotausgabe von u* und theta* korrigiert. Hier wurden bisher falsche Felder herausgeschrieben.

Bei SOR-Aufrufen war bisher dp nicht initialisiert. Entsprechend geändert.

init_3d_model, plot_2d, pres

1.0

Ausgabeformate für Statistik-Ausgaben auf Datei RUN_CONTROL um einige Kommastellen erweitert.

Lieferung von ausführlicheren Informationen bei Programmabbruch durch Unterschreitung des minimal erlaubten Zeitschritts.

run_control, timestep

1.0

Rechnungen mit geneigter Ebene zur Simulation von Kaltluftabflüssen sind erlaubt. Gestartet und gesteuert werden sie mit dem Intialisierungsparameter alpha_surface. Realisiert wird die geneigte Ebene durch entsprechende Schrägstellung des Schwerkraftvektors. Dadurch erhält auch die u-Komponente der Windgeschwindigkeit einen Anteil der Auftriebskraft.

Diese Implementierung ist noch nicht abgeschlossen. Die Simulation von Kaltluftabflüssen erfordert bisher noch Eingriffe "von Hand".

buoyancy, check_parameters, header, leap_frog, modules, parin, prandtl_fluxes, read_var_list, write_var_list

15/12/98

1.0

Vereinbarung einer Cache-Layer zur schnelleren binären I/O auf T3E-Rechnern.

Ermittelung wahrer I/O-Zeiten für binäre Ein-/Ausgabe auf T3E-Rechnern mit TIMEF.

check_open,
cpu_zeitmessung, init_3d_model, write_3d_model

1.0

Bekanntgabe von run_description_header an alle PE's.

Diese Programmänderung erfordert auf T3E-Rechnern zwingend den Einsatz der MPI-Version mpt.1.2.0.1. In dieser neuen Version tritt auch der alte Fehler in MPI_ALLREDUCE nicht mehr auf, so daß ab jetzt mpt.1.1.0.1 nicht mehr verwendet werden muß.

parles

05/02/99

1.1

C/N

Implementierung des Upstream-Spline- Verfahrens zur Advektion von Impuls und skalaren Größen (neue UP's, s. rechts). Berechnung von Variablen für spezielle Advektionsverfahren in eigenständiges UP ausgelagert (init_advec + eigenes Modul advection). Reines Euler-/Upstream-Verfahren kann gerechnet werden.

Schnelle binäre I/O (s. 15/12/98) wieder abgeschaltet, da Dateien nicht wieder einlesbar sind. Druckstörung (p) wird zu Beginn auf 0 gesetzt, damit sie keine Tendenzen im ersten Zeitschritt hervorruft.

advec_s_bc, check_open, check_parameters, flow_statistics, header, init_3d_model, init_grid, leap_frog, modules, parin, plot_2d, plot_ts, read_var_list, timestep, write_3d_binary, write_var_list

Neu:
advec_s_ups, advec_u_ups, advec_v_ups, advec_w_ups, init_advec, long_filter, spline_x, spline_y, spline_z

24/02/99 SR 1.1a C/N Kleinere Ergänzungen / Änderungen für Upstream - Spline,
Diverse Verbesserungen bei Berechnung der Diffusionskoeffizienten (exaktere Formulierung von Scherungen, Dissipation gemaess Original - Deardorff - Ansatz, Mischungswegreduktion nur, wenn Schichtung sowohl oberhalb als auch unterhalb des entsprechenden Gitterpunktes stabil ist), Wertebegrenzung bei Spline - Berechnung zur Verhinderung von "Überschwingern" check_parameters, diffusion_e, diffusion_pt, diffusivities, header, init_advec, long_filter, modules, production_e, spline_x, spline_y, spline_z

" SR 1.1a N/B Erweiterung des des Bott - Chlond - Schemas auf Advektion der TKE, Korrektur der Dichteberechnung (war falsch bei Galilei-Transformation) advec_s_bc, leap_frog

01/03/99

1.1b

Kleinere Ergänzungen für Upstream - Spline:

Gradientenkontrolle für den nicht-parallelen Teil
statistische Auswertung über den prozentualen Anteil des Upstream-Verfahrens an der Gesamtadvektion fuer nicht-parallelen Teil

modules, spline_x, spline_y, spline_z

1.1b

Datenkompression für 3D-Plotausgaben

check_open, check_parameters, close_files, header, modules, parin, plot_3d

Neu:
write_compressed

03/03/99 SR 1.1c B/C Schnelle binäre I/O auf T3E-Rechnern funktioniert jetzt. Beim binären Einlesen dürfen Records nicht mit dem Trick "READ (..) idum" überlesen werden. Zeitmessungen auf T3E erfolgen jetzt grundsätzlich mit TIMEF. check_open, cpu_zeitmessung, init_3d_model, modules, write_3d_binary

25/03/99 SR 1.1d N Einbau einer Rayleigh-Dämpfungsschicht sowie flexiblere Wahl der numerischen Begrenzer für das Upstream - Spline - Verfahren. check_parameters, header, init_3d_model, leap_frog, modules, parin, read_var_list, spline_x, spline_y, spline_z, write_var_list

" SR 1.1d B Korrektur des Zeitschrittkriteriums bei z=zp im 1D-Modell init_1d_model

25/11/99

1.1e

Partikelplots mit vtk-Graphiksoftware sind auf t3eh möglich. Ausgabe erfolgt im dvr-Format. Bisher noch nicht nutzbar, da noch Einstellmöglichkeiten für die Partikelquelle(n) fehlen. Wichtig: Die Benutzung der vtk-Software erfordert die zusätzliche mrun-Option -g. Das Hauptprogramm parles wird in diesem Fall zum Unterprogramm und von einem c++-Programm aus aufgerufen.

header, init_3d_model, init_pegrid, leap_frog, modules, parin, parles, read_var_list, write_var_list

Neu:
advec_particles, init_particles, plot_particles

" SR 1.1e B/C TIMEF-Zeitmessungen von Subroutine- auf Function-Aufruf umgestellt. Laufindexkorrektur in spline_y. Fehler trat bei nx /= ny auf. cpu_zeitmessung, spline_y

28/12/99

1.1f

Plot von Isooberflächen mit vtk-Graphiksoftware sind auf t3eh möglich. Ausgabe erfolgt im dvr-Format. Bisher noch nicht allgemein nutzbar, da noch diverse manuelle Eingriffe nötig sind.

header, leap_frog, modules, parin, plot_particles, read_var_list, write_var_list

Neu:
plot_isosurface

10/01/00

1.2

Randbedingungen für w (=0) werden explizit gesetzt, und zwar nicht nur bei k = nzb, nzt+1 sondern auch bei k = nzt, wo bislang der vom vorigen Zeitschritt durch den Drucklöser ermittelte Wert stand. Auch jetzt wird der Wert bei k = nzt anschließend durch den Drucklöser modifiziert. Soweit bis jetzt abzusehen, hat diese Änderung nur minimale Auswirkungen auf den Prognoseverlauf. Evtl. gibt es stärkere Auswirkungen bei exzessivem Auftreten von Schwerewellen am oberen Rand (die dort aber eigentlich sowieso nicht hingehören).
Variablenübergabe an boundary_conds erfolgt
über Modul und nicht mehr per Parameterliste.

Bei Galilei-Transformation wird jetzt defaultmäßig der geostrophische Wind als Translationsgeschwindigkeit verwendet (neuer Initialisierungsparameter use_ug_for_galilei_tr). Dadurch wird das verstärkte Auftreten von Schwerewellen bei höheren geostrophischen Windgeschwindigkeiten unterdrückt.

3D-Feldvariablen (u, v, w, pt, e, km, kh) sowie einige 2D-Variablen (usws, vsws, rif, shf) sind nun als Pointer deklariert, denen nach jedem Zeitschritt neue Targets zugeordnet werden (für die benötigten 3 Zeitebenen existieren die Targets ..._1 (z.B. u_1), ..._2, ..._3). Durch die jeweilige Neuzuordnung der Targets entfällt das Umspeichern der Zeitebenen komplett (Zeitersparung: ca. 5% der Gesamtrechenzeit).
Die Umordnung der Zeitebenen wurde ins neue UP swap_timelevel ausgelagert.
Wichtig: Die Übersetzungszeiten haben durch die Einführung der Pointer t.w. sehr deutlich zugenommen (leap_frog benötigt jetzt z.B. ca. 60 Sekunden). Außerdem benötigt der Compiler jetzt wesentlich mehr Speicher. Der "memory"-Parameter in den Konfigurationsdateien (.mrun.config, .al.config) muss nun mindestens den Wert 60 (MB) (t3eh, t3ej2) bzw. 45 (t3eb) und 96 (vpp) haben.
Achtung: bislang wurde diese Änderung nur für das reine Leapfrog-Verfahren getestet!

Neue Laderoption "-D preset=nan" empfohlen (s. Beispiel-Konfigurationsdatei). Ebenso empfehlenswert für Debug-Läufe: Compileroption "-ei". Diese bewirkt einen Programmabbruch, falls mit nicht initialisierten Variablen gearbeitet wird (ist aber zeitaufwendig in der Ausführung!).

boundary_conds, check_parameters, header, init_3d_model, leap_frog, modules, parin, read_var_list, timestep, write_var_list

Neu:
swap_timelevel

10/01/00 SR 1.2 B Der Asselin-Filter wird vor Umordnung der Zeitebenen aufgerufen. Dies geschah bisher fälschlicherweise erst danach, sodass der Zeitfilter asymmetrisch arbeitete.
Achtung: diese Änderung hat Auswirkungen auf den Prognoseverlauf. Die Testergebnis-Datei wurde entsprechend geändert. leap_frog

14/02/00

1.2

B/C

1D-Modell:
Der Asselin-Filter im 1D-Modell wird nun auch vor der Umordnung der Zeitebenen aufgerufen (s.o.).

3D-Modell:
TKE wurde bisher bei scalar_advec=bc-scheme einer zeitlichen Filterung unterzogen. Dies wird nun unterbunden.

Die Divergenz der Impulsflüsse an der Prandtl - Schicht - Obergrenze wird nun über eine ganze Gitterweite gebildet. Die bisherige Methode lieferte im Vergleich mit Ergebnissen anderer LES-Modelle zu große Impulsflüsse.

Bei der Berechnung der Diffusion der Temperatur werden die Diffusionskoeffizienten nun wieder durch Mittelung für die entsprechenden Gitterpunkte bereitgestllt. Das Maximumkriterium, das im Zuge der Implementierung des Upstream-Spline Advektionsverfahrens eingebaut wurde, bewirkte ein zu rasches Anwachsen der Grenzschicht mit der Zeit.

Achtung: diese Änderungen haben Auswirkungen auf den Prognoseverlauf. Die Testergebnis-Datei wurde entsprechend geändert.

init_1d_model, asselin_filter, diffusion_u, diffusion_v,
diffusion_pt

17/02/00

2.0

Allgemeine Beschreibung der Änderungen:
vollständige Implementierung des wolkenphysikalischen Moduls

Zur Implementierung der Wolkenphysik waren umfangreiche Änderungen in zahlreichen Programmteilen notwendig (Beschreibung siehe unten ). Im einzelnen wurden folgende Unprogramme geändert:
modules:
1) Bereitstellung neuer 3D-, 2D- und 1D-Felder für die Prognose des Gesamtwassergehaltes
2) "cloud_parameters" als neues Modul mit wolkenphysikalischen Variablen
parin: Initialisierungsparameterliste wurde erweitert (s.u.)
read_var_list, write_var_list, write_3d_binary: In/Output der neuen Initialisierungsparamter und Felder für Fortsetzungsläufe
check_parameters: Initialisierung des Feuchteprofils und Ausgabe von neuen Vertikalprofilen:
vpt: virtuelle poteniellen Temperatur,
lpt: potentielle Flüssigwassertemperatur,
q: Gesamtwassergehalt,
qv: spezifische Feuchte,
ql: Flüssigwassergehalt,
w"vpt", w*vpt*, wvpt: Auftriebsfluss (subskalig, aufgelöst, gesamt),
w"q", w*q*, wq: Wasserfluss (subs., aufg., ges.),
w"qv", w*qv*, wqv: Feuchtefluss (subs, aufg., ges.)
init_3d_model: Initialisierung aller Feuchte/Wasserfelder
init_1d_model: Anpassung an Rechnungen mit Feuchte
header: Headerausgaben für Feuchtebehandlung und Wolkenphysik eingefügt, außerdem Formatnummern erweitert um Platz für weitere Ausgaben zu schaffen
leap_frog:
1) alle prognostischen Gleichungen wurden in prognostic_equations ausgelagert
2) Aufrufe der neuen Unterprogramme (UP) compute_vpt, calc_liquid_water_content
3) veraenderter Aufruf von diffusivities
boundary_conds, asselin_filter, swap_timelevel, flow_statistics: Anpassung an Feuchtebehandlung
plot2d, plot3d: Ausgabemöglichkeiten von q und ql
buoyancy: Anpassung des Autriebstermes an Feuchtebehandlung durch neue Übergabeparameter: Temperatur + Nummer des entsprechenden mittleren Temperaturprofils
diffusion_e: Anpassung der Mischungswegberechnung an Feuchtebehandlung durch Variation der Übergabeparameter
production_e: TKE-Produktion durch Auftriebskräfte an Feuchtebehandlung angepasst
diffusivities: siehe diffusion_e
prandtl-fluxes: Prandtl-Schicht fuer den Gesamtwassergehalt eingefügt

Zusätzlich wurden folgende Unterprogramme NEU eingeführt:
inti_cloud_physics: Initialisierung wolkenphysikalischer Parameter
prognostic_equations: Auslagerung der prognostischen Gleichungen aus leap_frog, zusätzliche porgnostische Gleichung für den Gesamtwassergehalt
diffusion_s: Berechnung der Diffusion skalarer Größen (Temperatur und Gesamtwassergehalt), ersetzt diffusion_pt
calc_liquid_water_content: Berechnung des Flüssigwassergehalts
calc_radiation: Parametrisierung langwelliger Strahlungsprozesse über des Schema der effektiven Emissivität
calc_precipitation:Parametrisierung von Niederschlagsprozessen mit Hilfe eines vereinfachten Kesslerschemas
impact_of_latent_heat: Berücksichtigung der durch Niederschlagsprozesse zurückbleibenden latenten Waerme innerhalb eines Gittervolumens
compute_vpt: Berechnung der virtuellen potentiellen Temperatur

Liste der neuen Initialisierungsparameter:

Parameter Typ Default Beschreibung

moisture L FALSE Ab-/Zuschalten des Gesamtwassergerhaltes als neue progn. Variable

cloud_physics L FALSE Ab-/Zuschalten des Kondensationsschemas

radiation L FALSE Ab-/Zuschalten des Strahlungsschemas

precipitation L FALSE Ab-/Zuschalten der Niderschlagsparametrisierung

bc_q_b C 'dirichlet' untere Randbedingung für q (siehe auch bc_pt_b)

bc_q_t C 'neumann' obere RB für q (siehe auch bc_pt_t)

q_surface R 0.0 Feuchtewert an der Erdoberfläche

q_surface_initial_change R 0.0 vgl. pt_surface_initial_change

q_vertical_gradient R(10) 0.0 vgl. pt_vertical_gradient

q_vertical_gradient_level R(10) 10000.0 vgl. pt_vertical_gradient_level

surface_pressure R 1013.25 Luftdruck an der Erdoberfläche

surface_waterflux R 0.0 oberflächennaher Wasser/Feuchtefluss

Fortsetzungsläufe mit Daten, die mit Modellversionen kleiner 2.0 erzeugt wurden, sind aufgrund der erweiterten Initialisierungsparameterliste INKOMPATIBEL zu den Einleseroutinen der Version 2.0. Ein Update auf Version 2.0 sollte deshalb nach Abschluss eines kompletten Modelllaufs erfolgen.

Achtung: diese Änderungen haben Auswirkungen auf den Prognoseverlauf. Die Testergebnis-Datei wurde entsprechend geändert.

modules, parin, read_var_list, check_parameters, init_3d_model, init_1d_model, header, leap_frog. buoyancy, diffusion_e, production_e, diffusivities, prandtl_fluxes, boundary_conds, asselin_filter, swap_timelevels, flow_statistics, plot_2d, plot_3d, write_var_list, write_3d_binary

Neu:
init_cloud_physics, prognostic_equations, diffusion_s, calc_liquid_water_content, calc_radiation, calc_precipitation, impact_of_latent_heat, comute_vpt

18/02/00

2.0

B/N

Im Zuge der Umstellung auf Modellversion 2.0 wurden noch kleine Änderungen an den Unterprogrammen diffusion_e, production_e und diffusivities vorgenommen. In allen genannten
Unterprogrammen wurde bisher ein strengeres Kriterium zur Bestimmung der Schichtungsverhältnisse verwendet. Ein Modellvergleich zeigte jedoch, dass dadurch die Diffusion im Bereich der Inversion zu gross wird.

Eine weitere Ergänzung betrifft die Advektionsverfahren:
Das Bott-Chlond Advektionsverfahren ist nun auch für Rechnungen mit Feuchte/Wolkenphysik verfügbar.

Achtung: diese Änderungen haben Auswirkungen auf den Prognoseverlauf. Die Testergebnis-Datei wurde entsprechend geändert.

diffusion_e, production_e, diffusivities, check_parameters

26/04/00

2.0a

Vollständige Umstellung von vtk-Grafik-Software auf dvrp-Software. Grafik-Ausgaben im dvr-Format sind mittlerweile begrenzt nutzbar (Isooberflächen sind darstellbar, siehe neue Laufparameter dt_dvrp, threshold), es wird aber in der Anwendung in naher Zukunft noch diverse Änderungen geben. Partikelausgabe vorerst nicht mehr möglich (Verlagerung schon).

header, init_particles, init_3d_model, leap_frog, modules, parin, parles, read_var_list, write_var_list
Gestrichen:
plot_isosurface, plot_particles

Neu:
init_dvrp, plot_dvrp

26/04/00 SR 2.0a C Durch zusätzlichen zyklischen Rand im Bott-Chlond-Schema sehr viele SENDRECV-Aufrufe eingespart.
Änderung des Namens einer eingelesenen Environment-Variablen in check_open von remote_addres nach return_addres.
Prozessor-Topologie kann durch Benutzer vorgegeben werden. Dazu neue Initialisierungsparameter npex und npey. Durch Änderungen am Header-Format wurde Testergebnis-Datei geändert. advec_s_bc, check_open, header, init_pegrid, modules, parin, read_var_list, write_var_list

26/04/00

2.0a

Rechnungen mit geneigter Oberfläche erlauben jetzt die Vorgabe eines stabil geschichteten Temperaturprofils mit konstantem Gradienten. Hinzuschalten von Feuchte funktioniert für Hangrechnungen noch nicht.

advec_s_bc, buoyancy, init_3d_model, modules, prognostic_equations

Neu:
init_slope

13/06/00 MS 2.0a B Es werden nun auch die seitlichen Ränder für ql gesetzt. Ein Nichtsetzen der Randbedingungen führte zu Fehlern bei der Ausgabe von pt calc_liquid_water_content

03/07/00 SR 2.0b C Für eine Reihe von Unterprogrammen, die innerhalb von prognostic_equation aufgerufen werden und an die Pointer-Variablen übergeben werden, sind jetzt explizite Interfaces vereinbart (neues Modul pointer_interfaces). Auch im Vereinbarungsteil der entsprechenden Unterprogramme sind diese Übergabeparameter jetzt aus Konsistenzgründen (und weil es sonst Laufzeitfehler gibt) als Pointer-Variablen deklariert. Mit dieser Änderung ist der große Speicher- und CPU-Zeit-Bedarf beim Übersetzen von prognostic_equations wieder auf ein erträgliches Maß reduziert.
Im Rahmen dieser Änderungen wurde auch an diffusion_e ein Hilfsfeld weniger übergeben.
Aus nicht vollständig gekärten Gründen haben sich die Zahlen in der Testergebnis-Datei minimal geändert (und zwar die Spalte mit der Gesamtdivergenz nach Aufruf des Drucklösers). advec_s_bc, buoyancy, diffusion_e, diffusion_s, diffusion_u, diffusion_v, diffusion_w, disturb_field, modules, production_e, prognostic_equations

04/07/00 SR 2.0b B Diriclet-Randbedingungen für Temperatur und Flüssigwassergehalt werden gesetzt. Dies wäre bereits nach Einführung der Pointer zwingend notwendig gewesen. Rechnungen mit vorgegebener Oberflächentemperatur schlugen deshalb bislang fehl. boundary_conds

07/09/00 MS 2.0b B Die virtuelle potenielle Temperatur wird nun auch für k=nzb und k=nzt+1 berechnet, um die Randwerte auch für Ausgabezwecke bereichtzustellen. compute_vpt

28/12/00

2.1

C/N

PALM erlaubt jetzt den Einsatz optionaler Software-Pakete, die zwar zum Modell gehören, standardmäßig aber nicht mit übersetzt werden, um so unter anderem Kompilationszeit einzusparen. Siehe neues Kapitel 3.7 in der Modelldokumentation.

Einsatz der dvrp-Software wurde vollständig überarbeitet. Sie ist nun als Software-Paket optional im Modell einsetzbar. Siehe überarbeitetes Kapitel 4.5.6 in der Modelldokumentation.

Die Prognose der Partikelverlagerung ist ebenfalls in ein optionales Software-Paket ausgelagert.

advec_particles, header, init_3d_model, init_dvrp, init_particles, init_pegrid, modules, parin, parles, plot_dvrp, prognostic_equations, read_var_list, write_var_list

Neu:
package_parin

28/12/00 SR 2.1 B MPI_FINALIZE ans Ende des Hauptprogramms geschoben (stand vorher vor cpu_auswertung, wo noch ein Barrier-Aufruf steht). comm2d war zu Beginn undefiniert und wird nun erst einmal gleich MPI_COMM_WORLD gesetzt. parles

02/01/01

2.1a

Offene Dateien werden sobald möglich geschlossen. Wiedereröffnung bestimmter Dateien mit POSITION='APPEND' möglich. In diesem Zusammenhang wurde das Unterprogramm close_files in close_file umbenannt und hat nun 1 Argument.
Anpassung der Diffusionsparametrisierung im 1D-Modell an das 3D-Modell.

advec_particles, check_cpu_time, check_open, cpu_auswertung, init_1d_model, init_3d_model, init_particles, modules.f90, parin, parles, plot_2d, write_3d_binary

Neu:
close_file

Eliminiert:
close_files

02/01/01 SR 2.1a B Beseitigung kleiner Fehler. check_cpu_time, diffusion_e, diffusivities, long_filter, production_e

05/01/01

2.1b

Neues Software-Paket zur Berechnung von Spektren im Ortsraum

check_open, header, init_3d_model, leap_frog, modules, package_parin, parles, read_var_list, write_3d_binary, write_var_list

Neu:
calc_spectra, plot_spectra

25/01/01

2.1c

C/N

Drucklöser kann jetzt die Singleton-FFT benutzen. Modul singleton wurde deshalb von calc_spectra.f90 nach modules.f90 geschoben (neuer Initialisierungsparameter fft_method).
Flüsse können jetzt bei k=1 unabhängig von der Verwendung einer Prandtl-Schicht vorgegeben werden (neuer Initialisierungsparameter use_surface_fluxes).
Modul test_variables wurde aus allen Programmteilen entfernt. Hauptprogramm wurde von parles nach palm umbenannt.
Modell kann auf DEC-Workstations eingesetzt werden (-D dec).
Zusätzliche Zeitmessungen mit dvrp-Software. Schreiben von Partikel-Informationen ist optional (neuer Parameter write_particle_informations).

advec_particles, calc_spectra, check_parameters, cpu_zeitmessung, diffusion_s, header, init_3d_model, init_dvrp, init_particles, leap_frog, modules, package_parin, parin, poisfft, read_var_list, swap_timelevel, write_var_list

Neu:
palm

Eliminiert:
parles, module_test

25/01/01 SR 2.1c B Fehler beim öffnen von Unit 23 beseitigt. Modul interface fehlte in disturb_field.
Wertebereich von theta* in Prandtl-Fluxes eingeschränkt, weil sonst auf einigen Rechnern overflow droht, wenn u*=0. check_open, disturb_field, prandtl_fluxes

30/01/01

2.1d

N/C

Prognose eines passiven Skalars ist möglich (anstatt Feuchte), dazu neuer Initialisierungsparameter passive_scalar.

Falls kein Pfad für ftpcopy existiert, werden in check_open die Dateinamen für AVS-Koordinaten- und Datendatei auf "unknown" gesetzt.

asselin_filter, boundary_conds, check_open, check_parameters, flow_statistics, header, init_1d_model, init_3d_model, modules, parin, plot_2d, plot_dvrp, prandtl_fluxes, prognostic_equations, read_var_list, swap_timelevel, write_3d_binary, write_var_list

30/01/01 SR 2.1d B String-Ausgabe für use_surface_fluxes korrigiert. Version vom 25/01/01 brach deswegen bei Fortsetzungsläufen ab.
xy-Plotausgabe von q und ql korrigiert. plot_2d, write_var_list

30/03/01

2.2

N/C

Sämtliche 3D-Felder der Zeitebene t+dt wurden entfernt. Ebenso die Arbeitsfelder work und work1. Hilfsfelder wurden soweit möglich eingespart und werden grundsätzlich lokal allokiert. Dadurch konnten eine Reihe von Übergabeparametern gestrichen werden (z.B. beim Aufruf von pres, advec_s_bc, poisfft, production_e, sor sowie den Spline-Unterprogrammen). Der Long-Filter wurde in das Upstream-Spline-Verfahren integriert. Da der Zeitfilter jetzt direkt in der prognostischen Gleichung angewendet wird (bisher geschah dies erst nach Aufruf des Drucklösers), haben sich die Zahlen in der Testergebnis-Datei geändert. Die mittleren Profile sind aber unbeeinflusst.

Eine weitere kleinere Änderung der Testergebnis-Daten wird dadurch verursacht, dass jetzt der Parameter adjust_mixing_length defaultmäßig .FALSE. ist und der Mischungsweg grundsätzlich zusätzlich auf 0.7 * zu begrenzt wird. Falls adjust_mixing_length=T, wird jetzt der Mischungsweg in diffusivities analog zu diffusion_e an allen Gitterpunkten modifiziert. Achtung: Diese Änderungen können bei anderen Simulationen t.w. erhebliche Auswirkungen auf die oberflächennahen Ergebnisse haben.

Kleinere Änderungen: In init_cloud_physics wird der Wert der Variablen surface_pressure in hPa belassen (bisher Umwandlung in Pa).
Die Namelist-Namen der Software-Pakete sind t.w. geändert.
Unit 14 (binäre Ausgabe für Fortsetzungsläufe) wird jetzt im Hauptprogramm geschlossen, damit auf diese Datei noch benutzergesteuerte Ausgaben erfolgen können.
Laufparameter werden in keinem Fall mehr auf Unit 14 ausgegeben bzw. von Unit 13 gelesen, d.h. sie gelten jetzt tatsächlich nur für den jeweils aktuellen Lauf.
Partikeladvektion funktioniert jetzt auch zusammen mit der Galilei-Transformation.

Restliche deutsche Variablen-, Unterprogramm- und Modulnamen wurden ins Englische übersetzt. Davon sind fast alle Programmteile betroffen (nicht in rechter Spalte aufgeführt).

advec_particles, advec_s_bc, advec_s_ups, advec_u_ups, advec_v_ups, advec_w_ups, boundary_conds, calc_spectra, check_parameters, diffusion_e, init_3d_model, init_cloud_physics, init_pegrid, init_rankine, leap_frog, modules, package_parin, palm, parin, poisfft, pres, production_e, prognostic_equations, read_var_list, sor, swap_timelevel, transpose_*, write_3d_binary, write_var_list

Neu:
cpu_log, cpu_statistics

Eliminiert:
asselin_filter, cpu_auswertung, cpu_zeitmessung, long_filter

12/07/01 SR 2.2a N/C Defaultwert der unteren Randbedingung für die TKE ist ab sofort bc_e_b = 'neumann'.
Partikelquelle ist über Paketparameter steuerbar. Partikeleigenschaften sind mittels benutzereigener Software steuerbar. Der dvrp-Ausgabeweg ist jetzt ebenfalls über Paketparameter steuerbar.
Allen Modulen (bis auf singleton) wurde eine SAVE-Anweisung hinzugefügt, damit die durch sie vereinbarten Variablen während der Rechnung in keinem Fall undefiniert werden können (diese Gefahr besteht z.B. auf SGI-Origin-Maschinen). advec_particles, header, init_dvrp, init_particles, modules.f90, package_parin, plot_dvrp, user_interface

12/07/01 SR 2.2a B Verschiebung der OPEN-Anweisung für Unit 33 (AVS-FLD-Datei). check_open

20/07/01

2.3

Einbau des Mehrgitterverfahrens zur Lösung der Poisson-Gleichung.

check_parameters, exchange_horiz, header, init_grid, init_pegrid, modules, parin, pres, run_control

Neu:
poismg

21/08/01

2.3a

N/C

Ergänzung der dvrp-Software: Partikel können mit Schwänzen versehen werden. Die Ausgabe der dvrp-Plotdaten durch einen separaten PE ist möglich (mrun-Option "-p dvrp_graphics+1PE"). Der Kommunikator MPI_COMM_WORLD wurde deshalb durch comm_palm ersetzt.

Reduzierung des Mischungsweges im Wandbereich kann wahlweise abgeschaltet werden. Für km und kh wird jetzt eine Neumann-Randbedingung am unteren Rand verwendet. Die TKE-Energieproduktionsberechnung durch Scherung des Grundstroms am unteren Rand wurde verbessert.

Generelle Verwendung der bodennahen vertikalen Impulsflüsse in den Diffusionstermen von u und v, immer wenn diese als Randbedingung vorgegeben sind (gemäß use_surface_fluxes, - bisher wurde dies nur bei eingeschalteter Prandtl-Schicht gemacht). Schubspannungen können jetzt als Randbedingungen vorgegeben werden (allerdings unter zuhilfenahme benutzereigener Software).

Zusätzliche Schnittstellen für benutzereigene Software in allen Tendenztermen, in flow_statistics, und für Partikel- bzw. dvrp-Programmteile (Bestimmung von Partikeleigenschaften, Festlegung von Farbtabellen).

Achtung: Diese Änderungen können bei Simulationen mit mittlerem Wind durch die Änderungen im bereich des unteren Randes t.w. erhebliche Auswirkungen auf die oberflächennahen Ergebnisse haben.

advec_particles, diffusion_e, diffusion_u, diffusion_v, diffusivities, flow_statistics, header, init_dvrp, init_particles, init_pegrid, leap_frog, modules, package_parin, palm, parin, plot_dvrp, production_e, prognostic_equations, read_var_list, swap_timelevel, user_interface, write_var_list

21/08/01 SR 2.3a B Falsche Positionierung von MPI_ALLREDUCE in check_cpu_time korrigiert. check_cpu_time

04/09/01 SR 2.3b N Zusätzliche Profilausgaben für Energieproduktionsterme möglich. Stördruck wird bei der FFT-Methode nicht mehr durch Aufsummieren der Werte zu den einzelnen Zeitschritten ermittelt. check_parameters, flow_statistics, modules, pres

04/09/01

2.3b

Austausch der Geisterränder für die zeitgefilterten Felder direkt nach Durchführung der Filterung (war fehlerhaft seit Entfernen der Zeitebene t+dt (Version 2.2) und führte zu sehr kleinen Störungen an den seitlichen Rändern der Teilgebiete).

Achtung: Der Inhalt der Testergebnis-Datei ändert sich durch diese Fehlerkorrektur.

prognostic_equations

09/11/01 SR 2.3c N Farbe entlang der Partikelschwänze kann sich ändern. Einschalten der Partikeladvektion bei Fortsetzungsläufen möglich. advec_particles, init_particles, modules, package_parin, plot_dvrp

16/04/02

2.3d

Vorgabe von Randbedingungen für Partikeladvektion möglich. Partikeldaten können für spätere Analysen auf Datei geschrieben werden (Unit 85). PARTICLE-Datentyp enthält Informationen über Partikelgeschwindigkeitskomponenten und Startposition.

Skalartransport sowie entsprechende Datenausgaben können durch eigene Parameter gesteuert werden (nicht mehr über die Feuchteparameter).

Im Fall von pdims(1)=1 (eindimensionales virtuelles Prozessornetz in y-Richtung) werden die zyklischen Randbedingungen in x-Richtung durch direktes Umspeichern anstatt durch sendrecv realisiert. Ebenso werden die Transponierungen xz, yz, zx und zy eingespart (es wird aber weiter umsortiert).

Im Fall von 3D-Plotausgaben Aufruf von ftpcopy-Script durch batch_scp-Script ersetzt (dient der Ermittelung von Dateizyklusnummern als Information für die AVS-fld-Datei).

Bei fehlerhafter Eröffnung einer CPU_MEASURES-Datei werden erneute Eröffnungsversuche durchgeführt.

advec_particles, check_open, check_parameters, exchange_horiz, flow_statistics, header, init_particles, modules, package_parin, parin, plot_2d, plot_3d, prognostic_equations, read_var_list, transpose_xz, transpose_yz, transpose_zx, transpose_zy, write_var_list

02/05/02

2.3e

Wiedereinführung der 3D-Felder für die Zeitebene t+dt, die notwendig ist, damit in den Diffusionstermen bei Leapfrog-Zeitschritten mit der korrekten Zeitebene t-dt gearbeitet wird. Seit Version 2.2 wurde fälschlicherweise die Zeitebene t+dt verwendet. Asselin-Filter ist jetzt wieder eigenständiges Unterprogramm.
Die Wiedereinführung dieser dritten Zeitebene ist auch Voraussetzung für die in der nächsten Version geplante skalare Optimierung.

Fehler bei Berechnung von Ausgabezeitpunkten eliminiert (trat auf, wenn bei Fortsetzungsläufen von Ausgabeabständen 0.0 auf von Null verschiedene Werte gewechselt werden sollte).

Achtung: Der Inhalt der Testergebnis-Datei ändert sich durch diese Fehlerkorrektur.

init_3d_model, leap_frog, modules, prognostic_equations, swap_timelevel

Neu:
asselin_filter

02/05/02

2.3e

N/C

Kleinere Anpassungen an IBM-Regatta-Systeme in check_parameters, cpu_log.

Modul singleton in eigenständige Datei überführt.

Global_min_max arbeitet mit REAL*4, um Komplikationen auf 32-bit-Rechnern zu vermeiden (Datentyp MPI_2REAL müsste dort sonst auf MPI_2DOUBLE_PRECISION geändert werden).
Horizontale Geschwindigkeitskomponenten für Partikeladvektion werden exakt zwischen den jeweils benachbarten horizontalen Gitterflächen interpoliert.

advec_particles, check_parameters, cpu_log, global_min_max, modules

Neu:
singleton

11/06/02

2.4

N/C

Optimierung und Anpassungen für einzelne Knoten der IBM-Regatta-Systeme.

Skalare (Cache) Optimierung: Tendenzterme werden innerhalb einer großen (i,j)-Schleife berechnet, die nun jeweils die gesamte prognostische Gleichung umfasst. Schleifenindices i und j werden an die Tendenzunterprogramme als Argumente übergeben. Die Tendenzunterprogramme sind als Module geschrieben, die mittels Technik der überladenen Funktionen auch ohne diese Indices i und j aufgerufen werden können, und dann wie in früheren Versionen funktionieren (d.h. in ihnen laufen die Schleifen über alle 3 Dimensionen). Solche Teile der prognostischen Gleichungen, die globale Kommunikation beinhalten, sind vor die (i,j)-Schleife gezogen. Falls ausschließlich mit Piascek-Williams-Advektion gearbeitet wird, werden alle prognostischen Gleichungen in einer einzigen (i,j)-Schleife gerechnet.

Kommunikationsoptimierung: Zusätzliche Implementierung einer 1D-Gebietszerlegung in x-Richtung. Mit dem neuen direkten Drucklöser (poisfft_hybrid) werden dadurch 4 von 6 notwendigen Transpositionen eingespart. Die 1D-Zerlegung wird automatisch bei Verwendung dieses neuen Lösers aktiviert. Austausch von Geisterrändern in y-Richtung wird bei dieser 1D-Zerlegung durch einfache zyklische Randbedingungen ersetzt.

Kleinere Anpassungen: String-Vergleiche erfordern beim IBM-Compiler in der Regel den Einsatz der TRIM-Funktion, um überschüssige Blanks am Ende des Strings abzuschneiden (sonst liefert der Vergleich .F.).
Lokal allokierte Felder t.w. in sogenannte automatische Felder umgewandelt (günstiger, um Gefahr von eventuellen Speicherlecks zu vermeiden). Einige Feldoperationen wurden wegen schlechter Performance auf der IBM entfernt (s. pres).

Achtung: Die Übersetzung des Modells erfordert aufgrund der Vielzahl nunmehr eingesetzter Module die Verwendung des make-Mechanismus, um die korrekte Reihenfolge bei der Übersetzung der Programmteile zu gewährleisten. Die Beschreibung dieser Abhängigkeiten erfolgt in der Datei Makefile, die zusammen mit den Programmdateien abgelegt ist und ebenfalls unter RCS-Verwaltung steht.

Im User-Interface ist das Unterprogramm user_actions nun ebenfalls als Modul geschrieben und bedient sich der Methode der überladenen Funktionen. Existierende benutzereigene Software muss entsprechend angepasst werden.

advec_s_pw, advec_s_up, advec_u_pw, advec_u_up, advec_v_pw, advec_v_up, advec_w_pw, advec_w_up, buoyancy, calc_precipitation, calc_radiation, calc_spectra, check_parameters, coriolis, diffusion_e, diffusion_s, diffusion_u, diffusion_v, diffusion_w, diffusivities, exchange_horiz, exchange_horiz_2d, global_min_max, header, impact_of_latent_heat, init_pegrid, leap_frog, modules, plot_2d, plot_3d, plot_spectra, poisfft, pres, production_e, prognostic_equations, user_interface,

Neu:
fft_for_1d_decomp, Makefile, poisfft_hybrid,

11/06/02 SR 2.4 B Fehlende Variablenvereinbarung in advec_particles ergänzt. advec_particles

12/09/02 SR 2.4a N Partikel können mit Trägheit versehen werden und einen Dichteunterschied zum umgebenden Fluid haben. Zur Steuerung dient der neue Paketparameter density_ratio. advec_particles, header, init_particles, modules, package_parin

12/09/02

2.4a

Fehler in Berechnung der Anfangsprofile von pt und q entfernt (obere Feldgrenze von 10 konnte in pt_vertical_gradient_level_ind überschritten werden). Fehler führte unter Umständen zu "segmentation fault" bei Verwendung sehr großer Gitterpunktzahlen in z-Richtung.

Fehler in Berechnung der v-Komponente am unteren Rand entfernt, die für den Scherungsproduktionsterm der TKE verwendet wird (Variable v_0). Fälschlicherweise wurde zur Berechnung u(k=1) statt v(k=1) verwendet.

check_parameters, modules, production_e

19/12/02

2.5

Restart times can be set by user with new runtime (d3par) parameters restart_time and dt_restart. Run description header is written on file CONTINUE_RUN. Output of cpu statistics changed (PE results are now collected on PE0 in order to calculate mean cpu statistics). Output of warnings in the job protocol in case of negative measured cpu-times. Remaining cpu-time is also evaluated on IBM-Regatta.

cpu_log, cpu_statistics, check_parameters,header, leap_frog, local_tremain, local_tremain_ini, modules, palm,

new:
check_for_restart

deleted:
check_cpu_time

Unit 14 (BINOUT) must be opened using a special process id string, because on IBM, the PE rank differs between communicators MPI_COMM_WORLD and comm_2d. This leads to a mismatch between filename-PE-number and PE-number written on file. As the consequence, these files could not be read by restart-jobs. Reading of array hom moved from init_3d_modelto read_var_list (binary version number incremented to 2.2).

Further optimization for IBM-Regatta-systems:
Additional optimization of the hybrid-solver for multinode usage (overlapping of communication and computation).

Further cache optimization by using strides and joining loops (hybrid_solver, pres and timestep). Joining of MPI_ALLREDUCE calls in timestep.

In case of 1d-decomposition along x only a part of the integral divisor conditions is checked, on IBM hosts a 1d-decomposition along x is the default, a switch from Poisson-FFT-solver to the hybrid-solver is made in case of 1d-decomposition along x, the hybrid-solver does not force a 1d-decomposition any more.

Array notation changed to do-loop constructs due to better performance.

Cyclic boundary conditions along y used instead of sendrecv in case of a 1d-decomposition along x. SENDRECV replaced by nonblocking routines ISEND and IRECV in exchange_horiz.f90.

Speed optimization by removing MINVAL/MAXVAL calls and by handling
the "abs" case in a different way than the min/max cases (routine global_min_max).

asselin_filter, check_open, exchange_horiz, global_min_max, init_3d_model, init_pegrid, parin, poisfft_hybrid, read_var_list, timestep, write_3d_binary, write_var_list

Correction of mixing length term (l(k)/ll(k)). The condition kh=3*km in the unstable case is now also exactly met in the wall adjustment region. Factor 0.7 in wall adjustment part replaced by variable wall_adjustment_factor, which is set to 1.8 in modules.f90. The factor 0.7 was the possible reason for 2-delta-x-waves, which were observed since version 2.1d.

Calculation of deformation tensor re-designed (production_e).

STOP statements replaced by call of new subroutine local_stop,where MPI_FINALIZE is called before STOP in case of a parallel environment.

ISO2D parameter dp set to true for ibm hosts.

tend=p added in pres after calling sor method.

advec_s_bc, buoyancy, check_open, check_parameters, close_file, coriolis, diffusion_e, diffusivities, fft_for_1d_decomp, flow_statistics, init_3d_model, init_dvrp, init_grid, init_particles, init_pegrid, parin, plot_2d, poisfft, pres, read_var_list, user_interface,

new:
local_stop, production_e

03/03/03 SR 2.5a N Particle velocities are also stored in array particles in case of zero density ratio. Steering of variables by dvrp browser included. advec_particles, init_dvrp, modules,

C AVS data format changed from float to xdr_float (needed on linux machines
due to the little/big endian problem). Updates for new version of dvrp software (e.g. using module dvrp is now mandatory). close_file, init_dvrp, leap_frog, plot_dvrp

B Error in particle inertia part removed (exp_arg must not contain the timestep).
Error in calculation of the vertical flux of resolved scale energy (profile 57) removed. Displacement for integers in mpi_particle_type reduced from 16 to 8 on ibm. advec_particles, flow_statistics, init_particles

12/03/03

2.6

Version optimized for NEC-SX6 parallel-vector machines.

There are two main changes. A new vectorizable routine prognostic_equations_vec is added, where cache optimizations are undone. The tendency subroutines called by prognostic_equations_vec contain the full 3d-loops (compared with the cache-optimized versions where only loops over k are carried out). Additionally, a new pressure solver poisfft_hybrid_vec is added, which requires a 1d-domain-decomposition (like poisfft_hybrid). In this solver, a very fast FFT from Clive Temperton (ECMWF) is optionally available by choosing fft_method = 'temperton-algorithm'.

Additional changes in flow_statistics to allow better vectorization.

Small changes in routines diffusivities and pres which caused run time errors on IBM and NEC due to compiler problems.

Reading of environment variable tasks_per_node moved from routine poisfft_hybrid to routine parin.

Changes in makefile: modules and user_interface now depend on the f90 files, dependency of singleton added, LDFLAGS moved to the end of PROG rule, temperton_fft added.

advec_s_pw, advec_s_up, advec_u_pw, advec_u_up, advec_v_pw, advec_v_up, advec_w_pw, advec_w_up, buoyancy, calc_precipitation, calc_radiation, check_parameters, coriolis, cpu_log, diffusion_e, diffusion_s, diffusion_u, diffusion_v, diffusion_w, diffusivities, fft_for_1d_decomp, header, impact_of_latent_heat, init_pegrid, leap_frog, local_tremain, modules, parin, poisfft, poisfft_hybrid, pres, production_e, prognostic_equations, user_interface

new:
temperton_fft

B tend=p added in routine pres after calling sor method. pres

14/03/03

2.6a

Additional vector optimization for NEC-SX6.

Optional system-specific random number generator available (for equal distributed numbers).

Define strings in all relevant subroutines changed from

#if defined( define_string )

#if defined( __define_string )

Caution: The relevant subroutines are NOT listed in the right column!

asselin_filter, check_parameters, diffusion_e, diffusivities, disturb_field, global_min_max, header, local_tremain, modules, parin, poisfft_hybrid, pres, read_var_list, write_var_list

B Error in particle boundary condition removed (velocity must be inverted in case of reflection) advec_particles

16/04/03

2.6b

Time series output of Monin Obukhov length.

Temperton fft can now be used for all hosts and every domain decomposition. Abort in case of Temperton fft, if number of gridpoints along x and/or y contain illegal factors.

Index values for the extrema found in global_min_max are limited to the range 0..nx, 0..ny (on IBM machines -1 and nx+1/ny+1 occured which produced different RUN_CONTROL output compared to other machines).

Output format of iteration count in routine run_control enlarged.

check_parameters, fft_for_1d_decomp, global_min_max, init_pegrid, modules, plot_ts, poisfft, run_control, temperton_fft

Additional checks of variables hybrid_solver and host (hybrid_solver must not be used for a 2d-domain-decomposition, host should be set by setting environment variable localhost).

Header output for mixing length limitations revised.

check_parameters, header

09/05/03

2.7

Version running on Linux Clusters using MPICH and Intel FORTRAN compiler (ifc)

So far, only absolutely neccessary changes have been done (possible optimizations will follow in a later version):

New time measurements for Linux (ifc) environment added.

Some parameters in MPI calls had to be modified (arrays had to be replaced by the first element of the regarding array) in order to fulfill f90 type and rank requirements, since on the MUK-cluster a FORTRAN90-version of MPI is used. This also required to replace "mpif.h" by using a special module (named mpi). I did not find out how character strings can be send with MPI. Therefore, these strings are transformed to integers, before they are send, and transformed back to characters on the receiving PE.

On the MUK-cluster, only PE0 is able to read the values of environment variables. Therefore, these values are communicated via broadcast to the other PEs.

1d-decomposition is set as the default on Linux Clusters. Character strings have to be transformed to integer

Smaller changes:

Batch_scp paths for IBM, NEC and Linux added

check_open, cpu_log, flow_statistics, init_pegrid, local_getenv, modules, palm, poisfft_hybrid

B Measurements on IBM are now using function irtc, which allows correct measurements for jobs running over the 24:00 timeline. cpu_log, local_tremain, local_tremain_ini

01/08/03

2.7a

Check that the number processors is also an integral divisor of the number of gridpoints along y in case of a 1d-decomposition along x.

Error concerning the multinode-version of poisfft_hybrid removed. It was caused by the Linux changes of version 2.7 and led to program abort due to MPI errors in MPI_ALLTOALL.

No abort on t3e-systems if system specific routines are used.

Arraysumsis initialized in init_3d_model before the first call of pres. Call ofinit_cloud_physicsmoved before init_particles.

fft_1dd_init, init_3d_model, init_pegrid, poisfft_hybrid

N Interactive steering of dvrp-graphic features by dvrp-browser plugin extended (position of slicers, threshold values of isosurfaces) . New dvrp_graphics package parameter slicer_range_limits_dvrp. init_dvrp, modules, package_parin, plot_dvrp

29/10/03

2.7b

In the multigrid method, on a defined level, data are gathered on PE0 and further calculations are carried out only on this PE. Newd3parparameter mg_switch_to_pe0_level.

Particle groups implemented. New data typeparticle_groups_typeand data type particle_type modified. Version numbers are output on the particle files. New particles_par parameterdiameter. Parameter name uniform_psize changed todvrp_psize. Parameters density_ratio is now an array.

advec_particles, check_open, exchange_horiz, header, init_particles, init_pegrid, modules, package_parin, parin, poismg

Random number generator from numerical recipes is now rewritten as a module. In restart runs the seed of this generator is set to the last values of the previous run in order to keep the sequence of the random numbers. New binary version 2.2.

Buffer is flushed for file RUN_CONTROL immediately after every output on IBM and Linux cluster.

advec_particles, disturb_field, init_3d_model, init_particles, random_function, random_gauss, run_control, write_3d_binary

In routine buoyance, horizontal mean temperature is now taken from array hom instead of array sums (otherwise inconsistence in case of using more than one statistical region).

Information about the time of the last timestep change added to the restart file.

System call of batch_scp on IBM corrected.

buoyancy, check_open, read_var_list, write_var_list

28/01/04

2.8

Runge-Kutta schemes (2nd and 3rd order) for time integration implemented. The third order scheme is the new default. It allows the timestep to be 0.9*CFL, which is much larger than for the former default leapfrog scheme. The intermediate steps, which are part of these schemes, are realized by an additional loop within routine time_integration (currently leap_frog, but this routine will be renamed soon). Steering variables at and bt have been replaced by array sct. , which is particularly used in the prognostic equations. Values of this steering array are calculated within the new routine timestep_scheme_steering.

When using Runge-Kutta schemes, the timestep increment is freely allowed to adjust after each timestep (the older schemes are using some restrictions, see routine timestep). Also, routine run_control is not automatically called in case of timestep changes, when Runge-Kutta schemes are switched on.

The old leapfrog scheme is still implemented and should produce the same results as in the previous version(s)!

Depending on the user experience, further adjustments to the Runge-Kutta schemes may be necessary within the next minor versions.

Attention:
The content of the testresult-file (run control output) has significantly changed due to the new default timestep scheme!

check_parameters, header, init_3d_model, init_rankine, leap_frog, modules, prognostic_equations, read_var_list, swap_timelevel, timestep, write_var_list

new:
timestep_scheme_steering

Output of particle infos in subroutine allocate_prt_memory on demand only.

Type log (used for cpu time measurements) changed to logpoint due to name conflict with FORTRAN intrinsic log.

advec_particles, cpu_log,

E In case of opening unit 80, a barrier is set only for the first call of check_open (from routine init_particles), in order to avoid the possibility of hanging jobs, which may occur if unit 80 is opened within routine allocate_prt_memory. check_open

30/01/04

2.8a

In order to prepare the code for the simulation of flow around buildings (to be realized in one of the next major versions), the lower k index nzb has been replaced by a two-dimensional array nzb_2d in many of the three-dimensional loops. So far, all elements of this array are set to nzb.

This change should not effect simulation results and cpu time in any way.

advec_s_pw, advec_s_up, advec_u_pw, advec_u_up, advec_v_pw, advec_v_up, advec_w_pw, advec_w_up, asselin_filter, buoyancy, calc_liquid_water_content, calc_precipitation, calc_radiation, coriolis, diffusion_e, diffusion_s, diffusion_u, diffusion_v, diffusion_w, diffusivities, impact_of_latent_heat, init_3d_model, modules, production_e, prognostic_equations,

Euler step informations are removed in the run control output in case of Runge-Kutta schemes. Also, the timestep increment limitation is removed for the Runge-Kutta schemes.

Attention:
The content of the testresult-file (run control output) has changed due to this modification.

run_control, timestep

E Velocity gradients at the surface calculated in routine production_e are now limited. In case of inhomogeneous surface heating small diffusivities (km) sometimes caused very small timesteps due to this problem. production_e

30/04/04 SR 2.8b N The number of processors along the respective directions of the virtual processor grid must not be integral divisors of the number of gridpoints along x, y or z any more. In case that they are not integral divisors, the subdomains on the processors at the upper borders of the virtual grid will contain less gridpoints than the other subdomains (which are all of equal size). Although this causes a load imbalance, only runs with very small numbers of processors will feel a significant decrease of performance. In order to be able to carry out the transpositions, array d (and also working array tend) have the same size on all processors. This new feature is switched on by setting the inipar-parameter grid_matching = 'match', which is the default now.

Routine poisfft is now the default for solving the Poisson-equation for the case of 2d- as well as 1d-domain-decompositions and has been changed from a subroutine to a module. This routine now includes most of the optimization from the (Ketelsen) routine poisfft_hybrid and additionally allows 1d-decompositions along y.
ATTENTION: Routine poisfft_hybrid might still be faster when running PALM on more than one node of the IBM-Regatta, although the difference is not very big. The poisfft-code for a decomposition along y has still a poor performance and needs further optimization.

The spectra-package is now able to use all the available FFT-methods (selected by inipar-parameter fft_method ). Performance of the transpositions needed for calculating the spectra has been improved by using new direct transpositions from z to y and from y to x instead of two subsequent transpositions.

Runge-Kutta scheme implemented in the 1d-version (init_1d_model). Diffusion quantity arrays are allocated for previous timelevels only in case of leapfrog scheme.

This version is adapted for use on COMPAQ-DECALPHA systems (main changes: calling of SHAPE function within subroutine arguments (e.g. singleton) causes compiler errors, particle data type modified for better alignment, DATA statements from modules removed because they caused internal compiler errors). advec_s_ups, advec_u_ups, advec_v_ups, advec_w_ups, calc_spectra, check_parameters, fft_xy, header, init_1d_model, init_3d_model, init_particles, init_pegrid, modules, parin, plot_2d, poisfft, prandtl_fluxes, pres, read_var_list, singleton, spline_x, spline_y, spline_z, time_integration, transpose, write_3d_binary, write_var_list

C FFT routines renamed. Module fft_for_1d_decomp renamed fft_xy. Subroutine leap_frog renamed time_integration.

For all transpositions, MPI_ALLTOALL is used instead of MPI_ALLTOALLV. The names of many transposition indices have been changed.

Inipar-parameter impulse_advec renamed momentum_advec.

Reading of restart data is moved from init_3d_model to new routine read_3d_binary.

All former transpose routines collected into one file. check_parameters, flow_statistics, header, init_1d_model, init_3d_model, init_advec, init_pegrid, palm, parin, poisfft, poisfft_hybrid, production_e, prognostic_equations, read_var_list, sline_x, spline_y, write_var_list

new:
fft_xy, read_3d_binary, time_integration, transpose

deleted:
fft_for_1d_decomp, leap_frog, transpose_xy, transpose_xz, transpose_yx, transpose_yz, transpose_zx, transpose_zy

E Small errors in the particle-package removed, which sometimes caused aborts of restart runs on IBM.

Small error in calculating the time series profiles removed (array bounds of some arrays needed for profil-output had to be increased).

Function fft (singleton method) replaced by subroutine fftn due to problems with 64-bit mode on IBM advec_particles, check_parameters,

26/03/05 SR 2.9 N Non-cyclic lateral boundary conditions implemented.
Dirichlet/Neumann conditions at the inflow/outflow can be used either along x or along y. Cyclic conditions have to be used along the remaining direction. In case of non-cyclic conditions along x (y), the upper bound nxl (nyn) of array u (v) is enlarged by one gridpoint on the rightmost (northest) processor. These array index informations are passed to routine exchange_horiz and disturb_field by two additional arguments. Boundary conditions are set with new inipar-parameters bc_lr and bc_ns. A damping layer is needed at the outflow in order to damp reflections (see new parameters km_damp_max and outflow_damping_width) and an additional disturbance is imposed near the inflow (see new parameters inflow_disturbance_begin and inflow_disturbance_end). Non-cyclic conditions require the use of the multigrid-method for solving the Poisson equation. Argument range added to routine boundary_conds (where the non-cyclic conditions are set).

Minor changes:
Output of parameter datform on &GLOBAL-Namelist (iso2d). This is set to 'big_endian' on IBM- and NEC-machines.

Number of particle groups are additionally calculated in header (formerly unknown). advec_u_pw, advec_u_up, advec_v_pw, advec_v_up, boundary_conds, calc_liquid_water_content, check_parameters, close_file, coriolis, diffusion_u, diffusion_v, diffusion_w, diffusivities, disturb_field, exchange_horiz, header, init_3d_model, init_pegrid, init_pt_anomaly, init_rankine, modules, parin, poismg, pres, prognostic_equations, read_var_list, sor, time_integration, timestep, write_var_list

C pres is called by default only at the last Runge-Kutta-substep, which spares a big amount of CPU-time (see also new parameter call_psolver_at_all_substeps). No pressure term in the momentum equations in case of Runge-Kutta-schemes.
Attention:
The content of the testresult-file (run control output) has changed due to this modification.

Program speedup by removing the exchange of ghost points in routines calc_liquid_water_content, diffusivities, .

Default value for residual limit increased from 1E-6 to 1E-4. Routine prolong in poismg simplified (one call of exchange_horiz spared). calc_liquid_water_content, diffusivities, modules, poismg, pres, time_integration, timestep_scheme_steering

E Calculation of vertical particle velocity (with inertia) corrected, exp_arg had a wrong sign.

(e)**1.5 replaced by e*SQRT(e) in init_1d_model because of wrong results on NEC machines (as a side effect, the new calculation is much faster).

Two errors concerning switch to PE0 (multigrid method) removed from init_pegrid.

Two errors in plot_2d removed: local_2d is allocated with upper bound nzt (former nzt+1 was wrong), additional barrier in case that PE0 gathers data from the other PEs, this barrier had to be set at the end of the gathering because otherwise PE0 sometimes received wrong data on tag 0.

Use of module indices removed from routine split_1dd in poisfft since it caused errors in case of nx /= ny and a 1D-decomposition along y.

Wrong re-definition of dx2 and dy2 in sor removed. advec_particles, init_1d_model, init_pegrid, plot_2d, poisfft, sor

23/04/05 SR 2.9a C Additional check for incompatibilities between non-cyclic lateral boundary conditions and other schemes.

fcl_factor renamed cfl_factor.

New local array sums_ll declared in flow_statistics instead of temporarily using sums_l.

crmax (maximum number of crosses allowed in a plot of vertical profiles) increased from 20 to 100. calc_spectra, check_parameters, flow_statistics, header, modules, parin, timestep

E Revised calculation of output time counters regarding a possible decrease of the output time interval in case of restart runs.

Error removed in Dirichlet bottom boundary conditions for pt and q in case of Runge-Kutta schemes.

batch_scp calls (needed for determining correct filenames for AVS-fld-files) is given the remote username as an additional argument.

Default setting of outflow_damping_width corrected.

Initial horizontal velocities at the lowest grid levels in the 1d-model are set to a very small value in order to avoid wrong results and the resulting too small timesteps.

Implicit counters i renamed i9 in modules due to declaration conflicts with i in other subroutines (reported by Portland compiler). advec_particles, boundary_conds, check_open, check_parameters, init_1d_model, modules, time_integration

18/05/05 SR 2.10 N NetCDF support implemented.
Profiles, time series, spectra, cross-sections, 3d-volume-data and particle data can now be output in NetCDF format (file handles 101-108). This output can be switched on with the new d3par-parameter data_output_format. So far, this support is available on IBM systems at HLRN and on the NEC system at DKRZ. Output in format suitable for graphic software profil, iso2d and avs is still possible.

Output of cross sections for qv, vpt and lwp (liquid water path). advec_particles, check_open, check_parameters, close_file, header, modules, parin, plot_1d, plot_2d, plot_3d, plot_spectra, plot_ts, read_var_list, write_var_list

new:
netcdf

C In calc_spectra, n is assigned the number of spectra (formerly, the number of spectra was given by n-1)

Abort if poisfft_hybrid is called in a non-parallel environment.

Default values of spectra package parameters pl_spectra and spectra_direction changed. calc_spectra, check_parameters, modules, pres, read_3d_binary, write_3d_binary

E Missing argument in ffty (non-parallel case) added.

Error in output of particle inertia information removed. poisfft, header

05/07/05 SR 2.11 N/C New cloud physics code implemented, which explicitly simulates single cloud droplets, including droplet growth by condensation and collision. Using this code requires to set the new inipar-parameter cloud_droplets = .TRUE. and to use the particle package (by using the mrun-option -p particles). Coupling between droplets and the thermodynamic quantities (potential temperature and specific humidity) is regarded in the new subroutine interaction_droplets_ptq. The real number of droplets in a grid cell can steered by the new inipar-parameter initial_weighting_factor.

This code will be further extended and probably changed in the near future, so the current status is that of a test version!

In connection with this new code implementation, several changes has been made to the existing code:

The particle (droplet) size is now given as radius (previously diameter). E.g., the name of the respective package parameter diameter has changed to radius.

The current weighting_factor and radius are stored as additional particle attributes in the type structure particle_type. The MPI data type mpi_particle_type, needed for exchanging particles between subdomains, has been changed correspondingly.

gas_constant is renamed r_d, latent_heat is renamed r_v, the allowed string length of pl2d and pl3d is increased from 6 to 10.

New 3d arrays ql_1, ql_2, ql_v and ql_vp, ql is now a pointer, new pointer ql_c.

Data format for unit 85 (particle data output) is changed. advec_particles.f90 check_open.f90, header.f90, init_3d_model.f90, init_cloud_physics.f90, init_particles.f90, modules.f90, package_parin.f90, parin.f90, plot_2d.f90, plot_3d.f90, read_var_list.f90, time_integration.f90, write_var_list.f90

new:
interaction_droplets_ptq.f90

C Default value of call_psolver_at_all_substeps is .TRUE., because small scale waves occured with .FALSE. modules.f90

E 2d- and 3d-NetCDF-files are not opened if there is no output for them (pl2d = pl3d = ' ').

No opening of units 101 - 108 in case of missing NetCDF support. check_open.f90, netcdf.f90

30/06/05 GS 2.11a C Scalars ug and vg have been changed into arrays in order to allow the specification of a geostrophic wind that depends on height (baroclinicity). The initial profiles of the the u- and v-component of the geostrophic wind are initialized by specifying the new initialization parameters ug_surface, ug_vertical_gradient, ug_vertical_gradient_level and vg_surface, vg_vertical_gradient, vg_vertical_gradient_level, respectively.

The former initialization parameters ug and vg have been removed. boundary_conds, check_parameters, coriolis, header, init_1d_model, modules, parin, prognostic_equations, read_var_list, spline_z, write_var_list

E Error in the calculation of u_0 and v_0 in production_e.f90 removed.

Attention:
The content of the testresult-file (run control output) has changed due to this modification. production_e

20/10/05 SR 2.11b N/C Collision process for droplet growth completed.

The droplet code is still under development and requires further speed optimization!

Number of particles really used is additionally output on the netcdf particle data file. advec_particles, modules, netcdf

E The last PE in a row is not allowed to have more grid points than the other PES (only less). Jobs crashed in these cases that the last PE has more grid points. Therefore, the number of gridpoints along x (and y) must now meet a special condition which is checked in case that grid_matching = 'match' is used.

Error removed in calculating y axis data for yz cross sections. Error in output of netcdf yz-cross-sections removed.

2*r replaced by r in the exponential term of the particle momentum equation.
Error in output of yz-slice information removed. advec_particles, header, init_pegrid, netcdf, plot_2d

06/12/05 SR 2.11c N Output of ql profile is allowed in case of using cloud droplets. check_parameters, flow_statistics, modules

E The horizontally averaged pt profile is calculated within buoyancy only in case of the first respective intermediate timestep. This is done in order to spare communication time and to produce identical model results with jobs which are calling flow_statistics at different time intervals.

Error in netcdf variable declaration for statistic_regions /= 0 removed. buoyancy, netcdf

23/02/06

3.0

Version for simulating flows over topography (buildings, mountains, etc.)

Besides the implementation of topography, this version contains several other improvements. The most important ones are listed here:

The cloud droplet code has been optimized (it now runs about 100 times faster than before).
Output of time-averaged 2d/3d data (sections / volume data) is possible.
Output of 2d-section data averaged along the direction normal to this section can be done.
The user-interface has been extended in order to allow 2d-/3d-output of user-defined quantities.

The kind of topography to be used is controlled by the new initialization parameter topography. Allowed values are 'flat', 'single_building', and 'read_from_file'. The user can define his own special topography setting within the new user-interface routine user_init_grid. For the single building case, the building (rectangular) size and position can be controlled with parameters building_height, building_length_x, building_length_y, building_wall_left, and building_wall_south.

The topography code is already optimized (vectorized) for use on the NEC-SX6.

Output of time-averaged 2d-/3d-data can be enabled by appending the string '_av' to the respective output quantities given with the new runtime-parameter data_output (see further below for the (name)changes in the parameters for data output). Example: if a time-average of the vertical cross-section of potential temperature is required, the user has to set data_output = 'pt_xy_av'. In case of NetCDF-output, the time-averaged data is output to additional files (seperate from the non-averaged output). The length of the averaging interval and the temporal distance of the single instantaneous sections (volumes) entering into this averaging is controlled by the new runtime-parameters averaging_intervaland dt_averaging_input.

An average normal to the direction of a cross-section can be output by setting the respective cross section index to -1 (example: section_xz = -1).

New user-interface routines allowing the output of user-defined quantities are user_check_data_output, user_define_netcdf_grid, user_data_output_2d, user_data_output_3d, and user_3d_data_averaging. These quantities can be calculated using the new CASE-Entry 'after integration' in the user-interface routine user_actions.

Optimization of the cloud droplet code is mainly done by re-sorting all particles after every timestep in a way that now all particles which belong to the same grid-box (k,j,i) are stored consecutively (wthout any strides) in memory.

Code adaptation for the IBM at the department of atmospheric sciences, Yonsei university (ibmy). Affected subroutines: fft_xy, netcdf,

Further new features:

Use of particle tails now requires setting of the new package-parameter use_particle_tails. The new parameter skip_particles_for_tail allows only every skip_particle_for_tail'th parameter having a tail.
Initialization-parameter e_min can be used to set a lower limit for the subgrid-scale TKE (i.e. to guarantee a lower limit for the flow's Reynolds-number).
A conservation of the volume flow (through the complete xz- and yz-sections of the total domain) can be enforced by the new initialization-parameter conserve_volume_flow.
Data output can be skipped for a given time interval from simulation start (t=0) using new runtime-parameters skip_time_dopr, skip_time_dosp, skip_time_do2d_xy, skip_time_do2d_xz, skip_time_do2d_yz, and skip_time_do3d.
By default, NetCDF output is now using 64-bit offset format (large file support) on most machines. The user can switch to the 32-bit offset format with the new runtime-parameter netcdf_64bit. Units 111:113 and 116 are opened for NetCDF output of time-averaged 2d-3d data.
Data logging routines are added (see file data_log.f90), which can be used for debugging purposes. Output is done on unit 20 (local file name/directory name DATA_LOG).
New routine exchange_horiz_2d_int for ghostpoint exchange of 2d-integer arrays. Routine exchange_horiz_2d is extended for non-cyclic boundary conditions in the non-parallel case.
Two different methods for calculating the mixing length and the dissipation can be used in the 1d-model. These are steered by the new initializing-parameters dissipation_1d and mixing_length_1d.

advec_particles, advec_s_pw, advec_s_up, advec_u_pw, advec_u_up, advec_v_pw, advec_v_up, advec_w_pw, advec_w_up, boundary_conds, buoyancy, check_open, check_parameters, close_file, coriolis, diffusion_e, diffusion_s, diffusion_u, diffusion_v, diffusion_w, diffusivities, disturb_field, disturb_heatflux, exchange_horiz_2d, exchange_horiz, flow_statistics, header, init_1d_model, init_3d_model, init_grid, init_particles, init_pegrid, modules, netcdf, package_parin, parin, poisfft, prandtl_fluxes, pres, production_e, prognostic_equations, read_3d_binary, read_var_list, run_control, time_integration, user_interface, write_3d_binary, write_var_list

new:
average_3d_data, data_log, sum_up_3d_data

renamed:
plot_dvrp -> data_output_dvrp
plot_spectra -> data_output_spectra
plot_ts -> data_output_tseries
plot_1d -> data_output_profiles
plot_2d -> data_output_2d
plot_3d -> data_output_3d

In parallel mode, all PEs are opening the same copy of the NAMELIST-parameter file PARIN, i.e. from now on, the file attribute "npe" must be removed from the corresponding file connection statements in the mrun configuration file (.mrun.config).

A large number of parameter names and local filenames have been changed (see list below). Most of them are parameters concerning data output (do). The former denotations "plot" and "pl" have been changed to "data_output" and "do", respectively. In addition to these changes, the names of many internal PALM parameters and variables have also changed.

Restart runs now require setting of the initialization-parameter initializing_actions = 'read_restart_data'.

Concerning particles, the parameters psl, psr, pdx, etc., which are controlling the position of the particle source and the number of particles within this source, are now 1d-arrays. This allows to define different particle sources for different particle groups.

The number of particle groups has to be set by the user (no automatic calculation from the value of parameter density_ratio any more). Variable idum in type particle_type has been renamed tail_id.

Output for units 15 and 17 flushed in init_1d_model.

List of changed parameter names:

Old name	New name	Comments
average_period_pl1d	averaging_interval_pr
average_period_pr1d	---	averaging of printed profiles is not possible any more
average_period_sp	averaging_interval_sp
dt_average	dt_averaging_input_pr
dt_pl1d	dt_dopr
dt_plsp	dt_dosp
dt_plts	dt_dots
dt_pr1d	dt_dopr_listing
dt_pl2d_xy	dt_do2d_xy
dt_pl2d_xz	dt_do2d_xz
dt_pl2d_yz	dt_do2d_yz
ebene_xy	section_xy
ebene_xz	section_xz
ebene_yz	section_yz
nz_plot3d	nz_do3d
plts	data_output_ts
pl1d	data_output_pr
pl2d	data_output	old parameters pl2d and pl3d are joined to new parameter data_output
pl3d	data_output
pl2d_at_begin	do2d_at_begin
pl3d_at_begin	do3d_at_begin
pl3d_compress	do3d_compress
pl3d_precision	do3d_comp_prec
pl_spectra	data_output_sp
read_fields_from_prior_run	read_restart_data
write_plot2d_on_each_pe	data_output_2d_on_each_pe
z_max_pl1d	z_max_do1d
z_max_pl2d	z_max_do2d

advec_particles, advec_s_bc, check_parameters, check_open, close_file, data_output_dvrp, data_output_profiles, data_output_tseries, data_output_2d, data_output_3d, flow_statistics, header, init_1d_model, init_3d_model, init_dvrp, init_particles, init_pegrid, init_slope, modules, netcdf, package_parin, palm, parin, print_1d, read_var_list, run_control, time_integration, timestep, write_compressed, write_var_list

E Error in the particle release (defaults of psl, psr, psn, pss) at the PE boundary has been fixed.

Error removed getting the variable ids for _ext for profiles and time series.

NetCDF calls in data_output_tseries only if NetCDF output is switched on.

Output of the initial geostrophic wind profile corrected.

Maximum possibel timestep for Runge-Kutta-schemes reduced, error concerning initialization of l_black removed (both in init_1d_model). advec_particles, data_output_tseries, header, init_particles, init_1d_model, netcdf

26/04/06 SR 3.0a N OpenMP parallelization for the cache-optimized PALM code.

OpenMP directives have been added where necessary. Still further tuning will be necessary to get optimum performance. Other parts of the code than the cache-optimized code still need to be (OpenMP-) parallelized.

So far, threads are activated by setting the mrun-options -T <number_of_threads> and -O, i.e. when the Option -O is given, the -T argument is interpreted as the number of threads to be used. When -O is given, the number of MPI tasks per node is automatically set to 1.

Beside the additional compiler directives, the main code change required by the OpenMP parallelization is that global sums are now calculated by first calculating local sums on each thread. As the next step, local sums are calculated from these thread sums for each MPI task before the global sum is computed via MPI_ALLREDUCE. For storing the thread sums, arrays sums_l and sums_l_l now have three dimensions where the number of elements of the third dimension is equal to the number of OpenMP threads. buoyancy, cpu_statistics, diffusivities, flow_statistics, header, init_pegrid, init_3d_model, modules, prandtl_fluxes, pres, production_e

ML C Provisional correction for Piacsek & Williams advection scheme: keep u and v zero one layer below the topography (in case of set_1d-model_profiles). init_3d_model

SR/ML E In flow_statistics.f90 nzb_s_inner(j,i) replaced by nzb in determination of z_i.

Errors removed in the computation of the diabatic mixing length (init_1d_model.f90).

Error removed in extend mode when checking whether the selected cross sections match those in the already existing NetCDF file (netcdf.f90). flow_statistics, init_1d_model, netcdf

02/06/06 SR 3.0b N This version is adapted to the Sun Fire X4600 system at TIT (to be used by setting the mrun option -h lctit). cpu_log, init_1d_model, local_tremain, locaö_tremain_ini, run_control

C NetCDF 2d- and 3d-datasets now contain the exact coordinates of the variables along x and y with respect to the positions where they are defined on the staggered grid.

NetCDF datasets of cross sections (e.g. DATA_2D_XY_NETCDF) are only opened if the respective cross sections are really requested by the settings of parameter data_output. check_parameters, data_output_2d, data_output_3d, modules, netcdf, user_interface

E p is assigned to to_be_resorted instead of w data_output_2d

04/08/06 SR 3.1 N Subgrid scale velocities can (optionally) be included for calculating the particle advection, using the method of Weil et al. (2004, JAS, 61, 2877-2887). This method is switched on by the new particle package parameter use_sgs_for_particles. This also forces the Euler/upstream method to be used for time advancement of the TKE (see new parameter use_upstream_for_tke). The minimum timestep during the sub-timesteps is controlled by parameter dt_min_part. The data type particle_type has additional new attributes e_m, dt_sum, and speed_x/y/z_sgs.

Other new particle features: output of particle quantities as timeseries in NetCDF format (on local file DATA_1D_PTS_NETCDF). The output time interval is controlled by parameter dt_dopts. Particle advection can be switched off after some time using parameter end_time_prel. More than one particle per point can be started with parameter particles_per_point.

Additional parameter in function random_gauss which limits the range of the created random numbers to five times the standard deviation (=1). Seeds iran and iran_part are stored for restart runs. advec_particles, check_open, check_parameters, close_file, diffusion_e, disturb_heatflux, header, init_3d_model, init_particles, modules, netcdf, package_parin, parin, prognostic_equations, random_gauss, read_3d_binary, read_var_list, time_integration, write_3d_binary, write_var_list

new:
data_output_ptseries

C Call of subroutine header is moved after call of subroutine init_3d_model. Generation of run_description_header is moved from routines palm and header to check_parameters. Determination of the number of particle groups is moved from header to init_particles.

izuf renamed iran. check_parameters, disturb_field, disturb_heatflux, header, init_particles, modules, palm

E Variables do2d_unit and do3d_unit now defined as 2d-arrays. Before, in some cases, the units of variables in the 2d- and 3d-NetCDF-files have been wrong.

In routine poisfft default setting of the thread number tn in case of not using OpenMP.

Error removed in the non-parallel part of routine flow_statistics (number of arguments of array sums_l).

Error removed which appeared if the user had decreased the value of dt_dopr within a job chain and if simulataneous time averaging of profiles was switched on. check_parameters, data_output_profiles, flow_statistics, modules, netcdf, poisfft

22/08/06 SR 3.1a N In case of more than one particle group, seperate output of timeseries for each of the groups.

New initial parameter dz_max, which limits the vertical gridspacing in case of a vertically stretched grid. data_output_ptseries, header, init_grid, modules, netcdf, parin, read_var_list, write_var_list

C Disturbances are imposed only for the last Runge-Kutta-substep.
Attention:
The content of the testresult-file (run control output) has changed due to this modification.

Output of xz and yz cross sections now up to gridpoint nzt+1.

Default settings of particle start positions changed. check_open, close_file, data_output_2d, init_particles, netcdf, time_integration

E Initialisation of all tendency arrays (t.._m) needed for the Runge-Kutta schemes.

Bugfix in index array initialization for line- or point-like topography structures.

Bugfix: yv coordinates are now used for yz cross sections, where neccessary. init_3d_model, init_grid, netcdf

12/02/07 SR 3.1b N Attention:
This is the last revision before switching to the subversion revison control system!

Informative output to the job protocoll in case of restarts. check_for_restart

C Values of environment variables are now read from local file ENVPAR instead of getting them by a system call.

For interpolation in advec_particles, allways level k is used. advec_particles, local_tremain_ini, parin

E Several bugfixes in advec_particles: new particles are released only if end_time_prel > simulated_time, transfer of particles when x < -0.5*dx (0.0 before), etc., index i,j used instead of cartesian (x,y) coordinate to check for transfer because this failed under very rare conditions, calculation of number of particles with same radius as the current particle (cloud droplet code).

Allocation of tail_mask and new_tail_id in case of restart-runs. "__" added in a cpp-directive. (both init_particles) advec_particles, init_particles

02/03/07 SR 3.1c N A heatflux can be prescribed at the top with new inipar parameters top_heatflux and use_top_fluxes. New 2d-arrays qswst, qswst_m, tswst and tswst_m are used to store this flux. Use of fluxes are controlled with new index variable nzt_diff. A Neumann boundary condition for temperature can be applied under these conditions.

Additionally, a Dirichlet condition for temperature can be used at the top.

Attention:
The content of the testresult-file (run control output) has changed due to this modification. check_parameters, diffusion_s, flow_statistics, header, init_grid, init_3d_model, modules, parin, production_e, prognostic_equations, read_var_list, read_3d_binary, swap_timelevel, write_var_list, write_3d_binary

C Prognostic equations for all scalars are now solved up to gridpoint nzt (formerly nzt-1). Boundary conditions for scalars at top adjusted respectively (now applied only at nzt+1).

The default top boundary condition for temperature has been renamed to 'initial_gradient'.

Calls of dvrp_output_local, which were commented out for a long time, are now activated for all streams. advec_s_pw, boundary_conds, calc_precipitation, check_parameters, diffusion_e, diffusion_s, impact_of_latent_heat, init_dvrp, init_pt_anomaly, modules, production_e, prognostic_equations, spline_z

E Bugfix: 3d-array p is not a pointer any more.

Bugfix in init_particles: MPI_REAL argument in MPI_ALLREDUCE replaced by
MPI_INTEGER (caused error on NEC only).

Bugfix: ddzw now dimensioned 1:nzt"+1". diffusion_e, diffusion_s, diffusion_u, diffusion_v, diffusion_w, init_particles, modules

29/03/07 SR 3.2 N The code, including scripts, utility programs and documentation is now under full control of subversion. This revision includes a quite large number of new features, changes and bugfixes.

Particle reflection from vertical walls implemented. The particle SGS model is also adjusted to these vertical walls.

Wall functions for vertical walls now include diabatic conditions. New subroutines wall_fluxes, wall_fluxes_e. New 4D-array rif_wall.

Calculation/output of precipitation amount, precipitation rate and z0 (by setting "pra*", "prr*", "z0*" with data_output). The time interval on which the precipitation amount is defined is set by new d3par-parameter precipitation_amount_interval.

New inipar-parameter loop_optimization to control the loop optimization method.

New inipar-parameter pt_reference. If given, this value is used as the reference that is used in the buoyancy terms (otherwise, the instantaneous horizontally averaged temperature is used).

New d3par-parameter netcdf_64bit_3d to switch on and especially to switch off 64bit offset for 3D files only (because still some pd-software is unable to handle 64bit offset).

New d3par-parameter dt_max to set the maximum value allowed for the timestep.

New user interfaces user_advec_particles and user_init_3d_model. The new initializing action "by_user" calls user_init_3d_model and allows the initial setting of all 3d arrays under complete user control.

Samples added to the user interface which show how to add user-define time series quantities.

Topography height informations are stored on arrays zu_s_inner and zw_w_inner and output to the 2d/3d NetCDF files.

Unit 9 is generally opened for output of debug informations (file DEBUG_<pe#>). Makefile

advec_particles, average_3d_data, buoyancy, calc_precipitation, check_open, check_parameters, data_output_2d, diffusion_e, diffusion_u, diffusion_v, diffusion_w, diffusivities, header, impact_of_latent_heat, init_particles, init_3d_model, modules, netcdf, parin, production_e, read_var_list, read_3d_binary, sum_up_3d_data, user_interface, write_var_list, write_3d_binary

new:
wall_fluxes

C General revision of non-cyclic horizontal boundary conditions:
Radiation boundary conditions are now used instead of Neumann conditions at the outflow (calculation needs velocity values for t-dt, which are stored on new arrays u_m_l, u_m_r, etc.). The calculation of a mean outflow is not needed any more. Volume flow control is added for the outflow boundary (currently only for the north boundary!!). Additional gridpoints along x and y (uxrp, vynp) are not needed any more and removed from the code. Routine "boundary_conds" now operates on timelevel t+dt and is not split in two parts (main, uvw_outflow) any more. Neumann boundary conditions are used at inflow/outflow in case of non-cyclic boundary conditions for all 2d-arrays that are handled by exchange_horiz_2d.
Attention: the non-cyclic boundary conditions of this revision still contain some bugs which will be removed within the next revisions!

The Bott-Chlond-scheme can now be used in the vectorization-optimized part (NEC, etc.). Equations are solved up to gridpoint nzt.

The FFT-method for solving the Poisson-equation is now working with Neumann boundary conditions both at the bottom and the top. This requires adjustments of the tridiagonal coefficients and subtracting the horizontally averaged mean from the vertical velocity field.

Particles-package is now part of the default code ("-p particles" is not needed as an mrun-option any more). New attribute age_m in particle_type.

Move call of user_actions( 'after_integration' ) below the increments of times
and counters. user_actions is now called for each statistic region and has as an argument the number of the respective region (sr).

d3par-parameter data_output_ts is removed. Timeseries output for profil-software removed. Timeseries are now switched on by setting dt_dots. Timeseries data is collected in flow_statistics.

Initial velocities at nzb+1 are regarded for volume flow control in case they have been set zero before (to avoid small timesteps); see new internal parameters u/v_nzb_p1_for_vfc.

q is not allowed any more to become negative (prognostic_equations).

poisfft_init is only called if the fft-solver is switched on (init_pegrid).

d3par-parameter moisture renamed to humidity.

Subversion global revision number is read from mrun and added to the run description header and to the run control (_rc) file.

__vtk directives removed from main program.

The uitility routine interpret_config now reads PALM environment variables from NAMELIST instead using the system call GETENV. advec_s_bc, advec_u_pw, advec_u_up, advec_v_pw, advec_v_up, asselin_filter, boundary_conds, check_parameters, coriolis, data_output_dvrp, data_output_ptseries, data_output_ts, data_output_2d, data_output_3d, diffusion_u, diffusion_v, disturb_field, exchange_horiz, exchange_horiz_2d, flow_statistics, header, init_grid, init_particles, init_pegrid, init_rankine, init_pt_anomaly, init_1d_model, init_3d_model, modules, palm, package_parin, parin, poisfft, poismg, prandtl_fluxes, pres, production_e, prognostic_equations, read_var_list, read_3d_binary, sor, swap_timelevel, time_integration, write_var_list, write_3d_binary

E Bugfix: preset of tendencies te_em, te_um, and te_vm (init_1d_model).

Bugfix in sample for reading user defined data from restart file (user_init).

Bugfix in setting diffusivities for cases with the outflow damping layer extending over more than one subdomain (init_3d_model).

Check for possible negative humidities in the initial humidity profile.

In Makefile, default suffixes are removed from the suffix list to avoid calling of m2c in cases of existing .mod files (problem on Linux systems). Makefile

check_parameters, init_1d_model, init_3d_model, user_interface

19/04/07 SR 3.2a C All system relevant compile and link options (e.g. cpp-directives, library paths, etc.) as well as the host identifier (local_host) are now specified in the configuration file. This generally allows to install PALM on any kind of Linux-, IBM-AIX-, or NEC-SX-system by adding appropriate settings to the configuration file. A description is added to chapter 5.0 of the online documentation.

Scripts are also running under the public domain ksh.

Filetransfer by ftp removed (options -f removed from mrun and mbuild).

Call of (system-)FLUSH routine moved to new routine local_flush.

return_addres and return_username are read from ENVPAR-NAMELIST-file instead of using local_getenv.

Preprocessor strings for different linux clusters changed to "lc", some preprocessor directives renamed (new: __intel_openmp_bug which is used for the bug in tridia_1dd and the OpenMP directives). Preprocessor directives for old systems have been removed. advec_particles, check_open, cpu_log, cpu_statistics, data_output_dvrp, flow_statistics, header, init_dvrp, init_particles, init_1d_model, init_dvrp, init_pegrid, local_getenv, local_system, local_tremain, local_tremain_ini, modules, palm, parin, run_control

new:
local_flush

changed scripts:
mbuild, mrun, .mrun.config.default

30/05/07 SR 3.2b N Calculation and output of user-defined profiles: New userpar-parameter data_output_pr_user. In the user-interface, routine user_check_data_output_pr has been added and user_statistics has been extended. check_parameters, flow_statistics, modules, parin, read_var_list, user_interface, write_var_list

C In pres.f90, division through dt_3d replaced by multiplication of the inverse. For performance optimisation, this is done in the loop calculating the divergence instead of using a seperate loop.

Variables var_hom and var_sum are both renamed pr_palm. data_output_profiles, flow_statistics, init_3d_model, modules, parin, pres, read_var_list, run_control, time_integration

E Bugfix: work_fft*_vec removed from some PRIVATE-declarations (poisfft).

Bugfix: field_chr renamed field_char (user_interface).

Bugfix: output of use_upstream_for_tke (header). header, poisfft, user_interface

21/06/07

3.3

This version allows runs for the ocean. These runs can be switched on with the new inipar-parameter ocean.

Setting this switch has several effects:

An additional prognostic equation for salinity is solved.
Potential temperature in buoyancy and stability-related terms is replaced by potential density.
Potential density is calculated from the equation of state for seawater after each timestep, using the algorithm proposed by Jackett et al. (2006, J. Atmos. Oceanic Technol., 23, 1709-1728).
So far, only the initial hydrostatic pressure is entered into this equation.
z=0 (sea surface) is assumed at the model top (vertical grid index k=nzt on the w-grid), with negative values of z indicating the depth.
Initial profiles are constructed (e.g. from pt_vertical_gradient / pt_vertical_gradient_level) starting from the sea surface, using surface values given by pt_surface, sa_surface, ug_surface, and vg_surface.
Zero salinity flux is used as default boundary condition at the bottom of the sea.
If switched on, random perturbations are by default imposed to the upper model domain from zu(nzt*2/3) to zu(nzt-3).

Relevant new inipar-parameters to be exclusively used for steering ocean runs are bc_sa_t, bottom_salinityflux, sa_surface, sa_vertical_gradient, sa_vertical_gradient_level, and top_salinityflux.

Salinity (sa) and potential density (rho) are included as new 2d/3d output quantities. Vertical profiles of salinity (sa), salinity fluxes (w"sa", w*sa*, wsa), and potential density (rho) can also be output.

advec_s_bc, average_3d_data,
boundary_conds, buoyancy,
check_parameters, data_output_2d,
data_output_3d, diffusion_e,
diffusivities, flow_statistics, header,
init_grid, init_3d_model, modules,
netcdf, parin, production_e,
prognostic_equations,
read_3d_binary, read_var_list,
sum_up_3d_data, swap_timelevel,
time_integration, user_interface,
write_var_list, write_3d_binary

new:
eqn_state_seawater, init_ocean

C Inipar-parameter use_pt_reference renamed use_reference.
Internal variable hydro_press renamed hyp, routine calc_mean_pt_profile renamed calc_mean_profile.

The format of the RUN_CONTROL file has been adjusted for ocean runs.

Defaults of .._vertical_gradient_levels changed from -1.0 to -9999999.9. advec_particles, buoyancy,
calc_liquid_water_content,
check_parameters, diffusion_e,
diffusivities, header,
init_cloud_physics, modules,
production_e, prognostic_equations,
run_control

E Bugfix: height above topography instead of height above level k=0 is used for calculating the mixing length (diffusion_e and diffusivities).

Bugfix: error in boundary condition for TKE removed (advec_s_bc). advec_s_bc, diffusion_e, diffusivities

05/10/07 SR 3.4 N This version allows runs for a coupled atmosphere-ocean LES.

Coupling frequency is controlled by new d3par-parameter dt_coupling, the coupling mode ('atmosphere_to_ocean' or 'ocean_to_atmosphere') for the respective processes is read from environment variable coupling_mode, which is set by the mpiexec-command in mrun. Communication between the two models is handled by the intercommunicator comm_inter. Local files opened by the ocean model get the additional suffic "_O". Saturation at k=nzb is assumed for the atmosphere coupled to ocean. Usage of the coupled model is described in the new section 3.8.

A momentum flux can be set as top boundary condition using the new inipar parameters top_momentumflux_u|v.

Non-cyclic boundary conditions can be used along all horizontal directions.

Quantities w*p* and w"e can be output as vertical profiles.

Initial profiles are reset to constant profiles in case that initializing_actions /= 'set_constant_profiles'. (init_rankine)

Optionally calculate km and kh from initial TKE e_init. boundary_conds, check_open, check_parameters, diffusion_u, diffusion_v, flow_statistics, header, init_pegrid, init_rankine, init_3d_model, local_stop, modules, palm, parin, prandtl_fluxes, pres, prognostic_equations, read_var_list, read_3d_binary, swap_timelevel, timestep, time_integration, write_var_list, write_3d_binary

new:
surface_coupler

C Remaining variables iran changed to iran_part (advec_particles, init_particles).

In case that the presure solver is not called for every Runge-Kutta substep (call_psolver_at_all_substeps = .F.), it is called after the first substep instead of the last. In that case, random perturbations are also added to the velocity field after the first substep.

Initialization of km, kh = 0.00001 for ocean = .T. (for ocean = .F. it remains 0.01).

Allow data_output_pr = 'q', 'wq', 'w"q"', 'w*q*' for humidity = .T. (instead of cloud_physics = .T.). advec_particles, check_for_restart, check_parameters, init_particles, init_3d_model, time_integration

E Bugs from code parts for non-cyclic boundary conditions are removed:
loops for u and v are starting from index nxlu, nysv, respectively. The radiation boundary condition is used for every Runge-Kutta substep. Velocity phase speeds for the radiation boundary conditions are calculated for the first Runge-Kutta substep only and reused for the further substeps. New arrays c_u, c_v, and c_w are defined for this purpose. Several index errors are removed from the radiation boundary condition code parts. Upper bounds for calculating u_0 and v_0 (in production_e) are nxr+1 and nyn+1 because otherwise these values are not available in case of non-cyclic boundary conditions.

+dots_num_palm in module user, +module netcdf_control in user_init (both in user_interface)

Bugfix: wrong sign removed from the buoyancy production term in the case use_reference = .T. (production_e)

Bugfix: Error message concerning output of particle concentration (pc) modified (check_parameters).

Bugfix: Rayleigh damping for ocean fixed. advec_u_pw, advec_u_up, advec_v_pw, advec_v_up, boundary_conds, buoyancy, check_parameters, coriolis, diffusion_u, diffusion_v, header, init_pegrid, init_3d_model, modules, poismg, production_e, prognostic_equations, user_interface

29/11/07 SR 3.4a N Pressure boundary conditions for vertical walls added to the multigrid solver. They are applied using new wall flag arrays (wall_flags_..) which are defined for each grid level. New argument gls added to routine user_init_grid (user_interface).

Plant canopy model of Watanabe (2004,BLM 112,307-341) added. It can be switched on by the new inipar parameter plant_canopy. The inipar parameter canopy_mode can be used to prescribe a plant canopy type. The default case is a homogeneous plant canopy. Heterogeneous distributions of the leaf area density and the canopy drag coefficient can be defined in the new routine user_init_plant_canopy (user_interface). The inipar parameters lad_surface, lad_vertical_gradient and lad_vertical_gradient_level can be used to prescribe the vertical profile of leaf area density. The inipar parameter drag_coefficient determines the canopy drag coefficient. Finally, the inipar parameter pch_index determines the index of the upper boundary of the plant canopy.

Allow new case bc_uv_t = 'dirichlet_0' for channel flow.

For unknown variables (CASE DEFAULT) call new subroutine user_data_output_dvrp.

Frequence of sorting particles can be controlled with new particles_par parameter dt_sort_particles. Sorting is moved from the SGS timestep loop in advec_particles after the end of this loop. advec_particles, check_parameters, data_output_dvrp, header, init_3d_model, init_grid, init_particles, init_pegrid, modules, package_parin, parin, read_var_list, user_interface, write_var_list

new:
plant_canopy_model

C Redefine initial nzb_local as the actual total size of topography (later the extent of topography in nzb_local is reduced by 1dx at the E topography walls and by 1dy at the N topography walls to form the basis for nzb_s_inner); for consistency redefine 'single_building' case.

Vertical profiles now based on nzb_s_inner; they are divided by ngp_2dh_s_inner (scalars, procucts of scalars) and ngp_2dh (staggered velocity components and their products, procucts of scalars and velocity components), respectively.

Allow two instead of one digit to specify isosurface and slicer variables.

Status of 3D-volume NetCDF data file only depends on switch netcdf_64bit_3d (check_open).

prognostic_equations include the respective wall_*flux in the parameter list of calls of diffusion_s. Same as before, only the values of wall_heatflux(0:4) can be assigned. At present, wall_humidityflux, wall_qflux, wall_salinityflux, and wall_scalarflux are kept zero. diffusion_s uses the respective wall_*flux instead of wall_heatflux. This update serves two purposes: - it avoids errors in calculations with humidity/scalar/salinity and prescribed non-zero wall heatflux, - it prepares PALM for a possible assignment of wall fluxes of humidity/scalar/salinity in a future releases. buoyancy, check_open, data_output_dvrp, diffusion_s, diffusivities, flow_statistics, header, init_3d_model, init_dvrp, init_grid, modules, prognostic_equations

E Bugfix: summation of sums_l_l (diffusivities).

Several bugfixes in the ocean part: Initial density rho is calculated (init_ocean). Error in initializing u_init and v_init removed (check_parameters). Calculation of density flux now starts from nzb+1 (production_e).

Bugfix: pleft/pright changed to pnorth/psouth in sendrecv of particle tail numbers along y, small bugfixes in the SGS part (advec_particles).

Bugfix: model_string needed a default value (combine_plot_fields).

Bugfix: wavenumber calculation for even nx in routines maketri (poisfft).

Bugfix: assignment of fluxes at walls.

Bugfix: absolute value of f must be used when calculating the Blackadar mixing length (init_1d_model). advec_particles, check_parameters, combine_plot_fields, diffusion_s, diffusivities, init_ocean, init_1d_model, poisfft, production_e

16/09/08 SR 3.5 N The turbulence recycling method of Kataoka and Mizuno (2002) has been implemented. A precursor run must be used to initialize the main run with a quasi-stationary turbulence state (see initializing_actions = 'read_data_for_recycling'). The precursor run may have a smaller domain than the main run, which helps to spare CPU time. New parameters for steering the recycling method are turbulent_inflow, recycling_width, inflow_damping_height, and inflow_damping_width.

One new feature related with the recycling method is that within a job chain, the number of processors or the virtual processor grid is now allowed to change. Such a change modifies the domain decomposition and requires that the PEs within a restart job have to know from which file(s) they have to read their restart data (up to now there has been a one to one relation between the restart files and the PEs, i.e. processor 4 only reads data from file "_0004"). Informations about the subdomain index limits of all restart files are now written by PE0 (to file "_0000"). Also, all variables containing global values which do not depend on the subdomain are written to file "_0000" only. In a restart run, this file is opened by all PEs to read these global values and the informations about the subdomain index limits. Depending on these index limit information it is decided which restart file(s) has/have to be opened in order to read the complete data necessary for the current subdomain.

The plant canopy model has been extended to the prognostic equation for the potential temperature and passive scalar, as proposed by Watanabe (2004) and Shaw and Schumann (1992).

Particle advection adjusted for ocean runs.

Grid coarsening (multigrid method) is limited to subdomains (i.e. no collection of data on PE0 for further coarsening beyond subdomains) if mg_switch_to_pe0_level = -1.

User-defined spectra are allowed.

Output of q*2 profile and w"u"0, w"v"0, w"q"0 timeseries are added.

Argument -c introduced to most of the subjob calls, which allows the user to choose his own job catalog by setting the environment variable job_catalog in the configuration file (default is ~/job_queue).

This release contains first adjustments for the SGI-ICE system at HLRN.

Attention: On the SGI-ICE, runs are still aborted or are hanging for unknown reasons from time to time. advec_particles, calc_spectra, check_open, check_parameters, data_output_spectra, header, init_3d_model, init_particles, init_pegrid, modules, netcdf, parin, particle_boundary_conds, plant_canopy_model, prognostic_equations, read_3d_binary, read_var_list, time_integration, user_interface, write_3d_binary, write_var_list

new:
inflow_turbulence

C Allow 100 spectra levels instead of 10 for consistency with define_netcdf_header. (calc_spectra, data_output_spectra, header, modules, netcdf)

Modification of the integrated version of the profile function for momentum for unstable stratification; more consistent flux definitions. (wall_fluxes, production_e)

Strict grid matching along z is not needed for mg-solver. (check_parameters)

Sharp lateral edges of the plant canopy introduced. Before, smaller values of the leaf area density than those applied in the prognostic equation for the vertical velocity were used in the prognostic equations for u and v at the boundaries of the plant_canopy. (init_3d_model)

-cross_ts_* (read_var_list, write_var_list), -dopr_time_count, old profil parameters (cross_..., dopr_crossindex, profile_***) removed. (read_3d_binary, write_3d_binary)

Reading/writing of hom_sum, volume_flow_area, volume_flow_initial is moved from ..._3d_binary to ..._var_list, hom_sum is now allocated in parin instead of init_3d_model.

npex, npey moved from inipar- to d3par-namelist. (parin)

myid_char14 removed. This was used, since on IBM machines the process rank could be changed when the final communicator was defined, in order to save the preliminary processor-id for opening the binary output file for restarts (unit 14), because otherwise a mismatch occured when reading this file in a restart job. This mechanism is not needed any more since the subdomain informations are now contained in file _0000. (parin, check_open)

Transpositions for the 2D domain decomposition have been optimized by using f_inv as an automatic array instead of providing the memory by a dummy argument. This spares one copy loop per transposition. Order of indices in the 3D loops in some of the transpose routines have been rearranged for better cache utilization. Both have been suggested by Roland Richter (SGI) as part of the HLRN-II benchmark process. (transpose)

Workaround for getting information about the coupling mode, because the -env Option (mpiexec) is not available on many systems (SGI, lcxt4 at Bergen Center for Computational Science). (palm, mrun)

Format adjustments in order to allow CPU# > 999. (cpu_statistics)

Provisional solution for run_control_1d output: add 'CALL check_open( 15 )'. (init_1d_model)

Subdomain data are read into temporary arrays pf_tmp / pf3d_tmp in order to avoid INTEL compiler warnings about (automatic) creation of temporary arrays. (combine_plot_fields)

Time coordinate t=0 stored on netcdf-file only if an output is required for this time for at least one of the profiles.

Output of spline timeseries splptx etc. removed. (flow_statistics) advec_s_ups, advec_u_ups, advec_v_ups, advec_w_ups, calc_spectra, check_open, check_parameters, cpu_statistics, data_output_profiles, data_output_spectra, flow_statistics, header, init_1d_model, init_3d_model, modules, netcdf, palm, parin, poisfft, production_e, read_3d_binary, read_var_list, transpose, wall_fluxes, write_3d_binary, write_var_list

E Bugfix: modification of the calculation of the vertical turbulent momentum fluxes u'w' and v'w'; the first usws that is computed corresponds to -u'w'/u* and not as priorily assumed to (-u'w')**0.5, the first vsws that is computed corresponds to -v'w'/u* and not as priorily assumed to (-v'w')**0.5. Therefore, the intermediate result for usws has to be multiplied by -u* instead by itself in order to get u'w'. Accordingly, the intermediate result for vsws has to be multiplied by -u* instead by itself in order to get v'w'. As u* is calculated for the position of a scalar an additional interpolation of u* to the position of u and v, respectively, is necessary. As u* is not determined for the ghost points on each PE, an additional exchange of information from neighbouring PEs is necessary. Tests showed that this decreases u* by some 10% and increases TKE and momentum fluxes by some 10% because friction was underestimated and momentum fluxes were wrongly calculated due to the bug. (prandtl_fluxes, wall_fluxes)

Attention:
The content of the testresult-file (run control output) has changed due to the above bugfix.

Bugfix: change definition of us_wall from 1D to 2D.

Bugfix: exponent = 1.0 / ( ny + 1.0 ) in calc_spectra_x. (calc_spectra)

Bugfix: calculation of horizontal fluxes at vertical walls. (diffusion_s)

Bugfix: zero assignments to tendency arrays in case of restarts. (init_3d_model)

Bugfix: wrong grid level used in multigrid solver, routine restrict. (poismg)

Bugfix: Construction of vertical profiles when 10 gradients have been specified in the parameter list (ug, vg, pt, q, sa, lad). (check_parameters)

Bugfix: Determination of the number of vertical levels for which spectra shall be output - 100 replaced by 10. (netcdf)

Bugfix: Lengths of those characters that are needed for the output of the characteristic levels of potential temperature, the geostrophic wind, scalar concentration, humidity and leaf area density to the header file has been increased. (header)

Bugfix: add ')' in user_read_restart_data.

The leaf area density is explicitly set to its surface value at k=0. (check_parameters)

Introduce prefix_chr to ensure unique dvrp_file path.

small bugfixes for user_interface sample code (comments):
- initialize ustvst with 0.0 as it is now computed only until nxr and nyn
- two ALLOCATE statements moved from user_read_restart_data back to user_init
- remove 'READ (13) u2_av' statement in user_read_restart_data

Bugfix: remove IF statement in plant_canopy_model_ij.

Bugfix: divide sums(k,8) (e) and sums(k,34) (e*) by ngp_2dh_s_inner(k,sr) (like other scalars). (flow_statistics)

Bugfix: dopr_time_count was written on the binary file, which caused that NetCDF files newly created by restart files (no append of existing files!) contained uneccessary time levels. (read_3d_binary, write_3d_binary)

Bugfix: extra '*' removed in user_statistics sample code. (user_interface)

Bugfix: a stop command was missing in some cases of the parallel branch. (local_stop)

Bugfix in volume flow control for non-cyclic boundary conditions. (pres)

Bugfix: misplaced #endif directives. (combine_plot_fields)

Data are collected from PE0 in an ordered sequence which seems to avoid hanging of processes on SGI-ICE. (cpu_statistics) calc_spectra, check_parameters, cpu_statistics, diffusion_s, flow_statistics, init_3d_model, init_dvrp, local_stop, plant_canopy_model, poismg, prandtl_fluxes, pres, read_3d_binary, user_interface, wall_fluxes, write_3d_binary

02/02/09 SR 3.6
(r228) N This release contains further additions, changes, and bugfixes for the SGI-ICE system, as well as important changes like the splitting of the user-interface into single files, a complete revision of the pre-compiling mechanism allowing to use different make-depositories simultaneously, and a revision of the reading-mechanism from restart files, which has no more restrictions in case that the previous and current run have different domain/subdomain sizes. A new handling of all kind of output messages is introduced with this release and will be applied to all existing messages within the next releases.

New features are:

Restart runs on SGI-ICE are working. (mrun)
2d-decomposition is default on SGI-ICE systems. (init_pegrid)

Ocean-atmosphere coupling realized with MPI-1. mrun adjusted for this case (-Y option). Check that PALM is called with mrun-option "-K parallel" for coupling. Adjustments in mrun, mbuild, and subjob for lcxt4.

DVRP arguments changed to single precision, mode pathlines added.

User can add additional routines in files user_additional_routines.

User can check user parameters and deduce further quantities in user_check_parameters. check_for_restart, check_parameters, data_output_dvrp, init_dvrp, init_pegrid, local_stop, modules, package_parin, palm, surface_coupler, timestep

Makefile, mbuild, mrun, subjob

new:
init_coupling, user_additional_routines, user_check_parameters

C User interface has been split into single files.

Type of variables count and count_rate changed to INTEGER(8) in order to avoid "out of range" problems, which result in measured negative time intervals. (cpu_log, local_tremain, local_tremain_ini)

Reading mechanism from restart files completely revised. The subdomain/total domain size is now allowed to vary arbitrarily between the current and previous run. (read_var_list, read_3d_binary, user_read_restart_data)

Precompilation mechanism (mbuild) completely revised: the source-tarfile remains in the source directory (trunk/SOURCE). One depository per block (given in the config-file) is created. Always all files from the source directory are copied to the respective depository. No additional file checks are done any more (mrun version 2.1). The depository name used contains the conditions given by mrun-option "-K". (mbuild, mrun)

Output messages (including required stop of execution) can now be handled with the new subroutine message. All output messages will be replaced by this routine step by step within the next revisions.

Neumann boundary condition at k=nzb is explicitly set for better reading, although this has been already done in boundary_conds. (advec_s_bc)

Origin of the xy-coordinate system shifted from the center of the first grid cell (indices i=0, j=0) to the south-left corner of this cell. (netcdf) Topography definition according to new user parameter topography_grid_convention. (init_grid, modules, user_header, user_init_grid, user_parin) advec_s_bc, check_parameters, cpu_log, init_grid, local_stop, local_tremain, local_tremain_ini, modules, netcdf, read_3d_binary, read_var_list, user_read_restart_data

Makefile, mbuild, mrun

removed:
user_interface

new:
message, user_3d_data_averaging, user_actions, user_advec_particles, user_check_data_output, user_check_data_output_pr, user_data_output_2d, user_data_output_3d, user_data_output_dvrp, user_define_netcdf_grid, user_dvrp_coltab, user_header, user_init_3d_model, user_init, user_init_grid, user_init_particles, user_init_plant_canopy, user_last_actions, user_parin, user_parin, user_particle_attributes, user_read_restart_data, user_spectra, user_statistics

E Bugfix: array d is reallocated in case that multigrid is used. (calc_spectra)

Bugfixes for nonparallel execution. (check_for_restart, cpu_statistics, inflow_turbulence, timestep)

Size of pf3d adjusted to the required output size (1 gridpoint less, along all three dimensions), because output of a subset of the data (pf3d(nxa:nxe...)) in the NF90_PUT_VAR statement caused segmentation fault with the INTEL compiler. (combine_plot_fields)

Bugfix: error in zu index in case of section_xy=-1. (header)

Inconsistency removed: as the thermal stratification is not taken into account for the evaluation of the wall fluxes at vertical walls, the eddy viscosity km must not be used for the evaluation of the velocity gradients dudy, dwdy, dvdx and dwdx. (production_e)

Bugfix in calculating k index in case of oceans runs. (sort_particles)

Bugfix: no output of particle concentration and radius unless particles have been started. (data_output_2d)

Bugfix: reading of spectrum_x|y from restart files ignored if total numbers of grid points do not match. (read_3d_binary)

Bugfix: abort in case that absolute temperature is below zero. (init_cloud_physics) advec_particles, calc_spectra, check_for_restart, cpu_statistics, data_output_2d, header, inflow_turbulence, init_cloud_physics, production_e, read_3d_binary, timestep

combine_plot_fields

23/09/09 SR 3.7 (r392) N Most important changes: atmosphere-ocean coupling has been improved and steering of dvr-software is more user-friendly now. Beside that, the release contains a lot of optimizations, changes, and bugfixes which have accumulated over the last months.

The atmosphere-ocean coupling allows to use independent precursor runs in order to account for different spin-up times. The time when coupling has to be started is given by new inipar parameter coupling_start_time. The precursor ocean run has to be started using new mrun option "-y" in order to add the appendix "_O" to all output files. (check_for_restart, check_parameters, data_output_2d, data_output_3d, data_output_profiles, data_output_ptseries, data_output_spectra, data_output_tseries, header, init_coupling, modules, mrun, parin, read_var_list, surface_coupler, time_integration, write_var_list)

Clipping of dvrp output implemented. Default colourtable for particles implemented, particle attributes (color, dvrp_size) can be set with new parameters particle_color, particle_dvrpsize, color_interval, dvrpsize_interval. Slicer attributes (dvrp) are set with new routine set_slicer_attributes_dvrp and are controlled with existing parameters slicer_range_limits. (init_dvrp, data_output_dvrp, modules, set_slicer_attributes_dvrp, user_data_output_dvrp)

Polygon reduction for topography and ground plate isosurface. Reduction level for buildings can be chosen with parameter cluster_size. (init_dvrp)

Variables bc_lr / bc_ns in most subroutines replaced by LOGICAL variables bc_lr_cyc,
bc_ns_cyc for speed optimization. This gives a significant speedup of more than 10%. (check_parameters, diffusion_u, diffusion_v, diffusion_w, modules)

Scripts have been adapted for machine lck (Yonsei Univ.). (mrun, mbuild, subjob)

Additional timestep criterion in case of simulations with plant canopy. (timestep)

Check for illegal entries in section_xy|xz|yz that exceed nz+1|ny+1|nx+1. (check_parameters)

External pressure gradient can be used as driving force. (check_parameters, header, init_3d_model, modules, parin, prognostic_equations, read_var_list, write_var_list)

New topography case 'single_street_canyon'. (header, init_grid, modules, parin, read_var_list, user_check_parameters, user_header, user_init_grid, write_var_list)

Option to predefine a target bulk velocity for conserve_volume_flow. (check_parameters, header, init_3d_model, modules, parin, read_var_list, write_var_list)

Option for user defined 2D data output in xy cross sections at z=nzb+1. (data_output_2d, user_data_output_2d)

xy cross section output of surface heatfluxes (sensible and latent). (average_3d_data, check_parameters, data_output_2d, modules, read_3d_binary, sum_up_3d_data, write_3d_binary) average_3d_data, check_for_restart, check_parameters, data_output_2d, data_output_3d, data_output_dvrp, data_output_profiles, data_output_ptseries, data_output_spectra, data_output_tseries, diffusion_u, diffusion_v, diffusion_w, init_coupling, init_dvrp, init_grid, init_3d_model, header, mbuild, modules, mrun, package_parin, parin, prognostic_equations, read_3d_binary, read_var_list, subjob, sum_up_3d_data, surface_coupler, timestep, time_integration, user_check_parameters, user_data_output_2d, user_data_output_dvrp, user_header, user_init_grid, write_3d_binary, write_var_list

new: set_particle_attributes, set_slicer_attributes_dvrp

C Output of messages replaced by message handling routine. (advec_particles, advec_s_bc, buoyancy, calc_spectra, check_for_restart, check_open, coriolis, cpu_log, data_output_2d, data_output_3d, data_output_dvrp, data_output_profiles, data_output_spectra, fft_xy, flow_statistics, header, init_1d_model, init_3d_model, init_dvrp, init_grid, init_particles, init_pegrid, netcdf, parin, plant_canopy_model, poisfft_hybrid, poismg, read_3d_binary, read_var_list, surface_coupler, temperton_fft, timestep, user_actions, user_data_output_dvrp, user_dvrp_coltab, user_init_grid, user_init_plant_canopy, user_parin, user_read_restart_data, user_spectra)

Output of NetCDF messages with aid of message handling routine. (check_open, close_file, data_output_2d, data_output_3d, data_output_profiles, data_output_ptseries, data_output_spectra, data_output_tseries, netcdf)

__lcmuk changed to __lc to avoid problems with Intel compiler on sgi-ice. (poisfft)

For extended NetCDF files, the updated title attribute includes an update of time_average_text where appropriate. (netcdf)

In case of restart runs without extension, initial profiles are not written to NetCDF-file anymore. So far, time axis always started at t=0. (data_output_profiles, modules, read_var_list, write_var_list)

initializing_actions='read_data_for_recycling' renamed to 'cyclic_fill'. A cyclic fill can now
be used independent of turbulent_inflow. (check_parameters, header, init_3d_model)

2 NetCDF error numbers changed. (data_output_3d)

A link to the webspage appendix_a.html is printed for further information about possible reasons of the error that appeared. (message)

Temperature gradient criterion for estimating the boundary layer height replaced by the gradient criterion of Sullivan et al. (1998). (flow_statistics)

NetCDF unit attribute in timeseries output in case of statistic regions added. (netcdf)

Maximum number of tails is calculated from maximum number of particles and skip_particles_for_tail. (init_particles)

Value of vertical_particle_advection is allowed to differ for each particle group. (advec_particles, header, modules)

First constant in array den also defined as type double. (eqn_state_seawater)

Parameter dvrp_psize moved from particles_par to dvrp_graphics_par. (package_parin)

topography_grid_convention moved from userpar to inipar (check_parameters, header, parin, read_var_list, user_check_parameters, user_header, user_init_grid, user_parin, write_var_list)

Default value of grid_matching changed to 'strict'. (modules)

Adjustments for runs on lcxt4 (necessary due to a software update on CRAY) and for coupled runs on ibmy. (mrun, subjob) advec_particles, advec_s_bc, buoyancy, calc_spectra, check_for_restart, check_open, check_parameters, close_file, coriolis, cpu_log, data_output_2d, data_output_3d, data_output_dvrp, data_output_profiles, data_output_ptseries, data_output_spectra, data_output_tseries, eqn_state_seawater, fft_xy, flow_statistics, header, init_1d_model, init_3d_model, init_dvrp, init_grid, init_particles, init_pegrid, message, mrun, netcdf, package_parin, parin, plant_canopy_model, poisfft, poisfft_hybrid, poismg, read_3d_binary, read_var_list, subjob, surface_coupler, temperton_fft, timestep, user_actions,user_check_parameters, user_data_output_dvrp, user_dvrp_coltab, user_header, user_init_grid, user_init_plant_canopy, user_parin, user_read_restart_data, user_spectra, write_var_list

E Bugfix: Initial hydrostatic pressure profile in case of ocean runs is now calculated in 5 iteration steps. (init_ocean)

Bugfix: wrong sign in buoyancy production of ocean part in case of not using the reference density (only in 3D routine production_e). (production_e)

Bugfix: output of averaged 2d/3d quantities requires that an avaraging interval has been set, respective error message is included. (check_parameters)

Bugfix: Output on unit 14 only if requested by write_binary. (user_last_actions)

Bugfix to avoid zero division by km_neutral. (production_e)

Bugfix for extended NetCDF files: In order to avoid 'data mode' errors if updated attributes are larger than their original size, NF90_PUT_ATT is called in 'define mode' enclosed by NF90_REDEF and NF90_ENDDEF calls. This implies a possible performance loss; an alternative strategy would be to ensure equal attribute size in a job chain. (netcdf)

Bugfix: correction of initial volume flow for non-flat topography. (init_3d_model)

Bugfix: zero initialization of arrays within buildings for 'cyclic_fill'. (init_3d_model)

Bugfix: to_be_resorted => s_av for time-averaged scalars. (data_output_2d, data_output_3d)

Bugfix: avoid that ngp_2dh_s_inner becomes zero. (init_3_model)

Typographical error: unit of wpt in dots_unit. (modules)

Bugfix: error in check, if particles moved further than one subdomain length. This check must not be applied for newly released particles. (advec_particles)

Bugfix: several tail counters are initialized, particle_tail_coordinates is only written to file if its third index is > 0, arrays for tails are allocated with a minimum size of 10 tails if there is no tail initially. (advec_particles, init_particles)

Bugfix: pressure included for profile output. (check_parameters)

Bugfix: Type of count and count_rate changed to default INTEGER on NEC machines. (cpu_log)

Bugfix: output of particle time series only if particle advection is switched on. (time_integration)

Bugfix: qsws was calculated in case of constant heatflux = .FALSE.. (prandtl_fluxes)

Bugfix: averaging along z is not allowed for 2d quantities (e.g. u* and z0). (data_output_2d)

Typographical errors. (netcdf)

If the inversion height calculated by the prerun is zero, inflow_damping_height must be explicitly specified. (init_3d_model)

Small bugfix concerning 3d 64bit netcdf output format. (header)

Bugfix: dt_fixed removed from the restart file, because otherwise, no change from a fixed to a variable timestep would be possible in restart runs. (read_var_list, write_var_list)

Bugfix: initial setting of time_coupling in coupled restart runs. (time_integration) advec_particles, check_parameters, cpu_log, data_output_2d, data_output_3d, header, init_3d_model, init_particles, init_ocean, modules, netcdf, prandtl_fluxes, production_e, read_var_list, time_integration, user_last_actions, write_var_list

Attention: If make is not used for compiling, all routines have to be re-compiled after every change to modules.f90!

2.0 How to change the model source code

Currently (Sep 23rd, 2009), only selected users are allowed to commit changes to the repository. Some rules about how to check in modified code will be added later.

3.0 Description of selected parts of the model source code

This section gives links to existing descriptions about special parts of the PALM code, e.g. special numerical methods that are used (including mathematical background). Documents are stored in single files with different formats. Some are in German (sorry for the inconvenience).

upstream-spline advection scheme ( .ps )
cloud physics module ( .pdf )
Datenkompression mit dem Verfahren der Bitverschiebung
Runge-Kutta time integration scheme ( .pdf )