| [5] | 1 | <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> |
|---|
| [40] | 2 | <html><head> |
|---|
| 3 | <meta http-equiv="CONTENT-TYPE" content="text/html; charset=windows-1252"> |
|---|
| 4 | |
|---|
| 5 | <title>PALM chapter 5.0</title><meta name="GENERATOR" content="StarOffice 7 (Win32)"> |
|---|
| [5] | 6 | <meta name="AUTHOR" content="Siegfried Raasch"> |
|---|
| 7 | <meta name="CREATED" content="20041103;12380720"> |
|---|
| 8 | <meta name="CHANGED" content="20041123;15532356"> |
|---|
| 9 | <meta name="KEYWORDS" content="parallel LES model"> |
|---|
| 10 | <style> |
|---|
| 11 | <!-- |
|---|
| 12 | @page { size: 21cm 29.7cm } |
|---|
| 13 | --> |
|---|
| [40] | 14 | </style></head> |
|---|
| [5] | 15 | <body dir="ltr" lang="en-US"> |
|---|
| 16 | <h2 style="line-height: 100%;"><font size="4">5.0 Installation of the |
|---|
| 17 | model</font></h2> |
|---|
| [41] | 18 | <font color="#000000">This chapter |
|---|
| [40] | 19 | describes the installation of PALM on a Linux workstation (local host) and a suitable remote computer, on which the |
|---|
| 20 | model runs are to be carried out. The local host is used to |
|---|
| 21 | start batch jobs with <b>mrun</b> and to analyze the results |
|---|
| [41] | 22 | which are produced by the model on the remote host and send back to the local host. Alternatively, <span style="font-weight: bold;">mrun</span> can also be used to start PALM on the local host in interactive mode or as a batch job.</font><p style="line-height: 100%;"><font color="#000000"></font></p><p style="line-height: 100%;"><font color="#000000"><span style="font-weight: bold; text-decoration: underline;">Requirements</span></font></p><p style="line-height: 100%;"><font color="#000000">The installation and operation of PALM requires at mimimum (on both, the local and the remote host, unless stated otherwise):</font></p><ol><li><font color="#000000">The AT&T Korn-shell (ksh) must be available under<span style="font-family: Courier New,Courier,monospace;"> </span></font><font style="font-family: Courier New,Courier,monospace;" color="#000000">/bin/ksh</font><font color="#000000">. Most of the current Linux distributions are using the public domain Korn-shell (pdksh). The PALM scripts <span style="font-weight: bold;">mrun</span> and <span style="font-weight: bold;">mbuild</span> do not work with the pdksh! Please replace it by the AT&T ksh. If the AT&T ksh path is different from<span style="font-family: Courier New,Courier,monospace;"> </span></font><font style="font-family: Courier New,Courier,monospace;" color="#000000">/bin/ksh</font><font color="#000000">, please modify the first lines in <span style="font-weight: bold;">mrun</span> and <span style="font-weight: bold;">mbuild</span>, respectively.</font></li><li><font color="#000000">The NetCDF-library with version number not earlier than 3.6.0-p1 (for NetCDF, see under <a href="http://www.unidata.ucar.edu/software/netcdf/">www.unidata.ucar.edu</a>).</font></li><li><font color="#000000">A FORTRAN90/95 compiler.</font></li><li><font color="#000000">The Message Passing Interface (MPI), at least on the remote host, if the parallel version of PALM shall be used.</font></li><li><font color="#000000">On the local host, the revision control system <span style="font-weight: bold;">subversion</span> (see <a href="http://subversion.tigris.org/">subversion.tigris.org</a>). |
|---|
| 23 | This is already included in many Linux distributions (e.g. SuSe). <span style="font-weight: bold;">subversion requires port 3690 to be open for tcp/udp.</span> <span style="font-weight: bold;">If there are firewall restrictions concerning this port, the PALM code cannot be accessed.</span> The |
|---|
| [40] | 24 | user needs a permit to access the PALM repository. For getting a permit |
|---|
| 25 | please contact the PALM group (<a href="mailto:raasch@muk.uni-hannover.de">raasch@muk.uni-hannover.de</a>) |
|---|
| 26 | and define a username under which you like to access the repository. |
|---|
| 27 | You will then receive a password which allows the access under this |
|---|
| 28 | name.</font></li><li><font color="#000000">A job queueing system must be available on the remote host. Currently, <span style="font-weight: bold;">mrun</span> can handle LoadLeveler (IBM-AIX) and NQS/PBS (Linux-Clusters, NEC-SX).</font></li><li><font color="#000000">ssh/scp-connections to and from the remote host must not be blocked by a firewall.<br></font></li></ol><font color="#000000"><span style="font-weight: bold;"></span></font><font color="#000000"> Currently, <span style="font-weight: bold;">mrun</span> is configured to be used on a limited number of selected machines. These are IBM-Regatta at computing center HLRN in Hannover (<span style="font-style: italic;">ibmh</span>), Berlin (<span style="font-style: italic;">ibmb</span>), at KISTI, Korea (<span style="font-style: italic;">ibms</span>), at Yonsei University, Seoul (<span style="font-style: italic;">ibms</span>), on NEC-SX6/8 systems at DKRZ, Hamburg (<span style="font-style: italic;">nech</span>) and RIAM, Kyushu University, Fukuoka (<span style="font-style: italic;">neck</span>), as well as on the Linux cluster of IMUK (<span style="font-style: italic;">lcmuk</span>) and Tokyo Institute of Technology (<span style="font-style: italic;">lctit</span>). The strings given in brackets are the systems names under which <span style="font-weight: bold;">mrun</span> indentifies the different hosts. mrun should also work on other systems, but this requires some modifications in the scripts <span style="font-weight: bold;">mrun</span>, <span style="font-weight: bold;">mbuild</span>, and <span style="font-weight: bold;">subjob</span>. Future PALM versions will allow to configure the scripts for different kinds of systems by just using a configuration file.</font><p style="line-height: 100%;"><font color="#000000">The examples given in this chapter refer to an |
|---|
| 29 | installation of PALM on an IMUK Linux workstation and the IBM-Regatta |
|---|
| [5] | 30 | system of |
|---|
| [41] | 31 | the HLRN used as remote host. They are just called local and remote host from now on. </font></p><p style="line-height: 100%;"> |
|---|
| [5] | 32 | </p> |
|---|
| 33 | <p style="line-height: 100%;">The installation process requires a valid |
|---|
| [40] | 34 | account on the local and on the remote host as well.<br> |
|---|
| [5] | 35 | </p> |
|---|
| 36 | <ul> |
|---|
| [41] | 37 | All hosts (local as well as remote) are |
|---|
| [5] | 38 | accessed via the secure shell (ssh). The user must establish |
|---|
| 39 | passwordless login using the private/public-key mechanism (see e.g. the |
|---|
| 40 | <a href="http://www.hlrn.de/doc/ssh/index.html">HLRN documentation</a>). |
|---|
| 41 | <b>To ensure proper function of mrun, passwordless login must be |
|---|
| 42 | established in both directions, from the local to the remote host as |
|---|
| 43 | well as from the remote to the local host!</b> Test this by carrying |
|---|
| [41] | 44 | out e.g. on the local host: |
|---|
| [5] | 45 | <p style="line-height: 100%;"><span style="font-family: monospace;"> |
|---|
| [40] | 46 | </span><span style="font-family: Courier New,Courier,monospace;">ssh <username on remote host>@<remote IP-address></span><br> |
|---|
| [5] | 47 | </p> |
|---|
| 48 | <p style="line-height: 100%;">and on the remote host:<br> |
|---|
| 49 | </p> |
|---|
| 50 | <p style="line-height: 100%;"><span style="font-family: monospace;"> |
|---|
| [40] | 51 | </span><span style="font-family: Courier New,Courier,monospace;">ssh <username on local host>@<local IP-adddress></span><br> |
|---|
| [5] | 52 | </p> |
|---|
| [41] | 53 | In both cases you should not be |
|---|
| [5] | 54 | prompted for a password. <b>Before continuing the further installation |
|---|
| 55 | process, this must be absolutely guaranteed!</b> This must also be |
|---|
| [41] | 56 | guaranteed for <b>all</b> other remote hosts, on which PALM shall run. |
|---|
| [5] | 57 | </ul> |
|---|
| [40] | 58 | <p style="line-height: 100%;"></p><p style="line-height: 100%;"><span style="font-weight: bold; text-decoration: underline;">Package Installation</span></p><p style="line-height: 100%;">In the <b>first installation step</b> a |
|---|
| [5] | 59 | set of directories must be created both on the local and on the |
|---|
| [40] | 60 | remote host. These directories are:</p><div style="margin-left: 40px; font-family: Courier New,Courier,monospace;">~/job_queue<br></div><div style="margin-left: 40px; font-family: Courier New,Courier,monospace;">~/palm<br></div><div style="margin-left: 40px; font-family: Courier New,Courier,monospace;">~/palm/current_version<br></div><div style="margin-left: 40px;"><span style="font-family: Courier New,Courier,monospace;">~/palm/current_version/JOBS</span><br></div><p style="line-height: 100%;">The names of these directories are |
|---|
| [5] | 61 | freely selectable (except <tt><font size="2">~/job_queue</font></tt>), |
|---|
| 62 | however new users should use them as suggested, since many |
|---|
| 63 | examples in this documentation as well as all example files are |
|---|
| [40] | 64 | assuming these settings. The directory <span style="font-family: monospace;">~/palm/current_version</span> on the local host will be called the working directory from now on. </p> |
|---|
| [5] | 65 | <p style="line-height: 100%;"><font color="#000000">In the <b>second |
|---|
| [40] | 66 | step</b> |
|---|
| 67 | a working copy of the recent version of the PALM software package, |
|---|
| 68 | including the source code, scripts, documentation, etc. must be |
|---|
| 69 | copied to the working directory (</font><font color="#000000">local host!) by executing the following |
|---|
| 70 | commands. Replace <your username> by the name that you choosed to access the repository.</font></p><div style="margin-left: 40px;"><span style="font-family: Courier New,Courier,monospace;">cd ~/palm/current_version</span><br style="font-family: Courier New,Courier,monospace;"><span style="font-family: Courier New,Courier,monospace;">svn checkout --username <your username> svn://130.75.105.45/palm/trunk</span><br></div><p style="line-height: 100%;"><font color="#000000">You will then be prompted for your password. After finishing, the subdirectory</font><font style="font-family: Courier New,Courier,monospace;" color="#000000"> trunk </font><font color="#000000">should |
|---|
| 71 | appear in your working directory. It contains a number of further |
|---|
| 72 | subdirectories which contain e.g. the PALM source code (</font><font style="font-family: Courier New,Courier,monospace;" color="#000000">SOURCE</font><font color="#000000">) and the scripts for running PALM (</font><font style="font-family: Courier New,Courier,monospace;" color="#000000">SCRIPTS</font><font color="#000000">).</font></p><p style="line-height: 100%;">Alternatively, executing</p><pre style="line-height: 100%; margin-left: 40px; font-family: Courier New,Courier,monospace;">svn checkout --username <your username> svn://130.75.105.45/palm/trunk abcde</pre><p style="line-height: 100%;">will place your working copy in a directory named<span style="font-family: Courier New,Courier,monospace;"> abcde </span>instead of a directory named<span style="font-family: Courier New,Courier,monospace;"> trunk</span>. But keep in mind that you will have to adjust several paths given below, if you do not use the default directory<span style="font-family: Courier New,Courier,monospace;"> trunk</span>.</p><p style="line-height: 100%;">Please never touch any file in your working copy of PALM, unless you know exactly what you are doing.</p><p style="line-height: 100%; font-weight: bold;">In future PALM releases, repository access to the most recent PALM version will |
|---|
| 73 | probably be restricted because this is a test version. However, access |
|---|
| 74 | to the last (and earlier) stable version(s) will still be possible with |
|---|
| 75 | (e.g.)</p><pre style="line-height: 100%; margin-left: 40px; font-family: Courier New,Courier,monospace;">svn checkout --username <your username> svn://130.75.105.45/palm/tags/release-3.1c trunk</pre><p style="line-height: 100%;"><font color="#000000"></font></p><p style="line-height: 100%;"><font color="#000000"><span style="font-weight: bold; text-decoration: underline;">Package Configuration</span></font></p><p style="line-height: 100%;"><font color="#000000">To |
|---|
| 76 | use the PALM scripts, the PATH variable has to be extended and the |
|---|
| 77 | environment variable PALM_BIN has to be set (on local and remote host) |
|---|
| [41] | 78 | in the respective profile of the users default shell (e.g. in<span style="font-family: Courier New,Courier,monospace;"> .profile</span>, if<span style="font-family: Courier New,Courier,monospace;"> ksh </span>is used):</font></p><p style="line-height: 100%; margin-left: 40px;"><font style="font-family: Courier New,Courier,monospace;" color="#000000">export PATH=$HOME/palm/current_version/trunk/SCRIPTS:$PATH</font><font color="#000000"><br><span style="font-family: Courier New,Courier,monospace;">export PALM_BIN=</span></font><font style="font-family: Courier New,Courier,monospace;" color="#000000">$HOME/palm/current_version/trunk/SCRIPTS</font></p><p style="line-height: 100%;"><font color="#000000">You may habe to login again in order to activate these settings.</font></p><font color="#000000">On the local and on the remote host, some small helper/utility programs have to be installed, which are e.g. later used by <span style="font-weight: bold;">mrun</span> or for PALM data postprocessing. The installation is done by <span style="font-weight: bold;">mbuild</span>. This script requires a configuration file<span style="font-family: Courier New,Courier,monospace;"> .mrun.config</span>, which will be also used by <span style="font-weight: bold;">mrun</span> in the following. A copy has to be put into the working directory under the name<span style="font-family: Courier New,Courier,monospace;"> .mrun.config </span>by</font><p style="line-height: 100%; margin-left: 40px;"><font color="#000000"><span style="font-family: Courier New,Courier,monospace;">cp trunk/SCRIPTS/.mrun.config.default .mrun.config</span></font></p><font color="#000000">Beside many other things, this file contains typical installation parameters |
|---|
| [40] | 79 | like compiler name, compiler options, etc. |
|---|
| [41] | 80 | for a set of different remote hosts. Please edit this file, uncomment lines like</font><div style="margin-left: 40px; font-family: Courier New,Courier,monospace;">#%remote_username <replace by your ... username> <remote-computer-shortcut></div><p style="line-height: 100%;"><font color="#000000">by removing the first hash (<span style="font-family: Courier New,Courier,monospace;">#</span>) character and replace the string "<span style="font-family: Courier New,Courier,monospace;"><replace by ...></span>" by your username on the respective host given in the <span style="font-family: Courier New,Courier,monospace;"><remote-computer-shortcut></span>. You only have to uncomment lines for those hosts on which you intend to use PALM.</font></p><p style="line-height: 100%;"><font color="#000000">After modifying the configuration file, the respective executables are generated by executing</font></p><p style="line-height: 100%; margin-left: 40px; font-family: Courier New,Courier,monospace;"><font color="#000000">mbuild -u -h lcmuk<br>mbuild -u -h ibmh</font></p><p style="line-height: 100%;"><font color="#000000">The second call also copies the PALM scripts like <span style="font-weight: bold;">mrun</span> and <span style="font-weight: bold;">mbuild</span> to the remote host.</font></p><p style="line-height: 100%;"><font color="#000000"></font></p><p style="line-height: 100%;"><font color="#000000"><span style="font-weight: bold; text-decoration: underline;">Pre-Compilation of PALM Code</span></font></p><font color="#000000"> |
|---|
| [40] | 81 | </font> |
|---|
| [41] | 82 | <br><font color="#000000">To avoid the |
|---|
| [5] | 83 | re-compilation of the complete source code for each model run, PALM |
|---|
| [40] | 84 | willl be pre-compiled once on the remote host by again using the script <span style="font-weight: bold;">mbuild</span>. Due to the use of |
|---|
| [5] | 85 | FORTRAN modules in the source code, the subroutines must be compiled |
|---|
| [40] | 86 | in a certain order. Therefore the so-called <i>make</i> mechanism |
|---|
| [5] | 87 | is used (see the respective man-page of the Unix operating system), |
|---|
| [40] | 88 | requiring a<span style="font-family: Courier New,Courier,monospace;"> </span></font><font style="font-family: Courier New,Courier,monospace;" color="#000000">Makefile</font><font color="#000000">, |
|---|
| 89 | in which the dependencies are described. This file is found in subdirectory <span style="font-family: Courier New,Courier,monospace;">trunk/SOURCE<span style="font-family: Times New Roman,Times,serif;">, where also the PALM code is stored</span></span>. The compiled sources (object |
|---|
| [5] | 90 | files) are |
|---|
| [40] | 91 | stored on the remote computer in the default directory |
|---|
| [41] | 92 | </font><font style="font-family: Courier New,Courier,monospace;" color="#000000">~/palm/current_version/MAKE_DEPOSITORY</font>.<font color="#000000"><span style="font-weight: bold;"></span></font><p style="line-height: 100%;"><font color="#000000">The |
|---|
| [40] | 93 | pre-compilation for the remote host (here the IBM-Regatta of HLRN) is done by</font><span style="font-family: monospace;"></span></p><div style="margin-left: 40px;"><span style="font-family: Courier New,Courier,monospace;">mbuild -h ibmh </span></div><p style="line-height: 100%;"><span style="font-family: Courier New,Courier,monospace;">mbuild |
|---|
| 94 | </span>will prompt some queries<font color="#000000">, which must all be |
|---|
| 95 | answered "y" by the user. The compiling process </font><font color="#000000">will take some time.<span style="font-family: Courier New,Courier,monospace;"> </span></font><font style="font-family: Courier New,Courier,monospace;" color="#000000">mbuild</font><font color="#000000"><span style="font-family: Courier New,Courier,monospace;"> </span>transfers the respective compiler calls to the remote |
|---|
| [5] | 96 | host where they are carried out interactively. You can follow the |
|---|
| 97 | progress at the terminal window, where also error messages |
|---|
| 98 | are displayed (hopefully not for this standard installation). By just |
|---|
| [41] | 99 | entering </font></p><p style="line-height: 100%;"> |
|---|
| [5] | 100 | </p> |
|---|
| [40] | 101 | <blockquote style="line-height: 100%; font-family: Courier New,Courier,monospace;"><font color="#000000">mbuild</font></blockquote> |
|---|
| [5] | 102 | <p style="line-height: 100%;"><font color="#000000">PALM will |
|---|
| 103 | be (consecutively) pre-compiled for all remote hosts listed in |
|---|
| [40] | 104 | the configuration file. If you want to compile for the local host only, please enter</font></p><p style="line-height: 100%; margin-left: 40px;"><font color="#000000"><span style="font-family: Courier New,Courier,monospace;">mbuild -h lcmuk</span> </font> |
|---|
| [5] | 105 | </p> |
|---|
| [41] | 106 | <p style="line-height: 100%;"><font color="#000000"></font></p><p style="line-height: 100%; font-weight: bold; text-decoration: underline;"><font color="#000000">Installation Verification</font></p><font color="#000000">As a last step, |
|---|
| [5] | 107 | after the compilation has been finished, the PALM installation has to |
|---|
| 108 | be verified. For this |
|---|
| 109 | purpose a simple test run is carried out. This once again requires the <b>mrun |
|---|
| [40] | 110 | </b>configuration file (described in </font><a href="chapter_3.2.html"><font color="#000080">chapter |
|---|
| 111 | 3.2</font></a><font color="#000000">), as well as the </font><a href="http://www.muk.uni-hannover.de/%7Eraasch/PALM_group/INSTALL/example_p3d"><font color="#000080">parameter |
|---|
| 112 | file</font></a><font color="#000000"> (described in </font><a href="chapter_4.4.html"><font color="#000080">chapter |
|---|
| 113 | 4.4</font></a>)<font color="#000000">. The parameter file must be |
|---|
| 114 | copied from the PALM working copy by<br> |
|---|
| [41] | 115 | </font> |
|---|
| [40] | 116 | <div style="margin-left: 40px;"><span style="font-family: monospace;"></span><font style="font-family: Courier New,Courier,monospace;" color="#000000">mkdir -p JOBS/example/INPUT<br>cp trunk/INSTALL/example_p3d JOBS/example/INPUT/example_p3d</font><font color="#000000"><span style="font-family: Courier New,Courier,monospace;"></span></font></div> |
|---|
| [5] | 117 | <p style="line-height: 100%;"><font color="#000000">The test run can |
|---|
| [40] | 118 | now be started by executing the command </font> |
|---|
| [5] | 119 | </p> |
|---|
| [40] | 120 | <pre style="margin-left: 1cm; margin-right: 1cm; margin-bottom: 0.5cm; line-height: 100%; font-family: Courier New,Courier,monospace;"><font color="#000000">mrun -d example -h ibmh -K parallel -X 8 -T 8 -t 500 -q cdev -r “d3# pr#”</font></pre> |
|---|
| [41] | 121 | <font color="#000000">This specific run |
|---|
| [5] | 122 | will be carried out on 8 PEs and is allowed to use up to 500 seconds |
|---|
| 123 | CPU time. After pressing <return>, the most important settings of |
|---|
| 124 | the job are displayed at the terminal window |
|---|
| [40] | 125 | and the user is prompted for o.k. (“</font><font style="font-family: Courier New,Courier,monospace;" color="#000000">y</font><font color="#000000">”). Next, a message of the queuing system like “<i>Request</i> … |
|---|
| 126 | <i>Submitted to queue… by…” </i>should be displayed. Now the job is |
|---|
| 127 | queued and either started immediately or at a later time, depending on the |
|---|
| [5] | 128 | current workload of the remote host. Provided that it is executed |
|---|
| 129 | immediately and that all things work as designed, the job protocol of |
|---|
| [40] | 130 | this run will appear under the file name<span style="font-family: Courier New,Courier,monospace;"> </span></font><font style="font-family: Courier New,Courier,monospace;" color="#000000">~/job_queue/ibmh_example</font><font color="#000000"><span style="font-family: Courier New,Courier,monospace;"> </span>no more than a few minutes later. The content of this |
|---|
| [41] | 131 | file should be carefully examined for any error messages.</font><p style="line-height: 100%;"> |
|---|
| [5] | 132 | </p> |
|---|
| [40] | 133 | |
|---|
| 134 | |
|---|
| [5] | 135 | <p style="line-height: 100%;">Beside the job protocol and according to |
|---|
| 136 | the configuration file and arguments given for <b>mrun</b> |
|---|
| [40] | 137 | options<span style="font-family: Courier New,Courier,monospace;"> -d </span>and<span style="font-family: Courier New,Courier,monospace;"> -r</span><tt>,</tt>further files should be found in |
|---|
| [5] | 138 | the |
|---|
| [40] | 139 | directories</p><div style="margin-left: 40px; font-family: Courier New,Courier,monospace;">~/palm/current_version/JOBS/example/MONITORING</div><p style="line-height: 100%;">and</p> |
|---|
| [5] | 140 | <p style="line-height: 100%;"><span style="font-family: monospace;"> |
|---|
| [40] | 141 | </span><span style="font-family: Courier New,Courier,monospace;">~/palm/current_version/JOBS/example/OUTPUT</span><span style="font-family: monospace;"></span><br> |
|---|
| [5] | 142 | </p> |
|---|
| 143 | <p style="line-height: 100%;">Please compare the contents of file</p> |
|---|
| 144 | <p style="line-height: 100%;"><span style="font-family: monospace;"> |
|---|
| [40] | 145 | </span><span style="font-family: Courier New,Courier,monospace;">~/palm/current_version/JOBS/example/MONITORING/ibmh_example_rc</span><br> |
|---|
| [5] | 146 | </p> |
|---|
| [41] | 147 | <p style="line-height: 100%;">with those of the example result file which can be found under<span style="font-family: Courier New,Courier,monospace;"> trunk/INSTALL/example_rc</span>., e.g. by using the<span style="font-family: Courier New,Courier,monospace;"><span style="font-family: Times New Roman,Times,serif;"> standard</span> diff </span>command:</p><p style="line-height: 100%; margin-left: 40px; font-family: Courier New,Courier,monospace;">diff JOBS/example/MONITORING/ibmh_example_rc trunk/INSTALL/example_rc</p><p style="line-height: 100%;">where it is assumed that your working directory is<span style="font-family: Courier New,Courier,monospace;"> ~/palm/current_version</span>.</p><span style="font-weight: bold;">You should not find any difference between these two files</span>, except of the run date and time displayed at the top of the file header. <font color="#000000">If |
|---|
| [5] | 148 | the file contents are identical, the installation is successfully |
|---|
| [41] | 149 | completed.</font> |
|---|
| [5] | 150 | <p style="line-height: 100%;">If you have any problems with the PALM |
|---|
| 151 | installation, the members of the PALM working group are pleased to |
|---|
| 152 | help you. </p> |
|---|
| 153 | <hr> |
|---|
| [40] | 154 | <p style="line-height: 100%;"><font color="#000080"><font color="#000080"><a href="chapter_4.6.html"><font color="#000080"><img src="left.gif" name="Grafik1" align="bottom" border="2" height="32" width="32"></font></a><a href="Inhaltsverzeichnis.html"><font color="#000080"><img src="up.gif" name="Grafik2" align="bottom" border="2" height="32" width="32"></font></a><a href="chapter_5.1.html"><font color="#000080"><img src="right.gif" name="Grafik3" align="bottom" border="2" height="32" width="32"></font></a></font></font></p> |
|---|
| 155 | <p style="line-height: 100%;"><i>Last change: </i> 02/03/07 (SR) </p> |
|---|
| 156 | </body></html> |
|---|
