Index: /palm/trunk/DOC/tec/topography_flags.html
===================================================================
--- /palm/trunk/DOC/tec/topography_flags.html	(revision 145)
+++ /palm/trunk/DOC/tec/topography_flags.html	(revision 145)
@@ -0,0 +1,56 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
+<html><head>
+
+
+  
+  <meta http-equiv="CONTENT-TYPE" content="text/html; charset=windows-1252">
+
+  
+
+
+  
+  
+  <title>PALM chapter 3.8</title><meta name="GENERATOR" content="StarOffice 7 (Win32)">
+
+
+  
+  <meta name="AUTHOR" content="Siegfried Raasch">
+
+
+  
+  <meta name="CREATED" content="20040809;13460943">
+
+
+  
+  <meta name="CHANGED" content="20041112;15085727">
+
+
+  
+  <meta name="KEYWORDS" content="parallel LES model">
+
+
+  
+  <style>
+<!--
+@page { size: 21cm 29.7cm }
+-->
+  </style></head>
+<body style="direction: ltr;" lang="en-US">
+
+
+<h3 style="line-height: 100%;">Topography flags</h3>
+
+
+<p style="line-height: 100%;">Starting from version 3.4a
+a flag array has been introduced in PALM for representing the topography. The new array<span style="font-family: Courier New,Courier,monospace;">&nbsp;flags </span>is a 3D-<span style="font-family: Courier New,Courier,monospace;">INTEGER</span> array dimensioned (<span style="font-family: Courier New,Courier,monospace;">nzb:nzt+1,nys-1:nyn+1,nxl-1:nxr+1</span>) where each value describes the features of the respective grid point with respect to the topography. Strictly speaking,&nbsp;<span style="font-family: Courier New,Courier,monospace;">flags</span> is a <span style="font-family: Courier New,Courier,monospace;">POINTER</span> which by default points to the real array<span style="font-family: Courier New,Courier,monospace;"> wall_flags_1</span>. In case of using the multigrid method as pressure solver, additional arrays<span style="font-family: Courier New,Courier,monospace;"> wall_flags_2</span>,<span style="font-family: Courier New,Courier,monospace;"> wall_flags_3</span>,
+etc. are used containing the respective flag informations for the
+coarser grids. (Attention: The finest grid always has the largest
+number, i.e. if four multigrid levels are used, then the flags for the
+finest grid are stored in array<span style="font-family: Courier New,Courier,monospace;"> wall_flags_4</span>.)</p><p style="line-height: 100%;">Each
+INTEGER value has to be interpreted bitwise. As to the FORTRAN model
+bits are counted from right to left starting with bit 0. So far, the
+first seven bits are used for storing the following informations (a bit value of "<span style="font-style: italic;">1</span>" means that the appropriate flag is set):</p><br><table style="text-align: left; height: 220px; width: 496px;" border="1" cellpadding="2" cellspacing="2"><tbody><tr><td style="width: 85px; text-align: center;"><span style="font-weight: bold;">bit position</span></td><td style="width: 393px;"><span style="font-weight: bold;">meaning</span></td></tr><tr><td style="width: 85px; text-align: center;">0</td><td style="width: 393px;">wall to the bottom</td></tr><tr><td style="width: 85px; text-align: center;">1</td><td style="width: 393px;">wall to the top &nbsp;(this cannot be used in the code so far)</td></tr><tr><td style="width: 85px; text-align: center;">2</td><td style="width: 393px;">wall to the south</td></tr><tr><td style="width: 85px; text-align: center;">3</td><td style="width: 393px;">wall to the north</td></tr><tr><td style="width: 85px; text-align: center;">4</td><td style="width: 393px;">wall to the left</td></tr><tr><td style="width: 85px; text-align: center;">5</td><td style="width: 393px;">wall to the rigth</td></tr><tr><td style="width: 85px; text-align: center;">6</td><td style="width: 393px;">inside topography / building</td></tr></tbody></table><p style="line-height: 100%;"><span style="font-weight: bold;">Example:</span><br>If a gridpoint has two neighbouring walls to the bottom and to the left, then the flag bit value is "<span style="font-style: italic;">...010001</span>" which gives the<span style="font-family: Courier New,Courier,monospace;"> INTEGER </span>value "<span style="font-style: italic;">17</span>" = 1*2<sup>4</sup>+1*2<sup>0</sup>.</p><p style="line-height: 100%;">For a gridpoint in the free flow without any neighbouring wall no bits are set, i.e. the<span style="font-family: Courier New,Courier,monospace;"> INTEGER </span>value of the respective array element of<span style="font-family: Courier New,Courier,monospace;"> flags </span>is "<span style="font-style: italic;">0</span>".</p><p style="line-height: 100%;">The intrinsic FORTRAN functions <span style="font-family: Courier New,Courier,monospace;">IBSET</span>, <span style="font-family: Courier New,Courier,monospace;">IBITS</span>, <span style="font-family: Courier New,Courier,monospace;">IAND</span>, <span style="font-family: Courier New,Courier,monospace;">IOR</span>, etc. can be used to manipulate the bits.</p><p style="line-height: 100%; font-weight: bold;"></p><p style="line-height: 100%; font-weight: bold;">Examples:</p><div style="margin-left: 40px;"><span style="font-family: Courier New,Courier,monospace;">flags(k,j,i) = IBSET( flags(k,j,i), 6 )</span></div><p style="line-height: 100%;">sets the sixth bit of array element <span style="font-family: Courier New,Courier,monospace;">(k,j,i)</span> to "<span style="font-style: italic;">1</span>".</p><div style="margin-left: 40px; font-family: Courier New,Courier,monospace;">IBITS( flags(k,j,i), 4, 1 )</div><p style="line-height: 100%;">gives the<span style="font-family: Courier New,Courier,monospace;"> INTEGER </span>value "<span style="font-style: italic;">1</span>" if the fourth bit of array element <span style="font-family: Courier New,Courier,monospace;">(k,j,i)</span> is "<span style="font-style: italic;">1</span>", otherwise "<span style="font-style: italic;">0</span>".</p><p style="line-height: 100%;">Here is an example from routine <span style="font-family: Courier New,Courier,monospace;">redblack</span> (within <span style="font-family: Courier New,Courier,monospace;">poismg.f90</span>) which shows how <span style="font-family: Courier New,Courier,monospace;">IBITS</span> is used for implicit setting of Neumann boundary conditions:</p><p style="line-height: 100%; margin-left: 40px; font-family: Courier New,Courier,monospace;">p_mg(k,j,i+1) + IBITS( flags(k,j,i), 5, 1 ) * &amp;<br>&nbsp; &nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ( p_mg(k,j,i) - p_mg(k,j,i+1) )&nbsp;</p><p style="line-height: 100%;">If there is no wall to the right of gridpoint <span style="font-family: Courier New,Courier,monospace;">(k,j,i)</span> the term results to <span style="font-family: Courier New,Courier,monospace;">p_mg(k,j,i+1)</span>, otherwise it gives <span style="font-family: Courier New,Courier,monospace;">p_mg(k,j,i)</span>.</p><br>The flag array <span style="font-family: Courier New,Courier,monospace;">flags</span> is calculated in routine <span style="font-family: Courier New,Courier,monospace;">init_grid</span> using the topography information stored in <span style="font-family: Courier New,Courier,monospace;">nzb_local</span>.
+Here also the flag arrays for the coarser grids needed for the
+multigrid scheme are calculated. This requires informations from a
+distance of more than one grid spacing away from the current gridpoint.
+Therfore, array <span style="font-family: Courier New,Courier,monospace;">nzb_local</span> had to be extended for further ghost point levels (see <span style="font-family: Courier New,Courier,monospace;">init_grid</span>).</body></html>
Index: /palm/trunk/SCRIPTS/mrun
===================================================================
--- /palm/trunk/SCRIPTS/mrun	(revision 144)
+++ /palm/trunk/SCRIPTS/mrun	(revision 145)
@@ -207,5 +207,5 @@
  read_from_config=""
  restart_run=false
- return_addres=$(nslookup `hostname` 2>&1 | grep "Address:" | tail -1 | awk '{print $2}')
+# return_addres=$(nslookup `hostname` 2>&1 | grep "Address:" | tail -1 | awk '{print $2}')
  if [[ $return_addres = 130.75.105.158 ]]
  then
Index: /palm/trunk/SOURCE/CURRENT_MODIFICATIONS
===================================================================
--- /palm/trunk/SOURCE/CURRENT_MODIFICATIONS	(revision 144)
+++ /palm/trunk/SOURCE/CURRENT_MODIFICATIONS	(revision 145)
@@ -17,5 +17,5 @@
 User-defined spectra.
 
-calc_spectra, check_open, data_output_spectra, netcdf, read_var_list, read_3d_binary, user_interface, write_3d_binary
+calc_spectra, check_open, data_output_spectra, init_pegrid, modules, netcdf, read_var_list, read_3d_binary, user_interface, write_var_list, write_3d_binary
 
 
Index: /palm/trunk/SOURCE/init_grid.f90
===================================================================
--- /palm/trunk/SOURCE/init_grid.f90	(revision 144)
+++ /palm/trunk/SOURCE/init_grid.f90	(revision 145)
@@ -428,8 +428,8 @@
 !
 !-- Test output of nzb_local -1:ny+1,-1:nx+1
-    WRITE (9,*)  '*** nzb_local ***'
-    DO  j = ny+1, -1, -1
-       WRITE (9,'(194(1X,I2))')  ( nzb_local(j,i), i = -1, nx+1 )
-    ENDDO
+!    WRITE (9,*)  '*** nzb_local ***'
+!    DO  j = ny+1, -1, -1
+!       WRITE (9,'(194(1X,I2))')  ( nzb_local(j,i), i = -1, nx+1 )
+!    ENDDO
 
 !
@@ -865,12 +865,12 @@
 !
 !--       Test output of flag arrays
-          i = nxl_l
-          WRITE (9,*)  ' '
-          WRITE (9,*)  '*** mg level ', l, ' ***', mg_switch_to_pe0_level
-          WRITE (9,*)  '    inc=', inc, '  i =', nxl_l
-          WRITE (9,*)  '    nxl_l',nxl_l,' nxr_l=',nxr_l,' nys_l=',nys_l,' nyn_l=',nyn_l
-          DO  k = nzt_l+1, nzb, -1
-             WRITE (9,'(194(1X,I2))')  ( flags(k,j,i), j = nys_l-1, nyn_l+1 )
-          ENDDO
+!          i = nxl_l
+!          WRITE (9,*)  ' '
+!          WRITE (9,*)  '*** mg level ', l, ' ***', mg_switch_to_pe0_level
+!          WRITE (9,*)  '    inc=', inc, '  i =', nxl_l
+!          WRITE (9,*)  '    nxl_l',nxl_l,' nxr_l=',nxr_l,' nys_l=',nys_l,' nyn_l=',nyn_l
+!          DO  k = nzt_l+1, nzb, -1
+!             WRITE (9,'(194(1X,I2))')  ( flags(k,j,i), j = nys_l-1, nyn_l+1 )
+!          ENDDO
 
           inc = inc * 2
Index: /palm/trunk/SOURCE/init_pegrid.f90
===================================================================
--- /palm/trunk/SOURCE/init_pegrid.f90	(revision 144)
+++ /palm/trunk/SOURCE/init_pegrid.f90	(revision 145)
@@ -4,5 +4,5 @@
 ! Actual revisions:
 ! -----------------
-! 
+! Collect on PE0 horizontal index bounds from all other PEs
 ! TEST OUTPUT (TO BE REMOVED) logging mpi2 ierr values
 !
@@ -458,4 +458,34 @@
     DEALLOCATE( nxlf , nxrf , nynf , nysf )
 
+!
+!-- Collect index bounds from other PEs (to be written to restart file later)
+    ALLOCATE( hor_index_bounds(4,0:numprocs-1) )
+
+    IF ( myid == 0 )  THEN
+
+       hor_index_bounds(1,0) = nxl
+       hor_index_bounds(2,0) = nxr
+       hor_index_bounds(3,0) = nys
+       hor_index_bounds(4,0) = nyn
+
+!
+!--    Receive data from all other PEs
+       DO  i = 1, numprocs-1
+          CALL MPI_RECV( ibuf, 4, MPI_INTEGER, i, MPI_ANY_TAG, comm2d, status, &
+                         ierr )
+          hor_index_bounds(:,i) = ibuf(1:4)
+       ENDDO
+
+    ELSE
+!
+!--    Send index bounds to PE0
+       ibuf(1) = nxl
+       ibuf(2) = nxr
+       ibuf(3) = nys
+       ibuf(4) = nyn
+       CALL MPI_SEND( ibuf, 4, MPI_INTEGER, 0, myid, comm2d, ierr )
+
+    ENDIF
+
 #if defined( __print )
 !
@@ -606,4 +636,10 @@
     nzta = nz
     nnz  = nz
+
+    ALLOCATE( hor_index_bounds(4,0:0) )
+    hor_index_bounds(1,0) = nxl
+    hor_index_bounds(2,0) = nxr
+    hor_index_bounds(3,0) = nys
+    hor_index_bounds(4,0) = nyn
 
 !
Index: /palm/trunk/SOURCE/modules.f90
===================================================================
--- /palm/trunk/SOURCE/modules.f90	(revision 144)
+++ /palm/trunk/SOURCE/modules.f90	(revision 145)
@@ -5,5 +5,5 @@
 ! Actual revisions:
 ! -----------------
-! 
+! +hor_index_bounds, hor_index_bounds_previous_run, numprocs_previous_run
 !
 ! Former revisions:
@@ -959,5 +959,9 @@
     CHARACTER(LEN=5)       ::  myid_char = '', myid_char_14 = ''
     INTEGER                ::  myid=0, npex = -1, npey = -1, numprocs = 1, &
+                               numprocs_previous_run = -1,                 &
                                tasks_per_node = -9999, threads_per_task = 1
+
+    INTEGER, DIMENSION(:,:), ALLOCATABLE ::  hor_index_bounds, &
+                                             hor_index_bounds_previous_run
 
 #if defined( __parallel )
Index: /palm/trunk/SOURCE/parin.f90
===================================================================
--- /palm/trunk/SOURCE/parin.f90	(revision 144)
+++ /palm/trunk/SOURCE/parin.f90	(revision 145)
@@ -246,5 +246,5 @@
                  u_init(0:nz+1), v_init(0:nz+1),                            &
                  hom(0:nz+1,2,pr_palm+max_pr_user,0:statistic_regions),     &
-                 hom_sum(nzb:nzt+1,pr_palm+max_pr_user,0:statistic_regions) )
+                 hom_sum(0:nz+1,pr_palm+max_pr_user,0:statistic_regions) )
        hom = 0.0
 
Index: /palm/trunk/SOURCE/read_3d_binary.f90
===================================================================
--- /palm/trunk/SOURCE/read_3d_binary.f90	(revision 144)
+++ /palm/trunk/SOURCE/read_3d_binary.f90	(revision 145)
@@ -5,6 +5,7 @@
 ! -----------------
 ! Restart file has to be re-opened on all PEs except PE0
+! +call of user_read_restart_data,
 ! -dopr_time_count,
-! hom_sum moved, volume_flow_area, volume_flow_initial moved to read_var_list,
+! hom_sum, volume_flow_area, volume_flow_initial moved to read_var_list,
 ! reading of old profil parameters (cross_..., dopr_crossindex, profile_***)
 ! removed
@@ -73,5 +74,5 @@
 !-- First compare the version numbers
     READ ( 13 )  version_on_file
-    binary_version = '3.0'
+    binary_version = '3.1'
     IF ( TRIM( version_on_file ) /= TRIM( binary_version ) )  THEN
        IF ( myid == 0 )  THEN
@@ -405,4 +406,8 @@
 
 !
+!-- Read user-defined restart data
+    CALL user_read_restart_data
+
+!
 !-- End of time measuring for reading binary data
     CALL cpu_log( log_point_s(14), 'read_3d_binary', 'stop' )
Index: /palm/trunk/SOURCE/read_var_list.f90
===================================================================
--- /palm/trunk/SOURCE/read_var_list.f90	(revision 144)
+++ /palm/trunk/SOURCE/read_var_list.f90	(revision 145)
@@ -5,6 +5,7 @@
 ! -----------------
 ! After reading the global variables, unit 13 is closed for all PEs except PE0
-! -cross_ts_*,
-! hom_sum moved, volume_flow_area, volume_flow_initial moved from
+! +numprocs_previous_run, hor_index_bounds_previous_run,
+! -cross_ts_*, npex, npey,
+! hom_sum, volume_flow_area, volume_flow_initial moved from
 ! read_3d_binary to here
 !
@@ -72,5 +73,5 @@
 !-- Make version number check first
     READ ( 13 )  version_on_file
-    binary_version = '3.1'
+    binary_version = '3.2'
     IF ( TRIM( version_on_file ) /= TRIM( binary_version ) )  THEN
        IF ( myid == 0 )  THEN
@@ -84,4 +85,25 @@
        CALL local_stop
     ENDIF
+
+!
+!-- Read number of PEs and horizontal index bounds of all PEs used in previous
+!-- run
+    READ ( 13 )  variable_chr
+    IF ( TRIM( variable_chr ) /= 'numprocs' )  THEN
+       PRINT*, '+++ read_var_list: numprocs not found in data from prior run', &
+                    ' on PE ', myid
+       CALL local_stop
+    ENDIF
+    READ ( 13 )  numprocs_previous_run
+
+    ALLOCATE( hor_index_bounds_previous_run(4,0:numprocs_previous_run-1) )
+
+    READ ( 13 )  variable_chr
+    IF ( TRIM( variable_chr ) /= 'hor_index_bounds' )  THEN
+       PRINT*, '+++ read_var_list: hor_index_bounds not found in data from ', &
+                    'prior run on PE ', myid
+       CALL local_stop
+    ENDIF
+    READ ( 13 )  hor_index_bounds_previous_run
 
 !
@@ -124,6 +146,7 @@
     READ ( 13 )  statistic_regions
     ALLOCATE( ug(0:nz+1), u_init(0:nz+1), vg(0:nz+1), v_init(0:nz+1), & 
-              pt_init(0:nz+1), q_init(0:nz+1), sa_init(0:nz+1), &
-              hom(0:nz+1,2,pr_palm+max_pr_user,0:statistic_regions) )
+              pt_init(0:nz+1), q_init(0:nz+1), sa_init(0:nz+1),       &
+              hom(0:nz+1,2,pr_palm+max_pr_user,0:statistic_regions),  &
+              hom_sum(0:nz+1,pr_palm+max_pr_user,0:statistic_regions) )
 
 !
@@ -283,8 +306,4 @@
           CASE ( 'netcdf_precision' )
              READ ( 13 )  netcdf_precision
-          CASE ( 'npex' )
-             READ ( 13 )  npex
-          CASE ( 'npey' )
-             READ ( 13 )  npey
           CASE ( 'nsor_ini' )
              READ ( 13 )  nsor_ini
Index: /palm/trunk/SOURCE/user_interface.f90
===================================================================
--- /palm/trunk/SOURCE/user_interface.f90	(revision 144)
+++ /palm/trunk/SOURCE/user_interface.f90	(revision 145)
@@ -4,5 +4,5 @@
 ! Actual revisions:
 ! -----------------
-! new subroutine user_spectra
+! +routines user_read_restart_data, user_spectra
 !
 ! Former revisions:
@@ -177,16 +177,15 @@
 
 
- SUBROUTINE user_init
-
-!------------------------------------------------------------------------------!
-!
-! Description:
-! ------------
-! Execution of user-defined initializing actions
+ SUBROUTINE user_read_restart_data
+
+!------------------------------------------------------------------------------!
+!
+! Description:
+! ------------
+! Reading restart data from file(s)
 !------------------------------------------------------------------------------!
 
     USE control_parameters
     USE indices
-    USE netcdf_control
     USE pegrid
     USE user
@@ -196,5 +195,5 @@
     CHARACTER (LEN=20) :: field_char
 !
-!-- Here the user-defined initializing actions follow:
+!-- Here the reading of user-defined restart data follows:
 !-- Sample for user-defined output
 !    ALLOCATE( u2(nzb:nzt+1,nys-1:nyn+1,nxl-1:nxr+1) )
@@ -224,5 +223,28 @@
 !    ENDIF
 
-!
+ END SUBROUTINE user_read_restart_data
+
+
+
+ SUBROUTINE user_init
+
+!------------------------------------------------------------------------------!
+!
+! Description:
+! ------------
+! Execution of user-defined initializing actions
+!------------------------------------------------------------------------------!
+
+    USE control_parameters
+    USE indices
+    USE netcdf_control
+    USE pegrid
+    USE user
+
+    IMPLICIT NONE
+
+    CHARACTER (LEN=20) :: field_char
+!
+!-- Here the user-defined initializing actions follow:
 !-- Sample for user-defined time series
 !-- For each time series quantity you have to give a label and a unit,
Index: /palm/trunk/SOURCE/write_3d_binary.f90
===================================================================
--- /palm/trunk/SOURCE/write_3d_binary.f90	(revision 144)
+++ /palm/trunk/SOURCE/write_3d_binary.f90	(revision 145)
@@ -69,5 +69,5 @@
 !
 !-- Write arrays.
-    binary_version = '3.0'
+    binary_version = '3.1'
 
     WRITE ( 14 )  binary_version
Index: /palm/trunk/SOURCE/write_var_list.f90
===================================================================
--- /palm/trunk/SOURCE/write_var_list.f90	(revision 144)
+++ /palm/trunk/SOURCE/write_var_list.f90	(revision 145)
@@ -4,5 +4,5 @@
 ! Actual revisions:
 ! -----------------
-! -cross_ts_*,
+! +numprocs, hor_index_bounds, -cross_ts_*, npex, npey
 ! hom_sum, volume_flow_area, volume_flow_initial moved from write_3d_binary
 ! to here
@@ -44,5 +44,6 @@
 ! Description:
 ! ------------
-! Writing values of control variables to restart-file (binary format)
+! Writing values of control variables to restart-file (binary format).
+! These informations are only written to the file opened by PE0.
 !------------------------------------------------------------------------------!
 
@@ -63,9 +64,12 @@
 
 
-
-    binary_version = '3.1'
-
-    WRITE ( 14 )  binary_version    ! opened in write_3d_binary
-
+    binary_version = '3.2'
+
+    WRITE ( 14 )  binary_version
+
+    WRITE ( 14 )  'numprocs                      '
+    WRITE ( 14 )  numprocs
+    WRITE ( 14 )  'hor_index_bounds              '
+    WRITE ( 14 )  hor_index_bounds
     WRITE ( 14 )  'nz                            '
     WRITE ( 14 )  nz
@@ -227,8 +231,4 @@
     WRITE ( 14 )  'netcdf_precision              '
     WRITE ( 14 )  netcdf_precision
-    WRITE ( 14 )  'npex                          '
-    WRITE ( 14 )  npex
-    WRITE ( 14 )  'npey                          '
-    WRITE ( 14 )  npey
     WRITE ( 14 )  'nsor_ini                      '
     WRITE ( 14 )  nsor_ini
