Home


About us
PALM
PALM-4U
Installation
Release notes
Documentation
Getting Help
Gallery
Developers area

Changeset 622 for palm/trunk/SOURCE


Ignore:
Timestamp:
Dec 10, 2010 8:08:13 AM (14 years ago)
Author:
raasch
Message:

New:
---

Optional barriers included in order to speed up collective operations
MPI_ALLTOALL and MPI_ALLREDUCE. This feature is controlled with new initial
parameter collective_wait. Default is .FALSE, but .TRUE. on SGI-type
systems. (advec_particles, advec_s_bc, buoyancy, check_for_restart,
cpu_statistics, data_output_2d, data_output_ptseries, flow_statistics,
global_min_max, inflow_turbulence, init_3d_model, init_particles, init_pegrid,
init_slope, parin, pres, poismg, set_particle_attributes, timestep,
read_var_list, user_statistics, write_compressed, write_var_list)

Adjustments for Kyushu Univ. (lcrte, ibmku). Concerning hybrid
(MPI/openMP) runs, the number of openMP threads per MPI tasks can now
be given as an argument to mrun-option -O. (mbuild, mrun, subjob)

Changed:

Initialization of the module command changed for SGI-ICE/lcsgi (mbuild, subjob)

Errors:

Location:
palm/trunk/SOURCE
Files:
26 edited

Legend:

Unmodified
Added
Removed

  • palm/trunk/SOURCE/advec_particles.f90

    r559 r622  
    44! Current revisions:
    55! -----------------
     6! optional barriers included in order to speed up collective operations
    67! TEST: PRINT statements on unit 9 (commented out)
    78!
    … …  
    792793!
    793794!--       Compute total sum from local sums
     795          IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    794796          CALL MPI_ALLREDUCE( sums_l(nzb,1,0), sums(nzb,1), nzt+2-nzb, &
    795797                              MPI_REAL, MPI_SUM, comm2d, ierr )
     798          IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    796799          CALL MPI_ALLREDUCE( sums_l(nzb,2,0), sums(nzb,2), nzt+2-nzb, &
    797800                              MPI_REAL, MPI_SUM, comm2d, ierr )
    … …  
    830833!
    831834!--       Compute total sum from local sums
     835          IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    832836          CALL MPI_ALLREDUCE( sums_l(nzb,8,0), sums(nzb,8), nzt+2-nzb, &
    833837                              MPI_REAL, MPI_SUM, comm2d, ierr )
     838          IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    834839          CALL MPI_ALLREDUCE( sums_l(nzb,30,0), sums(nzb,30), nzt+2-nzb, &
    835840                              MPI_REAL, MPI_SUM, comm2d, ierr )
     841          IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    836842          CALL MPI_ALLREDUCE( sums_l(nzb,31,0), sums(nzb,31), nzt+2-nzb, &
    837843                              MPI_REAL, MPI_SUM, comm2d, ierr )
     844          IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    838845          CALL MPI_ALLREDUCE( sums_l(nzb,32,0), sums(nzb,32), nzt+2-nzb, &
    839846                              MPI_REAL, MPI_SUM, comm2d, ierr )
    … …  
    19481955!--    and set the switch corespondingly
    19491956#if defined( __parallel )
     1957       IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    19501958       CALL MPI_ALLREDUCE( dt_3d_reached_l, dt_3d_reached, 1, MPI_LOGICAL, &
    19511959                           MPI_LAND, comm2d, ierr )

  • palm/trunk/SOURCE/advec_s_bc.f90

    r392 r622  
    44! Current revisions:
    55! -----------------
    6 !
     6! optional barriers included in order to speed up collective operations
    77!
    88! Former revisions:
    … …  
    166166    ENDDO
    167167#if defined( __parallel )
     168    IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    168169    CALL MPI_ALLREDUCE( fmax_l, fmax, 2, MPI_REAL, MPI_MAX, comm2d, ierr )
    169170#else
    … …  
    463464    ENDDO
    464465#if defined( __parallel )
     466    IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    465467    CALL MPI_ALLREDUCE( fmax_l, fmax, 2, MPI_REAL, MPI_MAX, comm2d, ierr )
    466468#else
    … …  
    863865    ENDDO
    864866#if defined( __parallel )
     867    IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    865868    CALL MPI_ALLREDUCE( fmax_l, fmax, 2, MPI_REAL, MPI_MAX, comm2d, ierr )
    866869#else

  • palm/trunk/SOURCE/buoyancy.f90

    r516 r622  
    44! Currrent revisions:
    55! -----------------
    6 !
     6! optional barriers included in order to speed up collective operations
    77!
    88! Former revisions:
    … …  
    284284#if defined( __parallel )
    285285
     286          IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    286287          CALL MPI_ALLREDUCE( sums_l(nzb,pr,0), sums(nzb,pr), nzt+2-nzb, &
    287288                              MPI_REAL, MPI_SUM, comm2d, ierr )

  • palm/trunk/SOURCE/check_for_restart.f90

    r392 r622  
    44! Current revisions:
    55! -----------------
    6 !
     6! optional barriers included in order to speed up collective operations
    77!
    88! Former revisions:
    … …  
    6363!-- Make a logical OR for all processes. Stop the model run if at least
    6464!-- one processor has reached the time limit.
     65    IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    6566    CALL MPI_ALLREDUCE( terminate_run_l, terminate_run, 1, MPI_LOGICAL, &
    6667                        MPI_LOR, comm2d, ierr )

  • palm/trunk/SOURCE/cpu_statistics.f90

    r484 r622  
    44! Current revisions:
    55! -----------------
    6 !
     6! output of handling of collective operations
    77!
    88! Former revisions:
    … …  
    248248
    249249!
     250!--    Output handling of collective operations
     251       IF ( collective_wait )  THEN
     252          WRITE ( 18, 103 )
     253       ELSE
     254          WRITE ( 18, 104 )
     255       ENDIF
     256
     257!
    250258!--    Empty lines in order to create a gap to the results of the model
    251259!--    continuation runs
    252        WRITE ( 18, 103 )
     260       WRITE ( 18, 105 )
    253261
    254262!
    … …  
    275283
    276284102 FORMAT (A20,2X,F9.3,2X,F7.2,1X,I7,3(1X,F9.3))
    277 103 FORMAT (//)
     285103 FORMAT (/'Barriers are set in front of collective operations')
     286104 FORMAT (/'No barriers are set in front of collective operations')
     287105 FORMAT (//)
    278288
    279289 END SUBROUTINE cpu_statistics

  • palm/trunk/SOURCE/data_output_2d.f90

    r559 r622  
    44! Current revisions:
    55! -----------------
    6 !
     6! optional barriers included in order to speed up collective operations
    77!
    88! Former revisions:
    … …  
    899899!
    900900!--                   Now do the averaging over all PEs along y
     901                      IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    901902                      CALL MPI_ALLREDUCE( local_2d_l(nxl-1,nzb),              &
    902903                                          local_2d(nxl-1,nzb), ngp, MPI_REAL, &
    … …  
    942943!--                      Distribute data over all PEs along y
    943944                         ngp = ( nxr-nxl+3 ) * ( nzt-nzb+2 )
     945                         IF ( collective_wait ) CALL MPI_BARRIER( comm2d, ierr )
    944946                         CALL MPI_ALLREDUCE( local_2d_l(nxl-1,nzb),            &
    945947                                             local_2d(nxl-1,nzb), ngp,         &
    … …  
    11981200!
    11991201!--                   Now do the averaging over all PEs along x
     1202                      IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    12001203                      CALL MPI_ALLREDUCE( local_2d_l(nys-1,nzb),              &
    12011204                                          local_2d(nys-1,nzb), ngp, MPI_REAL, &
    … …  
    12411244!--                      Distribute data over all PEs along x
    12421245                         ngp = ( nyn-nys+3 ) * ( nzt-nzb+2 )
     1246                         IF ( collective_wait ) CALL MPI_BARRIER( comm2d, ierr )
    12431247                         CALL MPI_ALLREDUCE( local_2d_l(nys-1,nzb),            &
    12441248                                             local_2d(nys-1,nzb), ngp,         &

  • palm/trunk/SOURCE/data_output_ptseries.f90

    r392 r622  
    44! Current revisions:
    55! -----------------
    6 !
     6! optional barriers included in order to speed up collective operations
    77!
    88! Former revisions:
    … …  
    138138    inum = number_of_particle_groups + 1
    139139
     140    IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    140141    CALL MPI_ALLREDUCE( pts_value_l(0,1), pts_value(0,1), 14*inum, MPI_REAL, &
    141142                        MPI_SUM, comm2d, ierr )
     143    IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    142144    CALL MPI_ALLREDUCE( pts_value_l(0,15), pts_value(0,15), inum, MPI_REAL, &
    143145                        MPI_MAX, comm2d, ierr )
     146    IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    144147    CALL MPI_ALLREDUCE( pts_value_l(0,16), pts_value(0,16), inum, MPI_REAL, &
    145148                        MPI_MIN, comm2d, ierr )
    … …  
    239242    inum = number_of_particle_groups + 1
    240243
     244    IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    241245    CALL MPI_ALLREDUCE( pts_value_l(0,17), pts_value(0,17), inum*10, MPI_REAL, &
    242246                        MPI_SUM, comm2d, ierr )

  • palm/trunk/SOURCE/flow_statistics.f90

    r550 r622  
    44! Current revisions:
    55! -----------------
    6 !
     6! optional barriers included in order to speed up collective operations
    77!
    88! Former revisions:
    … …  
    237237!
    238238!--    Compute total sum from local sums
     239       IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    239240       CALL MPI_ALLREDUCE( sums_l(nzb,1,0), sums(nzb,1), nzt+2-nzb, MPI_REAL, &
    240241                           MPI_SUM, comm2d, ierr )
     242       IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    241243       CALL MPI_ALLREDUCE( sums_l(nzb,2,0), sums(nzb,2), nzt+2-nzb, MPI_REAL, &
    242244                           MPI_SUM, comm2d, ierr )
     245       IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    243246       CALL MPI_ALLREDUCE( sums_l(nzb,4,0), sums(nzb,4), nzt+2-nzb, MPI_REAL, &
    244247                           MPI_SUM, comm2d, ierr )
    245248       IF ( ocean )  THEN
     249          IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    246250          CALL MPI_ALLREDUCE( sums_l(nzb,23,0), sums(nzb,23), nzt+2-nzb, &
    247251                              MPI_REAL, MPI_SUM, comm2d, ierr )
    248252       ENDIF
    249253       IF ( humidity ) THEN
     254          IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    250255          CALL MPI_ALLREDUCE( sums_l(nzb,44,0), sums(nzb,44), nzt+2-nzb, &
    251256                              MPI_REAL, MPI_SUM, comm2d, ierr )
     257          IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    252258          CALL MPI_ALLREDUCE( sums_l(nzb,41,0), sums(nzb,41), nzt+2-nzb, &
    253259                              MPI_REAL, MPI_SUM, comm2d, ierr )
    254260          IF ( cloud_physics ) THEN
     261             IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    255262             CALL MPI_ALLREDUCE( sums_l(nzb,42,0), sums(nzb,42), nzt+2-nzb, &
    256263                                 MPI_REAL, MPI_SUM, comm2d, ierr )
     264             IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    257265             CALL MPI_ALLREDUCE( sums_l(nzb,43,0), sums(nzb,43), nzt+2-nzb, &
    258266                                 MPI_REAL, MPI_SUM, comm2d, ierr )
    … …  
    261269
    262270       IF ( passive_scalar )  THEN
     271          IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    263272          CALL MPI_ALLREDUCE( sums_l(nzb,41,0), sums(nzb,41), nzt+2-nzb, &
    264273                              MPI_REAL, MPI_SUM, comm2d, ierr )
    … …  
    796805!
    797806!--    Compute total sum from local sums
     807       IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    798808       CALL MPI_ALLREDUCE( sums_l(nzb,1,0), sums(nzb,1), ngp_sums, MPI_REAL, &
    799809                           MPI_SUM, comm2d, ierr )

  • palm/trunk/SOURCE/global_min_max.f90

    r484 r622  
    55! Current revisions:
    66! -----------------
    7 !
     7! optional barriers included in order to speed up collective operations
    88!
    99! Former revisions:
    … …  
    6161#if defined( __parallel )
    6262       fmin_l(2)  = myid
     63       IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    6364       CALL MPI_ALLREDUCE( fmin_l, fmin, 1, MPI_2REAL, MPI_MINLOC, comm2d, ierr )
    6465
    … …  
    100101#if defined( __parallel )
    101102       fmax_l(2)  = myid
     103       IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    102104       CALL MPI_ALLREDUCE( fmax_l, fmax, 1, MPI_2REAL, MPI_MAXLOC, comm2d, ierr )
    103105
    … …  
    158160#if defined( __parallel )
    159161       fmax_l(2)  = myid
     162       IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    160163       CALL MPI_ALLREDUCE( fmax_l, fmax, 1, MPI_2REAL, MPI_MAXLOC, comm2d, &
    161164                           ierr )

  • palm/trunk/SOURCE/inflow_turbulence.f90

    r484 r622  
    44! Current revisions:
    55! -----------------
    6 !
     6! optional barriers included in order to speed up collective operations
    77!
    88! Former revisions:
    … …  
    7777!
    7878!-- Now, averaging over all PEs
     79    IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    7980    CALL MPI_ALLREDUCE( avpr_l(nzb,1), avpr(nzb,1), ngp_pr, MPI_REAL, MPI_SUM, &
    8081                        comm2d, ierr )
    … …  
    195196
    196197#if defined( __parallel )   
     198!       IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    197199!       CALL MPI_ALLREDUCE( volume_flow_l(1), volume_flow(1), 1, MPI_REAL, &
    198200!                           MPI_SUM, comm1dy, ierr )   

  • palm/trunk/SOURCE/init_3d_model.f90

    r561 r622  
    77! Current revisions:
    88! -----------------
    9 !
     9! optional barriers included in order to speed up collective operations
    1010!
    1111! Former revisions:
    … …  
    860860
    861861#if defined( __parallel )
     862          IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    862863          CALL MPI_ALLREDUCE( volume_flow_initial_l(1), volume_flow_initial(1),&
    863864                              2, MPI_REAL, MPI_SUM, comm2d, ierr )
     865          IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    864866          CALL MPI_ALLREDUCE( volume_flow_area_l(1), volume_flow_area(1),      &
    865867                              2, MPI_REAL, MPI_SUM, comm2d, ierr )
    … …  
    11721174
    11731175#if defined( __parallel )
     1176          IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    11741177          CALL MPI_ALLREDUCE( volume_flow_initial_l(1), volume_flow_initial(1),&
    11751178                              2, MPI_REAL, MPI_SUM, comm2d, ierr )
     1179          IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    11761180          CALL MPI_ALLREDUCE( volume_flow_area_l(1), volume_flow_area(1),      &
    11771181                              2, MPI_REAL, MPI_SUM, comm2d, ierr )
    … …  
    15601564    sr = statistic_regions + 1
    15611565#if defined( __parallel )
     1566    IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    15621567    CALL MPI_ALLREDUCE( ngp_2dh_l(0), ngp_2dh(0), sr, MPI_INTEGER, MPI_SUM,   &
    15631568                        comm2d, ierr )
     1569    IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    15641570    CALL MPI_ALLREDUCE( ngp_2dh_outer_l(0,0), ngp_2dh_outer(0,0), (nz+2)*sr,  &
    15651571                        MPI_INTEGER, MPI_SUM, comm2d, ierr )
     1572    IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    15661573    CALL MPI_ALLREDUCE( ngp_2dh_s_inner_l(0,0), ngp_2dh_s_inner(0,0),         &
    15671574                        (nz+2)*sr, MPI_INTEGER, MPI_SUM, comm2d, ierr )
     1575    IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    15681576    CALL MPI_ALLREDUCE( ngp_3d_inner_l(0), ngp_3d_inner_tmp(0), sr, MPI_REAL, &
    15691577                        MPI_SUM, comm2d, ierr )

  • palm/trunk/SOURCE/init_particles.f90

    r392 r622  
    44! Current revisions:
    55! -----------------
    6 !
     6! optional barriers included in order to speed up collective operations
    77!
    88! Former revisions:
    … …  
    342342!--    Calculate the number of particles and tails of the total domain
    343343#if defined( __parallel )
     344       IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    344345       CALL MPI_ALLREDUCE( number_of_particles, total_number_of_particles, 1, &
    345346                           MPI_INTEGER, MPI_SUM, comm2d, ierr )
     347       IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    346348       CALL MPI_ALLREDUCE( number_of_tails, total_number_of_tails, 1, &
    347349                           MPI_INTEGER, MPI_SUM, comm2d, ierr )
    … …  
    436438
    437439#if defined( __parallel )
     440          IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    438441          CALL MPI_ALLREDUCE( uniform_particles_l, uniform_particles, 1, &
    439442                              MPI_LOGICAL, MPI_LAND, comm2d, ierr )

  • palm/trunk/SOURCE/init_pegrid.f90

    r482 r622  
    44! Current revisions:
    55! -----------------
    6 !
     6! optional barriers included in order to speed up collective operations
    77! ATTENTION: nnz_x undefined problem still has to be solved!!!!!!!!
    88! TEST OUTPUT (TO BE REMOVED) logging mpi2 ierr values
    … …  
    154154       CALL message( 'init_pegrid', 'PA0223', 1, 2, 0, 6, 0 )
    155155    ENDIF
     156
     157!
     158!-- For communication speedup, set barriers in front of collective
     159!-- communications by default on SGI-type systems
     160    IF ( host(3:5) == 'sgi' )  collective_wait = .TRUE.
    156161
    157162!
    … …  
    929934       id_inflow_l = 0
    930935    ENDIF
     936    IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    931937    CALL MPI_ALLREDUCE( id_inflow_l, id_inflow, 1, MPI_INTEGER, MPI_SUM, &
    932938                        comm1dx, ierr )
    … …  
    935941!-- Broadcast the id of the recycling plane
    936942!-- WARNING: needs to be adjusted in case of inflows other than from left side!
    937     IF ( ( recycling_width / dx ) >= nxl  .AND.  ( recycling_width / dx ) <= nxr ) &
    938     THEN
     943    IF ( ( recycling_width / dx ) >= nxl  .AND. &
     944         ( recycling_width / dx ) <= nxr )  THEN
    939945       id_recycling_l = myidx
    940946    ELSE
    941947       id_recycling_l = 0
    942948    ENDIF
     949    IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    943950    CALL MPI_ALLREDUCE( id_recycling_l, id_recycling, 1, MPI_INTEGER, MPI_SUM, &
    944951                        comm1dx, ierr )

  • palm/trunk/SOURCE/init_slope.f90

    r484 r622  
    44! Current revisions:
    55! -----------------
    6 !
     6! optional barriers included in order to speed up collective operations
    77!
    88! Former revisions:
    … …  
    100100             ENDDO
    101101          ENDDO
    102       ENDDO
     102       ENDDO
    103103
    104104#if defined( __parallel )
    105       CALL MPI_ALLREDUCE( pt_init_local, pt_init, nzt+2-nzb, MPI_REAL, &
    106                            MPI_SUM, comm2d, ierr )
     105       IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
     106       CALL MPI_ALLREDUCE( pt_init_local, pt_init, nzt+2-nzb, MPI_REAL, &
     107                            MPI_SUM, comm2d, ierr )
    107108#else
    108       pt_init = pt_init_local
     109       pt_init = pt_init_local
    109110#endif
    110111
    111       pt_init = pt_init / ngp_2dh(0)
    112       DEALLOCATE( pt_init_local )
     112       pt_init = pt_init / ngp_2dh(0)
     113       DEALLOCATE( pt_init_local )
    113114
    114    ENDIF
     115    ENDIF
    115116
    116117 END SUBROUTINE init_slope

  • palm/trunk/SOURCE/modules.f90

    r601 r622  
    55! Current revisions:
    66! -----------------
    7 !
     7! +collective_wait in pegrid
    88!
    99! Former revisions:
    … …  
    11631163    INTEGER, DIMENSION(:), ALLOCATABLE ::  ngp_yz, type_xz
    11641164
    1165     LOGICAL ::  reorder = .TRUE.
     1165    LOGICAL ::  collective_wait = .FALSE., reorder = .TRUE.
    11661166    LOGICAL, DIMENSION(2) ::  cyclic = (/ .TRUE. , .TRUE. /), &
    11671167                              remain_dims

  • palm/trunk/SOURCE/parin.f90

    r601 r622  
    44! Current revisions:
    55! -----------------
    6 !
     6! +collective_wait in inipar
    77!
    88! Former revisions:
    … …  
    119119             canyon_width_x, canyon_width_y, canyon_wall_left, &
    120120             canyon_wall_south, cfl_factor, cloud_droplets, cloud_physics, &
    121              conserve_volume_flow, conserve_volume_flow_mode, &
     121             collective_wait, conserve_volume_flow, conserve_volume_flow_mode, &
    122122             coupling_start_time, cthf, cut_spline_overshoot, &
    123123             cycle_mg, damp_level_1d, dissipation_1d, dp_external, dp_level_b, &

  • palm/trunk/SOURCE/poisfft.f90

    r484 r622  
    44! Current revisions:
    55! -----------------
    6 !
     6! optional barriers included in order to speed up collective operations
    77!
    88! Former revisions:
    … …  
    718718!--    Transpose array
    719719       CALL cpu_log( log_point_s(32), 'mpi_alltoall', 'start' )
     720       IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    720721       CALL MPI_ALLTOALL( work(nxl,1,0),      sendrecvcount_xy, MPI_REAL, &
    721722                          f_out(1,1,nys_x,1), sendrecvcount_xy, MPI_REAL, &
    … …  
    756757!--    Transpose array
    757758       CALL cpu_log( log_point_s(32), 'mpi_alltoall', 'start' )
     759       IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    758760       CALL MPI_ALLTOALL( f_in(1,1,nys_x,1), sendrecvcount_xy, MPI_REAL, &
    759761                          work(nxl,1,0),     sendrecvcount_xy, MPI_REAL, &
    … …  
    10731075!--    Transpose array
    10741076       CALL cpu_log( log_point_s(32), 'mpi_alltoall', 'start' )
     1077       IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    10751078       CALL MPI_ALLTOALL( work(nys,1,0),      sendrecvcount_xy, MPI_REAL, &
    10761079                          f_out(1,1,nxl_y,1), sendrecvcount_xy, MPI_REAL, &
    … …  
    11071110!--    Transpose array
    11081111       CALL cpu_log( log_point_s(32), 'mpi_alltoall', 'start' )
     1112       IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    11091113       CALL MPI_ALLTOALL( f_in(1,1,nxl_y,1), sendrecvcount_xy, MPI_REAL, &
    11101114                          work(nys,1,0),     sendrecvcount_xy, MPI_REAL, &

  • palm/trunk/SOURCE/poismg.f90

    r392 r622  
    88! Current revisions:
    99! -----------------
    10 !
     10! optional barriers included in order to speed up collective operations
    1111!
    1212! Former revisions:
    … …  
    106106          maxerror = SUM( r(nzb+1:nzt,nys:nyn,nxl:nxr)**2 )
    107107#if defined( __parallel )
     108          IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    108109          CALL MPI_ALLREDUCE( maxerror, residual_norm, 1, MPI_REAL, MPI_SUM, &
    109110                              comm2d, ierr)

  • palm/trunk/SOURCE/pres.f90

    r484 r622  
    44! Current revisions:
    55! -----------------
    6 !
     6! optional barriers included in order to speed up collective operations
    77!
    88! Former revisions:
    … …  
    105105
    106106#if defined( __parallel )   
     107       IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    107108       CALL MPI_ALLREDUCE( volume_flow_l(1), volume_flow(1), 1, MPI_REAL, &
    108109                           MPI_SUM, comm1dy, ierr )   
    … …  
    143144
    144145#if defined( __parallel )   
     146       IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    145147       CALL MPI_ALLREDUCE( volume_flow_l(2), volume_flow(2), 1, MPI_REAL, &
    146148                           MPI_SUM, comm1dx, ierr )   
    … …  
    172174          ENDDO
    173175#if defined( __parallel )   
     176          IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    174177          CALL MPI_ALLREDUCE( w_l_l(1), w_l(1), nzt, MPI_REAL, MPI_SUM, comm2d, &
    175178                              ierr )
    … …  
    537540
    538541#if defined( __parallel )   
     542       IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    539543       CALL MPI_ALLREDUCE( volume_flow_l(1), volume_flow(1), 2, MPI_REAL, &
    540544                           MPI_SUM, comm2d, ierr ) 

  • palm/trunk/SOURCE/read_var_list.f90

    r601 r622  
    33!------------------------------------------------------------------------------!
    44! Current revisions:
    5 ! -----------------_
    6 !
     5! ------------------
     6! +collective_wait
    77!
    88! Former revisions:
    … …  
    275275          CASE ( 'cloud_physics' )
    276276             READ ( 13 )  cloud_physics
     277          CASE ( 'collective_wait' )
     278             READ ( 13 )  collective_wait
    277279          CASE ( 'conserve_volume_flow' )
    278280             READ ( 13 )  conserve_volume_flow

  • palm/trunk/SOURCE/set_particle_attributes.f90

    r484 r622  
    44! Current revisions:
    55! -----------------
    6 !
     6! optional barriers included in order to speed up collective operations
    77!
    88! Former revisions:
    … …  
    140140#if defined( __parallel )
    141141
     142       IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    142143       CALL MPI_ALLREDUCE( sums_l(nzb,4,0), sums(nzb,4), nzt+2-nzb, &
    143144                           MPI_REAL, MPI_SUM, comm2d, ierr )

  • palm/trunk/SOURCE/timestep.f90

    r392 r622  
    44! Current revisions:
    55! -----------------
    6 !
     6! optional barriers included in order to speed up collective operations
    77!
    88! Former revisions:
    … …  
    118118          uv_gtrans_l = uv_gtrans_l / REAL( (nxr-nxl+1)*(nyn-nys+1)*(nzt-nzb) )
    119119#if defined( __parallel )
     120          IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    120121          CALL MPI_ALLREDUCE( uv_gtrans_l, uv_gtrans, 2, MPI_REAL, MPI_SUM, &
    121122                              comm2d, ierr )
    … …  
    164165!$OMP END PARALLEL
    165166#if defined( __parallel )
     167       IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    166168       CALL MPI_ALLREDUCE( dt_diff_l, dt_diff, 1, MPI_REAL, MPI_MIN, comm2d, &
    167169                           ierr )
    … …  
    252254!--       Determine the global minumum
    253255#if defined( __parallel )
     256          IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    254257          CALL MPI_ALLREDUCE( dt_plant_canopy_l, dt_plant_canopy, 1, MPI_REAL,  &
    255258                              MPI_MIN, comm2d, ierr )

  • palm/trunk/SOURCE/transpose.f90

    r484 r622  
    44! Current revisions:
    55! -----------------
    6 !
     6! optional barriers included in order to speed up collective operations
    77!
    88! Former revisions:
    … …  
    6969!-- Transpose array
    7070    CALL cpu_log( log_point_s(32), 'mpi_alltoall', 'start' )
     71    IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    7172    CALL MPI_ALLTOALL( f_inv(nys_x,nzb_x,0), sendrecvcount_xy, MPI_REAL, &
    7273                       work(1),              sendrecvcount_xy, MPI_REAL, &
    … …  
    143144!--    Transpose array
    144145       CALL cpu_log( log_point_s(32), 'mpi_alltoall', 'start' )
     146       IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    145147       CALL MPI_ALLTOALL( work(1),          sendrecvcount_zx, MPI_REAL, &
    146148                          f_inv(nys,nxl,1), sendrecvcount_zx, MPI_REAL, &
    … …  
    229231!-- Transpose array
    230232    CALL cpu_log( log_point_s(32), 'mpi_alltoall', 'start' )
     233    IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    231234    CALL MPI_ALLTOALL( work(1),              sendrecvcount_xy, MPI_REAL, &
    232235                       f_inv(nys_x,nzb_x,0), sendrecvcount_xy, MPI_REAL, &
    … …  
    291294!-- Transpose array
    292295    CALL cpu_log( log_point_s(32), 'mpi_alltoall', 'start' )
     296    IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    293297    CALL MPI_ALLTOALL( f_inv(nxl,1,nys), sendrecvcount_xy, MPI_REAL, &
    294298                       work(1),          sendrecvcount_xy, MPI_REAL, &
    … …  
    373377!-- Transpose array
    374378    CALL cpu_log( log_point_s(32), 'mpi_alltoall', 'start' )
     379    IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    375380    CALL MPI_ALLTOALL( f_inv(nxl_y,nzb_y,0), sendrecvcount_yz, MPI_REAL, &
    376381                       work(1),              sendrecvcount_yz, MPI_REAL, &
    … …  
    454459!-- Transpose array
    455460    CALL cpu_log( log_point_s(32), 'mpi_alltoall', 'start' )
     461    IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    456462    CALL MPI_ALLTOALL( f_inv(nys,nxl,1), sendrecvcount_zx, MPI_REAL, &
    457463                       work(1),          sendrecvcount_zx, MPI_REAL, &
    … …  
    528534!--    Transpose array
    529535       CALL cpu_log( log_point_s(32), 'mpi_alltoall', 'start' )
     536       IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    530537       CALL MPI_ALLTOALL( work(1),              sendrecvcount_yz, MPI_REAL, &
    531538                          f_inv(nxl_y,nzb_y,0), sendrecvcount_yz, MPI_REAL, &
    … …  
    627634!-- Transpose array
    628635    CALL cpu_log( log_point_s(32), 'mpi_alltoall', 'start' )
     636    IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    629637    CALL MPI_ALLTOALL( f_inv(nys,nxl,1), sendrecvcount_zyd, MPI_REAL, &
    630638                       work(1),          sendrecvcount_zyd, MPI_REAL, &

  • palm/trunk/SOURCE/user_statistics.f90

    r556 r622  
    44! Current revisions:
    55! -----------------
    6 !
     6! optional barriers included in order to speed up collective operations
    77!
    88! Former revisions:
    … …  
    9292!--           assign ts_value(dots_num_palm+1:,sr) = ts_value_l directly.
    9393!#if defined( __parallel )
     94!       IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    9495!       CALL MPI_ALLREDUCE( ts_value_l(dots_num_palm+1),                       &
    9596!                           ts_value(dots_num_palm+1,sr),                      &

  • palm/trunk/SOURCE/write_compressed.f90

    r484 r622  
    55! Current revisions:
    66! -----------------
    7 !
     7! optional barriers included in order to speed up collective operations
    88!
    99! Former revisions:
    … …  
    9090
    9191#if defined( __parallel )
     92    IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    9293    CALL MPI_ALLREDUCE( ifieldmax_l, ifieldmax, 1, MPI_INTEGER, MPI_MAX, &
    9394                        comm2d, ierr )
     95    IF ( collective_wait )  CALL MPI_BARRIER( comm2d, ierr )
    9496    CALL MPI_ALLREDUCE( ifieldmin_l, ifieldmin, 1, MPI_INTEGER, MPI_MIN, &
    9597                        comm2d, ierr )

  • palm/trunk/SOURCE/write_var_list.f90

    r601 r622  
    44! Current revisions:
    55! -----------------
    6 !
     6! +collective_wait
    77!
    88! Former revisions:
    … …  
    200200    WRITE ( 14 )  'cloud_physics                 '
    201201    WRITE ( 14 )  cloud_physics
     202    WRITE ( 14 )  'collective_wait               '
     203    WRITE ( 14 )  collective_wait
    202204    WRITE ( 14 )  'conserve_volume_flow          '
    203205    WRITE ( 14 )  conserve_volume_flow
Note: See TracChangeset for help on using the changeset viewer.
                                                                                                                                                                                                                                                                                                                                                                               
  | Impressum | ©Leibniz Universität Hannover |