source: palm/trunk/SCRIPTS/palmrun @ 4842

Last change on this file since 4842 was 4536, checked in by raasch, 5 years ago

messages and debug output converted to PALM routines (restart_data_mpi_io_mod), binary version number set to 5.0, heeader output for restart data format added, restart data filesize and I/O transfer speed added in cpu_measures, handling of single restart files (created with MPI-I/O) added to palmrun, bugfix: preprocessor directive adjusted (virtual_measurement_mod), location message format changed

  • Property svn:executable set to *
  • Property svn:keywords set to Id Rev
File size: 105.7 KB
Line 
1#!/bin/bash
2
3# palmrun - script for running PALM jobs
4
5#--------------------------------------------------------------------------------#
6# This file is part of the PALM model system.
7#
8# PALM is free software: you can redistribute it and/or modify it under the terms
9# of the GNU General Public License as published by the Free Software Foundation,
10# either version 3 of the License, or (at your option) any later version.
11#
12# PALM is distributed in the hope that it will be useful, but WITHOUT ANY
13# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
14# A PARTICULAR PURPOSE.  See the GNU General Public License for more details.
15#
16# You should have received a copy of the GNU General Public License along with
17# PALM. If not, see <http://www.gnu.org/licenses/>.
18#
19# Copyright 2017-2020  Leibniz Universitaet Hannover
20#--------------------------------------------------------------------------------#
21#
22# Current revisions:
23# ------------------
24#
25#
26# Former revisions:
27# -----------------
28# $Id: palmrun 4536 2020-05-17 17:24:13Z raasch $
29# handling of single restart files (created with MPI-I/O) added
30#
31# 4481 2020-03-31 18:55:54Z maronga
32# add cycle number to initial files too (000),
33# remove compatibility for cycle numbers which are less than three digits wide
34#
35# 4425 2020-02-26 15:35:00Z raasch
36# hard coded path to the fileconnection file replaced by a path that is based on
37# the source path which has been given in the configuration file
38#
39# 4303 2019-11-25 08:55:18Z raasch
40# Abort if temporary working catalog cannot be created
41#
42# 3982 2019-05-16 11:07:31Z raasch
43# bugfix: -b is not added in automatically generated batch jobs for remote hosts
44#
45# 3917 2019-04-18 11:46:37Z raasch
46# in case of batch jobs on remote machines, copying files from SOURCES_FOR_RUN folder has
47# been rearranged to avoid that input files with original names (e.g. ..._p3d) are copied
48# into the temporary working catalog
49#
50# 3860 2019-04-04 06:23:27Z raasch
51# check added, that it is not allowed to set option -b if the configuration file has been setup
52# for running batch jobs on remote hosts,
53# local time added to terminal output before and after saving output files
54#
55# 3725 2019-02-07 10:11:02Z raasch
56# error messages for failed restarts extended
57#
58# 3682 2019-01-18 17:01:54Z knoop
59# ssh-call for submitting batch jobs on remote systems modified again to avoid output
60# of login messages on specific systems
61#
62# 3574 2018-11-28 13:14:39Z raasch
63# syntax-error in last commit fixed
64#
65# 3573 2018-11-28 13:03:44Z raasch
66# bugfixes: don't check the .palm.iofiles in restart runs + wrong cd-command
67# removed,
68# r3570 reverted
69#
70# 3570 2018-11-27 17:44:21Z kanani
71# Rename job-id by run_id
72#
73# 3555 2018-11-22 13:50:37Z raasch
74# sed-command to remove repeated space-characters replaced by tr-command, to
75# guarantee mac compatible
76#
77# 3549 2018-11-21 15:44:44Z raasch
78# ssh-call for submitting batch jobs on remote systems modified to avoid output
79# of login messages on specific systems
80#
81# 3534 2018-11-19 15:35:16Z raasch
82# batch_scp for sending back the job protocol file is called via login-node if
83# a login-node has been set in the config-file
84# ssh-calls rearranged to avoid output of system/user-profile messages
85#
86# 3505 2018-11-08 09:14:30Z raasch
87# job folders are created in case that they do not exist
88#
89# 3455 2018-10-30 14:12:31Z raasch
90# options -h and -d renamed -c and -r, respectively
91#
92# 3402 2018-10-23 11:49:24Z knoop
93# job_id is set for compatibility reasons with older versions of config files
94#
95# 3313 2018-10-06 15:22:48Z knoop
96# more detailed error message for outdated fileconnection file
97#
98# 3236 2018-09-10 09:07:06Z raasch
99# warning issued in case of file connection file version mismatch,
100# host_configuration renamed configuration_identifier,
101# "host identifier" in header output renamed "config. identifier",
102# jobname renamed run_identifier,
103# run_id renamed run_id_number
104# job_id renamed run_id
105# option -r renamed -i
106#
107# 3151 2018-07-19 08:45:38Z raasch
108# removed additional listing of files to be compiled, print PALM code revision
109# in header
110#
111# 3132 2018-07-16 11:44:58Z raasch
112# create consistent error messages in case of failed restarts,
113# check existence of remote job catalog
114#
115# 3043 2018-05-25 10:47:54Z raasch
116# check for setting of fast_io_catalog
117#
118# 3033 2018-05-23 15:26:19Z raasch
119# -T option not required for interactive runs (tasks per node =
120# total number of cores is assumed as default, if option is not given),
121# reformatting of a few messages,
122# bugfix: destination output directory is created if attribute "pe" is set in
123# .palm.iofiles
124#
125# 2718 2018-01-02 08:49:38Z maronga
126# bugfix of r2990
127#
128# 2718 2018-01-02 08:49:38Z maronga
129# bugfix: temporary SOURCES_FOR_RUN_CATALOG now created in base_directory and
130#         not in the working directory from where palmrun is called
131#
132# 2718 2018-01-02 08:49:38Z maronga
133# svfout in the activation string will be replaced now by svfin in case of
134# automatic restarts
135#
136# 2718 2018-01-02 08:49:38Z maronga
137# Introduction of new ENVIRONMENT variables read_svf and write_svf to consider
138# the possible output or input of sky view factors
139#
140# 2718 2018-01-02 08:49:38Z maronga
141# "fname" renamed to "jobname"
142#
143# 2718 2018-01-02 08:49:38Z maronga
144# parameters are allowed in output commands (OC:)
145#
146# 2718 2018-01-02 08:49:38Z maronga
147# bugfix: exit if no queue has been given neither with option -q nor in the
148# configuration file
149#
150# 2718 2018-01-02 08:49:38Z maronga
151# file connection file from trunk/SCRIPTS is taken by default
152#
153# 2718 2018-01-02 08:49:38Z maronga
154# Corrected "Former revisions" section
155#
156# 2696 2017-12-14 17:12:51Z kanani
157# Change in file header (GPL part)
158#
159# 2693 2017-12-13 12:33:49Z raasch
160# replacement of variables given by {{<variable>}} in configuration file is now
161# done in a more general way, variable names are written in small letters
162#
163# 2670 2017-12-06 16:32:19Z raasch
164# test printouts removed
165#
166# 2669 2017-12-06 16:03:27Z raasch
167# file attributes in .palm.iofiles restructured, "loc" attribute completely
168# removed,
169# wildcard (*) allowed in .palm.iofiles as file activation string for output
170# files,
171# informative messages in case of missing optional input files shortened
172# bugfix: variable cycle explicitly interpreted with 10 as the number base
173#
174# 2638 2017-11-23 12:44:23Z raasch
175# use of wildcards in file connection statements enabled
176#
177# 2605 2017-11-09 15:31:46Z raasch
178# in case of remote jobs, input files with "job" or "jobopt" (new) attribute
179# will not be sent with the job file any more, but copied into the
180# SOURCES_FOR_RUN... folder on the remote host, before the job is submitted
181#
182# 2600 2017-11-01 14:11:20Z raasch
183# cycle numbers are made three digits wide
184#
185# 2566 2017-10-20 08:50:47Z raasch
186# execute command for combine_plot_fields added
187# "TEMPDIR" renamed "tempdir"
188# temporary working directory for local batch jobs is created immediately within
189# the user's palmrun call, due to a requirement of the "grid engine" batch
190# system, where the working directory is given with batch directive -wd and must
191# already exist when the job is submitted,
192# informative messages in non-trace mode reduced and partly reformatted
193#
194# 2551 2017-10-18 07:25:11Z raasch
195# TEMPDIR added as replacement string to be used in batch directives
196#
197# 2512 2017-10-04 08:26:59Z raasch
198# bugfix for determining cycle numbers of NetCDF input files
199#
200# 2506 2017-09-29 08:30:37Z raasch
201# option -V added to check for an existing SOURCES_FOR_RUN_... folder
202# host configuration added to SOURCES_FOR_RUN_... folder name
203# host_identifier renamed host_configuration
204# option -W added to allow for job dependencies
205#
206# 2501 2017-09-26 11:41:55Z raasch
207# default value for number of cores (option -X) set to 1
208# bugfix for mechanism which overwrites configuration file settings with values
209# provided by palmrun options
210#
211# 2499 2017-09-22 16:47:58Z kanani
212# option -h named configuration identifier
213#
214# 2480 2017-09-19 06:24:14Z maronga
215# bugfix for last revision
216#
217# 2479 2017-09-19 06:12:16Z raasch
218# option -A (project account number) added
219#
220# 2422 2017-09-08 08:25:41Z raasch
221# initial revision
222#
223#--------------------------------------------------------------------------------#
224# palmrun - script for running PALM jobs on local and remote hosts
225#--------------------------------------------------------------------------------#
226
227
228 
229    # DECLARATION OF VARIABLES AND THEIR DEFAULT VALUES
230
231 set +o allexport    # SUPPRESS EXPORT OF ALL VARIABLES, SINCE IN THE PAST THIS
232                     # LEAD TO PROBLEMS IN ROUTINES CALLED BY PALMRUN
233                     # (TOO MANY ARGUMENTS - PROBLEM)
234 set +o noclobber    # EXISTING FILES ARE ALLOWED TO BE OVERWRITTEN
235
236 typeset -i  ibd=0 ibdt=0 iec=0 iic=0 iin=0 ioc=0 iout=0 nr_of_input_files=0
237 typeset -i  nr_of_output_files=0
238 typeset -x -i  memory=0   # HAS TO BE EXPORTED HERE, OTHERWISE AN UNKNOWN SIDE
239                           # SIDE EFFECT MAY CAUSE DATA LOSS WHEN GETOPTS IS READING THE
240                           # SCRIPT-OPTION ARGUMENTS
241 typeset -i  cores cputime cpu_hours cpu_minutes cpu_seconds i ii iia iii iio
242 typeset -i  icycle inode ival maxcycle mpi_tasks
243 typeset -i  nodes remaining_cores run_number tasks_per_node threads_per_task
244
245 activation_string_list=""
246 AddFilenames=""
247 combine_plot_fields=true
248 compiler_name=""
249 compiler_name_ser=""
250 compiler_options=""
251 configuration_identifier="default"
252 cores=1
253 cores_atmos=0
254 cores_ocean=0
255 coupled_dist=""
256 cpp_options=""
257 cpumax=0
258 create_batch_job=false
259 create_jobfile_only=false
260 create_remote_batch_job=false
261 dashes="  ----------------------------------------------------------------------------"
262 defaultqueue=""
263 delete_temporary_catalog=true
264 do_compile=true
265 do_trace=false
266 executable=""
267 execute_command=""
268 execution_error=false
269 fileconnection_file=""
270 global_revision=""
271 hostfile=""
272 hp=""
273 keep_data_from_previous_run=false
274 link_local_input=false
275 link_local_output=false
276 linker_options=""
277 local_jobcatalog=""
278 locat=normal
279 makefile=""
280 max_par_io_str=""
281 prc=$0
282 while [[ $(echo $prc | grep -c "/") != 0 ]]
283 do
284    prc=`echo $prc | cut -f2- -d"/"`
285 done
286 module_calls=""
287 palmrun_memory=""
288 palmrun_script_name=$prc
289 openmp=false
290 previous_job=""
291 project_account=""
292 queue=none
293 read_svf=""
294 restart_run=false
295 return_address=""
296 remote_jobcatalog=""
297 remote_username=""
298 running_in_batch_mode=false
299 running_in_test_mode=false
300 run_coupled_model=false
301 run_id=""
302 run_id_number=""
303 run_identifier=test
304 silent=false
305 source_list=""
306 source_path=""
307 tasks_per_node=0
308 threads_per_task=1
309 transfer_problems=false
310 user_source_path=""
311 use_existing_sources_folder=""
312 use_openmp=false
313 version="palmrun  1.0 Rev$Rev: 4536 $"
314 working_directory=`pwd`
315 write_binary=""
316 write_svf=""
317
318
319    # ERROR HANDLING IN CASE OF EXIT
320 trap 'if [[ $locat != normal  &&  $locat != control_c ]]
321       then
322
323              # CARRY OUT ERROR-COMMANDS GIVEN IN THE CONFIGURATION FILE (EC:)
324          (( i = 0 ))
325          while (( i < iec ))
326          do
327             (( i = i + 1 ))
328             printf "\n  *** Execution of ERROR-command:\n"
329             printf "  >>> ${err_command[$i]}\n"
330             eval  ${err_command[$i]}
331          done
332          [[ $delete_temporary_catalog = true ]]  &&  (cd; rm -rf $tempdir)
333          printf "\n\n+++ palmrun crashed \n\n"
334          exit 1
335       elif [[ $locat != normal ]]
336       then
337          [[ $delete_temporary_catalog = true ]]  &&  (cd; rm -rf $tempdir)
338          printf "\n+++ palmrun killed by \"^C\" \n\n"
339          exit 2
340       else
341          printf "\n --> palmrun finished\n\n"
342          exit 0
343       fi' exit
344
345
346    # ACTIONS IN CASE OF TERMINAL-BREAK (CONTROL-C):
347 trap 'locat=control_c
348       exit 1
349      ' 2
350
351
352    # READ SHELLSCRIPT-OPTIONS AND REBUILD THE PALMRUN-COMMAND STRING (prc),
353    # WHICH WILL BE USED TO START RESTART-JOBS
354 while  getopts  :a:A:bBc:CFG:i:jkm:M:O:q:r:R:s:t:T:u:U:vVw:W:xX:yY:zZ option
355 do
356   case  $option  in
357       (a)   activation_string_list=$OPTARG; prc="$prc -a'$OPTARG'";;
358       (A)   project_account=$OPTARG; prc="$prc -A'$OPTARG'";;
359       (b)   create_batch_job=true; prc="$prc -b";;
360       (B)   delete_temporary_catalog=false; prc="$prc -B";;
361       (c)   configuration_identifier=$OPTARG; prc="$prc -c$OPTARG";;
362       (C)   restart_run=true; prc="$prc -C";;
363       (F)   create_jobfile_only=true;;
364       (G)   global_revision=$OPTARG; prc="$prc -G'$OPTARG'";;
365       (i)   run_id_number=$OPTARG;;
366       (j)   running_in_batch_mode=true;;
367       (k)   keep_data_from_previous_run=true; prc="$prc -k";;
368       (m)   palmrun_memory=$OPTARG; prc="$prc -m$OPTARG";;
369       (M)   makefile=$OPTARG; prc="$prc -M$OPTARG";;
370       (O)   use_openmp=true; threads_per_task=$OPTARG; prc="$prc -O$OPTARG";;
371       (q)   queue=$OPTARG; prc="$prc -q$OPTARG";;
372       (r)   run_identifier=$OPTARG; prc="$prc -r$OPTARG";;
373       (R)   return_address=$OPTARG;;
374       (s)   source_list=$OPTARG;;
375       (t)   palmrun_cpumax=$OPTARG; prc="$prc -t$OPTARG";;
376       (T)   palmrun_tasks_per_node=$OPTARG; prc="$prc -T$OPTARG";;
377       (u)   remote_username=$OPTARG; prc="$prc -u$OPTARG";;
378       (U)   return_username=$OPTARG; prc="$prc -U$OPTARG";;
379       (v)   silent=true; prc="$prc -v";;
380       (V)   use_existing_sources_folder="-V";;
381       (w)   max_par_io_str=$OPTARG; prc="$prc -w$OPTARG";;
382       (W)   previous_job=$OPTARG;;
383       (x)   do_trace=true;set -x; prc="$prc -x";;
384       (X)   palmrun_cores=$OPTARG; prc="$prc -X$OPTARG";;
385       (y)   ocean_file_appendix=true; prc="$prc -y";;
386       (Y)   run_coupled_model=true; coupled_dist=$OPTARG; prc="$prc -Y'$OPTARG'";;
387       (z)   running_in_test_mode=true;;
388       (Z)   combine_plot_fields=false; prc="$prc -Z";;
389       (\?)  printf "\n  +++ unknown option $OPTARG \n"
390             printf "\n  --> type \"$0 ?\" for available options \n"
391             locat=parameter;exit;;
392   esac
393 done
394
395
396    # SKIP GIVEN OPTIONS TO READ POSITIONAL PARAMETER, IF GIVEN
397    # CURRENTLY ONLY PARAMETER ? (TO OUTPUT A SHORT COMMAND INFO) IS ALLOWED
398 (( to_shift = $OPTIND - 1 ))
399 shift $to_shift
400
401    # PRINT SHORT DESCRIPTION OF PALMRUN OPTIONS
402 if [[ "$1" = "?" ]]
403 then
404   (printf "\n  *** Description of available palmrun options:\n"
405    printf "\n      Option  Description                              Default-Value"
406    printf "\n        -a    activation string list                   \"\" "
407    printf "\n        -A    project account number                   ---"
408    printf "\n        -b    batch-job on local machine               ---"
409    printf "\n        -B    do not delete temporary directory at end ---"
410    printf "\n        -c    configuration identifier                 \"default\" "
411    printf "\n        -F    create batch job file only               ---"
412    printf "\n        -k    keep data from previous run"
413    printf "\n        -m    memory demand in MB (batch-jobs)         0 MB"
414    printf "\n        -M    Makefile name                            Makefile"
415    printf "\n        -O    threads per openMP task                  ---"
416    printf "\n        -q    queue                                    \"$queue\" "
417    printf "\n        -r    run identifier                           test"
418    printf "\n        -s    filenames of routines to be compiled     \"\" "
419    printf "\n              must end with .f, .f90, .F, or .c !"
420    printf "\n              use \"..\" for more than one file and wildcards"
421    printf "\n              -s LM compiles all locally modified files"
422    printf "\n        -t    allowed cpu-time in seconds (batch)      0"
423    printf "\n        -T    tasks per node                           ---"
424    printf "\n        -u    username on remote machine               \"\" "
425    printf "\n        -v    no prompt for confirmation               ---"
426    printf "\n        -V    check if SOURCES_FOR_RUN_... exists      ---"
427    printf "\n        -w    maximum parallel io streams              as given by -X"
428    printf "\n        -W    name of job to wait for                  ---"
429    printf "\n        -x    tracing of palmrun for debug purposes    ---"
430    printf "\n        -X    # of processors (on parallel machines)   1"
431    printf "\n        -y    add appendix \"_O\" to all local output"
432    printf "\n              files (ocean precursor runs followed by"
433    printf "\n              coupled atmosphere-ocean runs)           ---"
434    printf "\n        -Y    run coupled model, \"#1 #2\" with" 
435    printf "\n              #1 atmosphere and #2 ocean processors    \"#/2 #/2\" depending on -X"
436    printf "\n        -Z    skip combine_plot_fields at the end of      "
437    printf "\n              the simulation                           ---"
438    printf "\n "
439    printf "\n      Possible values of positional parameter <modus>:"
440    printf "\n        \"?\"       -  this outline \n\n") | more
441    exit
442 elif [[ "$1" != "" ]]
443 then
444    printf "\n  +++ positional parameter $1 unknown \n"
445    locat=parameter; exit
446 fi
447
448
449
450    # SHORT STARTING MESSAGE
451 printf "\n*** $version "
452 printf "\n    will be executed.     Please wait ..."
453
454
455    # BUILD THE CONFIGURATION-FILE NAME AND THE SOURCES_FOR_RUN-FOLDER NAME
456 config_file=.palm.config.$configuration_identifier
457 sources_for_run_catalog=SOURCES_FOR_RUN_${configuration_identifier}_$run_identifier
458
459
460    # CHECK, IF CONFIGURATION FILE EXISTS
461 if [[ ! -f $config_file ]]
462 then
463    printf "\n\n  +++ configuration file: "
464    printf "\n           $config_file"
465    printf "\n      does not exist"
466    locat=connect; exit 
467 fi
468
469
470    # CHECK, IF THE ACTIVATION_STRING_LIST HAS BEEN GIVEN
471 if [[ "$activation_string_list" = "" ]]
472 then
473    printf "\n\n  +++ no activation string list given: "
474    printf "\n        please set palmrun option \"-a\" "
475    locat=palmrun_option; exit 
476 fi
477
478
479    # SET VARIABLE TO ACTIVATE PALM BINARY OUTPUT FOR RESTARTS
480 if [[ $(echo $activation_string_list | grep -c "restart") != 0 ]]
481 then
482    write_binary=true
483 else
484    write_binary=false
485 fi
486
487    # SET VARIABLE TO ACTIVATE WRITING OF SVF DATA
488 if [[ $(echo $activation_string_list | grep -c "svfout") != 0 ]]
489 then
490    write_svf=true
491 else
492    write_svf=false
493 fi
494
495    # SET VARIABLE TO ACTIVATE READING OF SVF DATA
496 if [[ $(echo $activation_string_list | grep -c "svfin") != 0 ]]
497 then
498    read_svf=true
499 else
500    read_svf=false
501 fi
502
503
504    # READ AND EVALUATE THE CONFIGURATION-FILE
505 [[ $silent = false ]]  &&  printf "\n\n    Reading the configuration file... "
506
507    # READ VARIABLE SETTINGS FROM CONFIG FILE LINE BY LINE
508 while  read line
509 do
510
511       # FIRST REPLACE ENVIRONMENT-VARIABLES BY THEIR RESPECTIVE VALUES
512    eval  line=\"$line\"
513
514
515       # INTERPRET THE LINE
516    if [[ "$(echo $line)" = "" ]]
517    then
518
519          # EMPTY LINE, NO ACTION
520       continue
521
522    elif [[ "$(echo $line | cut -c1)"  =  "#" ]]
523    then
524
525          # LINE IS A COMMENT LINE
526       continue
527
528    elif [[ "$(echo $line | cut -c1)"  =  "%" ]]
529    then
530
531          # LINE DEFINES AN ENVIRONMENT-VARIABLE
532       var=`echo $line | cut -d" " -s -f1 | cut -c2-`
533       value=`echo $line | cut -d" " -s -f2-`
534
535
536          # VALUE FROM THE CONFIGURATION-FILE IS ASSIGNED TO THE
537          # ENVIRONMENT-VARIABLE, BUT ONLY IF NO VALUE HAS BEEN ALREADY
538          # ASSIGNED WITHIN THIS SCRIPT (E.G. BY SCRIPT-OPTIONS).
539          # NON-ASSIGNED VARIABLES HAVE VALUE "" OR 0 (IN CASE OF INTEGER).
540          # HENCE THE GENERAL RULE IS: SCRIPT-OPTION OVERWRITES THE
541          # CONFIGURATION-FILE.
542       if [[ "$(eval echo \$$var)" = ""  ||  "$(eval echo \$$var)" = "0" ]]
543       then
544          eval  export  $var="\$value"
545
546             # TERMINAL OUTPUT OF ENVIRONMENT-VARIABLES, IF TRACEBACK IS SWITCHED on
547          if [[ $do_trace = true ]]
548          then
549             printf "\n*** ENVIRONMENT-VARIABLE $var = $value"
550          fi
551       fi
552
553    elif [[ "$(echo $line | cut -c1-3)" = "BD:" ]]
554    then
555
556          # LINE DEFINES BATCH-DIRECTIVE
557       (( ibd = ibd + 1 ))
558       line=$(echo $line | cut -c4-)
559       batch_directive[$ibd]="$line"
560
561    elif [[ "$(echo $line | cut -c1-4)" = "BDT:" ]]
562    then
563
564          # LINE DEFINES BATCH-DIRECTIVE FOR SENDING BACK THE JOBFILE FROM A
565          # REMOTE TO A LOCAL HOST
566       (( ibdt = ibdt + 1 ))
567       line=$(echo $line | cut -c5-)
568       batch_directive_transfer[$ibdt]="$line"
569
570    elif [[ "$(echo $line | cut -c1-3)" = "EC:" ]]
571    then
572
573          # LINE DEFINES ERROR-COMMAND
574       (( iec = iec + 1 ))
575       line=$(echo $line | cut -c4-)
576       err_command[$iec]="$line"
577
578    elif [[ "$(echo $line | cut -c1-3)" = "IC:" ]]
579    then
580
581          # LINE DEFINES INPUT-COMMAND
582       (( iic = iic + 1 ))
583       line=$(echo $line | cut -c4-)
584       in_command[$iic]="$line"
585
586    elif [[ "$(echo $line | cut -c1-3)" = "OC:" ]]
587    then
588
589          # LINE DEFINES OUTPUT-COMMAND
590       (( ioc = ioc + 1 ))
591       line=$(echo $line | cut -c4-)
592       out_command[$ioc]="$line"
593
594    else
595
596          # SKIP ALL OTHER LINES
597       continue
598
599    fi
600
601 done < $config_file
602
603
604    # CHECK SETTING OF REQUIRED PARAMETERS
605 if [[ "$fast_io_catalog" = "" ]]
606 then
607    printf "\n\n  +++ no fast_io_catalog found in $config_file"
608    printf "\n      Please add line \"fast_io_catalog ...\" to that file."
609    locat=config_file; exit
610 fi
611 if [[ "$compiler_name" = "" ]]
612 then
613    printf "\n\n  +++ no compiler name found in $config_file"
614    printf "\n      Please add line \"compiler_name ...\" to that file."
615    locat=config_file; exit
616 fi
617 if [[ "$compiler_name_ser" = "" ]]
618 then
619    printf "\n\n  +++ no compiler name for non-paralle compilation found in $config_file"
620    printf "\n      Please add line \"compiler_name_ser ...\" to that file."
621    locat=config_file; exit
622 fi
623 if [[ "$compiler_options" = "" ]]
624 then
625    printf "\n\n  +++ no compiler options found in $config_file"
626    printf "\n      Please add line \"compiler_options ...\" to that file."
627    locat=config_file; exit
628 fi
629 if [[ "$execute_command" = "" ]]
630 then
631    printf "\n\n  +++ no execute command found in $config_file"
632    printf "\n      Please add line \"execute_command ...\" to that file."
633    locat=config_file; exit
634 fi
635
636 if [[ "$hostfile" != "" ]]
637 then
638    if [[ $hostfile != auto  &&  ! -f $hostfile ]]
639    then
640       printf "\n\n  +++ no hostfile \"$hostfile\" found"
641       printf "\n      Please check line \"hostfile ...\" in $config_file"
642       locat=config_file; exit   
643    fi
644 fi
645
646
647    # DETERMINE THE CALL STATUS
648 if [[ "$return_address" != "" ]]
649 then
650 
651       # I AM RUNNING ON A REMOTE HOST, WHICH ALSO MEANS THAT I AM RUNNING IN
652       # BATCH MODE AND ...
653    running_on_remote=true
654 
655 else
656
657       # I HAVE BEEN CALLED INTERACTIVELY ON THIS HOST
658     if [[ "$remote_ip" != "" ]]
659     then
660
661           # I HAVE TO CREATE A BATCH JOB TO RUN PALM ON THE REMOTE HOST
662        create_remote_batch_job=true
663
664           # CHECK THAT THE REMOTE JOBCATALOG HAS BEEN SET
665        if [[ "$remote_jobcatalog" = "" ]]
666        then
667           printf "\n\n  +++ no remote jobcatalog found in $config_file"
668           printf "\n      Please add line \"remote_jobcatalog ...\" to that file."
669           locat=config_file; exit
670        fi
671
672           # CHECK THAT OPTION -b (SUBMITTING BATCH JOB ON LOCAL HOST) HASN'T BEEN SET
673        if [[ $create_batch_job = true ]]
674        then
675           printf "\n\n  +++ option -b must not be set because configuration file has been setup"
676           printf "\n      for running jobs on a remote host"
677           locat=config_file; exit
678        fi
679
680     fi
681     running_on_remote=false
682 fi
683
684
685    # SET THE DEFAULT FILE CONNECTION FILE (WHICH IS IN THE REPOSITORY)
686 fileconnection_file=${source_path}/../SCRIPTS/.palm.iofiles
687
688    # CHECK, IF USER PROVIDES OWN FILE CONNECTION FILE
689 if [[ -f .palm.iofiles ]]
690 then
691
692       # CHECK VERSION MISMATCH
693       # RESTART RUNS DON'T REQUIRE A CHECK, BECAUSE THEY USE A COPY OF THAT
694       # FILE WHICH HAS ALREADY BEEN CHECKED WITHIN THE INITAL PALMRUN CALL
695    if [[ $running_in_batch_mode != true  &&  $restart_run != true ]]
696    then
697       if [[ $(head -n1 $fileconnection_file) != $(head -n1 .palm.iofiles) ]]
698       then
699          printf "\n\n  +++ WARNING: A file connection file has been found in your"
700          printf "\n               working directory, but its revision does not match"
701          printf "\n               the revision of the default (trunk) version."
702          printf "\n               You may need to update your connection file"
703          printf "\n               \"${working_directory}/.palm.iofiles\" !"
704       fi
705    fi
706
707    fileconnection_file=.palm.iofiles
708
709 fi
710
711
712    # CHECK, IF FILE CONNECTION FILE EXISTS
713 if [[ ! -f $fileconnection_file ]]
714 then
715    printf "\n\n  +++ file connection file: "
716    printf "\n           $fileconnection_file"
717    printf "\n      does not exist"
718    locat=connect; exit 
719 fi
720
721
722    # READ AND EVALUATE THE I/O-FILE LIST
723 [[ $silent = false ]]  &&  printf "\n    Reading the I/O files... "
724
725    # READ THE FILE CONNECTION FILE LINE BY LINE
726 while  read line
727 do
728
729       # REPLACE REPEATING SPACES BETWEEN THE COLUMNS BY A SINGLE SPACE
730       # HERE, TR IS USED INSTEAD OF SED TO GUARANTEE MAC COMPATIBILITY
731    line=`echo "$line" | tr -s " "`
732
733       # INTERPRET THE LINE
734    if [[ "$(echo $line)" = "" ]]
735    then
736          # EMPTY LINE, NO ACTION
737       continue
738
739    elif [[ "$(echo $line | cut -c1)"  =  "#" ]]
740    then
741
742          # LINE IS A COMMENT LINE
743       true
744
745    else
746
747          # LINE DEFINES FILE CONNECTION. READ THE FILE ATTRIBUTES.
748          # s2a: in/out - field
749          # s2b: action - field (optional)
750       s1=`echo "$line" | cut -d" " -f1`
751       s2=`echo "$line" | cut -d" " -s -f2`
752       if [[ $(echo $s2 | grep -c ":") = 0 ]]
753       then
754          s2a=$s2
755          s2b=""
756       else
757          s2a=`echo $s2 | cut -d":" -f1`
758          s2b=`echo $s2 | cut -d":" -f2`
759       fi
760       s3=`echo "$line" | cut -d" " -f3 | sed 's/*/wildcard /g'`
761       s4=`echo "$line" | cut -d" " -s -f4`
762       eval s4=\"$s4\"    # REPLACE ENVIRONMENT-VARIABLES IN PATH BY THEIR RESPECTIVE VALUES
763       s5=`echo "$line" | cut -d" " -s -f5`
764       s6=`echo "$line" | cut -d" " -s -f6`
765
766       
767          # STORE FILE CONNECTION, IF ACTIVATED BY ACTIVATION-STRING FROM
768          # INPUT- OR OUTPUT-LIST.
769          # VARIABLE S3 MAY CONTAIN A LIST OF ACTIVATION STRINGS (FIELD-SEPERATOR ":").
770          # IF EXECUTION IS SCHEDULED FOR A REMOTE-MACHINE AND THE FILE IS ONLY
771          # LOCALLY REQUIRED ON THAT MACHINE (I.E. s2b != tr), THE FILE CONNECTION
772          # IS NOT CHECKED AND STORED.
773       IFSALT="$IFS"; IFS="$IFS:"      # ADD ":" AS FIELD SEPARATOR
774       if [[ ( "$s2a" = in  || "$s2a" = inopt )  &&  ! ( $create_remote_batch_job = true  &&  "$s2b" != tr ) ]]
775       then
776          found=false
777          for  actual  in  $activation_string_list
778          do
779             for  formal  in  $s3
780             do
781                [[ $actual = $formal  ||  "$formal" = "-" ]]  &&  found=true
782             done
783          done
784          if [[ $found = true ]]
785          then
786             (( iin = iin + 1 ))
787             localin_pre[$iin]=$s1; actionin_pre[$iin]=$s2b;
788             pathin_pre[$iin]=$s4; endin_pre[$iin]=$s5; extin_pre[$iin]=$s6
789             if [[ "$s2a" = inopt ]]
790             then
791                optin_pre[$iin]=yes
792             else
793                optin_pre[$iin]=no
794             fi 
795
796                # FILES WITH JOB-ATTRIBUTE ARE STORED IN THE SOURCES_FOR_RUN
797                # FOLDER IF THE JOB IS RUNNING ON A REMOTE HOST
798             if [[ $running_on_remote = true  &&  "$s2b" = tr ]]
799             then
800                pathin_pre[$iin]=${fast_io_catalog}/${sources_for_run_catalog}
801             fi
802
803                # CHECK FOR MULTIPLE FILES, SET A RESPECTIVE FLAG AND REMOVE
804                # THE WILDCARD FROM THE ENDING
805             if [[ "${s5: -1}" = "*" ]]
806             then
807                if [[ "$s2b" = "di" ]]
808                then
809                   printf "\n    +++ wildcards (*) not allowed with \"di\" file attribute."
810                   printf "\n        see file \"$fileconnection_file\", line"
811                   printf "\n$line"
812                   locat=iofiles_file; exit
813                fi
814                multin[$iin]=true
815                string=${endin_pre[$iin]}
816                endin_pre[$iin]="${string%?}"
817             else
818                multin[$iin]=false
819             fi
820          fi
821       elif [[ "$s2a" = out  &&  ! ( $create_remote_batch_job = true ) ]]
822       then
823          found=false
824          for  actual  in  $activation_string_list
825          do
826             for  formal  in  $s3 
827             do
828                [[ $actual = $formal  ||  $formal = wildcard  ]]  &&  found=true
829             done
830          done
831          if [[ $found = true ]]
832          then
833             (( iout = iout + 1 ))
834             localout_pre[$iout]=$s1; actionout_pre[$iout]=$s2b
835             pathout_pre[$iout]=$s4; endout_pre[$iout]=$s5; extout_pre[$iout]=$s6
836
837                # CHECK IF WILDCARD IS USED AS ACTIVATION STRING
838                # IN SUCH CASES, NO WARNING WILL LATER BE OUTPUT IF LOCAL FILES DO NOT EXIST
839             if [[ $formal = wildcard  ]]
840             then
841                warnout_pre[$iout]=false
842             else
843                warnout_pre[$iout]=true
844             fi
845
846                # CHECK FOR MULTIPLE FILES, SET A RESPECTIVE FLAG AND REMOVE
847                # THE WILDCARD FROM THE LOCAL FILENAME
848             if [[ "${s1: -1}" = "*" ]]
849             then
850                if [[ "$s2b" = "di" ]]
851                then
852                   printf "\n    +++ wildcards (*) not allowed with \"di\" file attribute."
853                   printf "\n        see file \"$fileconnection_file\", line"
854                   printf "\n$line"
855                   locat=iofiles_file; exit
856                fi
857                multout[$iout]=true
858                string=${localout_pre[$iout]}
859                localout_pre[$iout]="${string%?}"
860             else
861                multout[$iout]=false
862             fi
863          fi
864       elif [[ "$s2a" != in  &&  "$s2a" != inopt  &&  "$s2a" != out ]]
865       then
866          printf "\n  +++ I/O-attribute in file $fileconnection_file has invalid"
867          printf "\n      value \"$s2\". Only \"in\", \"inopt\", and \"out\" are allowed!"
868          locat=connect; exit
869       fi
870       IFS="$IFSALT"
871    fi
872
873 done < $fileconnection_file
874
875
876
877    # VALUES OF PALMRUN-OPTIONS OVERWRITE THOSE FROM THE CONFIGURATION-FILE
878 [[ "$palmrun_memory" != "" ]]  &&  memory=$palmrun_memory
879 [[ "$palmrun_cpumax" != "" ]]  &&  cpumax=$palmrun_cpumax
880 [[ "$palmrun_cores"  != "" ]]  &&  cores=$palmrun_cores
881 [[ "$max_par_io_str" != "" ]]  &&  maximum_parallel_io_streams=$max_par_io_str
882 [[ "$palmrun_tasks_per_node" != "" ]]  &&  tasks_per_node=$palmrun_tasks_per_node
883
884
885
886      # EVALUATE MODEL COUPLING FEATURES (OPTION -Y)
887 if [[ $run_coupled_model = true ]] 
888 then
889
890    cores_atmos=`echo $coupled_dist | cut -d" " -s -f1`
891    cores_ocean=`echo $coupled_dist | cut -d" " -s -f2`
892
893    if (( $cores_ocean + $cores_atmos != $cores ))
894    then
895
896       printf "\n  +++ number of processors does not fit to specification by \"-Y\"."
897       printf "\n      PEs (total)     : $cores"
898       printf "\n      PEs (atmosphere): $cores_atmos"
899       printf "\n      PEs (ocean)     : $cores_ocean"
900       locat=coupling; exit
901
902    fi
903
904 fi
905
906
907    # IF I AM IN BATCH MODE, CHECK IF EXECUTABLE AND OTHER REQUIRED FILES
908    # HAVE BEEN GENERATED BY PALMBUILD AND STORED IN THE SOURCES_FOR_RUN_...
909    # FOLDER
910 if [[ $running_in_batch_mode = true ]]
911 then
912 
913    if [[ ! -d ${fast_io_catalog}/${sources_for_run_catalog} ]]
914    then
915       printf "\n  +++ directory ${fast_io_catalog}/${sources_for_run_catalog} is missing"
916       printf "\n      Please check the output of the palmrun-call"
917       printf "\n      that you did on your local host."
918       locat=SOURCES_FOR_RUN; exit
919    fi
920
921 else
922
923       # CREATE THE SOURCES_FOR_RUN_... FOLDER, BUT NOT IF I AM PART OF AN
924       # AUTOMATIC RESTART RUN
925       # AUTOMATIC RESTART RUNS JUST ACCESS THE DIRECTORY CREATED BY THE INITIAL RUN
926    if [[ $restart_run = false ]]
927    then
928
929          # COLLECT FILES TO BE COMPILED IN THE SOURCES_FOR_RUN_... FOLDER ON
930          # THE LOCAL HOST
931       if [[ ! -d $source_path ]]
932       then
933          printf "\n\n  +++ source path \"$source_path\" on local host"
934          printf "\n      \"$(hostname)\" does not exist"
935          locat=source_path; exit
936       fi
937
938       rm -rf   ${base_directory}/${sources_for_run_catalog}
939       mkdir -p ${base_directory}/${sources_for_run_catalog}
940
941       if [[ "$source_list" = LM ]]
942       then
943
944             # DETERMINE MODIFIED FILES OF THE SVN WORKING COPY
945          source_list=""
946          cd  $source_path
947
948
949             # CHECK, IF TRUNK-DIRECTORY IS UNDER SVN CONTROL
950          if [[ ! -d ../.svn ]]
951          then
952             printf "\n\n  +++ source directory"
953             printf "\n         \"$source_path\" "
954             printf "\n         is not under control of \"subversion\"."
955             printf "\n         Please do not use palmrun-option \"-s LM\"\n"
956          fi
957
958
959             # LIST ALL MODIFIED SOURCE CODE FILES
960          Filenames=""
961          svn status  >  tmp_svnstatus
962          while  read line
963          do
964             firstc=`echo $line | cut -c1`
965             if [[ $firstc = M  ||  $firstc = "?" ]]
966             then
967                Name=`echo "$line" | cut -c8-`
968                extension=`echo $Name | cut -d. -f2`
969                if [[ "$extension" = f90 || "$extension" = F90 || "$extension" = f || "$extension" = F || "$extension" = c ]]
970                then
971                   Filenames="$Filenames "$Name
972                fi
973             fi
974          done < tmp_svnstatus
975          rm -rf  tmp_svnstatus
976
977
978             # COPY FILES TO SOURCES_FOR_RUN_...
979          for  filename  in  $Filenames
980          do
981             cp  $filename  ${base_directory}/${sources_for_run_catalog}
982             source_list=$source_list"$filename "
983          done
984
985          cd -  > /dev/null
986
987
988          # COPY FILES GIVEN BY OPTION -s TO DIRECTORY SOURCES_FOR_RUN_...
989       elif [[ "$source_list" != "" ]]
990       then
991
992          cd  $source_path
993
994          for  filename  in  $source_list
995          do
996
997                # SOURCE CODE FILE IS NOT ALLOWED TO INCLUDE PATH
998             if [[ $(echo $filename | grep -c "/") != 0 ]]
999             then
1000                printf "\n  +++ source code file:  $filename"
1001                printf "\n      must not contain (\"/\") "
1002                locat=source; exit
1003             fi
1004
1005             if [[ ! -f $filename ]]
1006             then
1007                printf "\n  +++ source code file:  $filename"
1008                printf "\n      does not exist"
1009                locat=source; exit
1010             else
1011                cp  $filename  ${base_directory}/${sources_for_run_catalog}
1012             fi
1013
1014          done
1015
1016          cd -  > /dev/null
1017
1018       fi
1019
1020          # CHECK, IF MAKEFILE EXISTS AND COPY IT TO THE SOURCES_FOR_RUN... DIRECTORY
1021       [[ "$makefile" = "" ]]  &&  makefile=$source_path/Makefile
1022       if [[ ! -f $makefile ]]
1023       then
1024          printf "\n  +++ file \"$makefile\" does not exist"
1025          locat=make; exit
1026       else
1027          cp  $makefile  ${base_directory}/${sources_for_run_catalog}/Makefile
1028       fi
1029
1030
1031          # COPY FILES FROM OPTIONAL SOURCE PATH GIVEN IN THE CONFIGURATION FILE
1032       if [[ "$user_source_path" != "" ]]
1033       then
1034
1035             # DOES THE DIRECTORY EXIST?
1036          if [[ ! -d $user_source_path ]]
1037          then
1038
1039             printf "\n\n  *** INFORMATIVE: additional source code directory"
1040             printf "\n      \"$user_source_path\" "
1041             printf "\n      does not exist or is not a directory."
1042             printf "\n      No source code will be used from this directory!\n"
1043             user_source_path=""
1044             if [[ $silent == false ]]
1045             then
1046                sleep 2
1047             fi
1048
1049          else
1050
1051             cd $user_source_path
1052
1053             Names=$(ls -1 *.f90 2>&1)
1054             [[ $(echo $Names | grep -c '*.f90') = 0 ]]  &&  AddFilenames="$Names"
1055             Names=$(ls -1 *.F90 2>&1)
1056             [[ $(echo $Names | grep -c '*.F90') = 0 ]]  &&  AddFilenames="$AddFilenames $Names"
1057             Names=$(ls -1 *.F 2>&1)
1058             [[ $(echo $Names | grep -c '*.F') = 0   ]]  &&  AddFilenames="$AddFilenames $Names"
1059             Names=$(ls -1 *.f 2>&1)
1060             [[ $(echo $Names | grep -c '*.f') = 0   ]]  &&  AddFilenames="$AddFilenames $Names"
1061             Names=$(ls -1 *.c 2>&1)
1062             [[ $(echo $Names | grep -c '*.c') = 0   ]]  &&  AddFilenames="$AddFilenames $Names"
1063
1064             cd -  > /dev/null
1065             cd  ${base_directory}/${sources_for_run_catalog}
1066
1067                # COPY MAKEFILE IF EXISTING
1068             if [[ -f $user_source_path/Makefile ]]
1069             then
1070                printf "\n\n  *** user Makefile from directory"
1071                printf "\n      \"$user_source_path\" is used \n"
1072                if [[ $silent == false ]]
1073                then
1074                   sleep 1
1075                fi
1076                cp  $user_source_path/Makefile  .
1077             fi
1078
1079             for  filename  in  $AddFilenames
1080             do
1081                if [[ -f $filename ]]
1082                then
1083                   printf "\n  +++ source code file \"$filename\" found in additional"
1084                   printf "\n      source code directory \"$user_source_path\" "
1085                   printf "\n      but was also given with option \"-s\" which means that it should be taken"
1086                   printf "\n      from directory \"$source_path\"."
1087                   locat=source; exit
1088                fi
1089
1090                cp  $user_source_path/$filename  .
1091                source_list="$source_list $filename"
1092
1093                   # CHECK IF FILE IS CONTAINED IN MAKEFILE
1094                if [[ $(grep -c $filename Makefile) = 0 ]]
1095                then
1096                   printf "\n\n  +++ user file \"$filename\" "
1097                   printf "\n      is not listed in Makefile \n"
1098                   locat=source; exit
1099                fi
1100             done
1101             cd -  > /dev/null
1102          fi
1103       fi
1104
1105          # COPY CONFIGURATION FILES
1106       cp  $config_file  ${base_directory}/${sources_for_run_catalog}
1107       cp  $fileconnection_file  ${base_directory}/${sources_for_run_catalog}
1108
1109          # COPY SHELLSCRIPTS
1110       cp  ${source_path}/../SCRIPTS/palmrun    ${base_directory}/${sources_for_run_catalog}
1111       cp  ${source_path}/../SCRIPTS/batch_scp  ${base_directory}/${sources_for_run_catalog}
1112
1113    fi
1114
1115 fi
1116
1117
1118    # GET THE GLOBAL REVISION-NUMBER OF THE SVN-REPOSITORY
1119    # (HANDED OVER TO RESTART-RUNS USING OPTION -G)
1120 if [[ "$global_revision" = "" ]]
1121 then
1122    global_revision=`svnversion $source_path  2>/dev/null`
1123    global_revision="Rev: $global_revision"
1124 fi
1125
1126
1127    # IN CASE OF PARALLEL EXECUTION, CHECK SOME SPECIFICATIONS CONCERNING PROCESSOR NUMBERS
1128 if [[ -n $cores ]]
1129 then
1130
1131       # CHECK, IF THE NUMBER OF CORES PER NODE HAS BEEN GIVEN UND IF IT IS AN
1132       # INTEGRAL DIVISOR OF THE TOTAL NUMBER OF CORES GIVEN BY OPTION -X
1133    if [[ $tasks_per_node = 0 ]]
1134    then
1135       if [[ $create_batch_job = true  ||  $create_remote_batch_job = true ]]
1136       then
1137          printf "\n"
1138          printf "\n  +++ option \"-T\" (tasks per node) is missing"
1139          printf "\n      set -T option or define tasks_per_node in the config file"
1140          locat=tasks_per_node; (( iec = 0 )); exit
1141       else
1142             # DEFAULT FOR INTERACTIVE RUN
1143          tasks_per_node=$cores
1144       fi
1145    fi
1146
1147    if (( cores < tasks_per_node ))
1148    then
1149       printf "\n"
1150       printf "\n  +++ tasks per node (-T) cannot exceed total number of cores (-X)"
1151       printf "\n      given values: -T $tasks_per_node  -X $cores"
1152       locat=tasks_per_node; (( iec = 0 )); exit
1153    fi
1154
1155    (( nodes = cores / ( tasks_per_node * threads_per_task ) ))
1156    (( mpi_tasks = cores / threads_per_task ))
1157    [[ $mpi_tasks = 0 ]]  &&  (( mpi_tasks = 1 ))
1158    (( ii = cores / tasks_per_node ))
1159    (( remaining_cores =  cores - ii * tasks_per_node ))
1160    if (( remaining_cores > 0 ))
1161    then
1162       printf "\n"
1163       printf "\n  +++ WARNING: tasks per node (option \"-T\") is not an integral"
1164       printf "\n      divisor of the total number of cores (option \"-X\")"
1165       printf "\n      values of this palmrun-call: \"-T $tasks_per_node\" \"-X $cores\""
1166       printf "\n      One of the nodes is filled with $remaining_cores instead of $tasks_per_node tasks"
1167       (( nodes = nodes + 1 ))
1168    fi
1169
1170 fi
1171
1172
1173    # SET DEFAULT VALUE FOR THE MAXIMUM NUMBER OF PARALLEL IO STREAMS
1174 if [[ "$maximum_parallel_io_streams" = "" ]]
1175 then
1176    maximum_parallel_io_streams=$cores
1177 fi
1178
1179
1180    # SET PORT NUMBER OPTION FOR CALLS OF SSH/SCP AND batch_scp SCRIPT
1181 if [[ "$scp_port" != "" ]]
1182 then
1183    PORTOPT="-P $scp_port"
1184    SSH_PORTOPT="-p $scp_port"
1185 fi
1186
1187
1188    # DETERMINE THE SSH-OPTION IN CASE THAT AN SSH-KEY IS EXPLICITLY GIVEN IN THE
1189    # CONFIG-FILE
1190 if [[ "$ssh_key" != "" ]]
1191 then
1192    ssh_key="-i $HOME/.ssh/$ssh_key"
1193 fi
1194
1195
1196    # SET QUEUE, IF NOT GIVEN
1197 if [[ $create_batch_job = true  ||  $create_remote_batch_job = true ]]
1198 then
1199
1200    if [[ $queue = none  &&  "$defaultqueue" = "" ]]
1201    then
1202       printf "\n"
1203       printf "\n  +++ no default queue given in configuration file and no queue"
1204       printf "\n      given with option -q"
1205       locat=queue; exit
1206    fi
1207    if [[ $queue = none ]]
1208    then
1209       queue=$defaultqueue
1210    fi
1211
1212 fi
1213
1214
1215    # GENERATE FULL FILENAMES OF INPUT-FILES, INCLUDING THEIR PATH
1216    # CHECK, IF INPUT-FILES EXIST, AND DETERMINE HIGHEST CYCLE NUMBER (IF CYCLES EXIST)
1217 (( i = 0 ))
1218 (( nr_of_input_files = 0 ))
1219 while (( i < iin ))
1220 do
1221    (( i = i + 1 ))
1222
1223       # GENERATE PATH AND FULL FILE NAME (then-BRANCH: FIXED FULL NAME IS GIVEN, I.E. THE
1224       # FILE IDENTIFIER IS NOT PART OF THE FILENAME))
1225    if [[ "${actionin_pre[$i]}" = di ]]
1226    then
1227       eval filename=${pathin_pre[$i]}/${endin_pre[$i]}
1228    else
1229       eval filename=${pathin_pre[$i]}/${run_identifier}${endin_pre[$i]}
1230    fi
1231
1232
1233       # CHECK IF FILE EXISTS
1234    if ! ls $filename* 1>/dev/null 2>&1
1235    then
1236
1237          # FILES WITH ATTRIBUTE opt ARE OPTIONAL. NO ABORT, IF THEY DO NOT EXIST.
1238       if [[ "${optin_pre[$i]}" != "yes" ]]
1239       then
1240          printf "\n\n  +++ INPUT-file: "
1241          if [[ "${extin_pre[$i]}" = ""  ||  "${extin_pre[$i]}" = " " ]]
1242          then
1243             printf "\n           $filename"
1244          else
1245             printf "\n           $filename.${extin_pre[$i]}"
1246          fi
1247          printf "\n      does not exist\n"
1248          locat=input; exit
1249       else
1250          (( nr_of_input_files = nr_of_input_files + 1 ))
1251          localin[$nr_of_input_files]="${localin_pre[$i]}"
1252          optin[$nr_of_input_files]="${optin_pre[$i]}"
1253          actionin[$nr_of_input_files]="unavailable"
1254          pathin[$nr_of_input_files]="${pathin_pre[$i]}"
1255          endin[$nr_of_input_files]="${endin_pre[$i]}"
1256          extin[$nr_of_input_files]="${extin_pre[$i]}"
1257       fi
1258
1259    else
1260
1261          # FIRST CHECK FOR MULTIPLE NAMES WITH THE SAME BASENAME
1262          # ($run_identifier) AND CREATE A LIST FOR THE DETECTED BASENAME
1263          # ENDINGS
1264       if [[ "${multin[$i]}" = true ]]
1265       then
1266             # DETERMINE THE EXISTING EXTENSIONS FROM THE LIST OF FILES
1267          ls -1 -d ${filename}    >   filelist  2>/dev/null
1268          ls -1 -d ${filename}.*  >>  filelist  2>/dev/null
1269          ls -1 -d ${filename}_*  >>  filelist  2>/dev/null
1270
1271          endings=""
1272          while  read line
1273          do
1274                 # filename without path (i.e. after the last "/")
1275              basefilename=$(basename ${line})
1276
1277                 # check if there is an extension and remove it
1278              ext=${basefilename##*.}
1279              if [[ "$ext" = "${extin_pre[$i]}" ]]
1280              then
1281                 basefilename=${basefilename%.*}
1282              fi
1283
1284                 # check for an existing cycle number and remove it
1285              cycle=${basefilename##*.}
1286              if [[ $cycle =~ ^-?[0-9]+$ ]]
1287              then
1288                 basefilename=${basefilename%.*}
1289              fi
1290
1291                 # remove the run_identifier from the beginning
1292              length_run_identifier=${#run_identifier}
1293              ending=${basefilename:${length_run_identifier}}
1294
1295                 # remove the ending given in the .iofiles from the beginning
1296              endingstring="${endin_pre[$i]}"
1297              length_ending=${#endingstring}
1298              ending=${ending:${length_ending}}
1299
1300              if [[ "$ending" = "" ]]
1301              then
1302                    # standard ending as given in the .iofiles
1303                 if [[ $(echo $endings | grep -c DEFAULT) = 0 ]]
1304                 then
1305                    endings="$endings DEFAULT"
1306                 fi
1307              else
1308                    # ending must start with "_", otherwise its a different file
1309                 if [[ "${ending:0:1}" = "_" ]]
1310                 then
1311                    if [[ $(echo $endings | grep -c "$ending") = 0 ]]
1312                    then
1313                       endings="$endings $ending"
1314                    fi
1315                 fi
1316              fi
1317 
1318          done <filelist
1319
1320          rm filelist
1321
1322       else
1323
1324             # SINGLE NAME
1325          endings=DEFAULT
1326
1327       fi
1328
1329          # FOR EACH BASENAME ENDING CREATE AN ENTRY IN THE FINAL INPUT FILE LIST
1330       for  ending  in  $endings
1331       do
1332
1333             # DEFAULT MEANS THAT THE ENDING GIVEN IN .iofiles SHALL BE USED
1334          if [[ $ending = DEFAULT ]]
1335          then
1336             ending=""
1337          fi
1338
1339             # NEW ENTRY (ENDING IS ALSO ADDED TO LOCAL FILENAME READ BY PALM!)
1340          (( nr_of_input_files = nr_of_input_files + 1 ))
1341          localin[$nr_of_input_files]="${localin_pre[$i]}"$ending
1342          optin[$nr_of_input_files]="${optin_pre[$i]}"
1343          actionin[$nr_of_input_files]="${actionin_pre[$i]}"
1344          pathin[$nr_of_input_files]="${pathin_pre[$i]}"
1345          endin[$nr_of_input_files]="${endin_pre[$i]}"$ending
1346          extin[$nr_of_input_files]="${extin_pre[$i]}"
1347
1348
1349             # GENERATE PATH AND FULL FILE NAME (then-BRANCH: FIXED FULL NAME IS GIVEN, I.E. THE
1350             # FILE IDENTIFIER IS NOT PART OF THE FILENAME))
1351          if [[ "${actionin[$nr_of_input_files]}" = di ]]
1352          then
1353             eval filename=${pathin[$nr_of_input_files]}/${endin[$nr_of_input_files]}
1354          else
1355             eval filename=${pathin[$nr_of_input_files]}/${run_identifier}${endin[$nr_of_input_files]}
1356          fi
1357
1358             # DETERMINE THE FILE'S CYCLE NUMBER
1359          (( maxcycle = 0 ))
1360          ls -1 -d $filename    >   filelist  2>/dev/null
1361          ls -1 -d $filename.*  >>  filelist  2>/dev/null
1362          while  read line
1363          do
1364                # filename without path (i.e. after the last "/")
1365             basefilename=$(basename ${line})
1366   
1367                # check if there is an extension
1368             extension=${basefilename##*.}
1369             if [[ "$extension" = "${extin[$nr_of_input_files]}" ]]
1370             then
1371                basefilename=${basefilename%.*}
1372             fi
1373   
1374                # check for an existing cycle number
1375             cycle=${basefilename##*.}
1376             if [[ $cycle =~ ^-?[0-9]+$ ]]
1377             then
1378                   # NUMBERS WITH LEADING ZEROS ARE INTERPRETED AS OCTAL NUMBERS
1379                   # 10# EXPLICITLY SPECIFIES THE NUMBER BASE AS 10
1380                (( icycle = $((10#$cycle)) ))
1381                explicit_cycle_number=true
1382             else
1383                (( icycle = 0 ))
1384                explicit_cycle_number=false
1385             fi
1386   
1387             if (( icycle > maxcycle ))
1388             then
1389                (( maxcycle = icycle ))
1390             fi
1391   
1392          done <filelist
1393          rm filelist
1394
1395   
1396             # MAKE CYCLE NUMBER THREE DIGITS WIDE
1397          cyclestring=`printf "%03d" $maxcycle`
1398
1399   
1400             # APPEND CYCLE NUMBER TO FILENAME
1401          if [[ $explicit_cycle_number = true ]]
1402          then
1403             if [[ "${extin[$nr_of_input_files]}" != " "  &&   "${extin[$nr_of_input_files]}" != "" ]]
1404             then
1405                filename=${filename}.$cyclestring.${extin[$nr_of_input_files]}
1406             else
1407                filename=${filename}.$cyclestring
1408             fi
1409          else
1410             if [[ "${extin[$nr_of_input_files]}" != " "  &&   "${extin[$nr_of_input_files]}" != "" ]]
1411             then
1412                filename=${filename}.${extin[$nr_of_input_files]}
1413             fi
1414          fi
1415         
1416             # STORE FILENAME WITHOUT PATH BUT WITH CYCLE NUMBER,
1417             # IS LATER USED FOR TRANSFERRING FILES WIHIN THE JOB (SEE END OF FILE)
1418          absnamein[$nr_of_input_files]=$filename
1419          if [[ $explicit_cycle_number = true ]]
1420          then
1421             if [[ "${actionin[$nr_of_input_files]}" = di ]]
1422             then
1423                frelin[$nr_of_input_files]=${endin[$nr_of_input_files]}.$cyclestring
1424             else
1425                frelin[$nr_of_input_files]=${run_identifier}${endin[$nr_of_input_files]}.$cyclestring
1426             fi
1427          else
1428             if [[ "${actionin[$nr_of_input_files]}" = di ]]
1429             then
1430                frelin[$nr_of_input_files]=${endin[$nr_of_input_files]}
1431             else
1432                frelin[$nr_of_input_files]=${run_identifier}${endin[$nr_of_input_files]}
1433             fi
1434          fi
1435
1436       done
1437
1438    fi
1439
1440 done
1441
1442
1443    # GENERATE FULL FILENAMES OF OUTPUT-FILES (WITHOUT $ OR ~),
1444    # CHECK, IF OUTPUT-FILES EXIST, AND DETERMINE HIGHEST CYCLE NUMBER (IF CYCLES EXIST),
1445    # OR, IN CASE THAT FILE DOES NOT EXIST, CHECK, IF IT CAN BE CREATED 
1446    # THESE ACTIONS ARE NOT CARRIED OUT, IF FILES SHALL BE TRANSFERRED FROM THE REMOTE TO
1447    # THE LOCAL HOST (BECAUSE THEIR IS NO DIRECT ACCESS TO THE LOCAL DIRECTORIES FROM THE
1448    # REMOTE HOST)
1449 (( i = 0 ))
1450 while (( i < iout ))
1451 do
1452    (( i = i + 1 ))
1453    if [[ ! ( $running_on_remote = true  &&  ( "${actionout_pre[$i]}" = tr || "${actionout_pre[$i]}" = tra || "${actionout_pre[$i]}" = trpe ) ) ]]
1454    then
1455       if [[ "${actionout_pre[$i]}" = tr ]]
1456       then
1457          actionout_pre[$i]=""
1458       elif [[ "${actionout_pre[$i]}" = trpe ]]
1459       then
1460          actionout_pre[$i]=pe
1461       elif [[ "${actionout_pre[$i]}" = tra ]]
1462       then
1463          actionout_pre[$i]=a
1464       fi
1465       (( maxcycle = 0 ))
1466       eval filename=${pathout_pre[$i]}/${run_identifier}${endout_pre[$i]}
1467       eval catalogname=${pathout_pre[$i]}
1468       if ! ls $filename* 1>/dev/null 2>&1
1469       then
1470     
1471             # IF OUTPUT-FILE DOES NOT EXIST CHECK, IF IT CAN BE CREATED
1472          if  cat /dev/null > $filename 
1473          then
1474             rm  $filename
1475          else
1476
1477                # CHECK, IF THE DIRECTORY WHERE FILE SHALL BE COPIED TO EXISTS
1478                # IF IT DOES NOT EXIST, TRY TO CREATE IT
1479             if [[ ! -d $catalogname ]]
1480             then
1481                if  mkdir -p  $catalogname
1482                then
1483                   printf "\n\n  *** directory:"
1484                   printf "\n           $catalogname"
1485                   printf "\n      was created\n"
1486                else
1487                   printf "\n\n  +++ OUTPUT-file:"
1488                   printf "\n           $filename"
1489                   printf "\n      cannot be created, because directory does not exist"
1490                   printf "\n      and cannot be created either"
1491                   printf "\n"
1492                   locat=output  ; exit
1493                fi 2>/dev/null
1494             else
1495                printf "\n\n  +++ OUTPUT-file:"
1496                printf "\n           $filename"
1497                printf "\n      cannot be created, although directory exists"
1498                printf "\n"
1499                locat=output  ; exit
1500             fi
1501          fi 2>/dev/null
1502
1503       fi
1504
1505    fi
1506 done
1507
1508
1509    # DETERMINE THE NAME OF PALMRUN'S TEMPORARY WORKING DIRECTORY
1510 if [[ $running_in_batch_mode = false ]]
1511 then
1512    run_id_number=$RANDOM
1513    run_id=${run_identifier}.$run_id_number
1514
1515    tempdir=$fast_io_catalog/$run_id
1516
1517       # FOR COMPATIBILITY REASONS WITH OLDER VERSIONS SET JOB_ID
1518    export job_id=$run_id
1519
1520 fi
1521
1522
1523    # CHECK SETTINGS REQUIRED FOR BATCH JOBS
1524 if [[ $create_batch_job = true  ||  $create_remote_batch_job = true ]]
1525 then
1526
1527       # CHECK, IF JOB DIRECTIVES HAVE BEEN GIVEN IN CONFIGURATION FILE
1528    if [[ $ibd = 0 ]]
1529    then
1530       printf "\n"
1531       printf "\n  +++ no batch directives found in configuration file"
1532       locat=config_file_batch_directives; (( iec = 0 )); exit
1533    fi
1534
1535       # CHECK IF CPUTIME IS GIVEN FOR JOB
1536    done=false
1537    cputime=$cpumax
1538    while [[ $done = false ]]
1539    do
1540       if (( cputime == 0 ))
1541       then
1542          printf "\n  +++ cpu-time is undefined"
1543          printf "\n  >>> Please type CPU-time in seconds as INTEGER:"
1544          printf "\n  >>> "
1545          read  cputime   1>/dev/null  2>&1
1546       else
1547          done=true
1548       fi
1549    done
1550    cpumax=$cputime
1551
1552       # CHECK THE MEMORY DEMAND
1553    done=false
1554    while [[ $done = false ]]
1555    do
1556       if (( memory == 0 ))
1557       then
1558          printf "\n  +++ memory demand is undefined"
1559          printf "\n  >>> Please type memory in  MByte per process  as INTEGER:"
1560          printf "\n  >>> "
1561          read  memory  1>/dev/null  2>&1
1562       else
1563          done=true
1564       fi
1565    done
1566
1567       # IN CASE OF REMOTE-JOBS CHECK, IF A USERNAME FOR THE REMOTE HOST IS GIVEN
1568    if [[ $create_remote_batch_job = true  &&  -z $remote_username ]]
1569    then
1570       while [[ -z $remote_username ]]
1571       do
1572          printf "\n  +++ username on remote host with IP \"$remote_ip\" is undefined"
1573          printf "\n  >>> Please type username:"
1574          printf "\n  >>> "
1575          read  remote_username
1576       done
1577    fi
1578
1579 else
1580
1581    if [[ $running_in_batch_mode = false ]]
1582    then
1583       cputime=10000000  # NO LIMT FOR INTERACTIVE RUNS
1584       cpumax=$cputime
1585    else
1586       cputime=$cpumax
1587    fi
1588
1589 fi
1590
1591
1592    # CALCULATE HOURS/MINUTES/SECONDS, E.G. FOR BATCH-DIRECTIVES
1593 (( cpu_hours  = cputime / 3600 ))
1594 (( resttime = cputime - cpu_hours * 3600 ))
1595 (( cpu_minutes  = resttime / 60 ))
1596 (( cpu_seconds = resttime - cpu_minutes * 60 ))
1597 timestring=${cpu_hours}:${cpu_minutes}:${cpu_seconds}
1598
1599
1600    # OUTPUT OF THE PALMRUN-HEADER
1601 calltime=$(date)
1602 printf "\n"
1603 printf "#------------------------------------------------------------------------# \n"
1604 printf "| %-35s%35s | \n" "$version" "$calltime"
1605 printf "| %-35s%35s | \n" "PALM code    $global_revision" " "
1606 printf "|                                                                        | \n"
1607 column1="called on:"; column2=$(hostname)
1608 printf "| %-25s%-45s | \n" "$column1" "$column2"
1609 if [[ $create_remote_batch_job = true ]]
1610 then
1611    column1="execution on:"; column2="$configuration_identifier (username: $remote_username)"
1612 else
1613    if [[ $running_on_remote = true ]]
1614    then
1615       column1="config. identifier:"; column2="$configuration_identifier (execute on IP: $remote_ip)"
1616    else
1617       column1="config. identifier:"; column2="$configuration_identifier (execute on IP: $local_ip)"
1618    fi
1619 fi
1620 printf "| %-25s%-45s | \n" "$column1" "$column2"
1621
1622 column1="running in:"
1623 if [[ $running_in_batch_mode = true ]]
1624 then
1625    column2="batch job mode"
1626 else
1627    if [[ $create_batch_job = true  ||  $create_remote_batch_job = true ]]
1628    then
1629       column2="job creation mode"
1630    else
1631       column2="interactive run mode"
1632    fi
1633 fi
1634 printf "| %-25s%-45s | \n" "$column1" "$column2"
1635
1636 if  [[ $running_in_batch_mode = true  ||  $create_batch_job = true  ||  $create_remote_batch_job = true ]]
1637 then
1638    if [[ "$project_account" != "" ]]
1639    then
1640       column1="project account number:"
1641       column2="$project_account"
1642       printf "| %-25s%-45s | \n" "$column1" "$column2"
1643    fi
1644 fi
1645
1646 if [[ -n $cores ]]
1647 then
1648    if [[ $run_coupled_model = false ]]
1649    then
1650       column1="number of cores:"; column2=$cores
1651    else
1652       column1="number of cores:"; column2="$cores  (atmosphere: $cores_atmos, ocean: $cores_ocean)"
1653    fi
1654    printf "| %-25s%-45s | \n" "$column1" "$column2"
1655 fi
1656 if [[ -n $tasks_per_node ]]
1657 then
1658    column1="tasks per node:"; column2="$tasks_per_node (number of nodes: $nodes)"
1659    printf "| %-25s%-45s | \n" "$column1" "$column2"
1660    if (( remaining_cores > 0 ))
1661    then
1662       column1=" "; column2="one of the nodes only filled with $remaining_cores tasks"
1663       printf "| %-25s%-45s | \n" "$column1" "$column2"
1664    fi
1665 fi
1666 if [[ $maximum_parallel_io_streams != $cores ]]
1667 then
1668    column1="max par io streams:"; column2="$maximum_parallel_io_streams"
1669    printf "| %-25s%-45s | \n" "$column1" "$column2"
1670 fi
1671 if [[ $use_openmp = true ]]
1672 then
1673    column1="threads per task:"; column2="$threads_per_task"
1674    printf "| %-25s%-45s | \n" "$column1" "$column2"
1675 fi
1676 if [[ $create_batch_job = true  ||  $create_remote_batch_job = true  ||  $running_in_batch_mode = true ]]
1677 then
1678    column1="memory demand / PE":; column2="$memory MB"
1679    printf "| %-25s%-45s | \n" "$column1" "$column2"
1680    column1="job cpu time (h:m:s):"; column2="$timestring"
1681    printf "| %-25s%-45s | \n" "$column1" "$column2"
1682 fi
1683 printf "|                                                                        | \n"
1684 if [[ "$source_list" != "" ]]
1685 then
1686    if [[ "$make_options" != "" ]]
1687    then
1688       column1="make options:"; column2=$(echo "$make_options" | cut -c-45)
1689       printf "| %-25s%-45s | \n" "$column1" "$column2"
1690       line=$(echo "$make_options" | cut -c46-)
1691       while [[ "$line" != "" ]]
1692       do
1693          column1=""
1694          column2=$(echo "$line" | cut -c-45)
1695          printf "| %-25s%-45s | \n" "$column1" "$column2"
1696          line=$(echo "$line" | cut -c46-)
1697       done
1698    fi
1699 fi
1700
1701 column1="cpp directives:"; column2=$(echo "$cpp_options" | cut -c-45)
1702 printf "| %-25s%-45s | \n" "$column1" "$column2"
1703 line=$(echo "$cpp_options" | cut -c46-)
1704 while [[ "$line" != "" ]]
1705 do
1706    column1=""
1707    column2=$(echo "$line" | cut -c-45)
1708    printf "| %-25s%-45s | \n" "$column1" "$column2"
1709    line=$(echo "$line" | cut -c46-)
1710 done
1711
1712 column1="compiler options:"; column2=$(echo "$compiler_options" | cut -c-45)
1713 printf "| %-25s%-45s | \n" "$column1" "$column2"
1714 line=$(echo "$compiler_options" | cut -c46-)
1715 while [[ "$line" != "" ]]
1716 do
1717    column1=""
1718    column2=$(echo "$line" | cut -c-45)
1719    printf "| %-25s%-45s | \n" "$column1" "$column2"
1720    line=$(echo "$line" | cut -c46-)
1721 done
1722
1723 column1="linker options:"; column2=$(echo "$linker_options" | cut -c-45)
1724 printf "| %-25s%-45s | \n" "$column1" "$column2"
1725 line=$(echo "$linker_options" | cut -c46-)
1726 while [[ "$line" != "" ]]
1727 do
1728    column1=""
1729    column2=$(echo "$line" | cut -c-45)
1730    printf "| %-25s%-45s | \n" "$column1" "$column2"
1731    line=$(echo "$line" | cut -c46-)
1732 done
1733
1734 if [[ "$login_init_cmd" != "" ]]
1735 then
1736    column1="login init commands:"; column2=$(echo "$login_init_cmd" | cut -c-45)
1737    printf "| %-25s%-45s | \n" "$column1" "$column2"
1738    line=$(echo "$login_init_cmd" | cut -c46-)
1739    while [[ "$line" != "" ]]
1740    do
1741       column1=""
1742       column2=$(echo "$line" | cut -c-45)
1743       printf "| %-25s%-45s | \n" "$column1" "$column2"
1744       line=$(echo "$line" | cut -c46-)
1745    done
1746 fi
1747
1748 if [[ "$module_commands" != "" ]]
1749 then
1750    column1="module commands:"; column2=$(echo "$module_commands" | cut -c-45)
1751    printf "| %-25s%-45s | \n" "$column1" "$column2"
1752    line=$(echo "$module_commands" | cut -c46-)
1753    while [[ "$line" != "" ]]
1754    do
1755       column1=""
1756       column2=$(echo "$line" | cut -c-45)
1757       printf "| %-25s%-45s | \n" "$column1" "$column2"
1758       line=$(echo "$line" | cut -c46-)
1759    done
1760 fi
1761 printf "|                                                                        | \n"
1762 column1="run identifier:"; column2=$run_identifier
1763 printf "| %-25s%-45s | \n" "$column1" "$column2"
1764 column1="activation string list:"; column2=$(echo $activation_string_list)
1765 printf "| %-25s%-45s | \n" "$column1" "$column2"
1766
1767 if [[ "$ocean_file_appendix" = true ]]
1768 then
1769    printf "| %-35s%-35s | \n" "suffix \"_O\" is added to local files" " "
1770 fi
1771
1772 if [[ "$source_list" != "" ]]
1773 then
1774    printf "|                                                                        | \n"
1775    printf "| Files to be compiled:                                                  | \n"
1776    line=$source_list
1777    while [[ "$line" != "" ]]
1778    do
1779       linestart=$(echo $line | cut -c-70)
1780       printf "| %-70s | \n" "$linestart"
1781       line=$(echo "$line" | cut -c71-)
1782    done
1783 fi
1784 printf "#------------------------------------------------------------------------#"
1785
1786
1787
1788    # OUTPUT OF FILE CONNECTIONS IN CASE OF TRACEBACK
1789 if [[ $do_trace = true ]]
1790 then
1791    (( i = 0 ))
1792    while (( i < nr_of_input_files ))
1793    do
1794       (( i = i + 1 ))
1795       if (( i == 1 ))
1796       then
1797          printf "\n\n >>> INPUT-file assignments:\n"
1798       fi
1799       printf "\n     ${localin[$i]} :  ${absnamein[$i]}"
1800    done
1801    (( i = 0 ))
1802    while (( i < iout ))
1803    do
1804       (( i = i + 1 ))
1805       if (( i == 1 ))
1806       then
1807          printf "\n\n >>> OUTPUT-file assignments:\n"
1808       fi
1809       printf "\n     ${localout[$i]} :  ${pathout[$i]}"
1810    done
1811    (( i = 0 ))
1812    while (( i < iic ))
1813    do
1814       (( i = i + 1 ))
1815       if (( i == 1 ))
1816       then
1817          printf "\n\n >>> INPUT-commands:\n"
1818       fi
1819       printf "\n     ${in_command[$i]}" 
1820    done
1821    (( i = 0 ))
1822    while (( i < ioc ))
1823    do
1824       (( i = i + 1 ))
1825       if (( i == 1 ))
1826       then
1827          printf "\n\n >>> OUTPUT-commands:\n"
1828       fi
1829       printf "\n     ${out_command[$i]}" 
1830    done
1831 fi
1832
1833    # QUERY FOR CONTINUE
1834 if [[ $silent = false  &&  $running_in_batch_mode = false ]]
1835 then
1836    antwort=dummy
1837    printf "\n\n"
1838    printf " >>> everything o.k. (y/n) ?  "
1839    while  read antwort
1840    do
1841       if [[ "$antwort" != y  &&  "$antwort" != Y  &&  "$antwort" != n  &&  "$antwort" != N ]]
1842       then
1843          printf " >>> everything o.k. (y/n) ?  "
1844       else
1845          break
1846       fi
1847    done
1848    if [[ $antwort = n  ||  $antwort = N ]]
1849    then
1850       locat=user_abort; (( iec = 0 )); exit
1851    fi
1852    if [[ $create_batch_job = true  ||  $create_remote_batch_job = true ]]
1853    then
1854       printf "\n ***  batch-job will be created and submitted"
1855    else
1856       printf "\n ***  PALMRUN will now continue to execute on this machine"
1857    fi
1858 fi
1859
1860 
1861
1862    # PROVIDE FILES TO EXECUTE PALM AND CREATE THE EXECUTABLE
1863 if [[ $restart_run = false  &&  $running_in_batch_mode = false ]]
1864 then
1865
1866    if [[ $create_batch_job = true  ||  $create_remote_batch_job  = true ]]
1867    then
1868       printf "\n\n  *** creating executable and other sources for the remote host\n"
1869    else
1870       printf "\n\n  *** creating executable and other sources for the local host\n"
1871    fi
1872
1873       # FIRST CHECK, IF A MAKE DEPOSITORY EXISTS, AND IF NOT, ASK THE USER IF
1874       # IT SHALL BE CREATED
1875    ask_for_make_depository=false
1876    if [[ $create_remote_batch_job = true ]]
1877    then
1878
1879       line=`grep %base_directory $config_file`
1880       make_depository=`echo $line | cut -d" " -s -f2`/MAKE_DEPOSITORY_${configuration_identifier}
1881       ssh  -q  $ssh_key  ${remote_username}@${remote_ip} "[[ ! -d ${make_depository} ]]  &&  echo depository not found" 2>&1  | tee ${configuration_identifier}_last_make_protokoll
1882
1883       if [[ $(grep -c "depository not found" ${configuration_identifier}_last_make_protokoll) != 0 ]]
1884       then
1885          printf "\n\n  +++ make depository \"${make_depository}\""
1886          printf "\n      on remote host not found!"
1887          ask_for_make_depository=true
1888       fi
1889       rm ${configuration_identifier}_last_make_protokoll
1890
1891    else
1892
1893          # CHECK FOR MAKE_DEPOSITORY ON THE LOCAL HOST
1894       make_depository=${base_directory}/MAKE_DEPOSITORY_${configuration_identifier}
1895       if [[ ! -d ${make_depository} ]]
1896       then
1897          printf "\n\n  +++ make depository \"${make_depository}\""
1898          printf "\n      on local host not found!"
1899          ask_for_make_depository=true
1900       fi
1901
1902    fi
1903
1904    if [[ $ask_for_make_depository = true ]]
1905    then
1906
1907       antwort=dummy
1908       printf "\n\n"
1909       printf "  >>> Create a new one (y/n) ?  "
1910       while  read antwort
1911       do
1912          if [[ "$antwort" != y  &&  "$antwort" != Y  &&  "$antwort" != n  &&  "$antwort" != N ]]
1913          then
1914             printf "  >>> Create a new one (y/n) ?  "
1915          else
1916             break
1917          fi
1918       done
1919       if [[ $antwort = n  ||  $antwort = N ]]
1920       then
1921          locat=user_abort; (( iec = 0 )); exit
1922       fi
1923
1924       if [[ $do_trace = true ]]
1925       then
1926          palmbuild  -c $configuration_identifier
1927       else
1928          palmbuild  -v  -c $configuration_identifier
1929       fi
1930
1931       if [[ ${PIPESTATUS[0]} != 0 ]]
1932       then
1933   
1934              # ABORT IN CASE OF COMPILATION PROBLEMS
1935          printf "\n  +++ error while compiling for the MAKE_DEPOSITORY"
1936          locat=make_depository
1937          exit
1938       else
1939          echo "  *** now continue with creating executable and other sources"
1940       fi
1941
1942    fi
1943
1944       # NOW CREATE THE SOURCES_FOR_RUN FOLDER
1945    palmbuild  -v  $use_existing_sources_folder  -c $configuration_identifier  -r $run_identifier
1946
1947    if [[ ${PIPESTATUS[0]} != 0 ]]
1948    then
1949
1950           # ABORT IN CASE OF COMPILATION PROBLEMS
1951       printf "\n  +++ error while creating executable and/or other sources"
1952       locat=execution
1953       rm -rf  ${base_directory}/${sources_for_run_catalog}
1954       exit
1955
1956    else
1957
1958       printf "  *** executable and other sources created\n"
1959       rm -rf  ${base_directory}/${sources_for_run_catalog}
1960
1961    fi
1962
1963 fi
1964
1965
1966    # WHEN CREATING A REMOTE BATCH JOB, THOSE INPUT FILES WITH TRANSFER-ATTRIBUT
1967    # WILL BE COPIED TO THE REMOTE HOST
1968 if [[ $create_remote_batch_job = true ]]
1969 then
1970    (( i = 0 ))
1971    while (( i < nr_of_input_files ))
1972    do
1973       (( i = i + 1 ))
1974       if [[ "${actionin[$i]}" = tr ]]
1975       then
1976          eval inputfile=${pathin[$i]}/${frelin[$i]}
1977          scp  -q $ssh_key  $PORTOPT  $inputfile  ${remote_username}@${remote_ip}:${fast_io_catalog}/${sources_for_run_catalog}/${frelin[$i]}
1978       fi
1979    done
1980    if (( i > 0 ))
1981    then
1982       printf "\n\n  *** input files have been copied to the remote host\n"
1983    fi
1984 fi
1985 
1986
1987    # NOW PERFORM THOSE ACTIONS REQUIRED TO EXECUTE THE PROGRAM (PALM) ON THIS MACHINE
1988    # (COMPILING/LINKING, EXECUTING, COPYING I/O FILES)
1989 if [[ $create_batch_job = false  &&  $create_remote_batch_job = false ]]
1990 then
1991
1992       # CHANGE TO THE TEMPORARY WORKING DIRECTORY
1993    if [[ $running_in_batch_mode = false ]]
1994    then
1995
1996          # CREATE THE DIRECTORY
1997       mkdir -p  $tempdir
1998       if [[ $? != 0 ]]
1999       then
2000          printf "\n  +++ temporary working directory cannot be created."
2001          printf "\n      Check setting of variable fast_io_catalog in your config file."
2002          locat=execution
2003          exit
2004       fi
2005       chmod  go+rx  $tempdir
2006       cd  $tempdir
2007       printf "\n  *** changed to temporary directory: $tempdir"
2008
2009    else
2010
2011          # IN BATCH MODE PALMRUN IS CALLED FROM TEMPDIR
2012       printf "\n  *** running in temporary directory: $tempdir"
2013
2014    fi
2015
2016
2017       # PROVIDE THE INPUT FILES
2018       # LOOP OVER ALL ACTIVATED FILES (LISTED IN THE CONFIGURATION FILE)
2019    optional_files_missing=false
2020    (( i = 0 ))
2021    while (( i < nr_of_input_files ))
2022    do
2023       (( i = i + 1 ))
2024       if (( i == 1 ))
2025       then
2026          printf "\n\n  *** providing INPUT-files:\n$dashes"
2027       fi
2028
2029
2030          # SKIP OPTIONAL FILES, IF THEY DO NOT EXIST
2031       if [[ "${actionin[$i]}" = unavailable ]]
2032       then
2033          optional_files_missing=true
2034          continue
2035       fi
2036
2037          # CHECK FOR SINGLE FILE (SERIAL RUN) OR DIRECTORY (ONE FILE PER CORE FOR PARELLEL EXECUTION)
2038       files_for_cores=false; filetype=file
2039       if [[ "${actionin[$i]}" = pe  &&  -n $cores ]]
2040       then
2041          files_for_cores=true; filetype=files
2042          actionin[$i]=""
2043       elif [[ "${actionin[$i]}" = pe  &&  ! -n $cores ]]
2044       then
2045          actionin[$i]=""
2046       elif [[ "${actionin[$i]}" = lnpe  &&  -n $cores ]]
2047       then
2048          files_for_cores=true; filetype=files
2049          actionin[$i]="ln"
2050             # FILE HAS BEEN FOUND INSTEAD OF DIRECTORY
2051          if [[ -f "${absnamein[$i]}" ]]
2052          then
2053             files_for_cores=false; filetype=file
2054          fi
2055       elif [[ "${actionin[$i]}" = lnpe  &&  ! -n $cores ]]
2056       then
2057          actionin[$i]="ln"
2058       fi
2059
2060       if [[ $files_for_cores = true ]]
2061       then
2062          printf "\n  >>> INPUT: ${absnamein[$i]}/....  to  ${localin[$i]}"
2063       else
2064          printf "\n  >>> INPUT: ${absnamein[$i]}  to  ${localin[$i]}"
2065       fi
2066
2067          # INPUT-FILES TO BE LINKED
2068       if [[ "${actionin[$i]}" = ln ]]
2069       then
2070
2071          printf "\n      $filetype will be linked"
2072          if [[ $files_for_cores = false ]]
2073          then
2074             if [[ -f "${absnamein[$i]}" ]]
2075             then
2076                ln  ${absnamein[$i]}  ${localin[$i]}
2077                got_tmp[$i]=true
2078             fi
2079          else
2080             if [[ -d "${absnamein[$i]}" ]]
2081             then
2082                mkdir -p ${localin[$i]}
2083                cd ${absnamein[$i]}
2084                for file in $(ls *)
2085                do
2086                   ln $file $tempdir/${localin[$i]}
2087                done >|/dev/null 2>&1
2088                cd $tempdir
2089             fi
2090
2091                # IF "ln -f" HAS FAILED DO A NORMAL COPY "cp -r"
2092             if [[ ! -f "${localin[$i]}/_000000" ]]
2093             then
2094                printf "\n  --- WARNING: ln failed, using cp instead (might be time consuming...)"
2095                cp -r  ${absnamein[$i]}/*  ${localin[$i]}
2096             fi
2097
2098             got_tmp[$i]=true
2099          fi
2100       fi
2101
2102          # FILE IS STORED IN THE RESPECTIVE DIRECTORY GIVEN IN THE CONFIGURATION FILE
2103       if [[ "${actionin[$i]}" = ""  ||  "${actionin[$i]}" = "di"  ||  "${actionin[$i]}" = "tr"  ||  "${actionin[$i]}" = "npe" ]]
2104       then
2105
2106          if [[ "${actionin[$i]}" = "npe"  &&  -n $cores ]]
2107          then
2108
2109                # FILE COPIES ARE PROVIDED FOR ALL CORES
2110                # EACH FILE GETS A UNIQUE FILENAME WITH A FOUR DIGIT NUMBER
2111             printf "\n      file will be provided for $cores processors"
2112             mkdir -p ${localin[$i]}
2113             ival=$cores
2114             (( ii = 0 ))
2115             while (( ii <= ival-1 ))
2116             do
2117                if (( ii < 10 ))
2118                then
2119                   cp  ${absnamein[$i]}  ${localin[$i]}/_000$ii
2120                elif (( ii < 100 ))
2121                then
2122                   cp  ${absnamein[$i]}  ${localin[$i]}/_00$ii
2123                elif (( ii < 1000 ))
2124                then
2125                   cp  ${absnamein[$i]}  ${localin[$i]}/_0$ii
2126                else
2127                   cp  ${absnamein[$i]}  ${localin[$i]}/_$ii
2128                fi
2129                (( ii = ii + 1 ))
2130             done
2131
2132          else
2133
2134             if [[ $files_for_cores = true ]]
2135             then
2136
2137                   # PROVIDE FILES FOR EACH CORE
2138                   # FIRST CREATE THE LOCAL DIRECTORY, THEN COPY FILES
2139                   # FROM THE PERMANENT DIRECTORY BY LINKING THEM TO THE LOCAL ONE
2140                printf "\n      providing $cores files for the respective cores"
2141                mkdir -p ${localin[$i]}
2142                        if [[ $link_local_input = true ]]
2143                              then
2144                                 printf "      files will be linked\n"
2145                                 cd ${absnamein[$i]}
2146                                 for file in $(ls *)
2147                                 do
2148                                    ln -f $file  ${localin[$i]}
2149                                 done
2150                                 cd $tempdir
2151                              fi
2152
2153                   # IF "ln -f" FAILED OR IF "$link_local_input = false" DO A NORMAL "cp -r"
2154                              if [[ ! -f "${localin[$i]}/_000000" ]]
2155                              then
2156                                 if [[ $link_local_input = true ]]
2157                                        then
2158                                        printf "\n  --- WARNING: ln failed, using cp instead (might be time consuming...)"
2159                                 fi
2160                                 cp -r  ${absnamein[$i]}/*  ${localin[$i]}
2161                              fi
2162
2163             else
2164
2165                   # PROVIDE FILE FOR RUNS ON A SINGLE CORE
2166                        if [[ $link_local_input = true ]]
2167                        then
2168                                 printf "      file will be linked\n"
2169                                 ln -f  ${absnamein[$i]}  ${localin[$i]}
2170                        fi
2171                                # IF "ln -f" FAILS OR IF "$link_local_input = false" DO A NORMAL "cp"
2172                        if [[ ! -f "${localin[$i]}" ]]
2173                then
2174                                 if [[ $link_local_input = true ]]
2175                          then
2176                      printf "\n  --- WARNING: ln failed, using cp instead (might be time consuming...)"
2177                                 fi
2178                   if [[ $running_on_remote = true  &&  "${actionin[$i]}" = tr ]]
2179                   then
2180                      mv  ${absnamein[$i]}  ${localin[$i]}
2181                   else
2182                                 cp  ${absnamein[$i]}  ${localin[$i]}
2183                   fi
2184                fi
2185             fi
2186          fi
2187       fi
2188
2189    done
2190    if (( i != 0 ))
2191    then
2192       if [[ $optional_files_missing = true ]]
2193       then
2194          printf "\n  *** INFORMATIVE: some optional INPUT-files are not present"
2195       fi
2196       printf "\n$dashes\n  *** all INPUT-files provided \n"
2197    fi
2198
2199       # NOW COPY FILES (*.f90, *.o, config files, etc.) FROM SOURCES_FOR_RUN_... TO THE TEMPORARY
2200       # WORKING DIRECTORY
2201    cp  ${fast_io_catalog}/${sources_for_run_catalog}/{*,.[!.]*}  $tempdir
2202
2203
2204       # EXECUTE INPUT-COMMANDS GIVEN IN THE CONFIGURATION FILE
2205    (( i = 0 ))
2206    while (( i < iic ))
2207    do
2208       (( i = i + 1 ))
2209       if (( i == 1 ))
2210       then
2211          printf "\n\n  *** execution of INPUT-commands:\n$dashes"
2212       fi
2213       printf "\n  >>> ${in_command[$i]}"
2214       eval  ${in_command[$i]}
2215       if (( i == iic ))
2216       then
2217          printf "\n$dashes\n"
2218       fi
2219    done
2220
2221
2222    # CHECK IF THE PROGRESS BAR NEEDS TO BE DISABLED
2223    if [[ $running_in_batch_mode = true  ||  $running_in_test_mode = true ]]
2224    then
2225       progress_bar_disabled=true
2226    else
2227       progress_bar_disabled=false
2228    fi
2229
2230
2231       # CREATE THE NAMELIST-FILE WITH VALUES OF ENVIRONMENT-VARIABLES REQUIRED BY PALM
2232       # (FILE ENVPAR WILL BE READ BY PALM)
2233    cat  >  ENVPAR  <<  EOF
2234 &envpar  run_identifier = '$run_identifier', host = '$configuration_identifier',
2235          write_svf = .${write_svf}., write_binary = .${write_binary}.,
2236          read_svf = .${read_svf}., tasks_per_node = $tasks_per_node,
2237          maximum_parallel_io_streams = $maximum_parallel_io_streams,
2238          maximum_cpu_time_allowed = ${cpumax}.,
2239          revision = '$global_revision',
2240          progress_bar_disabled = .${progress_bar_disabled}. /
2241
2242EOF
2243
2244
2245       # STARTING THE EXECUTABLE
2246    printf "\n\n  *** execution starts in directory\n      \"`pwd`\"\n$dashes\n"
2247    PATH=$PATH:$tempdir
2248
2249
2250       # REPLACE PARAMETERS IN THE EXECUTION COMMAND WITH REAL VALUES
2251    line=`echo  "${execute_command}" | sed 's/{{/$/g' | sed 's/}}//g'`
2252    line2=`echo  "${execute_command}" | sed 's/{{mpi_tasks}}/1/g' | sed 's/{{tasks_per_node}}/1/g' | sed 's/palm/combine_plot_fields.x/g'`
2253    eval line=\"$line\"
2254    execute_command="$line"
2255
2256
2257       # EXECUTION COMMAND FOR COMBINE_PLOT_FIELDS
2258    if [[ "$execute_command_for_combine" = "" ]]
2259    then
2260       eval line2=\"$line2\"
2261       execute_command_for_combine="$line2"
2262    fi
2263   
2264
2265
2266       # PROVIDE A HOSTFILE, IF REQUIRED
2267    if [[ "$hostfile" != "" ]]
2268    then
2269
2270       if [[ $hostfile = auto ]]
2271       then
2272             # CREATE A NEW HOSTFILE
2273          (( ii = 1 ))
2274          while (( ii <= cores / threads_per_task ))
2275          do
2276             echo  $(hostname)  >>  hostfile
2277             (( ii = ii + 1 ))
2278          done
2279          if (( cores / threads_per_task == 0 ))
2280          then
2281             echo  $(hostname)  >>  hostfile
2282          fi
2283       
2284       else
2285          cp  $hostfile  hostfile
2286       fi
2287       eval line=\"`head -n $ii  hostfile`\"
2288       printf "\n  *** running on: $line"
2289    fi
2290
2291
2292
2293       # SET THE NUMBER OF OPENMP-THREADS
2294    if [[ $use_openmp = true ]]
2295    then
2296       export OMP_NUM_THREADS=$threads_per_task
2297       printf "\n  *** number of OpenMP threads per MPI-task: $OMP_NUM_THREADS"
2298    else
2299       export OMP_NUM_THREADS=1
2300    fi
2301
2302
2303       # PROVIDE DATA FOR ATMOSPHERE OCEAN COUPLING
2304    if [[ $run_coupled_model = false ]]
2305    then
2306       if [[ "$ocean_file_appendix" = true ]]
2307       then
2308          echo "precursor_ocean"  >  coupling_steering
2309       else
2310          echo "precursor_atmos"  >  coupling_steering
2311      fi
2312    else
2313       (( iia = $cores_atmos / $threads_per_task ))
2314       (( iio = $cores_ocean / $threads_per_task ))
2315       printf "\n      coupled run ($iia atmosphere, $iio ocean)"
2316       printf "\n\n"
2317       echo "coupled_run $iia $iio"  >  coupling_steering
2318    fi
2319
2320    printf "\n  *** execute command:"
2321    printf "\n      \"$execute_command\" \n\n"
2322
2323
2324    if [[ $progress_bar_disabled = true ]]
2325    then
2326       $execute_command  <  coupling_steering  &> >(grep -v --line-buffered -e '^STOP 1$' -e '^1$' &> >(tee STDOUT) )
2327       exit_code=${PIPESTATUS[0]}
2328    else
2329       $execute_command  <  coupling_steering  &> >(tee STDOUT)
2330       exit_code=${PIPESTATUS[0]}
2331    fi
2332
2333    if [[ ${exit_code} != 0 ]]
2334    then
2335
2336           # ABORT IN CASE OF RUNTIME ERRORS
2337       printf "\n  +++ runtime error occured"
2338       locat=execution
2339       exit
2340
2341    else
2342
2343       printf "\n$dashes\n  *** execution finished \n"
2344
2345    fi
2346
2347
2348       # CALL OF combine_plot_fields IN ORDER TO MERGE SINGLE FILES WRITTEN
2349       # BY EACH CORE INTO ONE FILE
2350   if [[ ! -f combine_plot_fields.x ]]
2351   then
2352
2353      printf "\n\n\n  +++ WARNING: no combine_plot_fields found"
2354      printf "\n      2d- and/or 3d-data may be incomplete!"
2355      printf "\n      Your previous palmbuild may have failed. Please check.\n"
2356
2357   elif [[ "$combine_plot_fields" == true ]]
2358   then
2359
2360      printf "\n\n\n *** post-processing: now executing \"$execute_command_for_combine\" ..."
2361      $execute_command_for_combine
2362
2363   else
2364
2365         # TEMPORARY SOLUTION TO SKIP combine_plot_fields. THIS IS REQUIRED IN CASE OF HUGE AMOUNT OF
2366         # DATA OUTPUT
2367      printf "\n\n\n *** post-processing: skipping combine_plot_fields (-Z option set) ..."
2368   fi
2369
2370
2371
2372       # EXECUTE OUTPUT-COMMANDS GIVEN IN THE CONFIGURATION FILE
2373    (( i = 0 ))
2374    while (( i < ioc ))
2375    do
2376       (( i = i + 1 ))
2377       if (( i == 1 ))
2378       then
2379          printf "\n\n  *** execution of OUTPUT-commands:\n$dashes"
2380       fi
2381
2382          # REPLACE PARAMETERS IN THE OUTPUT COMMAND WITH REAL VALUES
2383       out_command[$i]=`echo  "${out_command[$i]}" | sed 's/{{/$/g' | sed 's/}}//g'`
2384
2385       printf "\n  >>> ${out_command[$i]}"
2386       eval  ${out_command[$i]}
2387       if (( i == ioc ))
2388       then
2389          printf "\n$dashes\n"
2390       fi
2391    done
2392
2393
2394       # IN A FIRST PASS, ADD ADDITIONAL OUTPUT FILE CONNECTIONS IN CASE OF
2395       # WILDCARDS
2396    (( i = 0 ))
2397    (( nr_of_output_files = 0 ))
2398
2399    while (( i < iout ))
2400    do
2401
2402       (( i = i + 1 ))
2403
2404          # FIRST CHECK FOR MULTIPLE NAMES WITH THE SAME LOCAL NAME AND
2405          # CREATE A LIST FOR THE DETECTED ENDINGS
2406       if [[ "${multout[$i]}" = true ]]
2407       then
2408             # DETERMINE THE EXISTING EXTENSIONS FROM THE LIST OF FILES
2409          ls -1 -d ${localout_pre[$i]}    >   filelist  2>/dev/null
2410          ls -1 -d ${localout_pre[$i]}_*  >>  filelist  2>/dev/null
2411
2412          endings="DEFAULT"
2413          while  read line
2414          do
2415                 # remove the local name from the beginning
2416              localnamestring="${localout_pre[$i]}"
2417              length_localname=${#localnamestring}
2418              ending=${line:${length_localname}}
2419
2420              if [[ "$ending" != "" ]]
2421              then
2422                 endings="$endings $ending"
2423              fi
2424 
2425          done <filelist
2426
2427          rm filelist
2428
2429       else
2430
2431             # SINGLE NAME
2432          endings=DEFAULT
2433
2434       fi
2435
2436          # FOR EACH BASENAME ENDING CREATE AN ENTRY IN THE FINAL OUTPUT FILE LIST
2437       for  ending  in  $endings
2438       do
2439
2440             # DEFAULT MEANS THAT THE ENDING GIVEN IN .iofiles SHALL BE USED
2441          if [[ $ending = DEFAULT ]]
2442          then
2443             ending=""
2444          fi
2445
2446             # NEW ENTRY (ENDING IS ALSO ADDED TO LOCAL FILENAME READ BY PALM!)
2447          (( nr_of_output_files = nr_of_output_files + 1 ))
2448          localout[$nr_of_output_files]="${localout_pre[$i]}"$ending
2449          transout[$nr_of_output_files]="${transout_pre[$i]}"
2450          actionout[$nr_of_output_files]="${actionout_pre[$i]}"
2451          pathout[$nr_of_output_files]="${pathout_pre[$i]}"
2452          endout[$nr_of_output_files]="${endout_pre[$i]}"$ending
2453          extout[$nr_of_output_files]="${extout_pre[$i]}"
2454          warnout[$nr_of_output_files]="${warnout_pre[$i]}"
2455
2456       done
2457
2458    done
2459
2460
2461
2462
2463       # COPY LOCAL OUTPUT-FILES TO THEIR PERMANENT DESTINATIONS
2464    (( i = 0 ))
2465    while (( i < nr_of_output_files ))
2466    do
2467       (( i = i + 1 ))
2468       if (( i == 1 ))
2469       then
2470          timestring=`date +%H:%M:%S`
2471          printf "\n\n  *** saving OUTPUT-files:     local time: $timestring"
2472
2473             # GET RUN NUMBER ASSIGNED BY PALM
2474          if [[ -f RUN_NUMBER ]]
2475          then
2476              read  run_number  <  RUN_NUMBER
2477              printf "\n  *** PALM generated run_number = "$run_number" will be used as unified cycle number for all output files"
2478              usecycle_option="-U $run_number"
2479          else
2480              run_number=0
2481              usecycle_option=""
2482          fi
2483          if [[ $running_on_remote = true  &&  "$remote_loginnode" != "" ]]
2484          then
2485             printf "\n  *** in case of SCP transfers to local host"
2486             printf "\n      they will be done via remote login-node \"$remote_loginnode\" "
2487          fi
2488          printf "\n$dashes"
2489       fi
2490
2491       if [[ ! ( $running_on_remote = true  &&  ( "${actionout[$i]}" = tr || "${actionout[$i]}" = tra || "${actionout[$i]}" = trpe ) ) ]]
2492       then
2493
2494          eval filename=${pathout[$i]}/${run_identifier}${endout[$i]}
2495
2496             # DETERMINE THE CYCLE NUMBER
2497          ls -1 -d $filename    >   filelist  2>/dev/null
2498          ls -1 -d $filename.*  >>  filelist  2>/dev/null
2499          (( maxcycle = 0 ))
2500          while  read line
2501          do
2502   
2503                # filename without path (i.e. after the last "/")
2504             basefilename=$(basename ${line})
2505   
2506                # check if there is an extension
2507             extension=${basefilename##*.}
2508             if [[ "$extension" = "${extout[$i]}" ]]
2509             then
2510                basefilename=${basefilename%.*}
2511             fi
2512   
2513                # check for an existing cycle number
2514             cycle=${basefilename##*.}
2515             if [[ $cycle =~ ^-?[0-9]+$ ]]
2516             then
2517                   # NUMBERS WITH LEADING ZEROS ARE INTERPRETED AS OCTAL NUMBERS
2518                   # 10# EXPLICITLY SPECIFIES THE NUMBER BASE AS 10
2519                (( icycle = $((10#$cycle)) + 1 ))
2520             else
2521                (( icycle = 1 ))
2522             fi
2523   
2524             if (( icycle > maxcycle ))
2525             then
2526                (( maxcycle = icycle ))
2527             fi
2528   
2529          done <filelist
2530          rm filelist
2531
2532   
2533             # SET THE CYCLE NUMBER
2534             # IN CASE OF FILE-APPEND, IT MUST BE THE HIGHEST EXISTING CYCLE NUMBER
2535          if [[ "${actionout[$i]}" = a ]]
2536          then
2537             if (( maxcycle > 0 ))
2538             then
2539                (( maxcycle = maxcycle - 1 ))
2540             fi
2541          fi
2542         
2543          (( cycnum[$i] = maxcycle ))
2544          pathout[$i]=$filename
2545
2546
2547             # ADD CYCLE NUMBER TO FILENAME
2548             # IN APPEND MODE, FILES KEEP THEIR CURRENT CYCLE NUMBER
2549          if [[ "${actionout[$i]}" != "a" ]]
2550          then
2551                # SET RUN NUMBER AS CYCLE NUMBER, IF THERE IS NOT A CONFLICT
2552                # WITH AN EXISTING CYCLE NUMBER
2553             if (( run_number >= cycnum[$i] ))
2554             then
2555                (( cycnum[$i] = run_number ))
2556             else
2557                if (( run_number > 0 ))
2558                then
2559                   printf "\n  --- INFORMATIVE: The following file cannot get a unified cycle number"
2560                fi
2561             fi
2562          fi
2563          cyclestring=`printf "%03d" ${cycnum[$i]}`
2564          pathout[$i]=${pathout[$i]}.$cyclestring
2565       fi
2566
2567          # CHECK FOR SINGLE FILE (SERIAL RUN) OR DIRECTORY (ONE FILE PER CORE FOR PARELLEL EXECUTION)
2568       files_for_cores=false; filetype=file
2569       link_local_output=false
2570       if [[ "${actionout[$i]}" = pe  &&  -n $cores ]]
2571       then
2572          files_for_cores=true; filetype=directory
2573          actionout[$i]=""
2574       elif [[ "${actionout[$i]}" = pe  &&  ! -n $cores ]]
2575       then
2576          actionout[$i]=""
2577       elif [[ "${actionout[$i]}" = lnpe  &&  -n $cores ]]
2578       then
2579          files_for_cores=true; filetype=directory
2580          link_local_output=true
2581          actionout[$i]=""
2582             # FILE HAS BEEN FOUND INSTEAD OF DIRECTORY
2583          if [[ -f ${localout[$i]} ]]
2584          then
2585             files_for_cores=false; filetype=file
2586          fi
2587       elif [[ "${actionout[$i]}" = lnpe  &&  ! -n $cores ]]
2588       then
2589          link_local_output=true
2590          actionout[$i]=""
2591       elif [[ "${actionout[$i]}" = trpe  &&  -n $cores ]]
2592       then
2593          files_for_cores=true; filetype=directory
2594          actionout[$i]="tr"
2595       elif [[ "${actionout[$i]}" = trpe  &&  ! -n $cores ]]
2596       then
2597          actionout[$i]="tr"
2598       fi
2599
2600       if [[ ! -f ${localout[$i]}  &&  $files_for_cores = false ]]
2601       then
2602          if [[ ${warnout[$i]} = true ]]
2603          then
2604             printf "\n  +++ temporary OUTPUT-file  ${localout[$i]}  does not exist\n"
2605          fi
2606       elif [[ ! -d ${localout[$i]}  &&  $files_for_cores = true ]]
2607       then
2608          if [[ ${warnout[$i]} = true ]]
2609          then
2610             printf "\n  +++ temporary OUTPUT-file  ${localout[$i]}/....  does not exist\n"
2611          fi
2612       else
2613
2614
2615             # COPY VIA SCP TO LOCAL HOST (ALWAYS IN BINARY MODE USING batch_scp option -m)
2616             # IF TARGET DIRECTORY DOES NOT EXISTS, TRY TO CREATE IT
2617          if [[ "${actionout[$i]}" = tr  ||  "${actionout[$i]}" = tra ]]
2618          then
2619             if [[ $running_on_remote = true ]]
2620             then
2621
2622                   # SET OPTIONS FOR TRANSFER
2623                if [[ "${actionout[$i]}" = tr ]]
2624                then
2625                   if [[ $files_for_cores = false ]]
2626                   then
2627                      catalog_option=""
2628                      catalog_string=""
2629                   else
2630                      catalog_option="-c"
2631                      catalog_string="/"
2632                   fi
2633                   append_option=""
2634                   append_string=""
2635                else
2636                   append_option="-A"
2637                   append_string="append"
2638                fi
2639
2640                transfer_failed=false
2641                printf "\n  >>> OUTPUT: ${localout[$i]}$catalog_string  $append_string by SCP to"
2642                printf "\n              ${pathout[$i]}/${configuration_identifier}_${run_identifier}${endout[$i]}$catalog_string\n"
2643
2644                   # TRANSFER VIA SCP
2645                if [[ "$remote_loginnode" != "" ]]
2646                then
2647                   ssh -q $remote_username@$remote_loginnode  "cd $tempdir; ${fast_io_catalog}/${sources_for_run_catalog}/batch_scp $PORTOPT $catalog_option $append_option -b -m $usecycle_option -u $local_username $return_address  ${localout[$i]} \"${pathout[$i]}\" ${configuration_identifier}_${run_identifier}${endout[$i]}  ${extout[$i]}"
2648                else
2649                   batch_scp $PORTOPT $catalog_option $append_option -b -m $usecycle_option -u $local_username $return_address  ${localout[$i]} "${pathout[$i]}" ${configuration_identifier}_${run_identifier}${endout[$i]}  ${extout[$i]}
2650                fi
2651                [[ ${PIPESTATUS[0]} != 0 ]]  &&  transfer_failed=true
2652
2653
2654                   # IF TRANSFER FAILED, CREATE BACKUP COPY ON THIS MACHINE
2655                if [[ $transfer_failed = true ]]
2656                then
2657                   printf "  +++ transfer failed. Trying to save a copy on this host under:\n"
2658                   printf "      ${pathout[$i]}/${configuration_identifier}_${run_identifier}${endout[$i]}_$run_id_number\n"
2659
2660                      # FIRST CHECK, IF DIRECTORY EXISTS, AND CREATE IT, IF NECESSARY
2661                   eval  local_catalog=${pathout[$i]}
2662                   if [[ ! -d $local_catalog ]]
2663                   then
2664                      printf "  *** local directory does not exist. Trying to create:\n"
2665                      printf "      $local_catalog \n"
2666                      mkdir -p  $local_catalog
2667                   fi
2668                   eval  cp  ${localout[$i]}  ${pathout[$i]}/${configuration_identifier}_${run_identifier}${endout[$i]}_$run_id_number
2669                   transfer_problems=true
2670                fi
2671
2672             else
2673
2674                   # UNSET actionout. DUE TO THIS SETTING, FILE WILL LATER JUST
2675                   # BE COPIED OR APPENDED ON THIS MACHINE
2676                if [[ "${actionout[$i]}" = tr ]]
2677                then
2678                   actionout[$i]=""
2679                else
2680                   actionout[$i]="a"
2681                fi
2682             fi
2683          fi
2684
2685
2686             # APPEND ON THIS MACHINE
2687          if [[ "${actionout[$i]}" = "a" ]]
2688          then
2689             if [[ "${extout[$i]}" != " "  &&  "${extout[$i]}" != "" ]]
2690             then
2691                printf "\n  >>> OUTPUT: ${localout[$i]}  append to"
2692                printf "\n              ${pathout[$i]}.${extout[$i]}\n"
2693                cat  ${localout[$i]}  >>  ${pathout[$i]}.${extout[$i]}
2694             else
2695                printf "\n  >>> OUTPUT: ${localout[$i]}  append to"
2696                printf "\n              ${pathout[$i]}\n"
2697                cat  ${localout[$i]}  >>  ${pathout[$i]}
2698             fi
2699          fi
2700
2701             # COPY ON THIS MACHINE
2702             # COPY HAS TO BE USED, BECAUSE MOVE DOES NOT WORK IF FILE-ORIGIN AND TARGET ARE
2703             # ON DIFFERENT FILE-SYSTEMS
2704          if [[ "${actionout[$i]}" = ""  &&  $files_for_cores = false ]]
2705          then
2706
2707                # COPY IN CASE OF RUNS ON SINGLE CORES
2708             if [[ "${extout[$i]}" != " "  &&  "${extout[$i]}" != "" ]]
2709             then
2710                printf "\n  >>> OUTPUT: ${localout[$i]}  to"
2711                printf "\n              ${pathout[$i]}.${extout[$i]}\n"
2712                if [[ $link_local_output = true ]]
2713                then
2714                   printf "      file will be linked\n"
2715                   ln -f  ${localout[$i]}  ${pathout[$i]}.${extout[$i]}
2716                fi
2717                   # If "ln -f" fails of if "$link_local_output = false" do a normal "cp"
2718                if [[ ! -f "${pathout[$i]}.${extout[$i]}" ]]
2719                then
2720                   if [[ $link_local_output = true ]]
2721                   then
2722                      printf "  --- WARNING: ln failed, using cp instead (might be time consuming...)\n"
2723                   fi
2724                   cp  ${localout[$i]}  ${pathout[$i]}.${extout[$i]}
2725                fi
2726             else
2727                printf "\n  >>> OUTPUT: ${localout[$i]}  to"
2728                printf "\n              ${pathout[$i]}\n"
2729                if [[ $link_local_output = true ]]
2730                then
2731                   printf "      file will be linked\n"
2732                   ln -f  ${localout[$i]}  ${pathout[$i]}
2733                fi
2734                   # If "ln -f" fails of if "$link_local_output = false" do a normal "cp"
2735                if [[ ! -f "${pathout[$i]}" ]]
2736                then
2737                   if [[ $link_local_output = true ]]
2738                   then
2739                      printf "  --- WARNING: ln failed, using cp instead (might be time consuming...)\n"
2740                   fi
2741                   cp  ${localout[$i]}  ${pathout[$i]}
2742                fi
2743             fi
2744
2745          elif [[ "${actionout[$i]}" = ""  &&  $files_for_cores = true ]]
2746          then
2747
2748                # FILES FROM THE DIFFERENT CORES ARE MOVED WITH ln-COMMAND TO THE PERMANENT DIRECTORY
2749                # AS A FIRST STEP, THE PERMANENT DIRECTORY IS CREATED
2750             printf "\n  >>> OUTPUT: ${localout[$i]}/_....  to"
2751             printf "\n              ${pathout[$i]}\n"
2752             if [[ $link_local_output = true ]]
2753             then
2754                printf "      files will be linked\n"
2755                mkdir -p ${pathout[$i]}
2756                cd ${localout[$i]}
2757                for file in $(ls *)
2758                do
2759                   ln -f $file  ${pathout[$i]}
2760                done >|/dev/null 2>&1
2761                cd $tempdir
2762             fi
2763
2764                # IF "ln -f" HAS FAILED OR IF "$link_local_output = false" DO A NORMAL COPY "cp -r"
2765             if [[ ! -f "${pathout[$i]}/_000000" ]]
2766             then
2767                if [[ $link_local_output = true ]]
2768                then
2769                   printf "  --- WARNING: ln failed, using cp instead (might be time consuming...)\n"
2770                fi
2771                [[ ! -d "${pathout[$i]}" ]]  &&  mkdir  ${pathout[$i]}
2772                cp -r  ${localout[$i]}/*  ${pathout[$i]}
2773             fi
2774
2775          fi
2776       fi
2777    done
2778
2779    if (( i != 0 ))
2780    then
2781       timestring=`date +%H:%M:%S`
2782       if [[ $transfer_problems = true ]]
2783       then
2784          printf "\n$dashes\n  *** OUTPUT-files saved       local time: $timestring"
2785          printf "\n  +++ WARNING: some data transfers failed! \n"
2786       else
2787          printf "\n$dashes\n  *** all OUTPUT-files saved       local time: $timestring \n"
2788       fi
2789    fi
2790
2791
2792       # IF REQUIRED, START A RESTART-JOB
2793       # FILE CONTINUE_RUN MUST HAVE BEEN CREATED BY THE EXECUTABLE (PALM)
2794    if [[ -f CONTINUE_RUN ]]
2795    then
2796
2797          # ADD RESTART-OPTIONS TO THE PALMRUN-CALL (IF THEY ARE NOT USED ALREADY):
2798          # -C TELLS PALMRUN THAT IT IS A RESTART-RUN
2799          # -v SILENT MODE WITHOUT INTERACTIVE QUERIES
2800          # -b START A BATCH JOB
2801       [[ $(echo $prc | grep -c "\-C") = 0 ]]  &&  prc="$prc -C"
2802       [[ $(echo $prc | grep -c "\-v") = 0 ]]  &&  prc="$prc -v"
2803       [[ $(echo $prc | grep -c "\-b") = 0  &&  $running_on_remote = false ]]  &&  prc="$prc -b"
2804
2805          # REPLACE SVFOUT IN THE ACTIVATION STRINGS (GIVEN WITH OPTION -a)
2806          # SO THAT RESTARTS USE SVF DATA CREATED BY THE INITIAL RUN
2807       if [[ $(echo $prc | grep -c "svfout") != 0 ]]
2808       then
2809          prc=`echo $prc | sed 's/svfout/svfin/g'`
2810       fi
2811
2812          # REPLACE THE HASH IN THE ACTIVATION STRINGS (GIVEN WITH OPTION -a)
2813          # SO THAT RESTARTS ACCESS DIFFERENT FILES THAN THE INITIAL RUN
2814       if [[ $(echo $prc | grep -c "#") != 0 ]]
2815       then
2816          prc=`echo $prc | sed 's/#/r/g'`
2817       fi
2818
2819
2820          # START THE RESTART-JOB
2821       printf "\n\n  *** initiating restart-run on \"$local_ip\" using command:\n"
2822       echo "      $prc"
2823       printf "\n$dashes\n"
2824       if [[ $running_on_remote = true ]]
2825       then
2826
2827          echo "*** ssh will be used to initiate restart-runs!"
2828          echo "    return_address=\"$return_address\" "
2829          echo "    return_username=\"$local_username\" "
2830
2831          if [[ "$remote_loginnode" != "" ]]
2832          then
2833             ssh -q $remote_username@$remote_loginnode  "ssh -q $SSH_PORTOPT  $local_username@$return_address \" PATH=\\\$PATH:$LOCAL_PALMRUN_PATH; cd $LOCAL_PWD; $prc\" " |  tee palmrun_restart.log
2834          else
2835             ssh -q $SSH_PORTOPT  $local_username@$return_address  " PATH=\\\$PATH:$LOCAL_PALMRUN_PATH; cd $LOCAL_PWD; $prc" |  tee palmrun_restart.log
2836          fi
2837
2838             # WAIT TO ALLOW THE RESTART-JOB TO BE QUEUED, BEFORE THE CURRENT JOB IS FINISHED
2839          sleep 30
2840
2841       else
2842
2843             # START THE RESTART JOB ON THE LOCAL HOST
2844          eval  $prc  |  tee palmrun_restart.log              # THE ' MUST BE EVALUATED
2845
2846       fi
2847
2848          # CHECK, IF RESTART JOB HAS BEEN STARTED
2849       if [[ $( grep -c "+++ palmrun crashed"  palmrun_restart.log ) != 0 ]]
2850       then
2851          printf "\n$dashes\n  +++ creating restart run failed \n"
2852          locat=create_restart
2853          rm  palmrun_restart.log
2854          exit
2855       elif [[ $( grep -c "*** palmrun finished"  palmrun_restart.log ) != 1 ]]
2856       then
2857          printf "\n$dashes\n  +++ creating restart run failed, probably due to network problems\n"
2858          locat=create_restart
2859          rm  palmrun_restart.log
2860          exit       
2861       else
2862          printf "\n$dashes\n  *** restart run initiated \n"
2863          rm  palmrun_restart.log
2864       fi
2865
2866
2867          # DELETE INPUT-(RESTART)FILES, WHICH HAVE BEEN FETCHED FROM THE TEMPORARY DATA
2868          # DIRECTORY, BACAUSE THEY ARE NOT REQUIRED BY THE RESTART-JOB.
2869          # THIS IS DONE IN ORDER TO AVOID EXCEEDING DISC QUOTAS OR DISC SPACE (RESTART-FILES
2870          # MAY BE VERY HUGE)
2871       (( i = 0 ))
2872       while (( i < nr_of_input_files ))
2873       do
2874          (( i = i + 1 ))
2875          if [[ "${got_tmp[$i]}" = true   &&  $keep_data_from_previous_run = false ]]
2876          then
2877             rm -r  ${absnamein[$i]}
2878          fi
2879       done
2880
2881    fi
2882
2883
2884       # ALL ACTIONS FINISHED, TEMPORARY WORKING-DIRECTORY CAN BE DELETED
2885    cd  $HOME
2886    [[ $delete_temporary_catalog = true ]]  &&  rm -rf $tempdir
2887
2888
2889 else
2890
2891
2892       # PREPARING ACTIONS,
2893       # IF A BATCH-JOB IS TO BE GENERATED AND TO BE STARTED ON A LOCAL OR REMOTE-MACHINE
2894
2895       # BUILD THE PALMRUN-COMMAND TO BE CALLED IN THE BATCH-JOB
2896    palmrun_com="$palmrun_script_name -r $run_identifier -c $configuration_identifier -m $memory -t $cpumax -q $queue -i $run_id_number -U $local_username"
2897    [[ "$activation_string_list" != "" ]]  &&  palmrun_com=${palmrun_com}" -a \"$activation_string_list\""
2898    [[ "$global_revision" != "" ]]  &&  palmrun_com=${palmrun_com}" -G \"$global_revision\""
2899    [[ $keep_data_from_previous_run = true ]]  &&  palmrun_com=${palmrun_com}" -k"
2900    [[ $do_trace = true ]]        &&  palmrun_com=${palmrun_com}" -x"
2901    [[ "$cores" != "" ]]       &&  palmrun_com=${palmrun_com}" -X $cores"
2902    [[ $use_openmp = true ]]  &&  palmrun_com=${palmrun_com}" -O $threads_per_task"
2903    [[ $tasks_per_node != 0 ]]  &&  palmrun_com=${palmrun_com}" -T $tasks_per_node"
2904    [[ $delete_temporary_catalog = false ]]  &&  palmrun_com=${palmrun_com}" -B"
2905    [[ "$ocean_file_appendix" = true ]]  &&  palmrun_com=${palmrun_com}" -y"
2906    [[ $run_coupled_model = true ]]  &&  palmrun_com=${palmrun_com}" -Y \"$coupled_dist\""
2907    [[ "$combine_plot_fields" = false ]]  &&  palmrun_com=${palmrun_com}" -Z"
2908    [[ "$max_par_io_str" != "" ]]  &&  palmrun_com=${palmrun_com}" -w $max_par_io_str"
2909    [[ "$project_account" != "" ]]  &&  palmrun_com=${palmrun_com}" -A $project_account"
2910    if [[ $create_remote_batch_job = true ]]
2911    then
2912       palmrun_com=${palmrun_com}" -j -u $remote_username -R $local_ip"
2913       if [[ $do_trace = true ]]
2914       then
2915          printf "\n *** PALMRUN-command on remote host:\n     $palmrun_com \n"
2916       fi
2917    elif [[ $create_batch_job = true ]]
2918    then
2919       palmrun_com=${palmrun_com}" -j"
2920       if [[ $do_trace = true ]]
2921       then
2922          printf "\n *** PALMRUN-command on local host:\n     $palmrun_com \n"
2923       fi
2924    fi
2925
2926
2927       # DETERMINE THE FULL PATHS FOR THE JOB PROTOCOL FILES ON THE LOCAL AND
2928       # REMOTE HOST
2929    job_protocol_file_local=${local_jobcatalog}/${configuration_identifier}_${run_id}
2930    job_protocol_file=$job_protocol_file_local
2931    if [[ $create_remote_batch_job = true ]]
2932    then
2933       job_protocol_file_remote=${remote_jobcatalog}/${configuration_identifier}_${run_id}
2934       job_protocol_file=$job_protocol_file_remote
2935       job_transfer_protocol_file=${remote_jobcatalog}/last_job_transfer_protocol
2936       scpjob_file=${remote_jobcatalog}/scpjob.$run_id_number
2937    fi
2938
2939
2940       # BUILD THE JOB-SCRIPTS ON FILE jobfile
2941    jobfile=jobfile.$run_id_number
2942
2943
2944       # FIRST CREATE THE BATCH DIRECTIVES
2945    (( i = 0 ))
2946    while (( i < ibd ))
2947    do
2948       (( i = i + 1 ))
2949       line=`echo  "${batch_directive[$i]}" | sed 's/{{/$/g' | sed 's/}}//g'`
2950       eval line=\"$line\"
2951       echo  "$line"                               >>  $jobfile
2952    done
2953    echo " "                                       >>  $jobfile
2954
2955
2956       # FOR BATCH JOBS ON REMOTE HOSTS, ADD THE JOBFILE TO SEND BACK THE JOB
2957       # PROTOCOL
2958    if [[ $create_remote_batch_job = true ]]
2959    then
2960       echo "set +vx"                                   >>  $jobfile
2961       echo "trap '"                                    >>  $jobfile
2962       echo "set +vx"                                   >>  $jobfile
2963       echo "cd ${remote_jobcatalog}"                   >>  $jobfile
2964       echo "cat >  scpjob.$run_id_number << %%END%%"   >>  $jobfile
2965
2966          # ADD THE BATCH DIRECTIVES
2967       (( i = 0 ))
2968       while (( i < ibdt ))
2969       do
2970          (( i = i + 1 ))
2971          line=`echo  "${batch_directive_transfer[$i]}" | sed 's/{{/$/g' | sed 's/}}//g'`
2972          eval line=\"$line\"
2973          echo  "$line"                            >>  $jobfile
2974       done
2975       echo " "                                    >>  $jobfile
2976
2977       echo "set -x"                               >>  $jobfile
2978       if [[ "$remote_loginnode" != "" ]]
2979       then
2980          echo "ssh -q $remote_username@$remote_loginnode  \"cd ${remote_jobcatalog}; ${fast_io_catalog}/${sources_for_run_catalog}/batch_scp  $PORTOPT  -d  -w 10  -u $local_username $local_ip  $job_protocol_file_remote \\\"$local_jobcatalog\\\"  ${configuration_identifier}_${run_identifier}\" "  >> $jobfile
2981       else
2982          echo "${fast_io_catalog}/${sources_for_run_catalog}/batch_scp  $PORTOPT  -d  -w 10  -u $local_username $local_ip  $job_protocol_file_remote  \"$local_jobcatalog\"  ${configuration_identifier}_${run_identifier}"  >>  $jobfile
2983       fi
2984       echo "%%END%%"                              >>  $jobfile
2985       echo "echo \" *** submitting job for transfering the job protocol file to $local_ip\" "  >>  $jobfile
2986       echo "$submit_command  $scpjob_file"     >>  $jobfile
2987       echo "rm $scpjob_file"                      >>  $jobfile
2988       echo "rm -rf $job_transfer_protocol_file"   >>  $jobfile
2989       echo "set -x"                               >>  $jobfile
2990       echo "     ' exit"                          >>  $jobfile
2991    fi
2992
2993
2994       # ACTIVATE ERROR-TRACEBACK
2995    if [[ $do_trace = true ]]
2996    then
2997       echo  "set -x"                                    >>  $jobfile
2998    else
2999       echo  "set +vx"                                   >>  $jobfile
3000    fi
3001
3002
3003       # INITIALIZE THE ENVIRONMENT AND LOAD MODULES
3004    if [[ "$login_init_cmd" != "" ]]
3005    then
3006       echo  "$login_init_cmd"        >>  $jobfile
3007    fi
3008    if [[ "$module_commands" != "" ]]
3009    then
3010       echo  "$module_commands"     >>  $jobfile
3011    fi
3012
3013
3014       # CREATE TEMPORARY DIRECTORY AND SWITCH TO IT
3015    if [[ $create_remote_batch_job = true ]]
3016    then
3017       echo  "mkdir  $tempdir"           >>  $jobfile
3018       echo  "if [[ \$? != 0 ]]"         >>  $jobfile
3019       echo  "then"                      >>  $jobfile
3020       echo  "   echo \"+++ temporary working directory cannot be created.\" "  >>  $jobfile
3021       echo  "   echo \"    Check setting of variable fast_io_catalog in your config file.\" "  >>  $jobfile
3022       echo  "   exit"                   >>  $jobfile
3023       echo  "fi"                        >>  $jobfile
3024       echo  "chmod  go+rx  $tempdir"    >>  $jobfile
3025    else
3026          # DIRECTORY FOR LOCAL BATCH JOBS IS CREATED NOW, DUE TO A
3027          # REQUIREMENT OF THE GRID ENGINE BATCH SYSTEM (WORKING DIR IS GIVEN IN
3028          # BATCH DIRECTIVE -wd AND MUST ALREADY EXIST WHEN THE JOB IS SUBMITTED)
3029       mkdir  $tempdir
3030       if [[ $? != 0 ]]
3031       then
3032          printf "\n  +++ temporary working directory cannot be created."
3033          printf "\n      Check setting of variable fast_io_catalog in your config file."
3034          locat=execution
3035          exit
3036       fi
3037       chmod  go+rx  $tempdir
3038    fi
3039    echo  "cd  $tempdir"                 >>  $jobfile
3040    echo  "export tempdir=$tempdir"      >>  $jobfile
3041    echo  "cp  ${fast_io_catalog}/${sources_for_run_catalog}/.[!.]*  ."           >>  $jobfile
3042    echo  "export PATH=.:${fast_io_catalog}/${sources_for_run_catalog}:\$PATH"    >>  $jobfile
3043    echo  "export execute_palmrun=true"  >>  $jobfile
3044
3045
3046       # PROVIDE NAME OF THE CURRENT WORKING-DIRECTORY ON THE LOCAL MACHINE (FROM WHERE THE JOB IS
3047       # STARTED) BY SETTING AN ENVIRONMENT-VARIABLE. THIS INFORMATION IS USED IN THE JOB BY PALMRUN
3048       # IN CASE THAT RESTART-RUNS HAVE TO BE GENERATED
3049    echo  "LOCAL_PWD=$working_directory"                >>  $jobfile
3050    echo  "export LOCAL_PWD"                            >>  $jobfile
3051
3052
3053       # PROVIDE THE PATH OF THE LOCAL PALMRUN-SCRIPT FOR THE SAME REASON
3054    echo  "LOCAL_PALMRUN_PATH=${source_path}/../SCRIPTS"   >>  $jobfile
3055    echo  "export LOCAL_PALMRUN_PATH"                      >>  $jobfile
3056
3057
3058       # CALL PALMRUN WITHIN THE JOB
3059       # AS FINAL ACTION, REMOVE THE TEMPORARY DIRECTORY CREATED AT THE BEGINNING OF THE JOB
3060    echo  "set -x"                                              >>  $jobfile
3061    echo  "[[ \$execute_palmrun = true ]]  &&  $palmrun_com"    >>  $jobfile
3062
3063
3064       # TRANSFER JOBFILE TO THE TARGET HOST
3065    if [[ $create_jobfile_only = false ]]
3066    then
3067
3068          # CREATE LOCAL JOB FOLDER
3069       eval  local_jobcatalog=$local_jobcatalog
3070       if [[ ! -d $local_jobcatalog ]]
3071       then
3072          echo " "
3073          echo " *** local jobcatalog \"$local_jobcatalog\" does not exist and will be created now"
3074          mkdir  $local_jobcatalog
3075       fi
3076 
3077       if [[ $create_remote_batch_job = true ]]
3078       then
3079
3080          echo " "
3081          echo " *** transfer of job to remote host via scp"
3082
3083             # CREATE REMOTE JOB FOLDER, IF IT DOES NOT EXIST
3084          ssh  -q  $ssh_key $SSH_PORTOPT  ${remote_username}@${remote_ip}  "[[ ! -d $remote_jobcatalog ]]  &&  mkdir $remote_jobcatalog"  2>&1
3085
3086             # COPY THE JOB FILE
3087          if [[ $do_trace = true ]]
3088          then
3089             echo "     scp $ssh_key $PORTOPT $jobfile ${remote_username}@${remote_ip}:${remote_jobcatalog}/${configuration_identifier}_${run_id}"
3090          fi
3091          scp $ssh_key $PORTOPT $jobfile ${remote_username}@${remote_ip}:${remote_jobcatalog}/${configuration_identifier}_${run_id}  >  /dev/null
3092
3093             # SUBMIT THE JOB
3094          printf " *** submit the job (output of submit command, \n"
3095          printf "     e.g. the job-id given by the batch-system, may follow) \n"
3096          if [[ $do_trace = true ]]
3097          then
3098             echo "     ssh  -q  $ssh_key $SSH_PORTOPT  ${remote_username}@${remote_ip} \"cd $remote_jobcatalog; $submit_command ${configuration_identifier}_${run_id}; rm ${configuration_identifier}_${run_id}\" "
3099          fi
3100          ssh  -q  $ssh_key $SSH_PORTOPT  ${remote_username}@${remote_ip} "cd $remote_jobcatalog; $submit_command ${configuration_identifier}_${run_id}; rm ${configuration_identifier}_${run_id}"
3101
3102       elif [[ $create_batch_job = true ]]
3103       then
3104
3105          eval  local_jobcatalog=$local_jobcatalog
3106          cp  $jobfile  ${local_jobcatalog}/${configuration_identifier}_${run_id}
3107          cd $local_jobcatalog
3108          echo " "
3109          echo " *** submit the job"
3110          if [[ $do_trace = true ]]
3111          then
3112             echo "$submit_command ${configuration_identifier}_${run_id}"
3113          fi
3114          $submit_command ${configuration_identifier}_${run_id}
3115          rm ${configuration_identifier}_${run_id}
3116          cd -  > /dev/null
3117
3118       fi
3119
3120       rm -rf  $jobfile
3121
3122    else
3123
3124       printf "\n *** jobfile created under name \"$jobfile\" "
3125       printf "\n     no batch-job has been sent!"
3126
3127    fi
3128
3129 fi  # END OF REMOTE-PART
Note: See TracBrowser for help on using the repository browser.