Home

Context Navigation

source: palm/trunk/SCRIPTS/mrun @ 2261

Last change on this file since 2261 was 2261, checked in by raasch, 7 years ago
changes in mrun: unified cycle numbers for output files are used, paths and filenames are allowed to contain arbitrary numbers of dots, archive feature completely removed from the script, nech related parts completely removed, OpenMP bugfix in prognostic_equations
Property svn:keywords set to `Id Rev`
File size: 142.8 KB

Line
1	#!/bin/bash
2
3	# mrun - script for running PALM jobs
4
5	#--------------------------------------------------------------------------------#
6	# This file is part of PALM.
7	#
8	# PALM is free software: you can redistribute it and/or modify it under the terms
9	# of the GNU General Public License as published by the Free Software Foundation,
10	# either version 3 of the License, or (at your option) any later version.
11	#
12	# PALM is distributed in the hope that it will be useful, but WITHOUT ANY
13	# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
14	# A PARTICULAR PURPOSE. See the GNU General Public License for more details.
15	#
16	# You should have received a copy of the GNU General Public License along with
17	# PALM. If not, see <http://www.gnu.org/licenses/>.
18	#
19	# Copyright 1997-2015 Leibniz Universitaet Hannover
20	#--------------------------------------------------------------------------------#
21	#
22	# Current revisions:
23	# ------------------
24	#
25	#
26	# Former revisions:
27	# -----------------
28	# $Id: mrun 2261 2017-06-08 14:25:57Z raasch $
29	# unified cycle numbers for output files are used,
30	# paths and filenames are allowed to contain arbitrary numbers of dots ".",
31	# creation of file OUTPUT_FILE_CONNECTIONS removed,
32	# archive feature completely removed from the script,
33	# nech related parts completely removed
34	#
35	# 2257 2017-06-07 14:07:05Z witha
36	# adjustments for lceddy, removed lcflow-specific code
37	#
38	# 2186 2017-03-21 05:36:31Z raasch
39	#
40	# 2185 2017-03-21 05:09:29Z raasch
41	# adjustment for lykyo*-hosts (PJM_ENVIRONMENT defines BATCH)
42	#
43	# 2148 2017-02-09 16:56:42Z scharf
44	# changed allocation of resources for lcbullhh in srun command
45	#
46	# 2141 2017-02-06 10:19:32Z gronemeier
47	# made check for existing files language independent
48	#
49	# 1944 2016-06-15 06:29:00Z raasch
50	# adjustments for using HLRN ssh-keys
51	#
52	# 1940 2016-06-14 05:15:20Z raasch $
53	# adjustments for lckiaps
54	#
55	# 1866 2016-04-15 06:50:59Z raasch
56	# adjusted for lcocean
57	#
58	# 1841 2016-04-07 19:14:06Z raasch
59	# script now running under bash
60	#
61	# 1804 2016-04-05 16:30:18Z maronga
62	# test: implementing an execute mechanism where the execute command is given in the
63	# configuration file
64	# ROPTS removed from execution commands
65	# Warnings about missing optional files or user code changed to informative messages
66	# Removed parameter file check
67	#
68	# 1755 2016-02-22 13:53:39Z maronga
69	# Minor bugfix to last commit
70	#
71	# 1753 2016-02-22 13:49:49Z maronga
72	# Bugfix: use random job number when PBS job nummer retrieval fails on lccray*
73	#
74	# 1622 2015-07-20 06:21:36Z heinze
75	# bugfix for Mistral (lcbullhh)
76	#
77	# 1621 2015-07-17 11:39:33Z heinze
78	# adjustments for Mistral at DKRZ Hamburg (lcbullhh)
79	#
80	# 1609 2015-07-03 15:37:58Z maronga
81	# Modified the random number of the temporary working directory to match the
82	# queuing system number on HLRN-III (lccray*).
83	#
84	# 1604 2015-06-23 15:48:03Z suehring
85	# Enable coupled runs for lcmuk
86	#
87	# 1598 2015-05-29 06:45:40Z raasch
88	# bugfix for creating hostfile and total number of MPI tasks if run in openmp-mode on one node
89	#
90	# 1549 2015-01-30 14:26:16Z suehring
91	# Enable interactive debug sessions with allinea debugger
92	#
93	# 1547 2015-01-29 15:09:12Z witha
94	# adjustments for ForWind computing cluster (lcflow)
95	#
96	# 1491 2014-11-12 08:12:57Z heinze
97	# correct setting of batch_job allowed for ibm-machines
98	#
99	# 1468 2014-09-24 14:06:57Z maronga
100	# Typo removed (addres->address)
101	# Bugfix: returning files to IMUK via ssh did not work for lccrayh and lcycrayb
102	# Added support for restart runs (remote) for lcxe6
103	#
104	# 1442 2014-07-28 07:09:10Z raasch
105	# default queues for HLRN-III (lccrayb/lccrayh) defined
106	#
107	# 1422 2014-07-02 10:01:25Z kanani
108	# automatic restarts enabled for lccrayb,
109	# adjustment of path to compilervars.sh at IMUK (see automatic restarts)
110	#
111	# 1420 2014-06-10 06:24:14Z raasch
112	# -j1 option added to aprung command for lccrayf (CSC)
113	#
114	# 1402 2014-05-09 14:25:13Z raasch
115	# batch_job added to envpar-NAMELIST
116	#
117	# 1357 2014-04-11 15:02:03Z witha
118	# adjustments for lccrayh (automatic restarts on lccrayh outside of IMUK are now
119	# possible)
120	#
121	# 1320 2014-03-20 08:40:49Z raasch
122	# check namelist file set false by default
123	#
124	# 1304 2014-03-12 10:29:42Z raasch
125	# ulimit option changed from -Ss to -s
126	#
127	# bugfix: missing "fi" in r1289
128	#
129	# 1289 2014-03-04 07:12:34Z raasch
130	# comments translated to English
131	# necriam-, fimm-, ibmy-, and sgi-specific code removed
132	# export of variables for palm and interpret_config removed
133	#
134	# 1281 2014-02-01 07:55:49Z raasch
135	# rsync-copy restricted to Cray machines, since command is unavailable on some
136	# other systems
137	#
138	# 1279 2014-01-28 12:10:14Z raasch
139	# tasks_per_node must not be an integral divisor of numprocs any more. This was done
140	# in order to remove annoying restrictions concerning the number of processors which
141	# appear on machines with larger nodes (e.g. containing 24 cores). Now without this
142	# restriction, one of the nodes will be filled with less than the given number of
143	# tasks per node. A respective warning is given.
144	#
145	# 1274 2014-01-09 13:14:54Z heinze
146	# adjustments for lccrayh
147	#
148	# 1272 2014-01-08 10:19:32Z witha
149	# small adjustment for lcflow
150	#
151	# 1270 2013-12-16 11:05:01Z fricke
152	# call of combine_plot_fields adjusted for lccrayb/lccrayh
153	#
154	# 1255 2013-11-07 14:43:35Z raasch
155	# further adjustments for lccrayb remote access
156	#
157	# 1241 2013-10-30 11:36:58Z heinze
158	# Enable use of nudging input and input of large scale forcing from
159	# external files
160	#
161	# 1229 2013-09-20 06:55:19Z raasch
162	# further adjustments for lccrayb
163	#
164	# 1224 2013-09-16 07:27:23Z raasch
165	# first adjustments for lccrayb
166	#
167	# 1210 2013-08-14 10:58:20Z raasch
168	# fftw support added
169	#
170	# 1201 2013-07-10 16:17:59Z witha
171	# adjustments for Forwind cluster (lcflow)
172	#
173	# 1199 2013-07-05 14:52:22Z raasch
174	# adjustments for CSC Helsinki (lccrayf),
175	# executables for batch jobs can be created in advance, in order to avoid calling
176	# the compiler within the batch job (only works if batch jobs are submitted on
177	# local host)
178	#
179	# 1190 2013-06-25 09:39:21Z heinze
180	# enable use of automatic restarts for ibmh
181	# use of cluster/express queue enabled (ibmh)
182	#
183	# 1124 2013-04-09 15:46:52Z raasch
184	# variable "memory" is exported via typeset option -x, because otherwise an unknown
185	# side effect may lead to data loss while getopts is reading the script-option arguments
186	#
187	# 1122 2013-04-09 08:37:16Z heinze
188	# Bugfix: change type of variable last_char
189	#
190	# 1119 2013-04-05 15:11:19Z raasch
191	# Bugfix for setting -T option for subjob
192	#
193	# 1108 2013-03-05 07:03:32Z raasch
194	# bugfix for coupled runs on lckyut/lckyuh
195	#
196	# 1106 2013-03-04 05:31:38Z raasch
197	# --stdin argument for mpiexec on lckyuh
198	# -y and -Y settings output to header
199	#
200	# 1103 2013-02-20 02:15:53Z raasch
201	# default script runs again under ksh, because of unsolved problems with read
202	# from stdin: when bash script is called from a ksh, message "read error: 0:
203	# Resource temporarily unavailable" appears and script does not stop,
204	# further bash compatibility adjustments,
205	# shebang line replaced by /bin/bash when running jobs on lckyuh; no restarts
206	# on lckyuh, but mrun does not terminate and issues a warning instead
207	#
208	# 1101 2013-02-17 10:20:21Z raasch
209	# script now running under bash instead of ksh, which required small adjustments
210	# (output formatting with printf instead "typeset -L/-R", print replaced by echo,
211	# read from stdin),
212	# cross compilername on lckyuh compute nodes replaced by real compiler name
213	#
214	# 1099 2013-02-10 01:47:43Z raasch
215	# adjustments for Kyushu-University computing center (lckyuh - hayaka)
216	# and for Forwind cluster (lcflow)
217	# small further adjustments for lckyut
218	#
219	# 1094 2013-02-03 01:52:12Z raasch
220	# explicit ssh/scp port can be set in config file with environment variable
221	# scp_port. This port is handled to all ssh/scp/batch_scp calls.
222	# decalpha parts (yonsei) removed
223	#
224	# 2013-02-02 07:06:13Z raasch
225	# adjustments for Kyushu-University computing center (lckyut - tatara)
226	#
227	# 1083 2013-01-04 10:22:09Z maronga
228	# bugfix in parameter file check (read %cpp_options was missing)
229	#
230	# 1069 2012-11-28 16:18:43Z maronga
231	# bugfix: coupling mode was always set to mpi2, typos removed
232	#
233	# 1058 2012-11-21 07:00:35Z raasch
234	# Intel inspector (inspxe) is given the number of PEs instead of the number of
235	# nodes
236	#
237	# 1046 2012-11-09 14:38:45Z maronga
238	# code put under GPL (PALM 3.9)
239	#
240	# 21/03/94 - Siggi - first version finished
241	# 03/03/94 - Siggi - script development started
242	#
243	#--------------------------------------------------------------------------------#
244	# mrun - script for running PALM jobs
245	#--------------------------------------------------------------------------------#
246
247
248
249	# DECLARATION OF VARIABLES AND THEIR DEFUALT VALUES
250
251	set +o allexport # SUPPRESS EXPORT OF ALL VARIABLES, SINCE IN THE PAST THIS
252	# LES TO PROBLEMS IN ROUTINES CALLED BY MRUN
253	# (TOO MANY ARGUMENTS - PROBLEM)
254	set +o noclobber # EXISTING FILES ARE ALLOWED TO BE OVERWRITTEN
255
256	AddFilenames=""
257	additional_conditions=""
258	add_source_path=""
259	afname=""
260	check_namelist_files=false
261	combine_plot_fields=true
262	compiler_name=""
263	cond1=""
264	cond2=""
265	config_file=.mrun.config
266	coupled_dist=""
267	coupled_mode="mpi1"
268	cpp_opts=""
269	cpp_options=""
270	cpumax=0
271	cpurest=0
272	create_executable_for_batch=false
273	delete_temporary_catalog=true
274	do_batch=false
275	do_compile=true
276	do_remote=false
277	do_stagein=true
278	do_stageout=true
279	do_trace=false
280	email_notification="none"
281	exclude=""
282	executable=""
283	execute_command="none"
284	execution_error=false
285	fftw_inc=""
286	fftw_lib=""
287	fftw_support=false
288	fname=test
289	fromhost=""
290	global_revision=""
291	group_number=none
292	host=""
293	host_file=""
294	hp=""
295	input_list=""
296	interpreted_config_file=""
297	job_catalog="~/job_queue"
298	job_on_file=""
299	keep_data_from_previous_run=false
300	link_local_input=false
301	link_local_output=false
302	localhost_realname=$(hostname)
303	local_dvrserver_running=.FALSE.
304	locat=normal
305	mainprog=""
306	makefile=""
307	max_par_io_str=""
308	mc=$0
309	while [[ $(echo $mc \| grep -c "/") != 0 ]]
310	do
311	mc=`echo $mc \| cut -f2- -d"/"`
312	done
313	module_calls=""
314	mrun_script_name=$mc
315	netcdf_inc=""
316	netcdf_lib=""
317	netcdf_support=false
318	node_usage=default
319	numprocs=""
320	numprocs_atmos=0
321	numprocs_ocean=0
322	OOPT=""
323	openmp=false
324	output_list=""
325	package_list=""
326	queue=none
327	read_from_config=""
328	restart_run=false
329	if [[ `hostname` = rte10 ]]
330	then
331	return_address=133.5.185.60
332	echo "+++ WARNING: fixed return_address = $return_address is used !!!!!"
333	elif [[ `hostname` = climate0 ]]
334	then
335	return_address=165.132.26.68
336	echo "+++ WARNING: fixed return_address = $return_address is used !!!!!"
337	elif [[ `hostname` = "schultzl-Latitude-E6540" ]]
338	then
339	return_address="schultzl-Latitude-E6540"
340	echo "+++ WARNING: fixed return_address = $return_address is used !!!!!"
341	elif [[ `hostname` = urban00 ]]
342	then
343	return_address=147.46.30.151
344	echo "+++ WARNING: fixed return_address = $return_address is used !!!!!"
345	else
346	return_address=$(nslookup `hostname` 2>&1 \| grep "Address:" \| tail -1 \| awk '{print $2}')
347	fi
348	return_password=""
349	return_username=$LOGNAME
350	remotecall=false
351	remote_username=""
352	run_coupled_model=false
353	run_mode=""
354	dashes=" ----------------------------------------------------------------------------"
355	silent=false
356	source_list=""
357	source_path=SOURCE
358	tasks_per_node=""
359	threads_per_task=1
360	tmpcreate=false
361	tmp_data_catalog=""
362	transfer_problems=false
363	usern=$LOGNAME
364	use_openmp=false
365	version="MRUN 2.1 Rev$Rev: 2261 $"
366	working_directory=`pwd`
367	TOPT=""
368	XOPT=""
369	zeit=$( date \| cut -c 12-19 )
370
371	typeset -i iec=0 iic=0 iin=0 ioc=0 iout=0 stagein_anz=0 stageout_anz=0
372	typeset -x -i memory=0 # HAS TO BE EXPORTED HERE, OTHERWISE AN UNKNOWN SIDE
373	# SIDE EFFECT MAY CAUSE DATA LOSS WHEN GETOPTS IS READING THE
374	# SCRIPT-OPTION ARGUMENTS
375	typeset -i cputime i ii iia iii iio icycle inode ival jobges jobsek last_char_int maxcycle minuten nodes pes remaining_pes run_number sekunden tp1
376
377	# ERROR HANDLING IN CASE OF EXIT
378	trap 'rm -rf $working_directory/tmp_mrun
379	if [[ $locat != normal && $locat != control_c ]]
380	then
381
382	# CARRY OUT ERROR-COMMANDS GIVEN IN THE CONFIGURATION FILE (EC:)
383	(( i = 0 ))
384	while (( i < iec ))
385	do
386	(( i = i + 1 ))
387	printf "\n *** Execution of ERROR-command:\n"
388	printf " >>> ${err_command[$i]}\n"
389	eval ${err_command[$i]}
390	done
391	if [[ -n $interpreted_config_file ]]
392	then
393	rm -rf $interpreted_config_file
394	fi
395	if [[ -n .mrun_environment ]]
396	then
397	rm -rf .mrun_environment
398	fi
399	if [[ $tmpcreate = true ]]
400	then
401	# printf "\n *** Contents of \"$TEMPDIR\":\n"
402	# ls -al; cd
403	[[ $delete_temporary_catalog = true ]] && rm -rf $TEMPDIR
404	fi
405	if [[ "$dvrserver_id" != "" ]]
406	then
407	echo "+++ killing dvrserver_id=$dvrserver_id"
408	kill $dvrserver_id
409	fi
410	if [[ -f ~/job_queue/JOBINFO.$QSUB_REQID ]]
411	then
412	rm -rf ~/job_queue/JOBINFO.$QSUB_REQID
413	fi
414	printf "\n\n+++ MRUN killed \n\n"
415	elif [[ $locat != control_c ]]
416	then
417	printf "\n\n --> all actions finished\n\n"
418	printf " Bye, bye $usern !!\n\n"
419	fi' exit
420
421
422	# ACTIONS IN CASE OF TERMINAL-BREAK (CONTROL-C):
423	trap 'rm -rf $working_directory/tmp_mrun
424	rm -rf $working_directory/tmp_check_namelist_files
425	[[ $tmpcreate = true ]] && (cd; rm -rf $TEMPDIR)
426	if [[ -f ~/job_queue/JOBINFO.$QSUB_REQID ]]
427	then
428	rm -rf ~/job_queue/JOBINFO.$QSUB_REQID
429	fi
430	if [[ "$dvrserver_id" != "" ]]
431	then
432	echo "+++ killing dvrserver_id=$dvrserver_id"
433	kill $dvrserver_id
434	fi
435	printf "\n+++ MRUN killed by \"^C\" \n\n"
436	locat=control_c
437	exit
438	' 2
439
440
441	# CHECK IF THE PATH FOR THE PALM BINARIES (SCRIPTS+UTILITY-PROGRAMS) HAS
442	# BEEN SET
443	if [[ "$PALM_BIN" = "" ]]
444	then
445	printf "\n +++ environment variable PALM_BIN has not been set"
446	printf "\n please set it to the directory where the PALM scripts are located"
447	locat=palm_bin; exit
448	fi
449	export PATH=$PALM_BIN:$PATH
450
451
452
453	# READ SHELLSCRIPT-OPTIONS AND REBUILD THE MRUN-COMMAND STRING (MC),
454	# WHICH WILL BE USED TO START RESTART-JOBS
455	while getopts :a:bBc:Cd:D:Fg:G:h:H:i:kK:m:M:n:o:O:p:P:q:r:R:s:St:T:u:U:vw:xX:yY:zZ option
456	do
457	case $option in
458	(a) afname=$OPTARG;;
459	(b) do_batch=true; mc="$mc -b";;
460	(B) delete_temporary_catalog=false; mc="$mc -B";;
461	(c) config_file=$OPTARG; mc="$mc -c$OPTARG";;
462	(C) restart_run=true; mc="$mc -C";;
463	(d) fname=$OPTARG; mc="$mc -d$OPTARG";;
464	(F) job_on_file="-D"; mc="$mc -F";;
465	(g) group_number=$OPTARG; mc="$mc -g$OPTARG";;
466	(G) global_revision=$OPTARG; mc="$mc -G'$OPTARG'";;
467	(h) host=$OPTARG; mc="$mc -h$OPTARG";;
468	(H) fromhost=$OPTARG; mc="$mc -H$OPTARG";;
469	(i) input_list=$OPTARG; mc="$mc -i'$OPTARG'";;
470	(k) keep_data_from_previous_run=true; mc="$mc -k";;
471	(K) additional_conditions="$OPTARG"; mc="$mc -K'$OPTARG'";;
472	(m) memory=$OPTARG; mc="$mc -m$OPTARG";;
473	(M) makefile=$OPTARG; mc="$mc -M$OPTARG";;
474	(n) node_usage=$OPTARG; mc="$mc -n$OPTARG";;
475	(o) output_list=$OPTARG; mc="$mc -o'$OPTARG'";;
476	(O) use_openmp=true; threads_per_task=$OPTARG; mc="$mc -O$OPTARG";;
477	(p) package_list=$OPTARG; mc="$mc -p'$OPTARG'";;
478	(P) return_password=$OPTARG; mc="$mc -P$OPTARG";;
479	(q) queue=$OPTARG; mc="$mc -q$OPTARG";;
480	(r) run_mode=$OPTARG; mc="$mc -r'$OPTARG'";;
481	(R) remotecall=true;return_address=$OPTARG; mc="$mc -R$OPTARG";;
482	(s) source_list=$OPTARG; mc="$mc -s'$OPTARG'";;
483	(S) read_from_config=false; mc="$mc -S";;
484	(t) cpumax=$OPTARG; mc="$mc -t$OPTARG";;
485	(T) mrun_tasks_per_node=$OPTARG; mc="$mc -T$OPTARG";;
486	(u) remote_username=$OPTARG; mc="$mc -u$OPTARG";;
487	(U) return_username=$OPTARG; mc="$mc -U$OPTARG";;
488	(v) silent=true; mc="$mc -v";;
489	(w) max_par_io_str=$OPTARG; mc="$mc -w$OPTARG";;
490	(x) do_trace=true;set -x; mc="$mc -x";;
491	(X) numprocs=$OPTARG; mc="$mc -X$OPTARG";;
492	(y) ocean_file_appendix=true; mc="$mc -y";;
493	(Y) run_coupled_model=true; coupled_dist=$OPTARG; mc="$mc -Y'$OPTARG'";;
494	(z) check_namelist_files=false; mc="$mc -z";;
495	(Z) combine_plot_fields=false; mc="$mc -Z";;
496	(\?) printf "\n +++ unknown option $OPTARG \n"
497	printf "\n --> type \"$0 ?\" for available options \n"
498	locat=parameter;exit;;
499	esac
500	done
501
502
503	# SKIP GIVEN OPTIONS TO READ POSITIONAL PARAMETER, IF GIVEN
504	# CURRENTLY ONLY PARAMETER ? (TO OUTPUT A SHORT COMMAND INFO) IS ALLOWED
505	(( to_shift = $OPTIND - 1 ))
506	shift $to_shift
507
508	# PRINT SHORT DESCRIPTION OF MRUN OPTIONS
509	if [[ "$1" = "?" ]]
510	then
511	(printf "\n *** mrun can be called as follows:\n"
512	printf "\n $mrun_script_name -b -c.. -d.. -D.. -f.. -F -h.. -i.. -I -K.. -m.. -o.. -p.. -r.. -R -s.. -t.. -T.. -v -x -X.. -y -Y.. -Z <modus> \n"
513	printf "\n Description of available options:\n"
514	printf "\n Option Description Default-Value"
515	printf "\n -a base name of input files equiv. -d"
516	printf "\n -b batch-job on local machine ---"
517	printf "\n -B do not delete temporary directory at end ---"
518	printf "\n -c configuration file .mrun.config"
519	printf "\n -d base name of files attached to program test"
520	printf "\n -D preprocessor(cpp)-directives \"\" "
521	printf "\n -F create remote job file only ---"
522	printf "\n -h execution host $localhost_realname"
523	printf "\n -i INPUT control list \"\" "
524	printf "\n -k keep data from previous run"
525	printf "\n -K additional conditions for controling"
526	printf "\n usage of conditional code and"
527	printf "\n env-variables in configuration file \"\" "
528	printf "\n -m memory demand in MB (batch-jobs) 0 MB"
529	printf "\n -M Makefile name Makefile"
530	printf "\n -n node usage (shared/not_shared) depending on -h"
531	printf "\n -o OUTPUT control list \"\" "
532	printf "\n -O threads per openMP task ---"
533	printf "\n -p software package list \"\" "
534	printf "\n -q queue \"$queue\" "
535	printf "\n -r run control list (combines -i -o) \"\" "
536	printf "\n -s filenames of routines to be compiled \"\" "
537	printf "\n must end with .f, .f90, .F, or .c !"
538	printf "\n use \"..\" for more than one file and wildcards"
539	printf "\n -s LM compiles all locally modified files"
540	printf "\n -S config file interpreted by shellscript ---"
541	printf "\n -t allowed cpu-time in seconds (batch) 0"
542	printf "\n -T tasks per node depending on -h"
543	printf "\n -u username on remote machine \"\" "
544	printf "\n -v no prompt for confirmation ---"
545	printf "\n -w maximum parallel io streams as given by -X"
546	printf "\n -x tracing of mrun for debug purposes ---"
547	printf "\n -X # of processors (on parallel machines) 1"
548	printf "\n -y add appendix \"_O\" to all local output"
549	printf "\n files (ocean precursor runs followed by"
550	printf "\n coupled atmosphere-ocean runs) ---"
551	printf "\n -Y run coupled model, \"#1 #2\" with"
552	printf "\n #1 atmosphere and #2 ocean processors \"#/2 #/2\" depending on -X"
553	printf "\n -Z skip combine_plot_fields at the end of "
554	printf "\n the simulation ---"
555	printf "\n "
556	printf "\n Possible values of positional parameter <modus>:"
557	printf "\n \"?\" - this outline \n\n") \| more
558	exit
559	elif [[ "$1" != "" ]]
560	then
561	printf "\n +++ positional parameter $1 unknown \n"
562	locat=parameter; exit
563	fi
564
565
566
567	# SHORT STARTING MESSAGE
568	printf "\n*** $version "
569	printf "\n will be executed. Please wait ..."
570
571
572
573	# CHECK, IF CONFIGURATION FILE EXISTS
574	if [[ ! -f $config_file ]]
575	then
576	printf "\n\n +++ configuration file: "
577	printf "\n $config_file"
578	printf "\n does not exist"
579	locat=connect; exit
580	fi
581
582
583	# DETERMINE THE HOST-IDENTIFIER (localhost) FROM THE CONFIGURATION FILE
584	line=""
585	grep "%host_identifier" $config_file > tmp_mrun
586	while read line
587	do
588	if [[ "$line" != "" && $(echo $line \| cut -c1) != "#" ]]
589	then
590	HOSTNAME=`echo $line \| cut -d" " -s -f2`
591	host_identifier=`echo $line \| cut -d" " -s -f3`
592	if [[ $localhost_realname = $HOSTNAME ]]
593	then
594	localhost=$host_identifier
595	break
596	fi
597	fi
598	done < tmp_mrun
599
600	if [[ "$localhost" = "" ]]
601	then
602	printf "\n\n +++ no host identifier found in configuration file \"$config_file\""
603	printf "\n for local host \"$localhost_realname\"."
604	printf "\n Please add line"
605	printf "\n \"\%host_identifier $localhost_realname <identifier>\""
606	printf "\n to the configuration file."
607	locat=localhost; exit
608	fi
609
610
611	# ??? THIS SHOULD BE DECRIBED IN THE MRUN DOCUMENTATION ???
612	# SET BASENAME OF THE INPUT-FILES TO THE GENERAL BASENAME (GIVEN BY OPTION -d),
613	# IF IT HAS NOT BEEN SET BY THE USER DIFFERENTLY (USING OPTION -a)
614	[[ "$afname" = "" ]] && afname=$fname
615
616
617	# ADD RUN-FILE ACTIVATION STRINGS (GIVEN BY OPTION -r) TO THE
618	# RESPECTIVE LISTS FOR ACTIVATING THE INPUT/OUTPUT FILE CONNECTION
619	# STATEMENTS IN THE CONFIGURATION FILE
620	if [[ "$run_mode" != "" ]]
621	then
622	input_list="$input_list $run_mode"
623	output_list="$output_list $run_mode"
624	fi
625
626
627	# ??? is this (and the respective option -H) really required ???
628	# STORE HOSTNAME, FROM WHICH THE JOB IS STARTED,
629	# BUT ONLY IF IT HASN'T BEEN ALREADY SET BY OPTION -H
630	# (MRUN IS AUTOMATICALLY SETTING THIS OPTION IN THE MRUN-CALLS WITHIN
631	# THOSE BATCH-JOBS, WHICH ARE CREATED BY MRUN ITSELF)
632	if [[ "$fromhost" = "" ]]
633	then
634	fromhost=$localhost
635	fi
636
637
638	# CHECK, IF EXECUTION SHALL TO BE DONE ON THE REMOTE-MACHINE.
639	# VALUE OF do_remote IS IMPORTANT FOR THE FILE CONNECTIONS.
640	# IN CASE OF EXECUTION ON A REMOTE-MACHINE, A BATCH-JOB HAS
641	# TO BE SUBMITTED (I.E. do_batch=true)
642	if [[ -n $host && "$host" != $localhost ]]
643	then
644	do_batch=true
645	do_remote=true
646	case $host in
647	(ibm\|ibmh\|ibmkisti\|ibmku\|ibms\|lcbullhh\|lccrayb\|lccrayh\|lccrayf\|lceddy\|lckyoto\|lcocean\|unics\|lcxe6\|lcxt5m\|lck\|lckiaps\|lckordi\|lckyuh\|lckyut\|lcsb) true;;
648	(*) printf "\n"
649	printf "\n +++ sorry: execution of batch jobs on remote host \"$host\""
650	printf "\n is not available"
651	locat=nqs; (( iec = 0 )); exit;;
652	esac
653	else
654	host=$localhost
655	fi
656
657
658	# EVALUATE ADDITIONAL CONDITIONS GIVEN BY OPTION -K
659	if [[ -n $additional_conditions ]]
660	then
661	cond1=`echo $additional_conditions \| cut -d" " -f1`
662	cond2=`echo $additional_conditions \| cut -d" " -s -f2`
663	dummy=`echo $additional_conditions \| cut -d" " -s -f3`
664	if [[ -n $dummy ]]
665	then
666	printf "\n +++ more than 2 additional conditions given for Option \"-K\""
667	locat=options; exit
668	fi
669	block=_$cond1
670	[[ -n $cond2 ]] && block=${block}_$cond2
671	fi
672
673
674	# EVALUATE MODEL COUPLING FEATURES (OPTION -Y) AND DETERMINE coupled_mode
675	if [[ $run_coupled_model = true ]]
676	then
677
678	numprocs_atmos=`echo $coupled_dist \| cut -d" " -s -f1`
679	numprocs_ocean=`echo $coupled_dist \| cut -d" " -s -f2`
680
681	if (( $numprocs_ocean + $numprocs_atmos != $numprocs ))
682	then
683
684	printf "\n +++ number of processors does not fit to specification by \"-Y\"."
685	printf "\n PEs (total) : $numprocs"
686	printf "\n PEs (atmosphere): $numprocs_atmos"
687	printf "\n PEs (ocean) : $numprocs_ocean"
688	locat=coupling; exit
689
690	fi
691
692
693	# GET coupled_mode FROM THE CONFIG FILE
694	line=""
695	grep "%cpp_options.-D__mpi2.$host" $config_file > tmp_mrun
696	while read line
697	do
698	echo line=\"$line\"
699	if [[ "$line" != "" && $(echo $line \| cut -c1) != "#" && $(echo $line \| cut -d" " -s -f4) = $cond1 && $(echo $line \| cut -d" " -s -f5) = $cond2 ]]
700	then
701	coupled_mode="mpi2"
702	fi
703	done < tmp_mrun
704
705	fi
706
707	# SAVE VALUES OF MRUN-OPTIONS SICHERN IN ORDER TO OVERWRITE
708	# THOSE VALUES GIVEN IN THE CONFIGURATION-FILE
709	mrun_memory=$memory
710	mrun_group_number=$group_number
711	mrun_cpumax=$cpumax
712	mrun_numprocs=$numprocs
713
714
715	# READ AND EVALUATE THE CONFIGURATION-FILE FROM WITHIN THIS SHELLSCRIPT
716	# (OPTION -S). THE DEFAULT IS USING THE ROUTINE <<<<
717	if [[ "$read_from_config" = false ]]
718	then
719	[[ $silent = false ]] && printf "\n Reading the configuration file... "
720
721	# READ THE CONFIGURATION FILE LINE BY LINE
722	while read zeile
723	do
724	[[ $silent = false ]] && printf "."
725
726
727	# FIRST REPLACE ENVIRONMENT-VARIABLES BY THEIR RESPECTIVE VALUES
728	eval zeile=\"$zeile\"
729
730
731	# INTERPRET THE LINE
732	if [[ "$(echo $zeile)" = "" ]]
733	then
734	# EMPTY LINE, NO ACTION
735	continue
736
737	elif [[ "$(echo $zeile \| cut -c1)" = "#" ]]
738	then
739
740	# LINE IS A COMMENT LINE
741	true
742
743	elif [[ "$(echo $zeile \| cut -c1)" = "%" ]]
744	then
745
746	# LINE DEFINES AN ENVIRONMENT-VARIABLE
747	zeile=$(echo $zeile \| cut -c2-)
748	var=`echo $zeile \| cut -d" " -f1`
749	value=`echo $zeile \| cut -d" " -s -f2`
750	for_host=`echo $zeile \| cut -d" " -s -f3`
751	for_cond1=`echo $zeile \| cut -d" " -s -f4`
752	for_cond2=`echo $zeile \| cut -d" " -s -f5`
753
754	if [[ "$for_host" = "" \|\| ( "$for_host" = $host && "$for_cond1" = "$cond1" && "$for_cond2" = "$cond2" ) \|\| $(echo "$input_list$output_list"\|grep -c "$for_host") != 0 ]]
755	then
756
757	# REPLACE ":" BY " " IN COMPILER- CPP- OR LINKER-OPTIONS,
758	# "::" IS REPLACED BY ":".
759	value=`echo $value \| sed 's/::/%DUM%/g' \| sed 's/:/ /g' \| sed 's/%DUM%/:/g'`
760
761
762	# VALUE FROM THE CONFIGURATION-FILE IS ASSIGNED TO THE
763	# ENVIRONMENT-VARIABLE, BUT ONLY IF NO VALUE HAS BEEN ALREADY
764	# ASSIGNED WITHIN THIS SCRIPT (E.G. BY SCRIPT-OPTIONS).
765	# NON-ASSIGNED VARIABLES HAVE VALUE "" OR 0 (IN CASE OF INTEGER).
766	# HENCE THE GENERAL RULE IS: SCRIPT-OPTION OVERWRITES THE
767	# CONFIGURATION-FILE.
768	if [[ "$(eval echo \$$var)" = "" \|\| "$(eval echo \$$var)" = "0" ]]
769	then
770	eval $var=\$value
771
772	# TERMINAL OUTPUT OF ENVIRONMENT-VARIABLES, IF TRACEBACK IS SWITCHED on
773	if [[ $do_trace = true ]]
774	then
775	printf "\n*** ENVIRONMENT-VARIABLE $var = $value"
776	fi
777	fi
778
779	# IF AN ENVIRONMENT-VARIABLE DETERMINES THE HOST, THEN EVALUATE IT IMMEDIATELY:
780	# VALUE OF do-remote IS REQUIRED FOR THE FILE CONNECTIONS (COPY OF I/O FILES).
781	# IF EXECUTION IS SCHEDULED FOR A REMOTE-MASCHINE, A BATCH-JOB
782	# MUST HAVE TO BE STARTED
783	if [[ $var = host ]]
784	then
785	if [[ -n $host && "$host" != $localhost ]]
786	then
787	do_batch=true
788	do_remote=true
789	case $host in
790	(ibm\|ibmh\|ibmkisti\|ibmku\|ibms\|lcbullhh\|lccrayb\|lccrayh\|lccrayf\|lceddy\|lckyoto\|lcocean\|unics\|lcxe6\|lcxt5m\|lck\|lckiaps\|lckordi\|lckyuh\|lckyut\|lcsb) true;;
791	(*) printf "\n +++ sorry: execution of batch jobs on remote host \"$host\""
792	printf "\n is not available"
793	locat=nqs; exit;;
794	esac
795	else
796	host=$localhost
797	fi
798	fi
799
800	# USER-DEFINED ENVIRONMENT VARIABLES MUST BE EXPORTED,
801	# BECAUSE THEY MAY BE REQUIRED IN OTHER SCRIPTS CALLED
802	# BY MRUN (E.G. subjob)
803	export $var
804	fi
805
806
807
808
809	elif [[ "$(echo $zeile \| cut -c1-3)" = "EC:" ]]
810	then
811
812	# LINE DEFINES ERROR-COMMAND
813	(( iec = iec + 1 ))
814	zeile=$(echo $zeile \| cut -c4-)
815	err_command[$iec]="$zeile"
816
817	elif [[ "$(echo $zeile \| cut -c1-3)" = "IC:" ]]
818	then
819
820	# LINE DEFINES INPUT-COMMAND
821	(( iic = iic + 1 ))
822	zeile=$(echo $zeile \| cut -c4-)
823	in_command[$iic]="$zeile"
824
825	elif [[ "$(echo $zeile \| cut -c1-3)" = "OC:" ]]
826	then
827
828	# LINE DEFINES OUTPUT-COMMAND
829	(( ioc = ioc + 1 ))
830	zeile=$(echo $zeile \| cut -c4-)
831	out_command[$ioc]="$zeile"
832
833	else
834
835	# LINE DEFINES FILE CONNECTION. READ THE FILE ATTRIBUTES.
836	# s2a: in/out - field
837	# s2b: loc - field (optional)
838	# s2c: tr/ar - field (optional)
839	s1=`echo $zeile \| cut -d" " -f1`
840	s2=`echo $zeile \| cut -d" " -s -f2`
841	s2a=$(echo $s2 \| cut -d":" -f1)
842	if [[ $(echo $s2 \| grep -c ":") = 0 ]]
843	then
844	s2b=""
845	s2c=""
846	else
847	s2b=`echo $s2 \| cut -d":" -f2 \| sed 's/:/ /g'`
848	s2c=`echo $s2 \| cut -d":" -s -f3 \| sed 's/:/ /g'`
849	fi
850	s3=`echo $zeile \| cut -d" " -f3`
851	s4=`echo $zeile \| cut -d" " -s -f4`
852	s5=`echo $zeile \| cut -d" " -s -f5`
853	s6=`echo $zeile \| cut -d" " -s -f6`
854
855
856	# STORE FILE CONNECTION, IF ACTIVATED BY ACTIVATION-STRING FROM
857	# INPUT- OR OUTPUT-LIST.
858	# VARIABLE S3 MAY CONTAIN A LIST OF ACTIVATION STRINGS (FIELD-SEPERATOR ":").
859	# IF EXECUTION IS SCHEDULED FOR A REMOTE-MACHINE AND THE FILE IS ONLY
860	# LOCALLY REQUIRED ON THAT MACHINE (I.E. s2b = loc), THE FILE CONNECTION
861	# IS NOT CHECKED AND STORED.
862	IFSALT="$IFS"; IFS="$IFS:"
863	if [[ "$s2a" = in && ! ( $do_remote = true && ( "$s2b" = loc \|\| "$s2b" = locopt ) ) ]]
864	then
865	found=false
866	for actual in $input_list
867	do
868	for formal in $s3
869	do
870	[[ $actual = $formal \|\| "$formal" = "-" ]] && found=true
871	done
872	done
873	if [[ $found = true ]]
874	then
875	(( iin = iin + 1 ))
876	localin[$iin]=$s1; transin[$iin]=$s2b; actionin[$iin]=$s2c;
877	typein[$iin]=$s3; pathin[$iin]=$s4; endin[$iin]=$s5;
878	extin[$iin]=$s6
879	fi
880	elif [[ "$s2a" = out && ! ( $do_remote = true && "$s2b" = loc ) ]]
881	then
882	found=false
883	for actual in $output_list
884	do
885	for formal in $s3
886	do
887	[[ $actual = $formal \|\| "$formal" = "-" ]] && found=true
888	done
889	done
890	if [[ $found = true ]]
891	then
892	(( iout = iout + 1 ))
893	localout[$iout]=$s1; actionout[$iout]=$s2c; typeout[$iout]=$s3;
894	pathout[$iout]=$s4; endout[$iout]=$s5; extout[$iout]=$s6
895	fi
896	elif [[ "$s2a" != in && "$s2a" != out ]]
897	then
898	printf "\n +++ I/O-attribute in configuration file $config_file has the invalid"
899	printf "\n value \"$s2\". Only \"in\" and \"out\" are allowed!"
900	locat=connect; exit
901	fi
902	IFS="$IFSALT"
903	fi
904	done < $config_file
905
906	else
907
908	# EVALUATE THE CONFIGURATION-FILE BY FORTRAN-PROGRAM
909	[[ $silent = false ]] && printf "..."
910	interpreted_config_file=.icf.$RANDOM
911
912
913
914	# PROVIDE VALUES OF ENVIRONMENT-VARIABLE FOR interpret_config VIA NAMELIST-FILE
915	cat > .mrun_environment << EOF
916	&mrun_environment cond1 = '$cond1', cond2 = '$cond2',
917	config_file = '$config_file', do_remote = '$do_remote',
918	do_trace = '$do_trace', host = '$host',
919	input_list = '$input_list', icf = '$interpreted_config_file',
920	localhost = '$localhost', output_list = '$output_list' /
921
922	EOF
923
924	if [[ "$host" != $localhost ]]
925	then
926
927	# REMOTE JOB FROM LOCAL HOST: JUST TAKE THE FIRST EXECUTABLE FOUND
928	interpret_config_executable=`ls -1 ${PALM_BIN}/interpret_config*.x 2>/dev/null`
929	if [[ $? != 0 ]]
930	then
931	printf "\n\n +++ no interpret_config found"
932	printf "\n run \"mbuild -u -h ...\" to generate utilities for this host"
933	locat=interpret_config; exit
934	fi
935	interpret_config_executable=`echo $interpret_config_executable \| cut -d" " -f1`
936	$interpret_config_executable
937
938	else
939
940	# CHECK, IF THERE IS AN EXECUTABLE FOR THE BLOCK
941	if [[ ! -f ${PALM_BIN}/interpret_config${block}.x ]]
942	then
943	printf "\n\n +++ no interpret_config found for given block \"$cond1 $cond2\""
944	printf "\n run \"mbuild -u -h ...\" to generate utilities for this block"
945	locat=interpret_config; exit
946	else
947	interpret_config${block}.x
948	fi
949
950	fi
951	rm .mrun_environment
952
953
954	# EXECUTE SHELL-COMMANDS GENERATED BY interpret_config WITHIN THIS SHELL
955	chmod u+x $interpreted_config_file
956	export PATH=$PATH:.
957	. $interpreted_config_file
958	rm $interpreted_config_file
959
960	fi
961
962
963	# VALUES OF MRUN-OPTIONS OVERWRITE THOSE FROM THE CONFIGURATION-FILE
964	[[ $mrun_memory != 0 ]] && memory=$mrun_memory
965	[[ "$mrun_group_number" != "none" ]] && group_number=$mrun_group_number
966	[[ $mrun_cpumax != 0 ]] && cpumax=$mrun_cpumax
967	[[ "$mrun_numprocs" != "" ]] && numprocs=$mrun_numprocs
968	[[ "$max_par_io_str" != "" ]] && maximum_parallel_io_streams=$max_par_io_str
969	[[ "$mrun_tasks_per_node" != "" ]] && tasks_per_node=$mrun_tasks_per_node
970
971
972	# ON THE LOCAL MACHINE, DETERMINE (FROM THE CONFIGURATION-FILE) THE PATH
973	# FOR SOURCE-FILES TO BE COMPILED.
974	# IN A BATCH-JOB, SOURCES (ROUTINES TO BE COMPILED) ARE COMPLETE ALREADY.
975	# BEFORE MRUN IS CALLED IN THE JOB, SOURCES_COMPLETE = true IS ASSIGNED (SEE FURTHER BELOW).
976	if [[ "$SOURCES_COMPLETE" = "" ]]
977	then
978
979	# FIRST CHECK, IF A GLOBAL SOURCE PATH (TO BE USED FOR ALL HOSTS)
980	# HAS BEEN DEFINED
981	source_path=""
982	line=""
983	grep "%source_path" $config_file > tmp_mrun
984	while read line
985	do
986	if [[ "$line" != "" && $(echo $line \| cut -c1) != "#" ]]
987	then
988	if [[ "$(echo $line \| cut -d" " -f3)" = "" ]]
989	then
990	global_source_path=`echo $line \| cut -d" " -f2`
991	fi
992	fi
993	done < tmp_mrun
994
995	line=""
996	found=false
997	grep " $localhost" $config_file \| grep "%source_path" > tmp_mrun
998	while read line
999	do
1000	if [[ "$line" != "" && $(echo $line \| cut -c1) != "#" ]]
1001	then
1002	if [[ $found = true ]]
1003	then
1004	printf "\n\n +++ more than one source path found in configuration file"
1005	printf "\n for local host \"$localhost\" "
1006	locat=source_path; exit
1007	fi
1008	source_path=`echo $line \| cut -d" " -f2`
1009	found=true
1010	fi
1011	done < tmp_mrun
1012	rm tmp_mrun
1013
1014	if [[ "$source_path" = "" ]]
1015	then
1016	if [[ "$global_source_path" != "" ]]
1017	then
1018	source_path=$global_source_path
1019	else
1020	printf "\n\n +++ no source path found in configuration file"
1021	printf "\n for local host \"$localhost\" "
1022	locat=source_path; exit
1023	fi
1024	fi
1025	eval source_path=$source_path
1026
1027	if [[ ! -d $source_path ]]
1028	then
1029	printf "\n\n +++ source path \"$source_path\" on local host"
1030	printf "\n \"$localhost\" does not exist"
1031	locat=source_path; exit
1032	fi
1033
1034	fi
1035
1036
1037	# GET THE GLOBAL REVISION-NUMBER OF THE SVN-REPOSITORY
1038	# (HANDED OVER TO RESTART-RUNS USING OPTION -G)
1039	if [[ "$global_revision" = "" && $host != "ibmkisti" ]]
1040	then
1041	global_revision=`svnversion $source_path 2>/dev/null`
1042	global_revision="Rev: $global_revision"
1043	fi
1044
1045
1046	# ??? das darf doch eigentlich garnicht passieren, weil optionen config-datei uebersteuern ???
1047	# CHECK AGAIN, IF EXECUTION SHALL BE DONE ON A REMOTE-HOST (BATCH-JOB).
1048	# (HOST MAY HAVE CHANGED DUE TO SETTINGS IN THE CONFIGURATION-FILE)
1049	if [[ -n $host && "$host" != $localhost ]]
1050	then
1051	do_batch=true
1052	do_remote=true
1053	case $host in
1054	(ibm\|ibmh\|ibmkisti\|ibmku\|ibms\|lcbullhh\|lccrayb\|lccrayh\|lccrayf\|lceddy\|lckyoto\|lcocean\|unics\|lcxe6\|lcxt5m\|lck\|lckiaps\|lckordi\|lckyuh\|lckyut\|lcsb) true;;
1055	(*) printf "\n"
1056	printf "\n +++ sorry: execution of batch jobs on remote host \"$host\""
1057	printf "\n is not available"
1058	locat=nqs; (( iec = 0 )); exit;;
1059	esac
1060	else
1061	host=$localhost
1062	fi
1063
1064
1065
1066	# IN CASE OF PARALLEL EXECUTION, CHECK SOME SPECIFICATIONS CONCERNING PROCESSOR NUMBERS
1067	if [[ "$cond1" = parallel \|\| "$cond2" = parallel ]]
1068	then
1069
1070	# CHECK, IF NUMBER OF CORES TO BE USED HAS BEEN GIVEN
1071	if [[ ! -n $numprocs ]]
1072	then
1073	printf "\n"
1074	printf "\n +++ option \"-K parallel\" requires additional specification"
1075	printf "\n of the number of processors to be used by"
1076	printf "\n mrun-option \"-X\" or by environment-variable"
1077	printf "\n \"numprocs\" in the configuration file"
1078	locat=numprocs; (( iec = 0 )); exit
1079	fi
1080
1081	# CHECK, IF THE NUMBER OF CORES PER NODE HAS BEEN GIVEN UND IF IT IS AN
1082	# INTEGRAL DIVISOR OF THE TOTAL NUMBER OF CORES GIVEN BY OPTION -X
1083	if [[ "$tasks_per_node" = "" && $host != lcxt5m ]]
1084	then
1085	printf "\n"
1086	printf "\n +++ option \"-T\" (tasks per node) is missing"
1087	printf "\n set -T option or define tasks_per_node in the config file"
1088	locat=tasks_per_node; (( iec = 0 )); exit
1089	fi
1090
1091	if (( numprocs < tasks_per_node ))
1092	then
1093	printf "\n"
1094	printf "\n +++ tasks per node (-T) cannot exceed total number of processors (-X)"
1095	printf "\n given values: -T $tasks_per_node -X $numprocs"
1096	locat=tasks_per_node; (( iec = 0 )); exit
1097	fi
1098
1099	if [[ $host != lcxt5m ]]
1100	then
1101	(( nodes = numprocs / ( tasks_per_node * threads_per_task ) ))
1102	(( ival = $tasks_per_node ))
1103	(( pes = numprocs ))
1104	(( ii = pes / ival ))
1105	(( remaining_pes = pes - ii * ival ))
1106	if (( remaining_pes > 0 ))
1107	then
1108	printf "\n"
1109	printf "\n +++ WARNING: tasks per node (option \"-T\") is not an integral"
1110	printf "\n divisor of the total number of processors (option \"-X\")"
1111	printf "\n values of this mrun-call: \"-T $tasks_per_node\" \"-X $numprocs\""
1112	printf "\n One of the nodes is filled with $remaining_pes instead of $tasks_per_node tasks"
1113	(( nodes = nodes + 1 ))
1114	fi
1115	fi
1116
1117
1118	# SETTINGS FOR SUBJOB-COMMAND
1119	OOPT="-O $threads_per_task"
1120
1121	# SET THE TOTAL NUMBER OF NODES, REQUIRED FOR THE SUBJOB-COMMAND (SEE FURTHER BELOW)
1122	if [[ "$tasks_per_node" != "" ]]
1123	then
1124	TOPT="-T $tasks_per_node"
1125	fi
1126
1127	# CHECK node_usage FOR ALLOWED VALUES AND SET DEFAULT VALUE, IF NECESSARY
1128	if [[ $node_usage = default ]]
1129	then
1130	if [[ $host = ibms ]]
1131	then
1132	node_usage=shared
1133	else
1134	node_usage=not_shared
1135	fi
1136	fi
1137
1138	if [[ $node_usage != shared && $node_usage != not_shared && $node_usage != singlejob && "$(echo $node_usage \| cut -c1-3)" != "sla" ]]
1139	then
1140	printf "\n"
1141	printf "\n +++ node usage (option \"-n\") is only allowed to be set"
1142	printf "\n \"shared\" or \"not_shared\""
1143	locat=tasks_per_node; (( iec = 0 )); exit
1144	fi
1145
1146	fi
1147
1148	# CHECK IF HOSTFILE EXISTS
1149	if [[ -n $hostfile ]]
1150	then
1151	if [[ ! -f $hostfile ]]
1152	then
1153	printf "\n"
1154	printf "\n +++ hostfile \"$hostfile\" does not exist"
1155	locat=hostfile; exit
1156	fi
1157	fi
1158
1159
1160	# SET DEFAULT VALUE FOR THE MAXIMUM NUMBER OF PARALLEL IO STREAMS
1161	if [[ "$maximum_parallel_io_streams" = "" ]]
1162	then
1163	maximum_parallel_io_streams=$numprocs
1164	fi
1165
1166
1167	# SET PORT NUMBER OPTION FOR CALLS OF SSH/SCP, subjob AND batch_scp SCRIPTS
1168	if [[ "$scp_port" != "" ]]
1169	then
1170	PORTOPT="-P $scp_port"
1171	SSH_PORTOPT="-p $scp_port"
1172	fi
1173
1174
1175	# DETERMINE THE SSH-OPTION IN CASE THAT AN SSH-KEY IS EXPLICITLY GIVEN IN THE
1176	# CONFIG-FILE
1177	if [[ "$ssh_key" != "" ]]
1178	then
1179	ssh_key="-i $HOME/.ssh/$ssh_key"
1180	fi
1181
1182
1183	# SET DEFAULT-QUEUE, IF NOT GIVEN
1184	if [[ $queue = none ]]
1185	then
1186	case $host in
1187	(ibmh) queue=cluster;;
1188	(ibmkisti) queue=class.32plus;;
1189	(lcbullhh) queue=compute;;
1190	(lccrayb) queue=mpp1q;;
1191	(lccrayh) queue=mpp1q;;
1192	(lckiaps) queue=normal;;
1193	(lckyoto) queue=ph;;
1194	(lckyuh) queue=fx-single;;
1195	(lckyut) queue=cx-single;;
1196	(lctit) queue=S;;
1197	(unics) queue=unics;;
1198	esac
1199	fi
1200
1201
1202	# GENERATE FULL FILENAMES OF INPUT-FILES, INCLUDING THEIR PATH
1203	# CHECK, IF INPUT-FILES EXIST, AND DETERMINE HIGHEST CYCLE NUMBER (IF CYCLES EXIST)
1204	(( i = 0 ))
1205	while (( i < iin ))
1206	do
1207	(( i = i + 1 ))
1208	(( maxcycle = 0 ))
1209
1210	# GENERATE PATH AND FULL FILE NAME (then-BRANCH: FIXED FULL NAME IS GIVEN, I.E. THE
1211	# FILE IDENTIFIER IS NOT PART OF THE FILENAME))
1212	if [[ "${actionin[$i]}" = di ]]
1213	then
1214	remotepathin[$i]=${pathin[$i]}/${endin[$i]} # EVALUATE REMOTE-PATH ON THE REMOTE
1215	# HOST ONLY
1216	eval filename=${pathin[$i]}/${endin[$i]}
1217	else
1218	remotepathin[$i]=${pathin[$i]}/${afname}${endin[$i]} # EVALUATE REMOTE-PATH ON THE REMOTE
1219	# HOST ONLY
1220	eval filename=${pathin[$i]}/${afname}${endin[$i]}
1221	fi
1222	eval pathname=${pathin[$i]}
1223
1224	# CHECK IF FILE EXISTS
1225	if ! ls $filename* 1>/dev/null 2>&1
1226	then
1227
1228	# FILES WITH ATTRIBUTE locopt ARE OPTIONAL. NO ABORT, IF THEY DO NOT EXIST.
1229	if [[ "${transin[$i]}" != "locopt" ]]
1230	then
1231	printf "\n\n +++ INPUT-file: "
1232	if [[ "${extin[$i]}" = "" \|\| "${extin[$i]}" = " " ]]
1233	then
1234	printf "\n $filename"
1235	else
1236	printf "\n $filename.${extin[$i]}"
1237	fi
1238	printf "\n does not exist\n"
1239	locat=input; exit
1240	else
1241	transin[$i]="unavailable"
1242	fi
1243	else
1244
1245	# DETERMINE THE FILE'S CYCLE NUMBER
1246	ls -1 -d $filename > filelist 2>/dev/null
1247	ls -1 -d $filename.* >> filelist 2>/dev/null
1248	while read zeile
1249	do
1250	# filename without path (i.e. after the last "/")
1251	basefilename=$(basename ${zeile})
1252
1253	# check if there is an extension
1254	extension=${basefilename##*.}
1255	if [[ "$extension" = "$extin[$i]" ]]
1256	then
1257	basefilename=${basefilename&.*}
1258	fi
1259
1260	# check for an existing cycle number
1261	cycle=${basefilename##*.}
1262	if [[ $cycle =~ ^-?[0-9]+$ ]]
1263	then
1264	(( icycle = $cycle ))
1265	else
1266	(( icycle = 0 ))
1267	fi
1268
1269	if (( icycle > maxcycle ))
1270	then
1271	(( maxcycle = icycle ))
1272	fi
1273
1274	# cycle=$(echo $zeile \| cut -f2 -d".")
1275	# if [[ "$cycle" = "$zeile" ]]
1276	# then
1277	# (( icycle = 0 ))
1278	# elif [[ "$cycle" = "${extin[$i]}" ]]
1279	# then
1280	# (( icycle = 0 ))
1281	# else
1282	# (( icycle = $cycle ))
1283	# fi
1284	# if (( icycle > maxcycle ))
1285	# then
1286	# (( maxcycle = icycle ))
1287	# fi
1288	done <filelist
1289	rm filelist
1290
1291	# APPEND CYCLE NUMBER TO FILENAME
1292	if (( maxcycle > 0 ))
1293	then
1294	if [[ "${extin[$i]}" != " " && "${extin[$i]}" != "" ]]
1295	then
1296	filename=${filename}.$maxcycle.${extin[$i]}
1297	else
1298	filename=${filename}.$maxcycle
1299	fi
1300	else
1301	if [[ "${extin[$i]}" != " " && "${extin[$i]}" != "" ]]
1302	then
1303	filename=${filename}.${extin[$i]}
1304	fi
1305	fi
1306
1307	# STORE FILENAME WITHOUT PATH BUT WITH CYCLE NUMBER,
1308	# IS LATER USED FOR TRANSFERRING FILES WIHIN THE JOB (SEE END OF FILE)
1309	absnamein[$i]=$filename
1310	if (( maxcycle > 0 ))
1311	then
1312	if [[ "${actionin[$i]}" = di ]]
1313	then
1314	frelin[$i]=${endin[$i]}.$maxcycle
1315	else
1316	frelin[$i]=${afname}${endin[$i]}.$maxcycle
1317	fi
1318	else
1319	if [[ "${actionin[$i]}" = di ]]
1320	then
1321	frelin[$i]=${endin[$i]}
1322	else
1323	frelin[$i]=${afname}${endin[$i]}
1324	fi
1325	fi
1326
1327	fi
1328	done
1329
1330	# GENERATE FULL FILENAMES OF OUTPUT-FILES (WITHOUT $ OR ~),
1331	# CHECK, IF OUTPUT-FILES EXIST, AND DETERMINE HIGHEST CYCLE NUMBER (IF CYCLES EXIST),
1332	# OR, IN CASE THAT FILE DOES NOT EXIST, CHECK, IF IT CAN BE CREATED
1333	# THESE ACTIONS ARE NOT CARRIED OUT, IF FILES SHALL BE TRASFERRED FROM THE REMOTE TO
1334	# THE LOCAL HOST (BECAUSE THEIR IS NO DIRECT ACCESS TO THE LOCAL DIRECTORIES FROM THE
1335	# REMOTE HOST)
1336	(( i = 0 ))
1337	while (( i < iout ))
1338	do
1339	(( i = i + 1 ))
1340	if [[ ! ( $fromhost != $localhost && ( "${actionout[$i]}" = tr \|\| "${actionout[$i]}" = tra \|\| "${actionout[$i]}" = trpe ) ) ]]
1341	then
1342	if [[ "${actionout[$i]}" = tr ]]
1343	then
1344	actionout[$i]=""
1345	elif [[ "${actionout[$i]}" = trpe ]]
1346	then
1347	actionout[$i]=pe
1348	elif [[ "${actionout[$i]}" = tra ]]
1349	then
1350	actionout[$i]=a
1351	fi
1352	(( maxcycle = 0 ))
1353	eval filename=${pathout[$i]}/${fname}${endout[$i]}
1354	eval catalogname=${pathout[$i]}
1355	if ! ls $filename* 1>/dev/null 2>&1
1356	then
1357
1358	# IF OUTPUT-FILE DOES NOT EXIST CHECK, IF IT CAN BE CREATED
1359	if cat /dev/null > $filename
1360	then
1361	rm $filename
1362	else
1363
1364	# CHECK, IF THE DIRECTORY WHERE FILE SHALL BE COPIED TO EXISTS
1365	# IF IT DOES NOT EXIST, TRY TO CREATE IT
1366	if [[ ! -d $catalogname ]]
1367	then
1368	if mkdir -p $catalogname
1369	then
1370	printf "\n\n *** directory:"
1371	printf "\n $catalogname"
1372	printf "\n was created\n"
1373	echo "### filename = $filename"
1374	else
1375	printf "\n\n +++ OUTPUT-file:"
1376	printf "\n $filename"
1377	printf "\n cannot be created, because directory does not exist"
1378	printf "\n and cannot be created either"
1379	printf "\n"
1380	locat=output ; exit
1381	fi 2>/dev/null
1382	else
1383	printf "\n\n +++ OUTPUT-file:"
1384	printf "\n $filename"
1385	printf "\n cannot be created, although directory exists"
1386	printf "\n"
1387	locat=output ; exit
1388	fi
1389	fi 2>/dev/null
1390	else
1391
1392	# DETERMINE THE CYCLE NUMBER
1393	ls -1 -d $filename > filelist 2>/dev/null
1394	ls -1 -d $filename.* >> filelist 2>/dev/null
1395	while read zeile
1396	do
1397
1398	# filename without path (i.e. after the last "/")
1399	basefilename=$(basename ${zeile})
1400
1401	# check if there is an extension
1402	extension=${basefilename##*.}
1403	if [[ "$extension" = "$extin[$i]" ]]
1404	then
1405	basefilename=${basefilename&.*}
1406	fi
1407
1408	# check for an existing cycle number
1409	cycle=${basefilename##*.}
1410	if [[ $cycle =~ ^-?[0-9]+$ ]]
1411	then
1412	(( icycle = $cycle + 1 ))
1413	else
1414	(( icycle = 1 ))
1415	fi
1416
1417	if (( icycle > maxcycle ))
1418	then
1419	(( maxcycle = icycle ))
1420	fi
1421
1422	# cycle=$(echo $zeile \| cut -f2 -d".")
1423	# if [[ "$cycle" = "$zeile" \|\| "$cycle" = ${extout[$i]} ]]
1424	# then
1425	# (( icycle = 1 ))
1426	# else
1427	# (( icycle = $cycle + 1 ))
1428	# fi
1429	# if (( icycle > maxcycle ))
1430	# then
1431	# (( maxcycle = icycle ))
1432	# fi
1433	done <filelist
1434	rm filelist
1435	fi
1436
1437	# APPEND CYCLE NUMBER TO FILENAME AND CHECK, IF FILE CAN BE CREATED
1438	# IN CASE OF FILE-APPEND, FILE MUST BE APPENDED TO THE ONE WITH HIGHEST CYCLE NUMBER
1439	if [[ "${actionout[$i]}" != a ]]
1440	then
1441	if (( maxcycle > 0 ))
1442	then
1443	filename_tmp=${filename}.$maxcycle
1444	if cat /dev/null > $filename_tmp
1445	then
1446	rm $filename_tmp
1447	else
1448	printf "\n +++ OUTPUT-file:"
1449	printf "\n $filename_tmp"
1450	printf "\n cannot be created"
1451	locat=output ; exit
1452	fi
1453	fi
1454	else
1455	(( maxcycle = maxcycle - 1 ))
1456	fi
1457
1458	(( cycnum[$i] = maxcycle ))
1459	pathout[$i]=$filename
1460
1461	fi
1462	done
1463
1464
1465	# THE DVR-PACKAGE REQUIRES ITS OWN LIBRARY
1466	if [[ $(echo $package_list \| grep -c dvrp_graphics) != 0 ]]
1467	then
1468	if [[ "$dvr_inc" = "" ]]
1469	then
1470	printf "\n\n +++ no value for \"dvr_inc\" given in configuration file"
1471	printf "\n This is required for the dvrp_graphics package.\n"
1472	locat=dvr; exit
1473	fi
1474	if [[ "$dvr_lib" = "" ]]
1475	then
1476	printf "\n\n +++ no value for \"dvr_lib\" given in configuration file"
1477	printf "\n This is required for the dvrp_graphics package.\n"
1478	locat=dvr; exit
1479	fi
1480	fi
1481
1482
1483	# CHECK, WETHER A MAIN PROGRAM OR AN EXECUTABLE HAVE BEEN DECLARED.
1484	# EXECUTABLES DO NOT NEED TO BE COMPILED.
1485	if [[ "$mainprog" = "" && "$executable" = "" ]]
1486	then
1487	printf "\n +++ neither main program nor executable defined"
1488	locat=source; exit
1489	elif [[ "$mainprog" != "" && "$executable" != "" ]]
1490	then
1491	printf "\n +++ main program as well as executable defined"
1492	locat=source; exit
1493	elif [[ "$mainprog" = "" && "$executable" != "" ]]
1494	then
1495	do_compile=false
1496	fi
1497
1498
1499	# CREATE SOURCE-DIRECTORY TO COLLECT ROUTINES TO BE COMPILED.
1500	# LATER THE MRUN-SCRIPT AND CONFIGURATION-FILE WILL ALSO BE COPIED TO THIS DIRECTORY.
1501	if [[ $restart_run != true && "$SOURCES_COMPLETE" = "" ]]
1502	then
1503	rm -rf SOURCES_FOR_RUN_$fname
1504	mkdir SOURCES_FOR_RUN_$fname
1505	fi
1506
1507
1508	# COLLECT ALL ROUTINES TO BE COMPILED
1509	# THIS IS NOT REQUIRED WITHIN BATCH-JOBS, BECAUSE ROUTINES HAVE ALREADY BEEN COLLECTED
1510	# BY THE MRUN-CALL WHICH CREATED THE BATCH-JOB.
1511	if [[ $do_compile = true && "$SOURCES_COMPLETE" = "" ]]
1512	then
1513
1514	[[ "$source_list" = LM ]] && source_list=LOCALLY_MODIFIED
1515
1516	if [[ "$source_list" = LOCALLY_MODIFIED ]]
1517	then
1518
1519	# DETERMINE MODIFIED FILES OF THE SVN WORKING COPY
1520	source_list=""
1521	cd $source_path
1522
1523
1524	# CHECK, IF DIRECTORY IS UNDER SVN CONTROL
1525	if [[ ! -d .svn ]]
1526	then
1527	printf "\n\n +++ source directory"
1528	printf "\n \"$source_path\" "
1529	printf "\n is not under control of \"subversion\"."
1530	printf "\n Please do not use mrun-option \"-s LOCALLY_MODIFIED\"\n"
1531	fi
1532
1533
1534	# LIST ALL MODIFIED SOURCE CODE FILES
1535	Filenames=""
1536	svn status > tmp_mrun
1537	while read line
1538	do
1539	firstc=`echo $line \| cut -c1`
1540	if [[ $firstc = M \|\| $firstc = "?" ]]
1541	then
1542	Name=`echo "$line" \| cut -c8-`
1543	extension=`echo $Name \| cut -d. -f2`
1544	if [[ "$extension" = f90 \|\| "$extension" = F90 \|\| "$extension" = f \|\| "$extension" = F \|\| "$extension" = c ]]
1545	then
1546	Filenames="$Filenames "$Name
1547	fi
1548	fi
1549	done < tmp_mrun
1550
1551
1552	# COPY FILES TO SOURCES_FOR_RUN_...
1553	for dateiname in $Filenames
1554	do
1555	cp $dateiname $working_directory/SOURCES_FOR_RUN_$fname
1556	source_list=$source_list"$dateiname "
1557	done
1558
1559	cd - > /dev/null
1560
1561
1562	# COPY FILES GIVEN BY OPTION -s TO DIRECTORY SOURCES_FOR_RUN_...
1563	# AUTOMATIC RESTART RUNS JUST ACCESS THE DIRECTORY CREATED BY THE INITIAL RUN
1564	elif [[ "$source_list" != "" && $restart_run != true ]]
1565	then
1566
1567	cd $source_path
1568
1569	for filename in $source_list
1570	do
1571
1572	# SOURCE CODE FILE IS NOT ALLOWED TO INCLUDE PATH
1573	if [[ $(echo $filename \| grep -c "/") != 0 ]]
1574	then
1575	printf "\n +++ source code file: $filename"
1576	printf "\n must not contain (\"/\") "
1577	locat=source; exit
1578	fi
1579
1580	if [[ ! -f $filename ]]
1581	then
1582	printf "\n +++ source code file: $filename"
1583	printf "\n does not exist"
1584	locat=source; exit
1585	else
1586	cp $filename $working_directory/SOURCES_FOR_RUN_$fname
1587	fi
1588
1589	done
1590
1591	cd - > /dev/null
1592
1593	fi
1594
1595
1596	# CHECK, IF A MAIN PROGRAM EXISTS AND IF IT IS PART OF THE LIST OF FILES
1597	# TO BE COMPILED. IF NOT, ADD IT TO THE LIST.
1598	if [[ $restart_run != true ]]
1599	then
1600
1601	if [[ ! -f "$source_path/$mainprog" ]]
1602	then
1603	printf "\n\n +++ main program: $mainprog"
1604	printf "\n does not exist in source directory"
1605	printf "\n \"$source_path\"\n"
1606	locat=source; exit
1607	else
1608	if [[ $(echo $source_list \| grep -c $mainprog) = 0 ]]
1609	then
1610	cp $source_path/$mainprog SOURCES_FOR_RUN_$fname
1611	source_list=${mainprog}" $source_list"
1612	fi
1613	fi
1614	fi
1615
1616
1617	# CHECK, IF MAKEFILE EXISTS AND COPY IT TO THE SOURCES_FOR_RUN... DIRECTORY
1618	# NOT REQUIRED FOR RESTART RUNS, SOURCES_FOR_RUN... HAS BEEN CREATED BY THE INITIAL RUN
1619	if [[ "$restart_run" != true ]]
1620	then
1621	[[ "$makefile" = "" ]] && makefile=$source_path/Makefile
1622	if [[ ! -f $makefile ]]
1623	then
1624	printf "\n +++ file \"$makefile\" does not exist"
1625	locat=make; exit
1626	else
1627	cp $makefile SOURCES_FOR_RUN_$fname/Makefile
1628	fi
1629	fi
1630
1631
1632	# COPY FILES FROM OPTIONAL SOURCE PATH GIVEN IN THE CONFIGURATION FILE
1633	if [[ $restart_run != true && "$add_source_path" != "" ]]
1634	then
1635
1636	# DOES THE DIRECTORY EXIST?
1637	if [[ ! -d $add_source_path ]]
1638	then
1639	printf "\n\n *** INFORMATIVE: additional source code directory"
1640	printf "\n \"$add_source_path\" "
1641	printf "\n does not exist or is not a directory."
1642	printf "\n No source code will be used from this directory!\n"
1643	add_source_path=""
1644	if [[ $silent == false ]]
1645	then
1646	sleep 3
1647	fi
1648	else
1649
1650	cd $add_source_path
1651	found=false
1652
1653	Names=$(ls -1 *.f90 2>&1)
1654	[[ $(echo $Names \| grep -c '*.f90') = 0 ]] && AddFilenames="$Names"
1655	Names=$(ls -1 *.F90 2>&1)
1656	[[ $(echo $Names \| grep -c '*.F90') = 0 ]] && AddFilenames="$AddFilenames $Names"
1657	Names=$(ls -1 *.F 2>&1)
1658	[[ $(echo $Names \| grep -c '*.F') = 0 ]] && AddFilenames="$AddFilenames $Names"
1659	Names=$(ls -1 *.f 2>&1)
1660	[[ $(echo $Names \| grep -c '*.f') = 0 ]] && AddFilenames="$AddFilenames $Names"
1661	Names=$(ls -1 *.c 2>&1)
1662	[[ $(echo $Names \| grep -c '*.c') = 0 ]] && AddFilenames="$AddFilenames $Names"
1663
1664	cd - > /dev/null
1665	cd SOURCES_FOR_RUN_$fname
1666
1667	# COPY MAKEFILE IF EXISTING
1668	if [[ -f $add_source_path/Makefile ]]
1669	then
1670	printf "\n\n *** user Makefile from directory"
1671	printf "\n \"$add_source_path\" is used \n"
1672	if [[ $silent == false ]]
1673	then
1674	sleep 1
1675	fi
1676	cp $add_source_path/Makefile .
1677	fi
1678
1679	for dateiname in $AddFilenames
1680	do
1681	if [[ -f $dateiname ]]
1682	then
1683	printf "\n +++ source code file \"$dateiname\" found in additional"
1684	printf "\n source code directory \"$add_source_path\" "
1685	printf "\n but was also given with option \"-s\" which means that it should be taken"
1686	printf "\n from directory \"$source_path\"."
1687	locat=source; exit
1688	fi
1689
1690	cp $add_source_path/$dateiname .
1691	source_list="$source_list $dateiname"
1692
1693	# CHECK IF FILE IS CONTAINED IN MAKEFILE
1694	if [[ $(grep -c $dateiname Makefile) = 0 ]]
1695	then
1696	printf "\n\n +++ user file \"$dateiname\" "
1697	printf "\n is not listed in Makefile \n"
1698	locat=source; exit
1699	else
1700
1701	if [[ $found = false ]]
1702	then
1703	found=true
1704	printf "\n\n *** following user file(s) added to the"
1705	printf " files to be translated:\n "
1706	fi
1707	printf "$dateiname "
1708	if [[ $silent == false ]]
1709	then
1710	sleep 0.5
1711	fi
1712
1713	fi
1714	done
1715	[[ $found = true ]] && printf "\n"
1716	cd - > /dev/null
1717	fi
1718	fi
1719
1720
1721	# ADD ALL ROUTINES BELONGING TO SOFTWARE PACKAGES (GIVEN BY OPTION -p)
1722	# TO THE LIST OF FILES TO BE COMPILED
1723	if [[ $restart_run != true && -n $package_list ]]
1724	then
1725
1726	cd $source_path
1727
1728	for package in $package_list
1729	do
1730
1731	[[ $package = "dvrp_graphics+1PE" ]] && package=dvrp_graphics
1732
1733	# DETERMINE FILES BELONGING TO THE PACKAGE
1734	# ERROR MESSAGE ARE REDIRECTED TO /dev/null, BECAUSE WILDCARD (*) ALSO GIVES
1735	# THE NAME OF THE DIRECTORY
1736	package_source_list=`grep "defined( __$package " * 2>/dev/null \| cut -f1 -d:`
1737
1738
1739	# ADD THESE FILES TO THE LIST OF FILES TO BE COMPILED,
1740	# IF THEY ARE NOT ALREADY PART OF THE LIST
1741	for source_list_name in $package_source_list
1742	do
1743	if [[ $(echo $source_list \| grep -c $source_list_name) = 0 ]]
1744	then
1745
1746	# ONLY TAKE FILES WITH VALID SUFFIX
1747	ending=`echo $source_list_name \| cut -f2 -d.`
1748	if [[ "$ending" = f90 \|\| "$ending" = F90 \|\| "$ending" = f \|\| "$ending" = F \|\| "$ending" = c ]]
1749	then
1750	cp $source_list_name $working_directory/SOURCES_FOR_RUN_$fname
1751	source_list="$source_list $source_list_name"
1752	fi
1753	fi
1754	done
1755	done
1756
1757	cd - > /dev/null
1758	fi
1759
1760	fi # do_compile=true
1761
1762
1763	# IF SOURCE CODE IS TO BE COMPILED, DO SOME MORE CHECKS
1764	# AND SET PRE-PROCESSOR DIRECTIVES
1765	if [[ $do_compile = true \|\| $create_executable_for_batch = true ]]
1766	then
1767
1768	# SET PREPROCESSOR-DIRECTIVES TO SELECT OPERATING SYSTEM SPECIFIC CODE
1769	if [[ $(echo $localhost \| cut -c1-3) = ibm ]]
1770	then
1771	cpp_options="${cpp_options},-D__ibm=__ibm"
1772	elif [[ $(echo $localhost \| cut -c1-3) = nec ]]
1773	then
1774	cpp_options="$cpp_options -D__nec"
1775	elif [[ $(echo $localhost \| cut -c1-2) = lc ]]
1776	then
1777	cpp_options="$cpp_options -D__lc"
1778	else
1779	cpp_options="$cpp_options -D__$localhost"
1780	fi
1781
1782	# SET DIRECTIVES GIVEN BY OPTION -K (E.G. parallel)
1783	if [[ $(echo $localhost \| cut -c1-3) = ibm ]]
1784	then
1785	[[ -n $cond1 ]] && cpp_options="${cpp_options},-D__$cond1=__$cond1"
1786	[[ -n $cond2 ]] && cpp_options="${cpp_options},-D__$cond2=__$cond2"
1787	else
1788	[[ -n $cond1 ]] && cpp_options="$cpp_options -D__$cond1"
1789	[[ -n $cond2 ]] && cpp_options="$cpp_options -D__$cond2"
1790	fi
1791
1792	# SET DIRECTIVES FOR ACTIVATING SOFTWARE-PACKAGES (OPTION -p)
1793	if [[ -n $package_list ]]
1794	then
1795	for package in $package_list
1796	do
1797	if [[ $(echo $localhost \| cut -c1-3) = ibm ]]
1798	then
1799	if [[ $package != "dvrp_graphics+1PE" ]]
1800	then
1801	cpp_options="${cpp_options},-D__$package=__$package"
1802	else
1803	cpp_options="${cpp_options},-D__dvrp_graphics=__dvrp_graphics"
1804	export use_seperate_pe_for_dvrp_output=true
1805	fi
1806	else
1807	if [[ $package != "dvrp_graphics+1PE" ]]
1808	then
1809	cpp_options="$cpp_options -D__$package"
1810	else
1811	cpp_options="$cpp_options -D__dvrp_graphics"
1812	export use_seperate_pe_for_dvrp_output=true
1813	fi
1814	fi
1815	done
1816	fi
1817
1818	# SET DIRECTIVES GIVEN BY OPTION -D
1819	if [[ -n $cpp_opts ]]
1820	then
1821	for popts in $cpp_opts
1822	do
1823	if [[ $(echo $localhost \| cut -c1-3) = ibm ]]
1824	then
1825	cpp_options="${cpp_options},-D__$popts=__$popts"
1826	else
1827	cpp_options="$cpp_options -D__$popts"
1828	fi
1829	done
1830	fi
1831
1832	else
1833
1834
1835	# FOR LOCAL RUNS CHECK AGAIN, IF EXECUTABLE EXISTS
1836	if [[ $do_remote = false ]]
1837	then
1838	if [[ ! -f $executable ]]
1839	then
1840	printf "\n +++ executable file: $executable"
1841	printf "\n does not exist"
1842	locat=executable; exit
1843	fi
1844	fi
1845	fi
1846
1847
1848	# DETERMINE THE JOB MODE
1849	if [[ $(echo $localhost \| cut -c1-3) = ibm ]]
1850	then
1851	if [[ "$LOADLBATCH" = yes ]]
1852	then
1853	batch_job=.TRUE.
1854	jobmo=BATCH
1855	else
1856	batch_job=.FALSE.
1857	jobmo=INTERACTIVE
1858	fi
1859	else
1860	if [[ "$ENVIRONMENT" = BATCH \|\| "$PJM_ENVIRONMENT" = BATCH ]]
1861	then
1862	batch_job=.TRUE.
1863	jobmo=BATCH
1864	else
1865	batch_job=.FALSE.
1866	jobmo=INTERACTIVE
1867	fi
1868	fi
1869
1870	# NO INTERACTIVE RUNS ALLOWED ON LCTIT
1871	if [[ $host = lctit && $jobmo = INTERACTIVE && $do_batch = false ]]
1872	then
1873	printf "\n +++ no interactive runs allowed on host \"$host\" "
1874	printf "\n please submit batch job using mrun option \"-b\" \n"
1875	locat=normal; exit
1876	fi
1877
1878
1879	# CHECK, IF USER DEFINED A COMPILER
1880	if [[ "$compiler_name" = "" ]]
1881	then
1882
1883	printf "\n +++ no compiler specified for \"$host $cond1 $cond2\""
1884	locat=compiler_name; exit
1885
1886	fi
1887
1888
1889	# DETERMINE THE NAME OF MRUN'S TEMPORARY WORKING DIRECTORY
1890
1891	# ON HLRN-III, USE THE QUEUING NAME. OTHERWISE USE USERNAME AND RANDOM NUMBER
1892	if [[ $do_batch = false && $(echo $host \| cut -c1-6) = lccray ]]
1893	then
1894	kennung=$(checkjob $PBS_JOBID \| grep Reservation \| cut -d" " -s -f2 \| cut -d"." -s -f2 \| sed "s/['\"]//g")
1895	if [[ "$kennung" = "" ]]
1896	then
1897	kennung=$RANDOM
1898	fi
1899	else
1900	kennung=$RANDOM
1901	fi
1902
1903	if [[ "$tmp_user_catalog" = "" ]]
1904	then
1905	if [[ $localhost = ibmh ]]
1906	then
1907	tmp_user_catalog=$SCRATCH
1908	else
1909	tmp_user_catalog=/tmp
1910	fi
1911	fi
1912	TEMPDIR=$tmp_user_catalog/${usern}.$kennung
1913
1914
1915	# DETERMINE THE NAME OF THE DIRECTORY WHICH IS USED TO TEMPORARILY STORE DATA FOR RESTART RUNS
1916	if [[ "$tmp_data_catalog" = "" ]]
1917	then
1918	tmp_data_catalog=/tmp/mrun_restart_data
1919	fi
1920
1921
1922	# IN CASE OF LOCAL RUNS REPLACE ENVIRONMENT VARIABLES BY THEIR VALUES
1923	if [[ $do_remote = false && $do_compile = true \|\| $create_executable_for_batch = true ]]
1924	then
1925	eval fopts=\"$fopts\"
1926	eval lopts=\"$lopts\"
1927	fi
1928
1929
1930	# DETERMINE COMPILE- AND LINK-OPTIONS
1931	fopts="$fopts $netcdf_inc $fftw_inc $dvr_inc"
1932	lopts="$lopts $netcdf_lib $fftw_lib $dvr_lib"
1933	XOPT="-X $numprocs"
1934
1935
1936	# CHECK THE CPU-LIMIT. IT MUST BE GIVEN FOR BATCH-JOBS AND IS COMMUNICATED TO THE
1937	# EXECUTABLE VIA NAMELIST-PARAMETER cputime
1938	done=false
1939	while [[ $done = false ]]
1940	do
1941	cputime=$cpumax
1942	if (( $cputime == 0 ))
1943	then
1944	if [[ $do_batch = true ]]
1945	then
1946	printf "\n +++ cpu-time is undefined"
1947	printf "\n >>> Please type CPU-time in seconds as INTEGER:"
1948	printf "\n >>> "
1949	read cputime 1>/dev/null 2>&1
1950	else
1951	cputime=10000000 # NO CPU LIMIT FOR INTERACTIVE RUNS
1952	fi
1953	else
1954	done=true
1955	fi
1956	cpumax=$cputime
1957	done
1958
1959	(( minuten = cputime / 60 ))
1960	(( sekunden = cputime - minuten * 60 ))
1961
1962
1963	# CHECK THE MEMORY DEMAND
1964	if [[ $do_batch = true ]]
1965	then
1966	done=false
1967	while [[ $done = false ]]
1968	do
1969	if (( memory == 0 ))
1970	then
1971	printf "\n +++ memory demand is undefined"
1972	printf "\n >>> Please type memory in MByte per process as INTEGER:"
1973	printf "\n >>> "
1974	read memory 1>/dev/null 2>&1
1975	else
1976	done=true
1977	fi
1978	done
1979	fi
1980
1981
1982	# IN CASE OF REMOTE-JOBS CHECK, IF A USERNAME FOR THE REMOTE HOST IS GIVEN
1983	if [[ $do_remote = true && -z $remote_username ]]
1984	then
1985	while [[ -z $remote_username ]]
1986	do
1987	printf "\n +++ username on remote host \"$host\" is undefined"
1988	printf "\n >>> Please type username:"
1989	printf "\n >>> "
1990	read remote_username
1991	done
1992	mc="$mc -u$remote_username"
1993	fi
1994
1995	# CHECK FOR INITIAL COMMANDS AFTER LOGIN
1996	if [[ "$login_init_cmd" != "" ]]
1997	then
1998	export init_cmds="${login_init_cmd};"
1999	fi
2000
2001
2002	# SET THE MODULE-LOAD COMMAD AND EXPORT IT FOR subjob
2003	if [[ "$modules" != "" ]]
2004	then
2005	if [[ $host = lctit ]]
2006	then
2007	export module_calls=". $modules"
2008	else
2009	export module_calls="module load ${modules};"
2010	fi
2011	fi
2012
2013	# OUTPUT OF THE MRUN-HEADER
2014	calltime=$(date)
2015	printf "\n"
2016	printf "#------------------------------------------------------------------------# \n"
2017	printf "\| %-35s%35s \| \n" "$version" "$calltime"
2018	printf "\| \| \n"
2019	spalte1="called on:"; spalte2=$localhost_realname
2020	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2021	if [[ $do_remote = true ]]
2022	then
2023	spalte1="execution on:"; spalte2="$host (username: $remote_username)"
2024	else
2025	spalte1="execution on:"; spalte2="$host ($localhost_realname)"
2026	fi
2027	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2028	if [[ -n $numprocs ]]
2029	then
2030	if [[ $run_coupled_model = false ]]
2031	then
2032	spalte1="number of PEs:"; spalte2=$numprocs
2033	else
2034	spalte1="number of PEs:"; spalte2="$numprocs (atmosphere: $numprocs_atmos, ocean: $numprocs_ocean)"
2035	fi
2036	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2037	fi
2038	if [[ -n $tasks_per_node ]]
2039	then
2040	spalte1="tasks per node:"; spalte2="$tasks_per_node (number of nodes: $nodes)"
2041	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2042	if (( remaining_pes > 0 ))
2043	then
2044	spalte1=" "; spalte2="one of the nodes only filled with $remaining_pes tasks"
2045	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2046	fi
2047	fi
2048	if [[ $maximum_parallel_io_streams != $numprocs ]]
2049	then
2050	spalte1="max par io streams:"; spalte2="$maximum_parallel_io_streams"
2051	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2052	fi
2053	if [[ $use_openmp = true ]]
2054	then
2055	spalte1="threads per task:"; spalte2="$threads_per_task"
2056	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2057	fi
2058	printf "\| \| \n"
2059	if [[ $do_compile = true ]]
2060	then
2061	if [[ "$mopts" != "" ]]
2062	then
2063	spalte1="make options:"; spalte2=$(echo "$mopts" \| cut -c-45)
2064	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2065	zeile=$(echo "$mopts" \| cut -c46-)
2066	while [[ "$zeile" != "" ]]
2067	do
2068	spalte1=""
2069	spalte2=$(echo "$zeile" \| cut -c-45)
2070	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2071	zeile=$(echo "$zeile" \| cut -c46-)
2072	done
2073	fi
2074
2075	spalte1="cpp directives:"; spalte2=$(echo "$cpp_options" \| cut -c-45)
2076	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2077	zeile=$(echo "$cpp_options" \| cut -c46-)
2078	while [[ "$zeile" != "" ]]
2079	do
2080	spalte1=""
2081	spalte2=$(echo "$zeile" \| cut -c-45)
2082	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2083	zeile=$(echo "$zeile" \| cut -c46-)
2084	done
2085
2086	spalte1="compiler options:"; spalte2=$(echo "$fopts" \| cut -c-45)
2087	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2088	zeile=$(echo "$fopts" \| cut -c46-)
2089	while [[ "$zeile" != "" ]]
2090	do
2091	spalte1=""
2092	spalte2=$(echo "$zeile" \| cut -c-45)
2093	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2094	zeile=$(echo "$zeile" \| cut -c46-)
2095	done
2096
2097	spalte1="linker options:"; spalte2=$(echo "$lopts" \| cut -c-45)
2098	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2099	zeile=$(echo "$lopts" \| cut -c46-)
2100	while [[ "$zeile" != "" ]]
2101	do
2102	spalte1=""
2103	spalte2=$(echo "$zeile" \| cut -c-45)
2104	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2105	zeile=$(echo "$zeile" \| cut -c46-)
2106	done
2107
2108	spalte1="modules to be load:"; spalte2=$(echo "$modules" \| cut -c-45)
2109	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2110	zeile=$(echo "$modules" \| cut -c46-)
2111	while [[ "$zeile" != "" ]]
2112	do
2113	spalte1=""
2114	spalte2=$(echo "$zeile" \| cut -c-45)
2115	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2116	zeile=$(echo "$zeile" \| cut -c46-)
2117	done
2118
2119	spalte1="main program:"; spalte2=$mainprog
2120	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2121	else
2122	spalte1=executable:; spalte2=$executable
2123	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2124	fi
2125	printf "\| \| \n"
2126	spalte1="base name of files:"; spalte2=$fname
2127	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2128	if [[ $fname != $afname ]]
2129	then
2130	spalte1="base name of input files:"; spalte2=$afname
2131	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2132	fi
2133	spalte1="INPUT control list:"; spalte2=$(echo $input_list)
2134	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2135	spalte1="OUTPUT control list:"; spalte2=$(echo $output_list)
2136	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2137
2138	if [[ "$ocean_file_appendix" = true ]]
2139	then
2140	printf "\| %-35s%-35s \| \n" "suffix \"_O\" is added to local files" " "
2141	fi
2142
2143	if [[ $do_batch = true \|\| "$LOADLBATCH" = yes ]]
2144	then
2145	spalte1="memory demand / PE":; spalte2="$memory MB"
2146	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2147	spalte1=CPU-time:; spalte2="$minuten:$sekunden"
2148	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2149	fi
2150
2151	if [[ $do_compile = true ]]
2152	then
2153	printf "\| \| \n"
2154	printf "\| Files to be compiled: \| \n"
2155	zeile=$source_list
2156	while [[ "$zeile" != "" ]]
2157	do
2158	linestart=$(echo $zeile \| cut -c-70)
2159	printf "\| %-70s \| \n" "$linestart"
2160	zeile=$(echo "$zeile" \| cut -c71-)
2161	done
2162	fi
2163	printf "#------------------------------------------------------------------------#"
2164
2165
2166
2167	# OUTPUT OF FILE CONNECTIONS IN CASE OF TRACEBACK
2168	if [[ $do_trace = true ]]
2169	then
2170	(( i = 0 ))
2171	while (( i < iin ))
2172	do
2173	(( i = i + 1 ))
2174	if (( i == 1 ))
2175	then
2176	printf "\n\n >>> INPUT-file assignments:\n"
2177	fi
2178	printf "\n ${localin[$i]} : ${absnamein[$i]}"
2179	done
2180	(( i = 0 ))
2181	while (( i < iout ))
2182	do
2183	(( i = i + 1 ))
2184	if (( i == 1 ))
2185	then
2186	printf "\n\n >>> OUTPUT-file assignments:\n"
2187	fi
2188	printf "\n ${localout[$i]} : ${pathout[$i]}"
2189	done
2190	(( i = 0 ))
2191	while (( i < iic ))
2192	do
2193	(( i = i + 1 ))
2194	if (( i == 1 ))
2195	then
2196	printf "\n\n >>> INPUT-commands:\n"
2197	fi
2198	printf "\n ${in_command[$i]}"
2199	done
2200	(( i = 0 ))
2201	while (( i < ioc ))
2202	do
2203	(( i = i + 1 ))
2204	if (( i == 1 ))
2205	then
2206	printf "\n\n >>> OUTPUT-commands:\n"
2207	fi
2208	printf "\n ${out_command[$i]}"
2209	done
2210	fi
2211
2212	# QUERY FOR CONTINUE (ON LOCAL MACHINES ONLY)
2213	if [[ $remotecall = false && $silent = false && $jobmo != BATCH ]]
2214	then
2215	antwort=dummy
2216	printf "\n\n"
2217	printf " >>> everything o.k. (y/n) ? "
2218	while read antwort
2219	do
2220	if [[ "$antwort" != y && "$antwort" != Y && "$antwort" != n && "$antwort" != N ]]
2221	then
2222	printf " >>> everything o.k. (y/n) ? "
2223	else
2224	break
2225	fi
2226	done
2227	if [[ $antwort = n \|\| $antwort = N ]]
2228	then
2229	locat=user_abort; (( iec = 0 )); exit
2230	fi
2231	if [[ $do_batch = true ]]
2232	then
2233	printf " >>> batch-job will be created and submitted"
2234	else
2235	printf " >>> MRUN will now continue to execute on this machine"
2236	fi
2237	fi
2238
2239
2240	# DETERMINE PATH FOR MAKE DEPOSITORY
2241	if [[ $do_batch = false \|\| $create_executable_for_batch = true ]]
2242	then
2243
2244	line=""
2245	grep "%depository_path" $config_file > tmp_mrun
2246	while read line
2247	do
2248	if [[ "$line" != "" && $(echo $line \| cut -c1) != "#" ]]
2249	then
2250	if [[ "$(echo $line \| cut -d" " -s -f3)" = "" ]]
2251	then
2252	global_depository_path=`echo $line \| cut -d" " -s -f2`
2253	fi
2254	fi
2255	done < tmp_mrun
2256
2257	line=""
2258	grep " $localhost" $config_file \| grep "%depository_path" > tmp_mrun
2259	while read line
2260	do
2261	if [[ "$line" != "" && $(echo $line \| cut -c1) != "#" ]]
2262	then
2263	if [[ "$(echo $line \| cut -d" " -s -f4)" = "$cond1" && "$(echo $line \| cut -d" " -s -f5)" = "$cond2" ]]
2264	then
2265	local_depository_path=`echo $line \| cut -d" " -s -f2`
2266	fi
2267	fi
2268	done < tmp_mrun
2269
2270	if [[ "$local_depository_path" = "" ]]
2271	then
2272	if [[ "$global_depository_path" != "" ]]
2273	then
2274	local_depository_path=$global_depository_path
2275	else
2276	printf "\n\n +++ no depository path found in configuration file"
2277	printf "\n for local host \"$localhost\" "
2278	printf "\n please set \"\%depository_path\" in configuration file\n"
2279	locat=config_file; exit
2280	fi
2281	fi
2282	eval local_depository_path=$local_depository_path
2283	[[ "$cond1" != "" ]] && local_depository_path=${local_depository_path}_$cond1
2284	[[ "$cond2" != "" ]] && local_depository_path=${local_depository_path}_$cond2
2285
2286
2287	basename=`echo $mainprog \| cut -f1 -d"."`
2288	eval make_depository=${local_depository_path}/${basename}_current_version.tar
2289	if [[ ! -f $make_depository ]]
2290	then
2291	printf "\n"
2292	printf "\n *** WARNING: make depository \"$make_depository\" not found"
2293	printf "\n \"make\" will fail, if the Makefile or other source files are missing\n"
2294	fi
2295
2296	fi
2297
2298
2299	# NOW PERFORM THOSE ACTIONS REQUIRED TO EXECUTE THE PROGRAM (PALM) ON THIS MACHINE
2300	# (COMPILING/LINKING, EXECUTING, COPYING I/O FILES)
2301	if [[ $do_batch = false ]]
2302	then
2303
2304
2305	# CREATE THE TEMPORARY WORKING DIRECTORY
2306	mkdir -p $TEMPDIR
2307	chmod go+rx $TEMPDIR
2308	tmpcreate=true
2309
2310
2311	# COPY EITHER THE COMPLETE SOURCE CODE FILES TO BE COMPILED OR THE EXECUTABLE
2312	# INTO THE TEMPORARY WORKING DIRECTORY
2313	if [[ $do_compile = true ]]
2314	then
2315
2316	TEMPDIR_COMPILE=$TEMPDIR
2317
2318	cp $make_depository $TEMPDIR_COMPILE
2319	cd $TEMPDIR_COMPILE
2320	tar -xf $make_depository > /dev/null 2>&1
2321	cd - > /dev/null
2322
2323	# DUE TO UNKNOWN REASONS, COPY WITH cp COMMAND CREATES CORRUPT
2324	# FILES ON CRAY XC30 SYSTEMS (HLRN III), rsync IS USED INSTEAD
2325	if [[ $(echo $host \| cut -c1-6) = lccray ]]
2326	then
2327	rsync -av -t SOURCES_FOR_RUN_$fname/* $TEMPDIR_COMPILE > /dev/null
2328	else
2329	cp SOURCES_FOR_RUN_$fname/* $TEMPDIR_COMPILE > /dev/null
2330	fi
2331
2332	else
2333
2334	cp $executable ${TEMPDIR}/a.out
2335
2336	fi
2337
2338
2339	# CHANGE TO THE TEMPORARY WORKING DIRECTORY
2340	cd $TEMPDIR
2341	printf "\n *** changed to temporary directory: $TEMPDIR"
2342
2343
2344	# IF REQUIRED, START WITH COMPILING
2345	if [[ $do_compile = true ]]
2346	then
2347
2348
2349	if [[ -f a.out ]]
2350	then
2351
2352	# EXECUTABLE WAS CREATED DURING INTERACTIVE CALL OF MRUN
2353	printf "\n\n\n *** executable a.out found"
2354	printf "\n no compilation required \n"
2355
2356	else
2357
2358	# COMPILING WITH MAKE (ON NEC COMPILER IS CALLED ON HOST CROSS)
2359	printf "\n\n\n *** compilation starts \n$dashes\n"
2360	printf " *** compilation with make using following options:\n"
2361	printf " make depository: $make_depository"
2362	if [[ "$mopts" != "" ]]
2363	then
2364	printf " make options: $mopts\n"
2365	fi
2366	printf " compilername: $compiler_name\n"
2367	printf " compiler options: $fopts\n"
2368	printf " preprocessor directives: $cpp_options \n"
2369	printf " linker options: $lopts \n"
2370	if [[ "$modules" != "" ]]
2371	then
2372	printf " modules to be load: $modules \n"
2373	fi
2374	printf " source code files: $source_list \n"
2375
2376	if [[ $localhost = ibmh ]]
2377	then
2378	printf " compiler is called via ssh on \"plogin1\" \n"
2379	ssh $SSH_PORTOPT plogin1 -l $usern "$init_cmds export PATH=/sw/ibm/xlf/13.1.0.8/usr/bin:$PATH; $module_calls cd $TEMPDIR; make $mopts -f Makefile PROG=a.out F90=$compiler_name COPT=\"$cpp_options\" F90FLAGS=\"$fopts\" LDFLAGS=\"$lopts\" "
2380	[[ ! -f a.out ]] && compile_error=true
2381	continue # STATUS=1, IF a.out EXISTS
2382	elif [[ $localhost = lcocean ]]
2383	then
2384	printf " compiler is called via ssh on \"ocean\" \n"
2385	echo $PWD
2386	ssh $SSH_PORTOPT ocean -l $usern "$init_cmds $module_calls cd $TEMPDIR; make $mopts -f Makefile PROG=a.out F90=$compiler_name COPT=\"$cpp_options\" F90FLAGS=\"$fopts\" LDFLAGS=\"$lopts\" "
2387	[[ ! -f a.out ]] && compile_error=true
2388	continue # STATUS=1, IF a.out EXISTS
2389	elif [[ $localhost = lcbullhh \|\| $localhost = lccrayb \|\| $localhost = lccrayf \|\| $localhost = lccrayh ]]
2390	then
2391	make $mopts -f Makefile PROG=a.out F90=$compiler_name COPT="$cpp_options" F90FLAGS="$fopts" LDFLAGS="$lopts"
2392	elif [[ $localhost = lceddy ]]
2393	then
2394	which mpifc
2395	make $mopts -f Makefile PROG=a.out F90=$compiler_name COPT="$cpp_options" F90FLAGS="$fopts" LDFLAGS="$lopts"
2396	else
2397	[[ "$init_cmds" != "" ]] && eval $init_cmds
2398	[[ "$module_calls" != "" ]] && eval $module_calls
2399	make $mopts -f Makefile PROG=a.out F90=$compiler_name COPT="$cpp_options" F90FLAGS="$fopts" LDFLAGS="$lopts"
2400	fi
2401
2402	if [[ $? != 0 \|\| "$compile_error" = true \|\| "$module_compile_error" = true ]]
2403	then
2404	printf "\n +++ error occured while compiling or linking"
2405	locat=compile
2406
2407	exit
2408	else
2409	printf "$dashes\n *** compilation finished \n"
2410	fi
2411
2412	fi
2413
2414	fi
2415
2416
2417	# PROVIDE THE INPUT FILES
2418	# LOOP OVER ALL ACTIVATED FILES (LISTED IN THE CONFIGURATION FILE)
2419	(( i = 0 ))
2420	while (( i < iin ))
2421	do
2422	(( i = i + 1 ))
2423	if (( i == 1 ))
2424	then
2425	printf "\n\n *** providing INPUT-files:\n$dashes"
2426	fi
2427
2428
2429	# SKIP OPTIONAL FILES, IF THEY DO NOT EXIST
2430	if [[ "${transin[$i]}" = unavailable ]]
2431	then
2432	if [[ "${extin[$i]}" = "" \|\| "${extin[$i]}" = " " ]]
2433	then
2434	printf "\n *** INFORMATIVE: input file \"${pathin[$i]}/${afname}${endin[$i]}\" "
2435	printf "\n is not available!"
2436	else
2437	printf "\n *** INFORMATIVE: input file \"${pathin[$i]}/${afname}${endin[$i]}.${extin[$i]}\" "
2438	printf "\n is not available!"
2439	fi
2440	continue
2441	fi
2442
2443	# CHECK FOR SINGLE FILE (SERIAL RUN) OR DIRECTORY (ONE FILE PER CORE FOR PARELLEL EXECUTION)
2444	files_for_pes=false; datentyp=file
2445	if [[ "${actionin[$i]}" = pe && -n $numprocs ]]
2446	then
2447	files_for_pes=true; datentyp=files
2448	actionin[$i]=""
2449	elif [[ "${actionin[$i]}" = pe && ! -n $numprocs ]]
2450	then
2451	actionin[$i]=""
2452	elif [[ "${actionin[$i]}" = lnpe && -n $numprocs ]]
2453	then
2454	files_for_pes=true; datentyp=files
2455	actionin[$i]="ln"
2456	elif [[ "${actionin[$i]}" = lnpe && ! -n $numprocs ]]
2457	then
2458	actionin[$i]="ln"
2459	fi
2460
2461	if [[ $files_for_pes = true ]]
2462	then
2463	printf "\n >>> INPUT: ${absnamein[$i]}/.... to ${localin[$i]}"
2464	else
2465	printf "\n >>> INPUT: ${absnamein[$i]} to ${localin[$i]}"
2466	fi
2467
2468	# INPUT-FILES TO BE LINKED
2469	if [[ "${actionin[$i]}" = ln ]]
2470	then
2471
2472	printf "\n $datentyp will be linked"
2473	if [[ $files_for_pes = false ]]
2474	then
2475	if [[ -f "${absnamein[$i]}" ]]
2476	then
2477	ln ${absnamein[$i]} ${localin[$i]}
2478	got_tmp[$i]=true
2479	fi
2480	else
2481	if [[ -d "${absnamein[$i]}" ]]
2482	then
2483	mkdir ${localin[$i]}
2484	cd ${absnamein[$i]}
2485	for file in $(ls *)
2486	do
2487	ln $file $TEMPDIR/${localin[$i]}
2488	done >\|/dev/null 2>&1
2489	cd $TEMPDIR
2490	fi
2491
2492	# IF "ln -f" HAS FAILED DO A NORMAL COPY "cp -r"
2493	if [[ ! -f "${localin[$i]}/_000000" ]]
2494	then
2495	printf "\n --- WARNING: ln failed, using cp instead (might be time consuming...)"
2496	cp -r ${absnamein[$i]}/* ${localin[$i]}
2497	fi
2498
2499	got_tmp[$i]=true
2500	fi
2501	fi
2502
2503	# FILE IS STORED IN THE RESPECTIVE DIRECTORY GIVEN IN THE CONFIGURATION FILE
2504	if [[ "${actionin[$i]}" = "" \|\| "${actionin[$i]}" = "di" \|\| "${actionin[$i]}" = "npe" ]]
2505	then
2506
2507	if [[ "${actionin[$i]}" = "npe" && -n $numprocs ]]
2508	then
2509
2510	# FILE COPIES ARE PROVIDED FOR ALL CORES
2511	# EACH FILE GETS A UNIQUE FILENAME WITH A FOUR DIGIT NUMBER
2512	printf "\n file will be provided for $numprocs processors"
2513	mkdir ${localin[$i]}
2514	ival=$numprocs
2515	(( ii = 0 ))
2516	while (( ii <= ival-1 ))
2517	do
2518	if (( ii < 10 ))
2519	then
2520	cp ${absnamein[$i]} ${localin[$i]}/_000$ii
2521	elif (( ii < 100 ))
2522	then
2523	cp ${absnamein[$i]} ${localin[$i]}/_00$ii
2524	elif (( ii < 1000 ))
2525	then
2526	cp ${absnamein[$i]} ${localin[$i]}/_0$ii
2527	else
2528	cp ${absnamein[$i]} ${localin[$i]}/_$ii
2529	fi
2530	(( ii = ii + 1 ))
2531	done
2532
2533	else
2534
2535	if [[ $files_for_pes = true ]]
2536	then
2537
2538	# PROVIDE FILES FOR EACH CORE
2539	# FIRST CREATE THE LOCAL DIRECTORY, THEN COPY FILES
2540	# FROM THE PERMANENT DIRECTORY BY LINKING THEM TO THE LOCAL ONE
2541	printf "\n providing $numprocs files for the respective processors"
2542	mkdir ${localin[$i]}
2543	if [[ $link_local_input = true ]]
2544	then
2545	printf " files will be linked\n"
2546	cd ${absnamein[$i]}
2547	for file in $(ls *)
2548	do
2549	ln -f $file ${localin[$i]}
2550	done
2551	cd $TEMPDIR
2552	fi
2553
2554	# IF "ln -f" FAILED OR IF "$link_local_input = false" DO A NORMAL "cp -r"
2555	if [[ ! -f "${localin[$i]}/_000000" ]]
2556	then
2557	if [[ $link_local_input = true ]]
2558	then
2559	printf "\n --- WARNING: ln failed, using cp instead (might be time consuming...)"
2560	fi
2561	cp -r ${absnamein[$i]}/* ${localin[$i]}
2562	fi
2563
2564	else
2565
2566	# PROVIDE FILE FOR RUNS ON A SINGLE CORE
2567	if [[ $link_local_input = true ]]
2568	then
2569	printf " file will be linked\n"
2570	ln -f ${absnamein[$i]} ${localin[$i]}
2571	fi
2572	# If "ln -f" fails of if "$link_local_input = false" do a normal "cp"
2573	if [[ ! -f "${localin[$i]}" ]]
2574	then
2575	if [[ $link_local_input = true ]]
2576	then
2577	printf "\n --- WARNING: ln failed, using cp instead (might be time consuming...)"
2578	fi
2579	cp ${absnamein[$i]} ${localin[$i]}
2580	fi
2581	fi
2582	fi
2583	fi
2584
2585	done
2586	if (( i != 0 ))
2587	then
2588	printf "\n$dashes\n *** all INPUT-files provided \n"
2589	fi
2590
2591
2592	# EXECUTE INPUT-COMMANDS GIVEN IN THE CONFIGURATION FILE
2593	(( i = 0 ))
2594	while (( i < iic ))
2595	do
2596	(( i = i + 1 ))
2597	if (( i == 1 ))
2598	then
2599	printf "\n\n *** execution of INPUT-commands:\n$dashes"
2600	fi
2601	printf "\n >>> ${in_command[$i]}"
2602	eval ${in_command[$i]}
2603	if (( i == iic ))
2604	then
2605	printf "\n$dashes\n"
2606	fi
2607	done
2608
2609
2610	# SET THE REMAINING CPU-TIME
2611	cpurest=${cpumax}.
2612
2613
2614	# START DVR STREAMING SERVER
2615	if [[ $(echo $package_list \| grep -c dvrp_graphics) != 0 ]]
2616	then
2617	if [[ "$dvr_server" != "" ]]
2618	then
2619
2620	printf "\n\n *** preparing the dvr streaming server configuration file"
2621
2622	# CHECK, IF A DVR SERVER IS ALREADY RUNNING
2623	running_dvrserver_id=`echo $(ps -edaf \| grep .dvrserver.config \| grep -v grep) \| cut -d" " -f2`
2624	if [[ "$running_dvrserver_id" != "" ]]
2625	then
2626
2627	printf "\n\n +++ WARNING: A dvr server with id=$running_dvrserver_id is already running!"
2628	printf "\n This server is used instead starting a new one!"
2629	printf "\n If required, script \"process_dvr_output\" has to be run manually."
2630
2631	else
2632
2633	# COPY CONFIGURATION FILE FOR STREAMING SERVER FROM REPOSITORY TO HERE
2634	if [[ -f ${PALM_BIN}/.dvrserver.config ]]
2635	then
2636	cp ${PALM_BIN}/.dvrserver.config .
2637
2638	# ENTERING THE BASEDIR, UID AND GID INTO THIS FILE
2639	user_id=`id -u`
2640	group_id=`id -g`
2641
2642	# "&" IS REQUIRED AS A SEPERATOR, BECAUSE TEMPDIR CONTAINS "/"
2643	sed "s&<replace by dvr data directory>&${TEMPDIR}&g" .dvrserver.config > .dvrserver.1
2644	sed "s/<replace by user id>/$user_id/g" .dvrserver.1 > .dvrserver.2
2645	sed "s/<replace by group id>/$group_id/g" .dvrserver.2 > .dvrserver.3
2646	mv .dvrserver.3 .dvrserver.config
2647	rm .dvrserver.1 .dvrserver.2
2648
2649	# START DVR SERVER IN BACKGROUND, GET HIS ID AND PRINT ON TERMINAL
2650	$dvr_server .dvrserver.config >> DVR_LOGFILE 2>&1 &
2651	dvrserver_id=`echo $(ps -edaf \| grep .dvrserver.config) \| cut -d" " -f2`
2652	printf "\n *** streaming server with id=$dvrserver_id is started in background"
2653	local_dvrserver_running=.TRUE.
2654	else
2655	printf "\n +++ missing file \".dvrserver.config\" in directory:"
2656	printf "\n \"$PALM_BIN\" "
2657	locat=dvr
2658	exit
2659	fi
2660
2661	fi
2662
2663	else
2664	printf "\n\n --- INFORMATIVE: no dvr streaming server will be started"
2665	fi
2666	fi
2667
2668
2669	# CREATE THE NAMELIST-FILE WITH VALUES OF ENVIRONMENT-VARIABLES REQUIRED BY PALM
2670	# (FILE ENVPAR WILL BE READ BY PALM)
2671	cat > ENVPAR << EOF
2672	&envpar run_identifier = '$fname', host = '$localhost',
2673	write_binary = '$write_binary', tasks_per_node = $tasks_per_node,
2674	maximum_parallel_io_streams = $maximum_parallel_io_streams,
2675	maximum_cpu_time_allowed = ${cpumax}.,
2676	revision = '$global_revision',
2677	local_dvrserver_running = $local_dvrserver_running,
2678	batch_job = $batch_job /
2679
2680	EOF
2681
2682
2683	# STARTING THE EXECUTABLE
2684	printf "\n\n *** execution starts in directory\n \"`pwd`\"\n$dashes\n"
2685	PATH=$PATH:$TEMPDIR
2686
2687
2688	if [[ $execute_command != "none" ]]
2689	then
2690
2691	printf "\n +++ branch still not realized"
2692	locat=execution
2693	exit
2694
2695	else
2696
2697	# MPI DEBUG OPTION (ARGUMENT CHECKING, SLOWS DOWN EXECUTION DUE TO INCREASED LATENCY)
2698	if [[ "$mpi_debug" = true ]]
2699	then
2700	export MPI_CHECK_ARGS=1
2701	printf "\n +++ MPI_CHECK_ARGS=$MPI_CHECK_ARGS"
2702	fi
2703
2704	if [[ "$totalview" = true ]]
2705	then
2706	printf "\n *** totalview debugger will be used"
2707	tv_opt="-tv"
2708	else
2709	tv_opt=""
2710	fi
2711
2712	if [[ "$cond1" = debug \|\| "$cond2" = debug ]]
2713	then
2714
2715	#Interactive ALLINEA DEBUG seesion
2716	if [[ "$ENVIRONMENT" != BATCH ]]
2717	then
2718	if [[ $host = lccrayb \|\| $host = lccrayh ]]
2719	then
2720	if [[ "$allinea" = true ]]
2721	then
2722	echo "--- aprun -n $ii -N $tasks_per_node a.out < runfile_atmos"
2723
2724	ddt aprun -n $ii -N $tasks_per_node a.out
2725	wait
2726	fi
2727	fi
2728	fi
2729
2730	if [[ $localhost = ibmh ]]
2731	then
2732
2733	# SETUP THE IBM MPI ENVIRONMENT
2734	export MP_SHARED_MEMORY=yes
2735	export AIXTHREADS_SCOPE=S
2736	export OMP_NUM_THREADS=$threads_per_task
2737	export AUTHSTATE=files
2738	export XLFRTEOPTS="nlwidth=132:err_recovery=no" # RECORD-LENGTH OF NAMELIST-OUTPUT
2739
2740	# FOLLOWING OPTIONS ARE MANDATORY FOR TOTALVIEW
2741	export MP_ADAPTER_USE=shared
2742	export MP_CPU_USE=multiple
2743	export MP_TIMEOUT=1200
2744
2745	unset MP_TASK_AFFINITY
2746
2747	if [[ "$LOADLBATCH" = yes ]]
2748	then
2749	totalview poe a.out
2750	else
2751	echo totalview poe -a a.out -procs $numprocs -rmpool 0 -nodes 1
2752	export TVDSVRLAUNCHCMD=ssh
2753	totalview poe -a a.out -procs $numprocs -rmpool 0 -nodes 1
2754	fi
2755
2756	fi # END DEBUG MODE
2757
2758	else
2759
2760	# NORMAL EXECUTION
2761	if [[ -n $numprocs ]]
2762	then
2763
2764	# RUNNING THE PROGRAM ON PARALLEL MACHINES
2765	if [[ $(echo $host \| cut -c1-3) = ibm ]]
2766	then
2767
2768	# SETUP THE IBM MPI ENVIRONMENT
2769	if [[ $host != ibmh && $host != ibmkisti ]]
2770	then
2771	export MP_SHARED_MEMORY=yes
2772	export AIXTHREAD_SCOPE=S
2773	export OMP_NUM_THREADS=$threads_per_task
2774	export XLSMPOPTS="spins=0:yields=0:stack=20000000"
2775	export AUTHSTATE=files
2776	export XLFRTEOPTS="nlwidth=132:err_recovery=no" # RECORD-LENGTH OF NAMELIST-OUTPUT
2777	# export MP_PRINTENV=yes
2778
2779	# TUNING-VARIABLES TO IMPROVE COMMUNICATION SPEED
2780	# DO NOT SHOW SIGNIFICANT EFFECTS (SEP 04, FEDERATION-SWITCH)
2781	export MP_WAIT_MODE=poll
2782	[[ $node_usage = not_shared ]] && export MP_SINGLE_THREAD=yes
2783	fi
2784
2785	if [[ $host = ibmkisti ]]
2786	then
2787	export LANG=en_US
2788	export MP_SHARED_MEMORY=yes
2789	if [[ $threads_per_task = 1 ]]
2790	then
2791	export MP_SINGLE_THREAD=yes
2792	export MEMORY_AFFINITY=MCM
2793	else
2794	export OMP_NUM_THREADS=$threads_per_task
2795	fi
2796	fi
2797
2798	if [[ "$LOADLBATCH" = yes ]]
2799	then
2800	printf "\n--- Control: OMP_NUM_THREADS = \"$OMP_NUM_THREADS\" \n"
2801	if [[ "$cond1" = hpmcount \|\| "$cond2" = hpmcount ]]
2802	then
2803	/opt/optibm/HPM_2_4_1/bin/hpmcount a.out
2804	else
2805	if [[ $run_coupled_model = false ]]
2806	then
2807	if [[ "$ocean_file_appendix" = true ]]
2808	then
2809	echo "precursor_ocean" > runfile_atmos
2810	else
2811	echo "precursor_atmos" > runfile_atmos
2812	fi
2813	else
2814	(( iia = $numprocs_atmos / $threads_per_task ))
2815	(( iio = $numprocs_ocean / $threads_per_task ))
2816	printf "\n coupled run ($iia atmosphere, $iio ocean)"
2817	printf "\n using $coupled_mode coupling"
2818	printf "\n\n"
2819	echo "coupled_run $iia $iio" > runfile_atmos
2820	fi
2821	poe ./a.out < runfile_atmos
2822	fi
2823	else
2824	if [[ $localhost = ibmh \|\| $localhost = ibms ]]
2825	then
2826	poe a.out -procs $numprocs -nodes 1 -rmpool 0
2827	elif [[ $localhost = ibmkisti \|\| $localhost = ibmku ]]
2828	then
2829	if [[ -f $hostfile ]]
2830	then
2831	cp $hostfile hostfile
2832	else
2833	(( ii = 1 ))
2834	while (( ii <= $numprocs ))
2835	do
2836	echo $localhost_realname >> hostfile
2837	(( ii = ii + 1 ))
2838	done
2839	fi
2840	export MP_HOSTFILE=hostfile
2841	if [[ $run_coupled_model = false ]]
2842	then
2843	if [[ "$ocean_file_appendix" = true ]]
2844	then
2845	echo "precursor_ocean" > runfile_atmos
2846	else
2847	echo "precursor_atmos" > runfile_atmos
2848	fi
2849	else
2850	(( iia = $numprocs_atmos / $threads_per_task ))
2851	(( iio = $numprocs_ocean / $threads_per_task ))
2852	printf "\n coupled run ($iia atmosphere, $iio ocean)"
2853	printf "\n using $coupled_mode coupling"
2854	printf "\n\n"
2855	echo "coupled_run $iia $iio" > runfile_atmos
2856	fi
2857
2858	poe ./a.out -procs $numprocs < runfile_atmos
2859
2860	else
2861	if [[ "$host_file" = "" ]]
2862	then
2863	printf "\n +++ no hostfile given in configuration file"
2864	locat=config_file
2865	exit
2866	else
2867	eval host_file=$host_file
2868	fi
2869	export MP_HOSTFILE=$host_file
2870	poe a.out -procs $numprocs -tasks_per_node $numprocs
2871	fi
2872	fi
2873
2874	elif [[ $(echo $host \| cut -c1-2) = lc && $host != lckyoto && $host != lctit ]]
2875	then
2876
2877	# COPY HOSTFILE FROM SOURCE DIRECTORY OR CREATE IT, IF IT
2878	# DOES NOT EXIST
2879	if [[ $host != lcbullhh && $host != lccrayb && $host != lccrayf && $host != lccrayh && $host != lckyuh && $host != lckyut && $host != lcocean && $host != lceddy ]]
2880	then
2881	if [[ -f $hostfile ]]
2882	then
2883	cp $hostfile hostfile
2884	(( ii = $numprocs / $threads_per_task ))
2885	[[ $ii = 0 ]] && (( ii = 1 ))
2886	else
2887	(( ii = 1 ))
2888	while (( ii <= $numprocs / $threads_per_task ))
2889	do
2890	echo $localhost_realname >> hostfile
2891	(( ii = ii + 1 ))
2892	done
2893	if (( $numprocs / $threads_per_task == 0 ))
2894	then
2895	echo $localhost_realname >> hostfile
2896	fi
2897	fi
2898	eval zeile=\"`head -n $ii hostfile`\"
2899	printf "\n *** running on: $zeile"
2900	fi
2901
2902	(( ii = $numprocs / $threads_per_task ))
2903	[[ $ii = 0 ]] && (( ii = 1 ))
2904	export OMP_NUM_THREADS=$threads_per_task
2905
2906	if [[ $threads_per_task != 1 ]]
2907	then
2908	# INCREASE STACK SIZE TO UNLIMITED, BECAUSE OTHERWISE LARGE RUNS
2909	# MAY ABORT
2910	ulimit -s unlimited
2911	printf "\n threads per task: $threads_per_task stacksize: unlimited"
2912	fi
2913	if [[ $run_coupled_model = false ]]
2914	then
2915	if [[ "$ocean_file_appendix" = true ]]
2916	then
2917	echo "precursor_ocean" > runfile_atmos
2918	else
2919	echo "precursor_atmos" > runfile_atmos
2920	fi
2921	printf "\n\n"
2922
2923	if [[ $host = lccrayb \|\| $host = lccrayh ]]
2924	then
2925	echo "--- aprun -n $ii -N $tasks_per_node a.out < runfile_atmos"
2926	aprun -n $ii -N $tasks_per_node a.out < runfile_atmos
2927	elif [[ $host = lcbullhh ]]
2928	then
2929	export OMPI_MCA_pml=cm
2930	export OMPI_MCA_mtl=mxm
2931	export OMPI_MCA_coll=^ghc
2932	export OMPI_MCA_mtl_mxm_np=0
2933	export MXM_RDMA_PORTS=mlx5_0:1
2934	export MXM_LOG_LEVEL=ERROR
2935	export OMP_NUM_THREADS=$threads_per_task
2936	export KMP_AFFINITY=verbose,granularity=core,compact,1
2937	export KMP_STACKSIZE=64m
2938
2939	srun -n $ii --ntasks-per-node=$tasks_per_node ./a.out < runfile_atmos
2940
2941	elif [[ $host = lccrayf ]]
2942	then
2943	aprun -j1 -n $ii -N $tasks_per_node -m ${memory}M a.out < runfile_atmos
2944	elif [[ $host = lcxe6 \|\| $host = lcxt5m ]]
2945	then
2946	aprun -n $ii -N $tasks_per_node a.out < runfile_atmos
2947	elif [[ $host = lceddy ]]
2948	then
2949	echo $ii
2950	echo $tasks_per_node
2951	echo $nodes
2952	mpirun -n $ii a.out < runfile_atmos
2953	elif [[ $host = lcocean ]]
2954	then
2955	mpirun a.out $ROPTS < runfile_atmos
2956	elif [[ $host = lcsb ]]
2957	then
2958	mpirun_rsh -hostfile $PBS_NODEFILE -np `cat $PBS_NODEFILE \| wc -l` a.out < runfile_atmos
2959
2960	elif [[ $host = lckiaps ]]
2961	then
2962	mpirun -np $ii -machinefile $PBS_NODEFILE ./a.out < runfile_atmos
2963	elif [[ $host = lckyu* ]]
2964	then
2965	mpiexec -n $ii --stdin runfile_atmos ./a.out
2966	else
2967	mpiexec -machinefile hostfile -n $ii a.out < runfile_atmos
2968	fi
2969
2970	else
2971
2972	# CURRENTLY THERE IS NO FULL MPI-2 SUPPORT ON ICE AND XT4
2973	(( iia = $numprocs_atmos / $threads_per_task ))
2974	(( iio = $numprocs_ocean / $threads_per_task ))
2975	printf "\n coupled run ($iia atmosphere, $iio ocean)"
2976	printf "\n using $coupled_mode coupling"
2977	printf "\n\n"
2978
2979	if [[ $coupled_mode = "mpi2" ]]
2980	then
2981	echo "atmosphere_to_ocean $iia $iio" > runfile_atmos
2982	echo "ocean_to_atmosphere $iia $iio" > runfile_ocean
2983
2984	if [[ $host = lccrayf \|\| $host = lcxe6 \|\| $host = lcxt5m ]]
2985	then
2986
2987	aprun -n $iia -N $tasks_per_node a.out < runfile_atmos &
2988	aprun -n $iio -N $tasks_per_node a.out < runfile_ocean &
2989
2990	else
2991	# WORKAROUND BECAUSE mpiexec WITH -env option IS NOT AVAILABLE ON SOME SYSTEMS
2992	mpiexec -machinefile hostfile -n $iia a.out < runfile_atmos &
2993	mpiexec -machinefile hostfile -n $iio a.out < runfile_ocean &
2994	# mpiexec -machinefile hostfile -n $iia -env coupling_mode atmosphere_to_ocean a.out &
2995	# mpiexec -machinefile hostfile -n $iio -env coupling_mode ocean_to_atmosphere a.out &
2996	fi
2997	wait
2998
2999	else
3000
3001	echo "coupled_run $iia $iio" > runfile_atmos
3002
3003	if [[ $host = lccrayf \|\| $host = lcxe6 \|\| $host = lcxt5m ]]
3004	then
3005
3006	aprun -n $ii -N $tasks_per_node a.out < runfile_atmos
3007
3008	elif [[ $host = lck \|\| $host = lckordi ]]
3009	then
3010
3011	mpiexec -n $ii ./a.out < runfile_atmos &
3012
3013	elif [[ $host = lckyu* ]]
3014	then
3015
3016	mpiexec -n $ii --stdin runfile_atmos ./a.out
3017
3018	elif [[ $host = lcmuk ]]
3019	then
3020
3021	mpiexec -machinefile hostfile -n $ii a.out < runfile_atmos
3022
3023	fi
3024	wait
3025	fi
3026
3027	fi
3028
3029	elif [[ $host = lckyoto ]]
3030	then
3031	set -xv
3032	export P4_RSHCOMMAND=plesh
3033	echo " P4_RSHCOMMAND = $P4_RSHCOMMAND"
3034	if [[ "$ENVIRONMENT" = BATCH ]]
3035	then
3036	if [[ "$cond2" = fujitsu ]]
3037	then
3038	mpiexec -n $numprocs ./a.out # for fujitsu-compiler
3039	elif [[ "cond2" = pgi ]]
3040	then
3041	mpirun -np $numprocs -machinefile ${QSUB_NODEINF} ./a.out
3042	else
3043	mpirun_rsh -np $numprocs -hostfile ${QSUB_NODEINF} MV2_USE_SRQ=0 ./a.out \|\| /bin/true
3044	fi
3045	else
3046	if [[ "$cond2" = "" ]]
3047	then
3048	mpiruni_rsh -np $numprocs ./a.out # for intel
3049	else
3050	mpirun -np $numprocs ./a.out
3051	fi
3052	fi
3053	set +xv
3054
3055	elif [[ $host = lctit ]]
3056	then
3057	export OMP_NUM_THREADS=$threads_per_task
3058	echo "OMP_NUM_THREADS=$OMP_NUM_THREADS"
3059	if [[ "$threads_per_task" != 1 ]]
3060	then
3061	export MV2_ENABLE_AFFINITY=0
3062	fi
3063	echo "----- PBS_NODEFILE content:"
3064	cat $PBS_NODEFILE
3065	echo "-----"
3066	(( ii = $numprocs / $threads_per_task ))
3067	echo "mpirun -np $ii -hostfile $PBS_NODEFILE ./a.out"
3068	mpirun -np $ii -hostfile $PBS_NODEFILE ./a.out
3069
3070	else
3071	mpprun -n $numprocs a.out
3072	fi
3073	else
3074	a.out
3075	fi
3076
3077	fi # end normal (non-debug) execution
3078
3079	fi # end explicit execute_command or host-specific execute actions
3080
3081	if [[ $? != 0 ]]
3082	then
3083
3084	# ABORT IN CASE OF RUNTIME ERRORS
3085	printf "\n +++ runtime error occured"
3086	locat=execution
3087	exit
3088
3089	else
3090
3091	printf "\n$dashes\n *** execution finished \n"
3092
3093	# STOP THE DVR STREAMING SERVER AND PROCESS THE DVR OUTPUT IN ORDER
3094	# TO CREAT DVRS- AND HTML-FILES CONTAINING ALL STREAMS
3095	if [[ "$dvrserver_id" != "" ]]
3096	then
3097	kill $dvrserver_id
3098	printf "\n *** dvr server with id=$dvrserver_id has been stopped"
3099
3100	# IF THERE IS A DIRECTORY, DATA HAVE BEEN OUTPUT BY THE
3101	# STREAMING SERVER. OTHERWISE, USER HAS CHOSEN dvrp_output=local
3102	if [[ -d DATA_DVR ]]
3103	then
3104
3105	# ADD THE CURRENT DVR CONFIGURATION FILE TO THE DVR OUTPUT
3106	# DIRECTORY
3107	cp .dvrserver.config DATA_DVR
3108
3109	# PROCESS THE DVR OUTPUT (OPTION -s FOR GENERATING
3110	# SEQUENCE MODE DATA TOO)
3111	process_dvr_output -d DATA_DVR -f $fname -s
3112
3113	else
3114
3115	# PROCESS THE LOCAL OUTPUT
3116	process_dvr_output -l -d DATA_DVR -f $fname
3117
3118	fi
3119
3120	elif [[ $(echo $package_list \| grep -c dvrp_graphics) != 0 ]]
3121	then
3122
3123	# PROCESS DVR OUTPUT GENERATD IN LOCAL MODE (dvrp_output=local)
3124	process_dvr_output -l -d DATA_DVR -f $fname
3125
3126	fi
3127	fi
3128
3129
3130
3131	# CALL OF combine_plot_fields IN ORDER TO MERGE SINGLE FILES WRITTEN
3132	# BY EACH CORE INTO ONE FILE
3133	if [[ ! -f ${PALM_BIN}/combine_plot_fields${block}.x ]]
3134	then
3135	printf "\n\n\n +++ WARNING: no combine_plot_fields found for given block \"$cond1 $cond2\""
3136	printf "\n 2d- and/or 3d-data may be incomplete!"
3137	printf "\n Run \"mbuild -u -h $localhost\" to generate utilities for this block.\n"
3138	elif [[ "$combine_plot_fields" == true ]]
3139	then
3140	if [[ $localhost = lccrayh \|\| $localhost = lccrayb ]]
3141	then
3142	printf "\n\n\n *** post-processing: now executing \"aprun -n 1 -N 1 combine_plot_fields${block}.x\" ..."
3143	aprun -n 1 -N 1 combine_plot_fields${block}.x
3144	else
3145	printf "\n\n\n *** post-processing: now executing \"combine_plot_fields${block}.x\" ..."
3146	combine_plot_fields${block}.x
3147	fi
3148	else
3149
3150	# TEMPORARY SOLUTION TO SKIP combine_plot_fields. THIS IS REQUIRED IN CASE OF HUGE AMOUNT OF
3151	# DATA OUTPUT. TO DO: EXTEND THIS BRANCH BY CREATING A BATCH JOB for combine_plot_fields.
3152	# ??? koennen wir das streichen ???
3153	printf "\n\n\n *** post-processing: skipping combine_plot_fields (-Z option set) ..."
3154	fi
3155
3156
3157
3158	# EXECUTE OUTPUT-COMMANDS GIVEN IN THE CONFIGURATION FILE
3159	(( i = 0 ))
3160	while (( i < ioc ))
3161	do
3162	(( i = i + 1 ))
3163	if (( i == 1 ))
3164	then
3165	printf "\n\n *** execution of OUTPUT-commands:\n$dashes"
3166	fi
3167	printf "\n >>> ${out_command[$i]}"
3168	eval ${out_command[$i]}
3169	if (( i == ioc ))
3170	then
3171	printf "\n$dashes\n"
3172	fi
3173	done
3174
3175
3176	# IN TRACE-MODE PRINT CONTENTS OF THE CURRENT (TEMPORARY) WORKING DIRECTORY
3177	if [[ $do_trace = true ]]
3178	then
3179	printf "\n\n"
3180	ls -al
3181	fi
3182
3183
3184	# COPY LOCAL OUTPUT-FILES TO THEIR PERMANENT DESTINATIONS
3185	(( i = 0 ))
3186	while (( i < iout ))
3187	do
3188	(( i = i + 1 ))
3189	if (( i == 1 ))
3190	then
3191	printf "\n\n *** saving OUTPUT-files:"
3192
3193	# GET RUN NUMBER ASSIGNED BY PALM
3194	if [[ -f RUN_NUMBER ]]
3195	then
3196	read run_number < RUN_NUMBER
3197	printf "\n *** PALM generated run_number = "$run_number" will be used as unified cycle number for all output files"
3198	usecycle_option="-U $run_number"
3199	else
3200	run_number=0
3201	usecycle_option=""
3202	fi
3203	printf "\n$dashes"
3204	fi
3205
3206	# ADD CYCLE NUMBER TO FILENAME
3207	if [[ ! ( $fromhost != $localhost && ( "${actionout[$i]}" = tr \|\| "${actionout[$i]}" = tra \|\| "${actionout[$i]}" = trpe ) ) ]]
3208	then
3209
3210	# IN APPEND MODE, FILES KEEP THEIR CURRENT CYCLE NUMBER
3211	if [[ "${actionout[$i]}" != "a" ]]
3212	then
3213	# SET RUN NUMBER AS CYCLE NUMBER, IF THERE IS NOT A CONFLICT
3214	# WITH AN EXISTING CYCLE NUMBER
3215	if (( run_number >= cycnum[$i] ))
3216	then
3217	(( cycnum[$i] = run_number ))
3218	else
3219	if (( run_number > 0 ))
3220	then
3221	printf "\n --- INFORMATIVE: The following file cannot get a unified cycle number"
3222	fi
3223	fi
3224	fi
3225	if (( cycnum[$i] > 0 ))
3226	then
3227	pathout[$i]=${pathout[$i]}.${cycnum[$i]}
3228	fi
3229	fi
3230
3231	# CHECK FOR SINGLE FILE (SERIAL RUN) OR DIRECTORY (ONE FILE PER CORE FOR PARELLEL EXECUTION)
3232	files_for_pes=false; filetyp=file
3233	link_local_output=false
3234	if [[ "${actionout[$i]}" = pe && -n $numprocs ]]
3235	then
3236	files_for_pes=true; filetyp=directory
3237	actionout[$i]=""
3238	elif [[ "${actionout[$i]}" = pe && ! -n $numprocs ]]
3239	then
3240	actionout[$i]=""
3241	elif [[ "${actionout[$i]}" = lnpe && -n $numprocs ]]
3242	then
3243	files_for_pes=true; filetyp=directory
3244	link_local_output=true
3245	actionout[$i]=""
3246	elif [[ "${actionout[$i]}" = lnpe && ! -n $numprocs ]]
3247	then
3248	link_local_output
3249	actionout[$i]=""
3250	elif [[ "${actionout[$i]}" = trpe && -n $numprocs ]]
3251	then
3252	files_for_pes=true; filetyp=directory
3253	actionout[$i]="tr"
3254	elif [[ "${actionout[$i]}" = trpe && ! -n $numprocs ]]
3255	then
3256	actionout[$i]="tr"
3257	fi
3258
3259	if [[ ! -f ${localout[$i]} && $files_for_pes = false ]]
3260	then
3261	printf "\n +++ temporary OUTPUT-file ${localout[$i]} does not exist\n"
3262	elif [[ ! -d ${localout[$i]} && $files_for_pes = true ]]
3263	then
3264	printf "\n +++ temporary OUTPUT-file ${localout[$i]}/.... does not exist\n"
3265	else
3266
3267
3268	# COPY VIA SCP TO LOCAL HOST (ALWAYS IN BINARY MODE USING batch_scp option -m)
3269	# IF TARGET DIRECTORY DOES NOT EXISTS, TRY TO CREATE IT
3270	if [[ "${actionout[$i]}" = tr ]]
3271	then
3272	if [[ $localhost != $fromhost ]]
3273	then
3274	if [[ $files_for_pes = false ]]
3275	then
3276	cps=""
3277	cst=""
3278	else
3279	cps=-c
3280	cst="/"
3281	fi
3282
3283	transfer_failed=false
3284	printf "\n >>> OUTPUT: ${localout[$i]}$cst by SCP to"
3285	printf "\n ${pathout[$i]}/${localhost}_${fname}${endout[$i]}$cst\n"
3286	if [[ $localhost = lccrayb ]]
3287	then
3288	ssh $usern@blogin1 ". \\$HOME/.bashrc; cd $TEMPDIR; batch_scp $PORTOPT $cps -b -m $usecycle_option -u $return_username $return_address ${localout[$i]} \"${pathout[$i]}\" ${localhost}_${fname}${endout[$i]} ${extout[$i]}"
3289	elif [[ $localhost = lccrayh ]]
3290	then
3291	ssh $usern@hlogin1 ". \\$HOME/.bashrc; cd $TEMPDIR; batch_scp $PORTOPT $cps -b -m $usecycle_option -u $return_username $return_address ${localout[$i]} \"${pathout[$i]}\" ${localhost}_${fname}${endout[$i]} ${extout[$i]}"
3292	elif [[ $localhost = lcbullhh ]]
3293	then
3294	ssh $usern@mlogin101 ". \\$HOME/.bashrc; cd $TEMPDIR; batch_scp $PORTOPT $cps -b -m $usecycle_option -u $return_username $return_address ${localout[$i]} \"${pathout[$i]}\" ${localhost}_${fname}${endout[$i]} ${extout[$i]}"
3295	elif [[ $localhost = lcxe6 ]]
3296	then
3297	ssh $usern@hexagon ". \\$HOME/.bashrc; cd $TEMPDIR; batch_scp $PORTOPT $cps -b -m $usecycle_option -u $return_username $return_address ${localout[$i]} \"${pathout[$i]}\" ${localhost}_${fname}${endout[$i]} ${extout[$i]}"
3298	else
3299	batch_scp $PORTOPT $cps -b -m $usecycle_option -u $return_username $return_address ${localout[$i]} "${pathout[$i]}" ${localhost}_${fname}${endout[$i]} ${extout[$i]}
3300	fi
3301	[[ $? != 0 ]] && transfer_failed=true
3302
3303	# IF TRANSFER FAILED, CREATE BACKUP COPY ON THIS MACHINE
3304	if [[ $transfer_failed = true ]]
3305	then
3306	printf " +++ transfer failed. Trying to save a copy on this host under:\n"
3307	printf " ${pathout[$i]}/${localhost}_${fname}${endout[$i]}_$kennung\n"
3308
3309	# FIRST CHECK, IF DIRECTORY EXISTS, AND CREATE IT, IF NECESSARY
3310	eval local_catalog=${pathout[$i]}
3311	if [[ ! -d $local_catalog ]]
3312	then
3313	printf " *** local directory does not exist. Trying to create:\n"
3314	printf " $local_catalog \n"
3315	mkdir -p $local_catalog
3316	fi
3317	eval cp ${localout[$i]} ${pathout[$i]}/${localhost}_${fname}${endout[$i]}_$kennung
3318	transfer_problems=true
3319	fi
3320
3321	else
3322
3323	# UNSET actionout. DUE TO THIS SETTING, FILE WILL LATER JUST BE COPIED ON THIS MACHINE
3324	actionout[$i]=""
3325	fi
3326	fi
3327
3328
3329	# APPEND VIA SCP TO LOCAL HOST (ALWAYS IN BINARY MODE USING batch_scp option -m)
3330	# IF TARGET DIRECTORY DOES NOT EXISTS, TRY TO CREATE IT
3331	if [[ "${actionout[$i]}" = tra ]]
3332	then
3333	if [[ $localhost != $fromhost ]]
3334	then
3335	if [[ $localhost = ibmh ]]
3336	then
3337
3338	# TRANSFER IN SEPERATE JOB
3339	# FIRST COPY FILE TO TEMPORY DATA DIRECTORY
3340	[[ ! -d $tmp_data_catalog/TRANSFER ]] && mkdir -p $tmp_data_catalog/TRANSFER
3341	file_to_transfer=${fname}_${localout[$i]}_to_transfer_$kennung
3342	ln -f ${localout[$i]} $tmp_data_catalog/TRANSFER/$file_to_transfer
3343
3344	echo "set -x" > transfer_${localout[$i]}
3345	echo "cd $tmp_data_catalog/TRANSFER" >> transfer_${localout[$i]}
3346
3347	printf "\n >>> OUTPUT: ${localout[$i]} append by SCP in seperate job to"
3348	printf "\n ${pathout[$i]}/${localhost}_${fname}${endout[$i]}"
3349	printf "\n or higher cycle\n"
3350	echo "batch_scp $PORTOPT -A -b -m -u $return_username $return_address $file_to_transfer \"${pathout[$i]}\" ${localhost}_${fname}${endout[$i]} ${extout[$i]}" >> transfer_${localout[$i]}
3351
3352	echo "[[ \$? = 0 ]] && rm $file_to_transfer" >> transfer_${localout[$i]}
3353
3354	if [[ $LOGNAME = b323013 ]]
3355	then
3356	subjob -v -q c1 -X 0 -m 1000 -t 900 -c $job_catalog $PORTOPT transfer_${localout[$i]}
3357	else
3358	subjob -d -v -q c1 -X 0 -m 1000 -t 900 -c $job_catalog $PORTOPT transfer_${localout[$i]}
3359	fi
3360
3361	else
3362
3363	# TRANSFER WITHIN THIS JOB
3364	transfer_failed=false
3365	printf "\n >>> OUTPUT: ${localout[$i]} append by SCP to"
3366	printf "\n ${pathout[$i]}/${localhost}_${fname}${endout[$i]}\n"
3367	if [[ $localhost = lccrayb ]]
3368	then
3369	ssh $usern@blogin1 ". \\$HOME/.bashrc; cd $TEMPDIR; batch_scp $PORTOPT -A -b -m -u $return_username $return_address ${localout[$i]} \"${pathout[$i]}\" ${localhost}_${fname}${endout[$i]} ${extout[$i]}"
3370	elif [[ $localhost = lccrayh ]]
3371	then
3372	ssh $usern@hlogin1 ". \\$HOME/.bashrc; cd $TEMPDIR; batch_scp $PORTOPT -A -b -m -u $return_username $return_address ${localout[$i]} \"${pathout[$i]}\" ${localhost}_${fname}${endout[$i]} ${extout[$i]}"
3373	elif [[ $localhost = lcxe6 ]]
3374	then
3375	ssh $usern@hexagon ". \\$HOME/.bashrc; cd $TEMPDIR; batch_scp $PORTOPT -A -b -m -u $return_username $return_address ${localout[$i]} \"${pathout[$i]}\" ${localhost}_${fname}${endout[$i]} ${extout[$i]}"
3376	else
3377	batch_scp $PORTOPT -A -b -m -u $return_username $return_address ${localout[$i]} "${pathout[$i]}" ${localhost}_${fname}${endout[$i]} ${extout[$i]}
3378	fi
3379	[[ $? != 0 ]] && transfer_failed=true
3380
3381	# IF TRANSFER FAILED, CREATE BACKUP COPY ON THIS MACHINE
3382	if [[ $transfer_failed = true ]]
3383	then
3384	printf " +++ transfer failed. Trying to save a copy on this host under:\n"
3385	printf " ${pathout[$i]}/${localhost}_${fname}${endout[$i]}_$kennung\n"
3386
3387	# FIRST CHECK, IF DIRECTORY EXISTS, AND CREATE IT, IF NECESSARY
3388	eval local_catalog=${pathout[$i]}
3389	if [[ ! -d $local_catalog ]]
3390	then
3391	printf " *** local directory does not exist. Trying to create:\n"
3392	printf " $local_catalog \n"
3393	mkdir -p $local_catalog
3394	fi
3395
3396	eval cp ${localout[$i]} ${pathout[$i]}/${localhost}_${fname}${endout[$i]}_$kennung
3397	transfer_problems=true
3398	fi
3399	fi
3400	else
3401
3402	# RESET actionout. DUE TO THIS SETTING, FILE WILL LATER JUST BE APPENDED ON THIS MACHINE
3403	actionout[$i]=a
3404	fi
3405	fi
3406
3407
3408	# APPEND ON THIS MACHINE
3409	if [[ "${actionout[$i]}" = "a" ]]
3410	then
3411	if [[ "${extout[$i]}" != " " && "${extout[$i]}" != "" ]]
3412	then
3413	printf "\n >>> OUTPUT: ${localout[$i]} append to ${pathout[$i]}.${extout[$i]}\n"
3414	cat ${localout[$i]} >> ${pathout[$i]}.${extout[$i]}
3415	else
3416	printf "\n >>> OUTPUT: ${localout[$i]} append to ${pathout[$i]}\n"
3417	cat ${localout[$i]} >> ${pathout[$i]}
3418	fi
3419	fi
3420
3421	# COPY ON THIS MACHINE
3422	# COPY HAS TO BE USED, BECAUSE MOVE DOES NOT WORK IF FILE-ORIGIN AND TARGET ARE
3423	# ON DIFFERENT FILE-SYSTEMS
3424	if [[ "${actionout[$i]}" = "" && $files_for_pes = false ]]
3425	then
3426
3427	# COPY IN CASE OF RUNS ON SINGLE CORES
3428	if [[ "${extout[$i]}" != " " && "${extout[$i]}" != "" ]]
3429	then
3430	printf "\n >>> OUTPUT: ${localout[$i]} to ${pathout[$i]}.${extout[$i]}\n"
3431	if [[ $link_local_output = true ]]
3432	then
3433	printf " file will be linked\n"
3434	ln -f ${localout[$i]} ${pathout[$i]}.${extout[$i]}
3435	fi
3436	# If "ln -f" fails of if "$link_local_output = false" do a normal "cp"
3437	if [[ ! -f "${pathout[$i]}.${extout[$i]}" ]]
3438	then
3439	if [[ $link_local_output = true ]]
3440	then
3441	printf " --- WARNING: ln failed, using cp instead (might be time consuming...)\n"
3442	fi
3443	cp ${localout[$i]} ${pathout[$i]}.${extout[$i]}
3444	else
3445	printf "+++ no copy because file ${pathout[$i]}.${extout[$i]} exists\n"
3446	fi
3447	else
3448	printf "\n >>> OUTPUT: ${localout[$i]} to ${pathout[$i]}\n"
3449	if [[ $link_local_output = true ]]
3450	then
3451	printf " file will be linked\n"
3452	ln -f ${localout[$i]} ${pathout[$i]}
3453	fi
3454	# If "ln -f" fails of if "$link_local_output = false" do a normal "cp"
3455	if [[ ! -f "${pathout[$i]}" ]]
3456	then
3457	if [[ $link_local_output = true ]]
3458	then
3459	printf " --- WARNING: ln failed, using cp instead (might be time consuming...)\n"
3460	fi
3461	cp ${localout[$i]} ${pathout[$i]}
3462	else
3463	printf "+++ no copy because file ${pathout[$i]} exists\n"
3464	fi
3465	fi
3466
3467	elif [[ "${actionout[$i]}" = "" && $files_for_pes = true ]]
3468	then
3469
3470	# FILES FROM THE DIFFERENT CORES ARE MOVED WITH ln-COMMAND TO THE PERMANENT DIRECTORY
3471	# AS A FIRST STEP, THE PERMANENT DIRECTORY IS CREATED
3472	printf "\n >>> OUTPUT: ${localout[$i]}/_.... to ${pathout[$i]}\n"
3473	if [[ $link_local_output = true ]]
3474	then
3475	printf " files will be linked\n"
3476	mkdir ${pathout[$i]}
3477	cd ${localout[$i]}
3478	for file in $(ls *)
3479	do
3480	ln -f $file ${pathout[$i]}
3481	done >\|/dev/null 2>&1
3482	cd $TEMPDIR
3483	fi
3484
3485	# IF "ln -f" HAS FAILED OR IF "$link_local_output = false" DO A NORMAL COPY "cp -r"
3486	if [[ ! -f "${pathout[$i]}/_000000" ]]
3487	then
3488	if [[ $link_local_output = true ]]
3489	then
3490	printf " --- WARNING: ln failed, using cp instead (might be time consuming...)\n"
3491	fi
3492	cp -r ${localout[$i]}/* ${pathout[$i]}
3493	fi
3494
3495	fi
3496	fi
3497	done
3498	if (( i != 0 ))
3499	then
3500	if [[ $transfer_problems = true ]]
3501	then
3502	printf "\n$dashes\n *** OUTPUT-files saved"
3503	printf "\n +++ WARNING: some data transfers failed! \n"
3504	else
3505	printf "\n$dashes\n *** all OUTPUT-files saved \n"
3506	fi
3507	fi
3508
3509
3510	# IF REQUIRED, START A RESTART-JOB
3511	# FILE CONTINUE_RUN MUST HAVE BEEN CREATED BY THE EXECUTABLE (PALM)
3512	if [[ -f CONTINUE_RUN ]]
3513	then
3514
3515	# ADD RESTART-OPTIONS TO THE MRUN-CALL (IF THEY ARE NOT USED ALREADY):
3516	# -C TELLS MRUN THAT IT IS A RESTART-RUN
3517	# -v SILENT MODE WITHOUT INTERACTIVE QUERIES
3518	# -n BATCH-MODE (IMPORTANT ONLY IN CASE OF BATCH JOBS ON THE LOCAL MACHINE)
3519	[[ $(echo $mc \| grep -c "\-C") = 0 ]] && mc="$mc -C"
3520	[[ $(echo $mc \| grep -c "\-v") = 0 ]] && mc="$mc -v"
3521	[[ $(echo $mc \| grep -c "\-b") = 0 ]] && mc="$mc -b"
3522	if [[ $(echo $mc \| grep -c "#") != 0 ]]
3523	then
3524	mc=`echo $mc \| sed 's/#/f/g'`
3525	fi
3526
3527
3528	# START THE RESTART-JOB
3529	printf "\n\n *** initiating restart-run on \"$return_address\" using command:\n"
3530	echo " $mc"
3531	printf "\n$dashes\n"
3532	if [[ $localhost != $fromhost ]]
3533	then
3534
3535	if [[ $localhost = lcbullhh \|\| $localhost = lccrayb \|\| $localhost = lccrayh \|\| $localhost = ibmh \|\| $localhost = ibmkisti \|\| $localhost = ibmku \|\| $localhost = ibms \|\| $localhost = lceddy \|\| $localhost = lckyu* \|\| $localhost = lcxe6 ]]
3536	then
3537	echo "*** ssh will be used to initiate restart-runs!"
3538	echo " return_address=\"$return_address\" "
3539	echo " return_username=\"$return_username\" "
3540	if [[ $(echo $return_address \| grep -c "130.75.105") = 1 ]]
3541
3542	if [[ $localhost = lcbullhh \|\| $localhost = lccrayb \|\| $localhost = lccrayh \|\| $localhost = ibmh \|\| $localhost = ibmkisti \|\| $localhost = ibmku \|\| $localhost = ibms \|\| $localhost = lceddy \|\| $localhost = lckyu* \|\| $localhost = lcxe6 ]]
3543	then
3544	if [[ $localhost = ibmh ]]
3545	then
3546	ssh $SSH_PORTOPT $usern@136.172.40.15 "ssh $SSH_PORTOPT $return_address -l $return_username \". \\\$HOME/.profile; module load intel-compiler hdf5 netcdf; PATH=\\\$PATH:$LOCAL_MRUN_PATH;export PALM_BIN=$LOCAL_MRUN_PATH;cd $LOCAL_PWD; $mc\" "
3547	elif [[ $localhost = lcbullhh ]]
3548	then
3549	ssh $SSH_PORTOPT $usern@mlogin101 "ssh $SSH_PORTOPT $return_address -l $return_username \". \\\$HOME/.profile; module load intel-compiler hdf5 netcdf; PATH=\\\$PATH:$LOCAL_MRUN_PATH;export PALM_BIN=$LOCAL_MRUN_PATH;cd $LOCAL_PWD; $mc\" "
3550	elif [[ $localhost = lccrayb ]]
3551	then
3552	ssh $usern@blogin1 "ssh $SSH_PORTOPT $return_address -l $return_username \". \\\$HOME/.profile; module load intel-compiler hdf5 netcdf; PATH=\\\$PATH:$LOCAL_MRUN_PATH; export PALM_BIN=$LOCAL_MRUN_PATH;cd $LOCAL_PWD; $mc\" "
3553	elif [[ $localhost = lccrayh ]]
3554	then
3555	ssh $usern@hlogin1 "ssh $SSH_PORTOPT $return_address -l $return_username \". \\\$HOME/.profile; module load intel-compiler hdf5 netcdf; PATH=\\\$PATH:$LOCAL_MRUN_PATH; export PALM_BIN=$LOCAL_MRUN_PATH;cd $LOCAL_PWD; $mc\" "
3556	elif [[ $localhost = lcxe6 ]]
3557	then
3558	ssh $usern@hexagon "ssh $SSH_PORTOPT $return_address -l $return_username \". \\\$HOME/.profile; module load intel-compiler hdf5 netcdf; PATH=\\\$PATH:$LOCAL_MRUN_PATH; export PALM_BIN=$LOCAL_MRUN_PATH;cd $LOCAL_PWD; $mc\" "
3559	else
3560	ssh $SSH_PORTOPT $return_address -l $return_username ". \\\$HOME/.profile; module load intel-compiler hdf5 netcdf; PATH=\$PATH:$LOCAL_MRUN_PATH;export PALM_BIN=$LOCAL_MRUN_PATH;cd $LOCAL_PWD; $mc "
3561	fi
3562	else
3563	if [[ $localhost = ibmkisti ]]
3564	then
3565	ssh $SSH_PORTOPT $usern@gaiad "ssh $SSH_PORTOPT $return_address -l $return_username \"PATH=\\\$PATH:$LOCAL_MRUN_PATH;export PALM_BIN=$LOCAL_MRUN_PATH;cd $LOCAL_PWD; $mc\" "
3566	elif [[ $localhost = lceddy ]]
3567	then
3568	/usr/bin/ssh $SSH_PORTOPT $return_address -l $return_username "PATH=\$PATH:$LOCAL_MRUN_PATH;export PALM_BIN=$LOCAL_MRUN_PATH;cd $LOCAL_PWD; $mc "
3569	elif [[ $localhost = lcocean ]]
3570	then
3571	/usr/bin/ssh $SSH_PORTOPT $return_address -l $return_username "PATH=\$PATH:$LOCAL_MRUN_PATH;export PALM_BIN=$LOCAL_MRUN_PATH;cd $LOCAL_PWD; $mc "
3572	elif [[ $localhost = lccrayb ]]
3573	then
3574	ssh $usern@blogin1 "ssh $SSH_PORTOPT $return_address -l $return_username \"PATH=\\\$PATH:$LOCAL_MRUN_PATH;export PALM_BIN=$LOCAL_MRUN_PATH;cd $LOCAL_PWD; $mc\" "
3575	elif [[ $localhost = lccrayh ]]
3576	then
3577	ssh $usern@hlogin1 "ssh $SSH_PORTOPT $return_address -l $return_username \"PATH=\\\$PATH:$LOCAL_MRUN_PATH;export PALM_BIN=$LOCAL_MRUN_PATH;cd $LOCAL_PWD; $mc\" "
3578	else
3579	ssh $SSH_PORTOPT $return_address -l $return_username "PATH=\$PATH:$LOCAL_MRUN_PATH;export PALM_BIN=$LOCAL_MRUN_PATH;cd $LOCAL_PWD; $mc "
3580	fi
3581	fi
3582	else
3583	printf "\n +++ no restart mechanism available for host \"$localhost\" "
3584	locat=restart; exit
3585	fi
3586
3587	# WAIT TO ALLOW THE RESTART-JOB TO BE QUEUED, BEFORE THE CURRENT JOB IS FINISHED
3588	if [[ $queue = special1q ]]
3589	then
3590	sleep 120
3591	else
3592	sleep 30
3593	fi
3594
3595	else
3596
3597	# JOBS RUNNING (AND STARTED) ON THE LOCAL MACHINE CAN DIRECTLY CALL MRUN (WITHOUT
3598	# USING SSH)
3599	cd $LOCAL_PWD
3600	if [[ $localhost = lckyuh ]]
3601	then
3602	printf "\n +++ WARNING: no restart mechanism available for host \"$localhost\" "
3603	printf "\n please restart job manually using command:\n"
3604	printf "\n \"$mc\" "
3605	else
3606	eval $mc # THE ' MUST BE EVALUATED
3607	fi
3608	cd - > /dev/null
3609	fi
3610	if [[ $localhost = lckyuh ]]
3611	then
3612	printf "\n$dashes\n *** restart-run to be initiated manually\n"
3613	else
3614	printf "\n$dashes\n *** restart-run initiated \n"
3615	fi
3616
3617
3618	# DELETE INPUT-(RESTART)FILES, WHICH HAVE BEEN FETCHED FROM THE TEMPORARY DATA
3619	# DIRECTORY, BACAUSE THEY ARE NOT REQUIRED BY THE RESTART-JOB.
3620	# THIS IS DONE IN ORDER TO AVOID EXCEEDING DISC QUOTAS OR DISC SPACE (RESTART-FILES
3621	# MAY BE VERY HUGE)
3622	(( i = 0 ))
3623	while (( i < iin ))
3624	do
3625	(( i = i + 1 ))
3626	if [[ "${got_tmp[$i]}" = true && $keep_data_from_previous_run = false ]]
3627	then
3628	rm -r ${absnamein[$i]}
3629	fi
3630	done
3631
3632	fi
3633
3634
3635
3636	# SEND EMAIL NOTIFICATION ABOUT THE FINISHED RUN
3637	if [[ "$email_notification" != "none" ]]
3638	then
3639
3640	if [[ $localhost != $fromhost ]]
3641	then
3642	if [[ -f CONTINUE_RUN ]]
3643	then
3644	echo "PALM restart run necessary" > email_text
3645	echo "description header of actual run:" >> email_text
3646	cat CONTINUE_RUN >> email_text
3647	echo "mrun-command to restart:" >> email_text
3648	echo "$mc" >> email_text
3649	else
3650	echo "PALM run with base filename \"$fname\" on host \"$localhost\" finished" > email_text
3651	fi
3652	mail $email_notification < email_text
3653	printf "\n *** email notification sent to \"$email_notification\" "
3654	fi
3655	fi
3656
3657
3658
3659	# ALL ACTIONS FINISHED, TEMPORARY WORKING-DIRECTORY CAN BE DELETED
3660	cd $HOME
3661	[[ $delete_temporary_catalog = true ]] && rm -rf $TEMPDIR
3662
3663	else
3664
3665	# PREPARING ACTIONS,
3666	# IF A BATCH-JOB IS TO BE GENERATED AND STARTED ON A REMOTE-MACHINE GERECHNET
3667
3668	# BUILD THE MRUN-COMMAND TO BE CALLED IN THE BATCH-JOB ON THE REMOTE-MACHINE
3669	mrun_com="$mrun_script_name -a $afname -c $config_file -d $fname -h $host -H $fromhost -m $memory -t $cpumax -q $queue -R $return_address -U $return_username -u $remote_username"
3670	[[ "$cpp_opts" != "" ]] && mrun_com=${mrun_com}" -D \"$cpp_opts\""
3671	[[ "$global_revision" != "" ]] && mrun_com=${mrun_com}" -G \"$global_revision\""
3672	[[ $group_number != none ]] && mrun_com=${mrun_com}" -g $group_number"
3673	[[ $do_compile = true ]] && mrun_com=${mrun_com}" -s \"$source_list\""
3674	[[ "$input_list" != "" ]] && mrun_com=${mrun_com}" -i \"$input_list\""
3675	[[ $keep_data_from_previous_run = true ]] && mrun_com=${mrun_com}" -k"
3676	[[ "$additional_conditions" != "" ]] && mrun_com=${mrun_com}" -K \"$additional_conditions\""
3677	[[ "$output_list" != "" ]] && mrun_com=${mrun_com}" -o \"$output_list\""
3678	[[ "$read_from_config" = false ]] && mrun_com=${mrun_com}" -S"
3679	[[ $do_trace = true ]] && mrun_com=${mrun_com}" -x"
3680	[[ "$numprocs" != "" ]] && mrun_com=${mrun_com}" -X $numprocs"
3681	if [[ $use_openmp = true ]]
3682	then
3683	mrun_com=${mrun_com}" -O $threads_per_task"
3684	fi
3685	[[ "$tasks_per_node" != "" ]] && mrun_com=${mrun_com}" -T $tasks_per_node"
3686	[[ $package_list != "" ]] && mrun_com=${mrun_com}" -p \"$package_list\""
3687	[[ $return_password != "" ]] && mrun_com=${mrun_com}" -P $return_password"
3688	[[ $delete_temporary_catalog = false ]] && mrun_com=${mrun_com}" -B"
3689	[[ $node_usage != default && "$(echo $node_usage \| cut -c1-3)" != "sla" && $node_usage != novice ]] && mrun_com=${mrun_com}" -n $node_usage"
3690	[[ "$ocean_file_appendix" = true ]] && mrun_com=${mrun_com}" -y"
3691	[[ $run_coupled_model = true ]] && mrun_com=${mrun_com}" -Y \"$coupled_dist\""
3692	[[ "$check_namelist_files" = false ]] && mrun_com=${mrun_com}" -z"
3693	[[ "$combine_plot_fields" = false ]] && mrun_com=${mrun_com}" -Z"
3694	[[ "$max_par_io_str" != "" ]] && mrun_com=${mrun_com}" -w $max_par_io_str"
3695	if [[ $do_remote = true ]]
3696	then
3697	printf "\n>>>> MRUN-command on execution host:\n>>>> $mrun_com \n"
3698	fi
3699
3700
3701	# CREATE EXECUTABLE FOR BATCH JOB
3702	if [[ $create_executable_for_batch = true && $restart_run != true ]]
3703	then
3704
3705	printf "\n *** creating the executable for batch job\n"
3706
3707	# METHOD ONLY WORKS FOR BATCH JOBS ON LOCAL HOSTS
3708	if [[ $host != $localhost ]]
3709	then
3710	printf "\n +++ creation of executables is only allowed for batch jobs on local hosts."
3711	printf "\n Please set create_executable_for_batch = false in the config-file.\n"
3712	locat=create_executable; exit
3713	fi
3714
3715	mkdir $working_directory/SOURCES_FOR_RUN_${fname}/TMPDIR_FOR_CREATING_EXECUTABLE
3716	cd $working_directory/SOURCES_FOR_RUN_${fname}/TMPDIR_FOR_CREATING_EXECUTABLE
3717
3718	cp $make_depository .
3719	tar -xf $make_depository > /dev/null 2>&1
3720	cp ../* . > /dev/null 2>&1
3721
3722	make $mopts -f Makefile PROG=a.out F90=$compiler_name COPT="$cpp_options" F90FLAGS="$fopts" LDFLAGS="$lopts"
3723
3724	if [[ $? != 0 \|\| "$compile_error" = true \|\| "$module_compile_error" = true ]]
3725	then
3726	printf "\n +++ error occured while compiling or linking"
3727	locat=compile
3728	exit
3729	fi
3730
3731	mv a.out ..
3732	cd - > /dev/null 2>&1
3733	rm -rf $working_directory/SOURCES_FOR_RUN_${fname}/TMPDIR_FOR_CREATING_EXECUTABLE
3734
3735	printf " *** executable created in \"$working_directory/SOURCES_FOR_RUN_${fname}\"\n "
3736
3737	fi
3738
3739
3740	# BUILD THE JOB-SCRIPTS ON FILE jobfile
3741	jobfile=jobfile.$RANDOM
3742
3743
3744	# CREATE TEMPORARY DIRECTORY AND SWITCH TO IT
3745	echo "mkdir $TEMPDIR" >> $jobfile
3746	echo "cd $TEMPDIR" >> $jobfile
3747
3748
3749	# ACTIVATE ERROR-TRACEBACK
3750	if [[ $do_trace = true ]]
3751	then
3752	echo "set -x" >> $jobfile
3753	else
3754	echo "set +vx" >> $jobfile
3755	fi
3756
3757
3758	# INITIALIZE THE ENVIRONMENT AND LOAD MODULES
3759	if [[ "$init_cmds" != "" ]]
3760	then
3761	echo "$init_cmds" >> $jobfile
3762	fi
3763	if [[ "$module_calls" != "" ]]
3764	then
3765	echo "$module_calls" >> $jobfile
3766	fi
3767
3768
3769	# PROVIDE SOURCE-CODE FILES, MRUN-SCRIPT AND CONFIGURATION-FILE FOR THE JOB
3770	# then-CLAUSE: FILES ARE COLLECTED IN THE SOURCES_FOR_RUN_... DIRECTORY ON THE LOCAL HOST,
3771	# WHICH IS THEN FETCHED FROM THE BATCH-JOB BY USING cp/scp
3772	# THE SOURCE-CODE FILES ALREADY EXIST IN THIS DIRECTORY
3773	# else-CLAUSE: FILE-CONTENTS ARE PUT INTO THE JOB-FILE AS HERE-DOCUMENTS
3774	# THIS MAY CREATE A QUITE LARGE JOB-FILE, WHICH CAN CAUSE PROBLEMS WITH SOME
3775	# QUEUEING-SYSTEMS
3776	if [[ $host = ibmkisti \|\| $host = lcbullhh \|\| $host = lccrayb \|\| $host = lccrayf \|\| $host = lccrayh \|\| $host = lcocean ]]
3777	then
3778
3779	# COPY CONFIGURATION-FILE AND MRUN-SCRIPT INTO THE SOURCES_FOR_RUN... DIRECTORY
3780	if [[ $restart_run != true ]]
3781	then
3782	cp $config_file $working_directory/SOURCES_FOR_RUN_$fname
3783	cp ${PALM_BIN}/$mrun_script_name $working_directory/SOURCES_FOR_RUN_$fname
3784	fi
3785
3786	# COPY THE SOURCES_FOR_RUN... DIRECTORY FROM THE LOCAL HOST TO THE JOB VIA scp
3787	# (then-CLAUSE: JOBS ON THE LOCAL HOST CAN JUST USE cp)
3788	echo "set -x" >> $jobfile
3789	if [[ $host = $localhost ]]
3790	then
3791	# DUE TO UNKNOWN REASONS, COPY WITH cp COMMAND CREATES CORRUPT
3792	# FILES ON CRAY XC30 SYSTEMS (CSC HELSINKI), rsync IS USED INSTEAD
3793	if [[ $(echo $host \| cut -c1-6) = lccray ]]
3794	then
3795	echo "rsync -av -t $working_directory/SOURCES_FOR_RUN_$fname ." >> $jobfile
3796	else
3797	echo "cp -r $working_directory/SOURCES_FOR_RUN_$fname ." >> $jobfile
3798	fi
3799
3800	else
3801	if [[ $host = ibmkisti ]]
3802	then
3803	# ON KISTI'S IBM FIREWALL IS ONLY OPENED ON INTERACTIVE NODE
3804	echo "localdir=\`pwd\`" >> $jobfile
3805	echo "ssh $SSH_PORTOPT $remote_username@gaiad \"cd \$localdir; scp $PORTOPT -r $return_username@$return_address:$working_directory/SOURCES_FOR_RUN_$fname .\" " >> $jobfile
3806	elif [[ $host = lcbullhh ]]
3807	then
3808	echo "localdir=\`pwd\`" >> $jobfile
3809	echo "ssh $SSH_PORTOPT $remote_username@mlogin101 \"cd \$localdir; scp $PORTOPT -r $return_username@$return_address:$working_directory/SOURCES_FOR_RUN_$fname .\" " >> $jobfile
3810
3811	elif [[ $host = lccrayb ]]
3812	then
3813	echo "localdir=\`pwd\`" >> $jobfile
3814	echo "ssh $SSH_PORTOPT $remote_username@blogin1 \"cd \$localdir; scp $PORTOPT -r $return_username@$return_address:$working_directory/SOURCES_FOR_RUN_$fname .\" " >> $jobfile
3815	elif [[ $host = lccrayh ]]
3816	then
3817	echo "localdir=\`pwd\`" >> $jobfile
3818	echo "ssh $SSH_PORTOPT $remote_username@hlogin1 \"cd \$localdir; scp $PORTOPT -r $return_username@$return_address:$working_directory/SOURCES_FOR_RUN_$fname .\" " >> $jobfile
3819
3820	else
3821	echo "scp $PORTOPT -r $return_username@$return_address:$working_directory/SOURCES_FOR_RUN_$fname ." >> $jobfile
3822	fi
3823	fi
3824	echo "export SOURCES_COMPLETE=true" >> $jobfile
3825
3826	# MOVE MRUN-SCRIPT AND CONFIGURATION-FILE FROM THE SOURCES_FOR_RUN... DIRECTORY TO THE
3827	# WORKING DIRECTORY OF THE JOB
3828	echo "mv SOURCES_FOR_RUN_$fname/$config_file . " >> $jobfile
3829	echo "mv SOURCES_FOR_RUN_$fname/$mrun_script_name . " >> $jobfile
3830	echo "chmod u+rwx $mrun_script_name" >> $jobfile
3831	echo "execute_mrun=true" >> $jobfile
3832	echo " " >> $jobfile
3833
3834	else
3835
3836	# PROVIDE SOURCE-CODE FILES AND MAKEFILE AS HERE DOCUMENT
3837	if [[ $do_compile = true ]]
3838	then
3839
3840	source_catalog=SOURCES_FOR_RUN_$fname
3841
3842	# CREATE SOURCES_FOR_RUN... DIRECTORY TO STORE THE SOURCE CODE FILES AND THE MAKEFILE
3843	echo "mkdir SOURCES_FOR_RUN_$fname" >> $jobfile
3844	echo "export SOURCES_COMPLETE=true" >> $jobfile
3845	echo "cd SOURCES_FOR_RUN_$fname" >> $jobfile
3846
3847	for filename in $source_list
3848	do
3849	# BACKSLASH IS USED FOR MASKING
3850	echo "cat > $filename << \"%END%\"" >> $jobfile
3851	cat $source_catalog/$filename >> $jobfile
3852	echo " " >> $jobfile
3853	echo "%END%" >> $jobfile
3854	echo " " >> $jobfile
3855	done
3856
3857	# BACKSLASH IS USED FOR MASKING
3858	echo "cat > Makefile << \"%END%\"" >> $jobfile
3859	cat $source_catalog/Makefile >> $jobfile
3860	echo " " >> $jobfile
3861	echo "%END%" >> $jobfile
3862	echo " " >> $jobfile
3863
3864	echo "cd - > /dev/null" >> $jobfile
3865
3866	fi
3867
3868
3869	# PROVIDE THE CONFIGURATION-FILE AS HERE-DOCUMENT
3870	# BACKSLASH IS USED FOR MASKING
3871	# LINES WITH #$ IN THE CONFIGURATION-FILE, COMING FROM THE SVN KEYWORD SUBSTITUTION,
3872	# ARE REMOVED FROM THE FILE IN ORDER TO AVOID PROBLEMS WITH THE SGE BATCH SYSTEM
3873	echo "cat > $config_file << \"%END%\"" >> $jobfile
3874	if [[ $host = lckyuh ]]
3875	then
3876	# NO CROSS-COMPILER ON COMPUTE NODE
3877	sed 's/frtpx/frt/g' $config_file >> $jobfile
3878	else
3879	sed 's/#$.*//g' $config_file >> $jobfile
3880	fi
3881	echo "%END%" >> $jobfile
3882	echo " " >> $jobfile
3883
3884
3885	# PROVIDE THE MRUN-SCRIPTS AS HERE-DOCUMENT
3886	# BACKSLASH IS USED FOR MASKING
3887	echo "cat > $mrun_script_name <<\"%END%\"" >> $jobfile
3888	if [[ $host = lckyuh ]]
3889	then
3890	sed 's/\/bin\/ksh/\/bin\/bash/g' ${PALM_BIN}/$mrun_script_name >> $jobfile
3891	else
3892	cat ${PALM_BIN}/$mrun_script_name >> $jobfile
3893	fi
3894	echo "%END%" >> $jobfile
3895	echo "chmod u+x $mrun_script_name" >> $jobfile
3896	echo "execute_mrun=true" >> $jobfile
3897	echo " " >> $jobfile
3898
3899	fi
3900
3901
3902	# GET REQUIRED INPUT-FILES BY SCP OR BY SENDING THEM WITH THE JOB AS HERE-DOCUMENT
3903	# PUT THESE FILES INTO THE USER'S RESPECTIVE PERMANENT DIRECTORIES ON THE REMOTE-HOST
3904	# IF THE DIRECTORIES DO NOT EXIST, TRY TO CREATE THEM
3905	if [[ $do_remote = true ]]
3906	then
3907	(( i = 0 ))
3908	while (( i < iin ))
3909	do
3910	(( i = i + 1 ))
3911	echo "[[ ! -d ${pathin[$i]} ]] && mkdir -p ${pathin[$i]}" >> $jobfile
3912	if [[ "${transin[$i]}" = job ]]
3913	then
3914	echo "cat > ${remotepathin[$i]} <<\"%END%\"" >> $jobfile
3915	eval cat ${pathin[$i]}/${frelin[$i]} >> $jobfile
3916	echo " " >> $jobfile
3917	echo "%END%" >> $jobfile
3918	else
3919	echo "batch_scp $PORTOPT -b -o -g -s -u $return_username $return_address ${remotepathin[$i]} \"${pathin[$i]}\" ${frelin[$i]}" >> $jobfile
3920	fi
3921
3922	# CHECK, IF FILE COULD BE CREATED
3923	echo "if [[ \$? = 1 ]]" >> $jobfile
3924	echo "then" >> $jobfile
3925	echo " echo \" \" " >> $jobfile
3926	echo " echo \"+++ file ${remotepathin[$i]} could not be created\" " >> $jobfile
3927	echo " echo \" please check, if directory exists on $host!\" " >> $jobfile
3928	echo " echo \"+++ MRUN will not be continued\" " >> $jobfile
3929	echo " execute_mrun=false" >> $jobfile
3930	echo "fi" >> $jobfile
3931	done
3932	fi
3933
3934	# PROVIDE NAME OF THE CURRENT WORKING-DIRECTORY ON THE LOCAL MACHINE (FROM WHERE THE JOB IS
3935	# STARTED) BY SETTING AN ENVIRONMENT-VARIABLE. THIS INFORMATION IS USED IN THE JOB BY MRUN
3936	# IN CASE THAT RESTART-RUNS HAVE TO BE GENERATED
3937	echo "LOCAL_PWD=$working_directory" >> $jobfile
3938	echo "export LOCAL_PWD" >> $jobfile
3939
3940	# PROVIDE THE PATH OF THE LOCAL MRUN-SCRIPT FOR THE SAME REASON
3941	echo "LOCAL_MRUN_PATH=$PALM_BIN" >> $jobfile
3942	echo "export LOCAL_MRUN_PATH" >> $jobfile
3943
3944	# lceddy ALSO REQUIRES TO PROVIDE PATH FOR THE PALM-SCRIPTS
3945	if [[ $host = lceddy \|\| $localhost = lceddy ]]
3946	then
3947	echo "export PALM_BIN=$PALM_BIN" \| sed -e 's:'$HOME':$HOME:' >> $jobfile
3948	echo "export PATH=\$PATH:\$PALM_BIN" >> $jobfile
3949	fi
3950
3951	# CALL MRUN WITHIN THE JOB (SETTING QUEUE IS A WORKAROUND FOR ibmkisti)
3952	# AS FINAL ACTION, REMOVE THE TEMPORARY DIRECTORY CREATED AT THE BEGINNING OF THE JOB
3953	echo "set -x" >> $jobfile
3954	echo "queue=$queue" >> $jobfile
3955	echo "[[ \$execute_mrun = true ]] && ./$mrun_com" >> $jobfile
3956	echo 'ls -al; echo `pwd`' >> $jobfile
3957	echo "cd \$HOME" >> $jobfile
3958	echo "rm -rf $TEMPDIR" >> $jobfile
3959
3960
3961
3962
3963	# START THE JOB USING SUBJOB-COMMAND
3964	if [[ $silent = false ]]
3965	then
3966	printf "\n "
3967	else
3968	printf "\n\n"
3969	fi
3970
3971	subjob $job_on_file -h $host -u $remote_username -g $group_number -q $queue -m $memory -N $node_usage -t $cpumax $XOPT $TOPT $OOPT -n $fname -v -c $job_catalog -e $email_notification $PORTOPT $jobfile
3972	rm -rf $jobfile
3973
3974
3975	fi # END OF REMOTE-PART

Note: See TracBrowser for help on using the repository browser.

Download in other formats:

| Impressum | ©Leibniz Universität Hannover |