Home

Context Navigation

source: palm/trunk/SCRIPTS/mrun @ 1357

Last change on this file since 1357 was 1357, checked in by witha, 10 years ago
Adjustments for lccrayh (automatic restarts on lccrayh outside of IMUK are now possible)
Property svn:keywords set to `Id Rev`
File size: 174.7 KB

Line
1	#!/bin/ksh
2
3	# mrun - script for running PALM jobs
4
5	#--------------------------------------------------------------------------------#
6	# This file is part of PALM.
7	#
8	# PALM is free software: you can redistribute it and/or modify it under the terms
9	# of the GNU General Public License as published by the Free Software Foundation,
10	# either version 3 of the License, or (at your option) any later version.
11	#
12	# PALM is distributed in the hope that it will be useful, but WITHOUT ANY
13	# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
14	# A PARTICULAR PURPOSE. See the GNU General Public License for more details.
15	#
16	# You should have received a copy of the GNU General Public License along with
17	# PALM. If not, see <http://www.gnu.org/licenses/>.
18	#
19	# Copyright 1997-2014 Leibniz Universitaet Hannover
20	#--------------------------------------------------------------------------------#
21	#
22	# Current revisions:
23	# ------------------
24	# adjustments for lccrayh (automatic restarts on lccrayh outside of IMUK are now
25	# possible)
26	#
27	# Former revisions:
28	# -----------------
29	# $Id: mrun 1357 2014-04-11 15:02:03Z witha $
30	#
31	# 1320 2014-03-20 08:40:49Z raasch
32	# check namelist file set false by default
33	#
34	# 1304 2014-03-12 10:29:42Z raasch
35	# ulimit option changed from -Ss to -s
36	#
37	# bugfix: missing "fi" in r1289
38	#
39	# 1289 2014-03-04 07:12:34Z raasch
40	# comments translated to English
41	# necriam-, fimm-, ibmy-, and sgi-specific code removed
42	# export of variables for palm and interpret_config removed
43	#
44	# 1281 2014-02-01 07:55:49Z raasch
45	# rsync-copy restricted to Cray machines, since command is unavailable on some
46	# other systems
47	#
48	# 1279 2014-01-28 12:10:14Z raasch
49	# tasks_per_node must not be an integral divisor of numprocs any more. This was done
50	# in order to remove annoying restrictions concerning the number of processors which
51	# appear on machines with larger nodes (e.g. containing 24 cores). Now without this
52	# restriction, one of the nodes will be filled with less than the given number of
53	# tasks per node. A respective warning is given.
54	#
55	# 1274 2014-01-09 13:14:54Z heinze
56	# adjustments for lccrayh
57	#
58	# 1272 2014-01-08 10:19:32Z witha
59	# small adjustment for lcflow
60	#
61	# 1270 2013-12-16 11:05:01Z fricke
62	# call of combine_plot_fields adjusted for lccrayb/lccrayh
63	#
64	# 1255 2013-11-07 14:43:35Z raasch
65	# further adjustments for lccrayb remote access
66	#
67	# 1241 2013-10-30 11:36:58Z heinze
68	# Enable use of nudging input and input of large scale forcing from
69	# external files
70	#
71	# 1229 2013-09-20 06:55:19Z raasch
72	# further adjustments for lccrayb
73	#
74	# 1224 2013-09-16 07:27:23Z raasch
75	# first adjustments for lccrayb
76	#
77	# 1210 2013-08-14 10:58:20Z raasch
78	# fftw support added
79	#
80	# 1201 2013-07-10 16:17:59Z witha
81	# adjustments for Forwind cluster (lcflow)
82	#
83	# 1199 2013-07-05 14:52:22Z raasch
84	# adjustments for CSC Helsinki (lccrayf),
85	# executables for batch jobs can be created in advance, in order to avoid calling
86	# the compiler within the batch job (only works if batch jobs are submitted on
87	# local host)
88	#
89	# 1190 2013-06-25 09:39:21Z heinze
90	# enable use of automatic restarts for ibmh
91	# use of cluster/express queue enabled (ibmh)
92	#
93	# 1124 2013-04-09 15:46:52Z raasch
94	# variable "memory" is exported via typeset option -x, because otherwise an unknown
95	# side effect may lead to data loss while getopts is reading the script-option arguments
96	#
97	# 1122 2013-04-09 08:37:16Z heinze
98	# Bugfix: change type of variable last_char
99	#
100	# 1119 2013-04-05 15:11:19Z raasch
101	# Bugfix for setting -T option for subjob
102	#
103	# 1108 2013-03-05 07:03:32Z raasch
104	# bugfix for coupled runs on lckyut/lckyuh
105	#
106	# 1106 2013-03-04 05:31:38Z raasch
107	# --stdin argument for mpiexec on lckyuh
108	# -y and -Y settings output to header
109	#
110	# 1103 2013-02-20 02:15:53Z raasch
111	# default script runs again under ksh, because of unsolved problems with read
112	# from stdin: when bash script is called from a ksh, message "read error: 0:
113	# Resource temporarily unavailable" appears and script does not stop,
114	# further bash compatibility adjustments,
115	# shebang line replaced by /bin/bash when running jobs on lckyuh; no restarts
116	# on lckyuh, but mrun does not terminate and issues a warning instead
117	#
118	# 1101 2013-02-17 10:20:21Z raasch
119	# script now running under bash instead of ksh, which required small adjustments
120	# (output formatting with printf instead "typeset -L/-R", print replaced by echo,
121	# read from stdin),
122	# cross compilername on lckyuh compute nodes replaced by real compiler name
123	#
124	# 1099 2013-02-10 01:47:43Z raasch
125	# adjustments for Kyushu-University computing center (lckyuh - hayaka)
126	# and for Forwind cluster (lcflow)
127	# small further adjustments for lckyut
128	#
129	# 1094 2013-02-03 01:52:12Z raasch
130	# explicit ssh/scp port can be set in config file with environment variable
131	# scp_port. This port is handled to all ssh/scp/batch_scp calls.
132	# decalpha parts (yonsei) removed
133	#
134	# 2013-02-02 07:06:13Z raasch
135	# adjustments for Kyushu-University computing center (lckyut - tatara)
136	#
137	# 1083 2013-01-04 10:22:09Z maronga
138	# bugfix in parameter file check (read %cpp_options was missing)
139	#
140	# 1069 2012-11-28 16:18:43Z maronga
141	# bugfix: coupling mode was always set to mpi2, typos removed
142	#
143	# 1058 2012-11-21 07:00:35Z raasch
144	# Intel inspector (inspxe) is given the number of PEs instead of the number of
145	# nodes
146	#
147	# 1046 2012-11-09 14:38:45Z maronga
148	# code put under GPL (PALM 3.9)
149	#
150	# 21/03/94 - Siggi - first version finished
151	# 03/03/94 - Siggi - script development started
152	#
153	#--------------------------------------------------------------------------------#
154	# mrun - script for running PALM jobs
155	#--------------------------------------------------------------------------------#
156
157
158
159	# DECLARATION OF VARIABLES AND THEIR DEFUALT VALUES
160
161	set +o allexport # SUPPRESS EXPORT OF ALL VARIABLES, SINCE IN THE PAST THIS
162	# LES TO PROBLEMS IN ROUTINES CALLED BY MRUN
163	# (TOO MANY ARGUMENTS - PROBLEM)
164	set +o noclobber # EXISTING FILES ARE ALLOWED TO BE OVERWRITTEN
165
166	AddFilenames=""
167	additional_conditions=""
168	add_source_path=""
169	afname=""
170	archive_save=true
171	archive_system=none
172	check_namelist_files=false
173	combine_plot_fields=true
174	compiler_name=""
175	cond1=""
176	cond2=""
177	config_file=.mrun.config
178	coupled_dist=""
179	coupled_mode="mpi1"
180	cpp_opts=""
181	cpp_options=""
182	cpumax=0
183	cpurest=0
184	create_executable_for_batch=false
185	delete_temporary_catalog=true
186	do_batch=false
187	do_compile=true
188	do_remote=false
189	do_stagein=true
190	do_stageout=true
191	do_trace=false
192	email_notification="none"
193	exclude=""
194	executable=""
195	execution_error=false
196	fftw_inc=""
197	fftw_lib=""
198	fftw_support=false
199	fname=test
200	fromhost=""
201	global_revision=""
202	group_number=none
203	host=""
204	host_file=""
205	hp=""
206	ignore_archive_error=false
207	input_list=""
208	interpreted_config_file=""
209	job_catalog="~/job_queue"
210	job_on_file=""
211	keep_data_from_previous_run=false
212	link_local_input=false
213	link_local_output=false
214	localhost_realname=$(hostname)
215	local_dvrserver_running=.FALSE.
216	locat=normal
217	mainprog=""
218	makefile=""
219	max_par_io_str=""
220	mc=$0
221	while [[ $(echo $mc \| grep -c "/") != 0 ]]
222	do
223	mc=`echo $mc \| cut -f2- -d"/"`
224	done
225	module_calls=""
226	mrun_script_name=$mc
227	netcdf_inc=""
228	netcdf_lib=""
229	netcdf_support=false
230	node_usage=default
231	numprocs=""
232	numprocs_atmos=0
233	numprocs_ocean=0
234	OOPT=""
235	openmp=false
236	output_list=""
237	package_list=""
238	queue=none
239	read_from_config=""
240	restart_run=false
241	if [[ `hostname` = rte10 ]]
242	then
243	return_addres=133.5.185.60
244	echo "+++ WARNING: fixed return_addres = $return_addres is used !!!!!"
245	elif [[ `hostname` = climate0 ]]
246	then
247	return_addres=165.132.26.68
248	echo "+++ WARNING: fixed return_addres = $return_addres is used !!!!!"
249	elif [[ `hostname` = urban00 ]]
250	then
251	return_addres=147.46.30.151
252	echo "+++ WARNING: fixed return_addres = $return_addres is used !!!!!"
253	else
254	return_addres=$(nslookup `hostname` 2>&1 \| grep "Address:" \| tail -1 \| awk '{print $2}')
255	fi
256	return_password=""
257	return_username=$LOGNAME
258	remotecall=false
259	remote_username=""
260	run_coupled_model=false
261	run_mode=""
262	store_on_archive_system=false
263	dashes=" ----------------------------------------------------------------------------"
264	silent=false
265	source_list=""
266	source_path=SOURCE
267	tasks_per_node=""
268	threads_per_task=1
269	tmpcreate=false
270	tmp_data_catalog=""
271	transfer_problems=false
272	usern=$LOGNAME
273	use_openmp=false
274	version="MRUN 2.1 Rev$Rev: 1357 $"
275	working_directory=`pwd`
276	TOPT=""
277	XOPT=""
278	zeit=$( date \| cut -c 12-19 )
279
280	typeset -i iec=0 iic=0 iin=0 ioc=0 iout=0 stagein_anz=0 stageout_anz=0
281	typeset -x -i memory=0 # HAS TO BE EXPORTED HERE, OTHERWISE AN UNKNOWN SIDE
282	# SIDE EFFECT MAY CAUSE DATA LOSS WHEN GETOPTS IS READING THE
283	# SCRIPT-OPTION ARGUMENTS
284	typeset -i cputime i ii iia iii iio icycle inode ival jobges jobsek last_char_int maxcycle minuten nodes pes remaining_pes sekunden tp1
285
286
287
288	# ERROR HANDLING IN CASE OF EXIT
289	trap 'rm -rf $working_directory/tmp_mrun
290	if [[ $locat != normal && $locat != control_c ]]
291	then
292
293	# CARRY OUT ERROR-COMMANDS GIVEN IN THE CONFIGURATION FILE (EC:)
294	(( i = 0 ))
295	while (( i < iec ))
296	do
297	(( i = i + 1 ))
298	printf "\n *** Execution of ERROR-command:\n"
299	printf " >>> ${err_command[$i]}\n"
300	eval ${err_command[$i]}
301	done
302	if [[ -n $interpreted_config_file ]]
303	then
304	rm -rf $interpreted_config_file
305	fi
306	if [[ -n .mrun_environment ]]
307	then
308	rm -rf .mrun_environment
309	fi
310	if [[ $tmpcreate = true ]]
311	then
312	printf "\n *** Contents of \"$TEMPDIR\":\n"
313	ls -al; cd
314	[[ $delete_temporary_catalog = true ]] && rm -rf $TEMPDIR
315	fi
316	if [[ "$dvrserver_id" != "" ]]
317	then
318	echo "+++ killing dvrserver_id=$dvrserver_id"
319	kill $dvrserver_id
320	fi
321	if [[ -f ~/job_queue/JOBINFO.$QSUB_REQID ]]
322	then
323	rm -rf ~/job_queue/JOBINFO.$QSUB_REQID
324	fi
325	printf "\n\n+++ MRUN killed \n\n"
326	elif [[ $locat != control_c ]]
327	then
328	printf "\n\n --> all actions finished\n\n"
329	printf " Bye, bye $usern !!\n\n"
330	fi' exit
331
332
333	# ACTIONS IN CASE OF TERMINAL-BREAK (CONTROL-C):
334	trap 'rm -rf $working_directory/tmp_mrun
335	rm -rf $working_directory/tmp_check_namelist_files
336	[[ $tmpcreate = true ]] && (cd; rm -rf $TEMPDIR)
337	if [[ -f ~/job_queue/JOBINFO.$QSUB_REQID ]]
338	then
339	rm -rf ~/job_queue/JOBINFO.$QSUB_REQID
340	fi
341	if [[ "$dvrserver_id" != "" ]]
342	then
343	echo "+++ killing dvrserver_id=$dvrserver_id"
344	kill $dvrserver_id
345	fi
346	printf "\n+++ MRUN killed by \"^C\" \n\n"
347	locat=control_c
348	exit
349	' 2
350
351
352	# CHECK IF THE PATH FOR THE PALM BINARIES (SCRIPTS+UTILITY-PROGRAMS) HAS
353	# BEEN SET
354	if [[ "$PALM_BIN" = "" ]]
355	then
356	printf "\n +++ environment variable PALM_BIN has not been set"
357	printf "\n please set it to the directory where the PALM scripts are located"
358	locat=palm_bin; exit
359	fi
360	export PATH=$PALM_BIN:$PATH
361
362
363
364	# READ SHELLSCRIPT-OPTIONS AND REBUILD THE MRUN-COMMAND STRING (MC),
365	# WHICH WILL BE USED TO START RESTART-JOBS
366	while getopts :a:AbBc:Cd:D:Fg:G:h:H:i:IkK:m:M:n:o:O:p:P:q:r:R:s:St:T:u:U:vw:xX:yY:zZ option
367	do
368	case $option in
369	(a) afname=$OPTARG;;
370	(A) store_on_archive_system=true; mc="$mc -A";;
371	(b) do_batch=true; mc="$mc -b";;
372	(B) delete_temporary_catalog=false; mc="$mc -B";;
373	(c) config_file=$OPTARG; mc="$mc -c$OPTARG";;
374	(C) restart_run=true; mc="$mc -C";;
375	(d) fname=$OPTARG; mc="$mc -d$OPTARG";;
376	(D) cpp_opts="$cpp_opts $OPTARG"; mc="$mc -D'$OPTARG'";;
377	(F) job_on_file="-D"; mc="$mc -F";;
378	(g) group_number=$OPTARG; mc="$mc -g$OPTARG";;
379	(G) global_revision=$OPTARG; mc="$mc -G'$OPTARG'";;
380	(h) host=$OPTARG; mc="$mc -h$OPTARG";;
381	(H) fromhost=$OPTARG; mc="$mc -H$OPTARG";;
382	(i) input_list=$OPTARG; mc="$mc -i'$OPTARG'";;
383	(I) ignore_archive_error=true; mc="$mc -I";;
384	(k) keep_data_from_previous_run=true; mc="$mc -k";;
385	(K) additional_conditions="$OPTARG"; mc="$mc -K'$OPTARG'";;
386	(m) memory=$OPTARG; mc="$mc -m$OPTARG";;
387	(M) makefile=$OPTARG; mc="$mc -M$OPTARG";;
388	(n) node_usage=$OPTARG; mc="$mc -n$OPTARG";;
389	(o) output_list=$OPTARG; mc="$mc -o'$OPTARG'";;
390	(O) use_openmp=true; threads_per_task=$OPTARG; mc="$mc -O$OPTARG";;
391	(p) package_list=$OPTARG; mc="$mc -p'$OPTARG'";;
392	(P) return_password=$OPTARG; mc="$mc -P$OPTARG";;
393	(q) queue=$OPTARG; mc="$mc -q$OPTARG";;
394	(r) run_mode=$OPTARG; mc="$mc -r'$OPTARG'";;
395	(R) remotecall=true;return_addres=$OPTARG; mc="$mc -R$OPTARG";;
396	(s) source_list=$OPTARG; mc="$mc -s'$OPTARG'";;
397	(S) read_from_config=false; mc="$mc -S";;
398	(t) cpumax=$OPTARG; mc="$mc -t$OPTARG";;
399	(T) mrun_tasks_per_node=$OPTARG; mc="$mc -T$OPTARG";;
400	(u) remote_username=$OPTARG; mc="$mc -u$OPTARG";;
401	(U) return_username=$OPTARG; mc="$mc -U$OPTARG";;
402	(v) silent=true; mc="$mc -v";;
403	(w) max_par_io_str=$OPTARG; mc="$mc -w$OPTARG";;
404	(x) do_trace=true;set -x; mc="$mc -x";;
405	(X) numprocs=$OPTARG; mc="$mc -X$OPTARG";;
406	(y) ocean_file_appendix=true; mc="$mc -y";;
407	(Y) run_coupled_model=true; coupled_dist=$OPTARG; mc="$mc -Y'$OPTARG'";;
408	(z) check_namelist_files=false; mc="$mc -z";;
409	(Z) combine_plot_fields=false; mc="$mc -Z";;
410	(\?) printf "\n +++ unknown option $OPTARG \n"
411	printf "\n --> type \"$0 ?\" for available options \n"
412	locat=parameter;exit;;
413	esac
414	done
415
416
417	# SKIP GIVEN OPTIONS TO READ POSITIONAL PARAMETER, IF GIVEN
418	# CURRENTLY ONLY PARAMETER ? (TO OUTPUT A SHORT COMMAND INFO) IS ALLOWED
419	(( to_shift = $OPTIND - 1 ))
420	shift $to_shift
421
422	# PRINT SHORT DESCRIPTION OF MRUN OPTIONS
423	if [[ "$1" = "?" ]]
424	then
425	(printf "\n *** mrun can be called as follows:\n"
426	printf "\n $mrun_script_name -b -c.. -d.. -D.. -f.. -F -h.. -i.. -I -K.. -m.. -o.. -p.. -r.. -R -s.. -t.. -T.. -v -x -X.. -y -Y.. -z -Z <modus> \n"
427	printf "\n Description of available options:\n"
428	printf "\n Option Description Default-Value"
429	printf "\n -a base name of input files equiv. -d"
430	printf "\n -A archiving when using file-attribute fl"
431	printf "\n -b batch-job on local machine ---"
432	printf "\n -B do not delete temporary directory at end ---"
433	printf "\n -c configuration file .mrun.config"
434	printf "\n -d base name of files attached to program test"
435	printf "\n -D preprocessor(cpp)-directives \"\" "
436	printf "\n -F create remote job file only ---"
437	printf "\n -h execution host $localhost_realname"
438	printf "\n -i INPUT control list \"\" "
439	printf "\n -I archiving errors of previous batch-jobs"
440	printf "\n will be ignored"
441	printf "\n -k keep data from previous run"
442	printf "\n -K additional conditions for controling"
443	printf "\n usage of conditional code and"
444	printf "\n env-variables in configuration file \"\" "
445	printf "\n -m memory demand in MB (batch-jobs) 0 MB"
446	printf "\n -M Makefile name Makefile"
447	printf "\n -n node usage (shared/not_shared) depending on -h"
448	printf "\n -o OUTPUT control list \"\" "
449	printf "\n -O threads per openMP task ---"
450	printf "\n -p software package list \"\" "
451	printf "\n -q queue \"$queue\" "
452	printf "\n -r run control list (combines -i -o) \"\" "
453	printf "\n -s filenames of routines to be compiled \"\" "
454	printf "\n must end with .f, .f90, .F, or .c !"
455	printf "\n use \"..\" for more than one file and wildcards"
456	printf "\n -s LM compiles all locally modified files"
457	printf "\n -S config file interpreted by shellscript ---"
458	printf "\n -t allowed cpu-time in seconds (batch) 0"
459	printf "\n -T tasks per node depending on -h"
460	printf "\n -u username on remote machine \"\" "
461	printf "\n -v no prompt for confirmation ---"
462	printf "\n -w maximum parallel io streams as given by -X"
463	printf "\n -x tracing of mrun for debug purposes ---"
464	printf "\n -X # of processors (on parallel machines) 1"
465	printf "\n -y add appendix \"_O\" to all local output"
466	printf "\n files (ocean precursor runs followed by"
467	printf "\n coupled atmosphere-ocean runs) ---"
468	printf "\n -Y run coupled model, \"#1 #2\" with"
469	printf "\n #1 atmosphere and #2 ocean processors \"#/2 #/2\" depending on -X"
470	printf "\n -z disable a priori parameter file check ---"
471	printf "\n -Z skip combine_plot_fields at the end of "
472	printf "\n the simulation ---"
473	printf "\n "
474	printf "\n Possible values of positional parameter <modus>:"
475	printf "\n \"?\" - this outline \n\n") \| more
476	exit
477	elif [[ "$1" != "" ]]
478	then
479	printf "\n +++ positional parameter $1 unknown \n"
480	locat=parameter; exit
481	fi
482
483
484
485	# SHORT STARTING MESSAGE
486	printf "\n*** $version "
487	printf "\n will be executed. Please wait ..."
488
489
490
491	# CHECK, IF CONFIGURATION FILE EXISTS
492	if [[ ! -f $config_file ]]
493	then
494	printf "\n\n +++ configuration file: "
495	printf "\n $config_file"
496	printf "\n does not exist"
497	locat=connect; exit
498	fi
499
500
501	# DETERMINE THE HOST-IDENTIFIER (localhost) FROM THE CONFIGURATION FILE
502	line=""
503	grep "%host_identifier" $config_file > tmp_mrun
504	while read line
505	do
506	if [[ "$line" != "" && $(echo $line \| cut -c1) != "#" ]]
507	then
508	HOSTNAME=`echo $line \| cut -d" " -s -f2`
509	host_identifier=`echo $line \| cut -d" " -s -f3`
510	if [[ $localhost_realname = $HOSTNAME ]]
511	then
512	localhost=$host_identifier
513	break
514	fi
515	fi
516	done < tmp_mrun
517
518	if [[ "$localhost" = "" ]]
519	then
520	printf "\n\n +++ no host identifier found in configuration file \"$config_file\""
521	printf "\n for local host \"$localhost_realname\"."
522	printf "\n Please add line"
523	printf "\n \"\%host_identifier $localhost_realname <identifier>\""
524	printf "\n to the configuration file."
525	locat=localhost; exit
526	fi
527
528
529	# SET HOST-SPECIFIC VARIABLES
530	case $localhost_realname in
531	(r1\|r2\|h01\|b01) archive_system=tivoli;;
532	(cs*) archive_system=ut;;
533	esac
534
535
536	# ??? THIS SHOULD BE DECRIBED IN THE MRUN DOCUMENTATION ???
537	# SET BASENAME OF THE INPUT-FILES TO THE GENERAL BASENAME (GIVEN BY OPTION -d),
538	# IF IT HAS NOT BEEN SET BY THE USER DIFFERENTLY (USING OPTION -a)
539	[[ "$afname" = "" ]] && afname=$fname
540
541
542	# ADD RUN-FILE ACTIVATION STRINGS (GIVEN BY OPTION -r) TO THE
543	# RESPECTIVE LISTS FOR ACTIVATING THE INPUT/OUTPUT FILE CONNECTION
544	# STATEMENTS IN THE CONFIGURATION FILE
545	if [[ "$run_mode" != "" ]]
546	then
547	input_list="$input_list $run_mode"
548	output_list="$output_list $run_mode"
549	fi
550
551
552	# ??? is this (and the respective option -H) really required ???
553	# STORE HOSTNAME, FROM WHICH THE JOB IS STARTED,
554	# BUT ONLY IF IT HASN'T BEEN ALREADY SET BY OPTION -H
555	# (MRUN IS AUTOMATICALLY SETTING THIS OPTION IN THE MRUN-CALLS WITHIN
556	# THOSE BATCH-JOBS, WHICH ARE CREATED BY MRUN ITSELF)
557	if [[ "$fromhost" = "" ]]
558	then
559	fromhost=$localhost
560	fi
561
562
563	# CHECK, IF EXECUTION SHALL TO BE DONE ON THE REMOTE-MACHINE.
564	# VALUE OF do_remote IS IMPORTANT FOR THE FILE CONNECTIONS.
565	# IN CASE OF EXECUTION ON A REMOTE-MACHINE, A BATCH-JOB HAS
566	# TO BE SUBMITTED (I.E. do_bacth=true)
567	if [[ -n $host && "$host" != $localhost ]]
568	then
569	do_batch=true
570	do_remote=true
571	case $host in
572	(ibm\|ibmh\|ibmkisti\|ibmku\|ibms\|nech\|lccrayb\|lccrayh\|lccrayf\|lcflow\|lckyoto\|unics\|lcxe6\|lcxt5m\|lck\|lckiaps\|lckordi\|lckyuh\|lckyut\|lcsb) true;;
573	(*) printf "\n"
574	printf "\n +++ sorry: execution of batch jobs on remote host \"$host\""
575	printf "\n is not available"
576	locat=nqs; (( iec = 0 )); exit;;
577	esac
578	else
579	host=$localhost
580	fi
581
582
583	# EVALUATE ADDITIONAL CONDITIONS GIVEN BY OPTION -K
584	if [[ -n $additional_conditions ]]
585	then
586	cond1=`echo $additional_conditions \| cut -d" " -f1`
587	cond2=`echo $additional_conditions \| cut -d" " -s -f2`
588	dummy=`echo $additional_conditions \| cut -d" " -s -f3`
589	if [[ -n $dummy ]]
590	then
591	printf "\n +++ more than 2 additional conditions given for Option \"-K\""
592	locat=options; exit
593	fi
594	block=_$cond1
595	[[ -n $cond2 ]] && block=${block}_$cond2
596	fi
597
598
599	# EVALUATE MODEL COUPLING FEATURES (OPTION -Y) AND DETERMINE coupled_mode
600	if [[ $run_coupled_model = true ]]
601	then
602
603	numprocs_atmos=`echo $coupled_dist \| cut -d" " -s -f1`
604	numprocs_ocean=`echo $coupled_dist \| cut -d" " -s -f2`
605
606	if (( $numprocs_ocean + $numprocs_atmos != $numprocs ))
607	then
608
609	printf "\n +++ number of processors does not fit to specification by \"-Y\"."
610	printf "\n PEs (total) : $numprocs"
611	printf "\n PEs (atmosphere): $numprocs_atmos"
612	printf "\n PEs (ocean) : $numprocs_ocean"
613	locat=coupling; exit
614
615	fi
616
617
618	# GET coupled_mode FROM THE CONFIG FILE
619	line=""
620	grep "%cpp_options.-D__mpi2.$host" $config_file > tmp_mrun
621	while read line
622	do
623	if [[ "$line" != "" && $(echo $line \| cut -c1) != "#" && $(echo $line \| cut -d" " -s -f4) = $cond1 && $(echo $line \| cut -d" " -s -f4) = $cond2 ]]
624	then
625	coupled_mode="mpi2"
626	fi
627	done < tmp_mrun
628
629	fi
630
631
632	# CHECK, IF FILE-ARCHIVING HAS FAILED IN PREVIOUS JOB (OF A JOB-CHAIN)
633	if [[ -f ~/job_queue/ARCHIVE_ERROR_$fname ]]
634	then
635	if [[ $ignore_archive_error = false ]]
636	then
637	printf "\n +++ data archiving of previous run failed"
638	printf "\n see directory \~/job_queue on remote machine"
639	locat=archive; exit
640	else
641	printf "\n +++ warning: data archiving in a previous run failed"
642	printf "\n MRUN continues, trying to get backup copy"
643	fi
644	fi
645
646
647	# SAVE VALUES OF MRUN-OPTIONS SICHERN IN ORDER TO OVERWRITE
648	# THOSE VALUES GIVEN IN THE CONFIGURATION-FILE
649	mrun_memory=$memory
650	mrun_group_number=$group_number
651	mrun_cpumax=$cpumax
652	mrun_numprocs=$numprocs
653
654
655	# READ AND EVALUATE THE CONFIGURATION-FILE FROM WITHIN THIS SHELLSCRIPT
656	# (OPTION -S). THE DEFAULT IS USING THE ROUTINE interpret_config
657	if [[ "$read_from_config" = false ]]
658	then
659
660	[[ $silent = false ]] && printf "\n Reading the configuration file... "
661
662	# READ THE CONFIGURATION FILE LINE BY LINE
663	while read zeile
664	do
665	[[ $silent = false ]] && printf "."
666
667
668	# FIRST REPLACE ENVIRONMENT-VARIABLES BY THEIR RESPECTIVE VALUES
669	eval zeile=\"$zeile\"
670
671
672	# INTERPRET THE LINE
673	if [[ "$(echo $zeile)" = "" ]]
674	then
675	# EMPTY LINE, NO ACTION
676	continue
677
678	elif [[ "$(echo $zeile \| cut -c1)" = "#" ]]
679	then
680
681	# LINE IS A COMMENT LINE
682	true
683
684	elif [[ "$(echo $zeile \| cut -c1)" = "%" ]]
685	then
686
687	# LINE DEFINES AN ENVIRONMENT-VARIABLE
688	zeile=$(echo $zeile \| cut -c2-)
689	var=`echo $zeile \| cut -d" " -f1`
690	value=`echo $zeile \| cut -d" " -s -f2`
691	for_host=`echo $zeile \| cut -d" " -s -f3`
692	for_cond1=`echo $zeile \| cut -d" " -s -f4`
693	for_cond2=`echo $zeile \| cut -d" " -s -f5`
694
695	if [[ "$for_host" = "" \|\| ( "$for_host" = $host && "$for_cond1" = "$cond1" && "$for_cond2" = "$cond2" ) \|\| $(echo "$input_list$output_list"\|grep -c "$for_host") != 0 ]]
696	then
697
698	# REPLACE ":" BY " " IN COMPILER- CPP- OR LINKER-OPTIONS,
699	# "::" IS REPLACED BY ":".
700	value=`echo $value \| sed 's/::/%DUM%/g' \| sed 's/:/ /g' \| sed 's/%DUM%/:/g'`
701
702
703	# VALUE FROM THE CONFIGURATION-FILE IS ASSIGNED TO THE
704	# ENVIRONMENT-VARIABLE, BUT ONLY IF NO VALUE HAS BEEN ALREADY
705	# ASSIGNED WITHIN THIS SCRIPT (E.G. BY SCRIPT-OPTIONS).
706	# NON-ASSIGNED VARIABLES HAVE VALUE "" OR 0 (IN CASE OF INTEGER).
707	# HENCE THE GENERAL RULE IS: SCRIPT-OPTION OVERWRITES THE
708	# CONFIGURATION-FILE.
709	if [[ "$(eval echo \$$var)" = "" \|\| "$(eval echo \$$var)" = "0" ]]
710	then
711	eval $var=\$value
712
713	# TERMINAL OUTPUT OF ENVIRONMENT-VARIABLES, IF TRACEBACK IS SWITCHED on
714	if [[ $do_trace = true ]]
715	then
716	printf "\n*** ENVIRONMENT-VARIABLE $var = $value"
717	fi
718	fi
719
720	# IF AN ENVIRONMENT-VARIABLE DETERMINES THE HOST, THEN EVALUATE IT IMMEDIATELY:
721	# VALUE OF do-remote IS REQUIRED FOR THE FILE CONNECTIONS (COPY OF I/O FILES).
722	# IF EXECUTION IS SCHEDULED FOR A REMOTE-MASCHINE, A BATCH-JOB
723	# MUST HAVE TO BE STARTED
724	if [[ $var = host ]]
725	then
726	if [[ -n $host && "$host" != $localhost ]]
727	then
728	do_batch=true
729	do_remote=true
730	case $host in
731	(ibm\|ibmh\|ibmkisti\|ibmku\|ibms\|lccrayb\|lccrayh\|lccrayf\|lcflow\|lckyoto\|nech\|unics\|lcxe6\|lcxt5m\|lck\|lckiaps\|lckordi\|lckyuh\|lckyut\|lcsb) true;;
732	(*) printf "\n +++ sorry: execution of batch jobs on remote host \"$host\""
733	printf "\n is not available"
734	locat=nqs; exit;;
735	esac
736	else
737	host=$localhost
738	fi
739	fi
740
741	# USER-DEFINED ENVIRONMENT VARIABLES MUST BE EXPORTED,
742	# BECAUSE THEY MAY BE REQUIRED IN OTHER SCRIPTS CALLED
743	# BY MRUN (E.G. subjob)
744	export $var
745	fi
746
747
748
749
750	elif [[ "$(echo $zeile \| cut -c1-3)" = "EC:" ]]
751	then
752
753	# LINE DEFINES ERROR-COMMAND
754	(( iec = iec + 1 ))
755	zeile=$(echo $zeile \| cut -c4-)
756	err_command[$iec]="$zeile"
757
758	elif [[ "$(echo $zeile \| cut -c1-3)" = "IC:" ]]
759	then
760
761	# LINE DEFINES INPUT-COMMAND
762	(( iic = iic + 1 ))
763	zeile=$(echo $zeile \| cut -c4-)
764	in_command[$iic]="$zeile"
765
766	elif [[ "$(echo $zeile \| cut -c1-3)" = "OC:" ]]
767	then
768
769	# LINE DEFINES OUTPUT-COMMAND
770	(( ioc = ioc + 1 ))
771	zeile=$(echo $zeile \| cut -c4-)
772	out_command[$ioc]="$zeile"
773
774	else
775
776	# LINE DEFINES FILE CONNECTION. READ THE FILE ATTRIBUTES.
777	# s2a: in/out - field
778	# s2b: loc - field (optional)
779	# s2c: tr/ar - field (optional)
780	s1=`echo $zeile \| cut -d" " -f1`
781	s2=`echo $zeile \| cut -d" " -s -f2`
782	s2a=$(echo $s2 \| cut -d":" -f1)
783	if [[ $(echo $s2 \| grep -c ":") = 0 ]]
784	then
785	s2b=""
786	s2c=""
787	else
788	s2b=`echo $s2 \| cut -d":" -f2 \| sed 's/:/ /g'`
789	s2c=`echo $s2 \| cut -d":" -s -f3 \| sed 's/:/ /g'`
790	fi
791	s3=`echo $zeile \| cut -d" " -f3`
792	s4=`echo $zeile \| cut -d" " -s -f4`
793	s5=`echo $zeile \| cut -d" " -s -f5`
794	s6=`echo $zeile \| cut -d" " -s -f6`
795
796
797	# STORE FILE CONNECTION, IF ACTIVATED BY ACTIVATION-STRING FROM
798	# INPUT- OR OUTPUT-LIST.
799	# VARIABLE S3 MAY CONTAIN A LIST OF ACTIVATION STRINGS (FIELD-SEPERATOR ":").
800	# IF EXECUTION IS SCHEDULED FOR A REMOTE-MACHINE AND THE FILE IS ONLY
801	# LOCALLY REQUIRED ON THAT MACHINE (I.E. s2b = loc), THE FILE CONNECTION
802	# IS NOT CHECKED AND STORED.
803	IFSALT="$IFS"; IFS="$IFS:"
804	if [[ "$s2a" = in && ! ( $do_remote = true && ( "$s2b" = loc \|\| "$s2b" = locopt ) ) ]]
805	then
806	found=false
807	for actual in $input_list
808	do
809	for formal in $s3
810	do
811	[[ $actual = $formal \|\| "$formal" = "-" ]] && found=true
812	done
813	done
814	if [[ $found = true ]]
815	then
816	(( iin = iin + 1 ))
817	localin[$iin]=$s1; transin[$iin]=$s2b; actionin[$iin]=$s2c;
818	typein[$iin]=$s3; pathin[$iin]=$s4; endin[$iin]=$s5;
819	extin[$iin]=$s6
820	fi
821	elif [[ "$s2a" = out && ! ( $do_remote = true && "$s2b" = loc ) ]]
822	then
823	found=false
824	for actual in $output_list
825	do
826	for formal in $s3
827	do
828	[[ $actual = $formal \|\| "$formal" = "-" ]] && found=true
829	done
830	done
831	if [[ $found = true ]]
832	then
833	(( iout = iout + 1 ))
834	localout[$iout]=$s1; actionout[$iout]=$s2c; typeout[$iout]=$s3;
835	pathout[$iout]=$s4; endout[$iout]=$s5; extout[$iout]=$s6
836	fi
837	elif [[ "$s2a" != in && "$s2a" != out ]]
838	then
839	printf "\n +++ I/O-attribute in configuration file $config_file has the invalid"
840	printf "\n value \"$s2\". Only \"in\" and \"out\" are allowed!"
841	locat=connect; exit
842	fi
843	IFS="$IFSALT"
844	fi
845	done < $config_file
846
847	else
848
849
850	# EVALUATE THE CONFIGURATION-FILE BY FORTRAN-PROGRAM
851	[[ $silent = false ]] && printf "..."
852	interpreted_config_file=.icf.$RANDOM
853
854
855
856	# PROVIDE VALUES OF ENVIRONMENT-VARIABLE FOR interpret_config VIA NAMELIST-FILE
857	cat > .mrun_environment << %%END%%
858	&mrun_environment cond1 = '$cond1', cond2 = '$cond2',
859	config_file = '$config_file', do_remote = '$do_remote',
860	do_trace = '$do_trace', host = '$host',
861	input_list = '$input_list', icf = '$interpreted_config_file',
862	localhost = '$localhost', output_list = '$output_list' /
863
864	%%END%%
865
866	if [[ "$host" != $localhost ]]
867	then
868
869	# REMOTE JOB FROM LOCAL HOST: JUST TAKE THE FIRST EXECUTABLE FOUND
870	interpret_config_executable=`ls -1 ${PALM_BIN}/interpret_config*.x 2>/dev/null`
871	if [[ $? != 0 ]]
872	then
873	printf "\n\n +++ no interpret_config found"
874	printf "\n run \"mbuild -u -h ...\" to generate utilities for this host"
875	locat=interpret_config; exit
876	fi
877	interpret_config_executable=`echo $interpret_config_executable \| cut -d" " -f1`
878	$interpret_config_executable
879
880	else
881
882	# CHECK, IF THERE IS AN EXECUTABLE FOR THE BLOCK
883	if [[ ! -f ${PALM_BIN}/interpret_config${block}.x ]]
884	then
885	printf "\n\n +++ no interpret_config found for given block \"$cond1 $cond2\""
886	printf "\n run \"mbuild -u -h ...\" to generate utilities for this block"
887	locat=interpret_config; exit
888	else
889	interpret_config${block}.x
890	fi
891
892	fi
893	rm .mrun_environment
894
895
896	# EXECUTE SHELL-COMMANDS GENERATED BY interpret_config WITHIN THIS SHELL
897	chmod u+x $interpreted_config_file
898	export PATH=$PATH:.
899	. $interpreted_config_file
900	rm $interpreted_config_file
901
902	fi
903
904
905	# VALUES OF MRUN-OPTIONS OVERWRITE THOSE FROM THE CONFIGURATION-FILE
906	[[ $mrun_memory != 0 ]] && memory=$mrun_memory
907	[[ "$mrun_group_number" != "none" ]] && group_number=$mrun_group_number
908	[[ $mrun_cpumax != 0 ]] && cpumax=$mrun_cpumax
909	[[ "$mrun_numprocs" != "" ]] && numprocs=$mrun_numprocs
910	[[ "$max_par_io_str" != "" ]] && maximum_parallel_io_streams=$max_par_io_str
911	[[ "$mrun_tasks_per_node" != "" ]] && tasks_per_node=$mrun_tasks_per_node
912
913
914	# ON THE LOCAL MACHINE, DETERMINE (FROM THE CONFIGURATION-FILE) THE PATH
915	# FOR SOURCE-FILES TO BE COMPILED.
916	# IN A BATCH-JOB, SOURCES (ROUTINES TO BE COMPILED) ARE COMPLETE ALREADY.
917	# BEFORE MRUN IS CALLED IN THE JOB, SOURCES_COMPLETE = true IS ASSIGNED (SEE FURTHER BELOW).
918	if [[ "$SOURCES_COMPLETE" = "" ]]
919	then
920
921	# FIRST CHECK, IF A GLOBAL SOURCE PATH (TO BE USED FOR ALL HOSTS)
922	# HAS BEEN DEFINED
923	source_path=""
924	line=""
925	grep "%source_path" $config_file > tmp_mrun
926	while read line
927	do
928	if [[ "$line" != "" && $(echo $line \| cut -c1) != "#" ]]
929	then
930	if [[ "$(echo $line \| cut -d" " -f3)" = "" ]]
931	then
932	global_source_path=`echo $line \| cut -d" " -f2`
933	fi
934	fi
935	done < tmp_mrun
936
937	line=""
938	found=false
939	grep " $localhost" $config_file \| grep "%source_path" > tmp_mrun
940	while read line
941	do
942	if [[ "$line" != "" && $(echo $line \| cut -c1) != "#" ]]
943	then
944	if [[ $found = true ]]
945	then
946	printf "\n\n +++ more than one source path found in configuration file"
947	printf "\n for local host \"$localhost\" "
948	locat=source_path; exit
949	fi
950	source_path=`echo $line \| cut -d" " -f2`
951	found=true
952	fi
953	done < tmp_mrun
954	rm tmp_mrun
955
956	if [[ "$source_path" = "" ]]
957	then
958	if [[ "$global_source_path" != "" ]]
959	then
960	source_path=$global_source_path
961	else
962	printf "\n\n +++ no source path found in configuration file"
963	printf "\n for local host \"$localhost\" "
964	locat=source_path; exit
965	fi
966	fi
967	eval source_path=$source_path
968
969	if [[ ! -d $source_path ]]
970	then
971	printf "\n\n +++ source path \"$source_path\" on local host"
972	printf "\n \"$localhost\" does not exist"
973	locat=source_path; exit
974	fi
975
976	fi
977
978
979	# GET THE GLOBAL REVISION-NUMBER OF THE SVN-REPOSITORY
980	# (HANDED OVER TO RESTART-RUNS USING OPTION -G)
981	if [[ "$global_revision" = "" && $host != "ibmkisti" ]]
982	then
983	global_revision=`svnversion $source_path 2>/dev/null`
984	global_revision="Rev: $global_revision"
985	fi
986
987
988	# ??? das darf doch eigentlich garnicht passieren, weil optionen config-datei uebersteuern ???
989	# CHECK AGAIN, IF EXECUTION SHALL BE DONE ON A REMOTE-HOST (BATCH-JOB).
990	# (HOST MAY HAVE CHANGED DUE TO SETTINGS IN THE CONFIGURATION-FILE)
991	if [[ -n $host && "$host" != $localhost ]]
992	then
993	do_batch=true
994	do_remote=true
995	case $host in
996	(ibm\|ibmh\|ibmkisti\|ibmku\|ibms\|lccrayb\|lccrayh\|lccrayf\|lcflow\|lckyoto\|nech\|unics\|lcxe6\|lcxt5m\|lck\|lckiaps\|lckordi\|lckyuh\|lckyut\|lcsb) true;;
997	(*) printf "\n"
998	printf "\n +++ sorry: execution of batch jobs on remote host \"$host\""
999	printf "\n is not available"
1000	locat=nqs; (( iec = 0 )); exit;;
1001	esac
1002	else
1003	host=$localhost
1004	fi
1005
1006
1007	# IN CASE OF PARALLEL EXECUTION, CHECK SOME SPECIFICATIONS CONCERNING PROCESSOR NUMBERS
1008	if [[ "$cond1" = parallel \|\| "$cond2" = parallel ]]
1009	then
1010
1011	# CHECK, IF NUMBER OF CORES TO BE USED HAS BEEN GIVEN
1012	if [[ ! -n $numprocs ]]
1013	then
1014	printf "\n"
1015	printf "\n +++ option \"-K parallel\" requires additional specification"
1016	printf "\n of the number of processors to be used by"
1017	printf "\n mrun-option \"-X\" or by environment-variable"
1018	printf "\n \"numprocs\" in the configuration file"
1019	locat=numprocs; (( iec = 0 )); exit
1020	fi
1021
1022	# CHECK, IF THE NUMBER OF CORES PER NODE HAS BEEN GIVEN UND IF IT IS AN
1023	# INTEGRAL DIVISOR OF THE TOTAL NUMBER OF CORES GIVEN BY OPTION -X
1024	if [[ "$tasks_per_node" = "" && $host != lcflow && $host != lcxt5m ]]
1025	then
1026	printf "\n"
1027	printf "\n +++ option \"-T\" (tasks per node) is missing"
1028	printf "\n set -T option or define tasks_per_node in the config file"
1029	locat=tasks_per_node; (( iec = 0 )); exit
1030	fi
1031
1032	if (( numprocs < tasks_per_node ))
1033	then
1034	printf "\n"
1035	printf "\n +++ tasks per node (-T) cannot exceed total number of processors (-X)"
1036	printf "\n given values: -T $tasks_per_node -X $numprocs"
1037	locat=tasks_per_node; (( iec = 0 )); exit
1038	fi
1039
1040	if [[ $host != lcflow && $host != lcxt5m ]]
1041	then
1042	(( nodes = numprocs / ( tasks_per_node * threads_per_task ) ))
1043	(( ival = $tasks_per_node ))
1044	(( pes = numprocs ))
1045	(( ii = pes / ival ))
1046	(( remaining_pes = pes - ii * ival ))
1047	if (( remaining_pes > 0 ))
1048	then
1049	printf "\n"
1050	printf "\n +++ WARNING: tasks per node (option \"-T\") is not an integral"
1051	printf "\n divisor of the total number of processors (option \"-X\")"
1052	printf "\n values of this mrun-call: \"-T $tasks_per_node\" \"-X $numprocs\""
1053	printf "\n One of the nodes is filled with $remaining_pes instead of $tasks_per_node tasks"
1054	(( nodes = nodes + 1 ))
1055	fi
1056	fi
1057
1058
1059	# SETTINGS FOR SUBJOB-COMMAND
1060	OOPT="-O $threads_per_task"
1061
1062	# SET THE TOTAL NUMBER OF NODES, REQUIRED FOR THE SUBJOB-COMMAND (SEE FURTHER BELOW)
1063	if [[ "$tasks_per_node" != "" ]]
1064	then
1065	TOPT="-T $tasks_per_node"
1066	fi
1067
1068	# CHECK node_usage FOR ALLOWED VALUES AND SET DEFAULT VALUE, IF NECESSARY
1069	if [[ $node_usage = default ]]
1070	then
1071	if [[ $host = ibms ]]
1072	then
1073	node_usage=shared
1074	else
1075	node_usage=not_shared
1076	fi
1077	fi
1078
1079	if [[ $node_usage != shared && $node_usage != not_shared && $node_usage != singlejob && "$(echo $node_usage \| cut -c1-3)" != "sla" ]]
1080	then
1081	printf "\n"
1082	printf "\n +++ node usage (option \"-n\") is only allowed to be set"
1083	printf "\n \"shared\" or \"not_shared\""
1084	locat=tasks_per_node; (( iec = 0 )); exit
1085	fi
1086
1087	fi
1088
1089	# CHECK IF HOSTFILE EXISTS
1090	if [[ -n $hostfile ]]
1091	then
1092	if [[ ! -f $hostfile ]]
1093	then
1094	printf "\n"
1095	printf "\n +++ hostfile \"$hostfile\" does not exist"
1096	locat=hostfile; exit
1097	fi
1098	fi
1099
1100
1101	# SET DEFAULT VALUE FOR THE MAXIMUM NUMBER OF PARALLEL IO STREAMS
1102	if [[ "$maximum_parallel_io_streams" = "" ]]
1103	then
1104	maximum_parallel_io_streams=$numprocs
1105	fi
1106
1107
1108	# SET PORT NUMBER OPTION FOR CALLS OF SSH/SCP, subjob AND batch_scp SCRIPTS
1109	if [[ "$scp_port" != "" ]]
1110	then
1111	PORTOPT="-P $scp_port"
1112	SSH_PORTOPT="-p $scp_port"
1113	fi
1114
1115
1116	# SET DEFAULT-QUEUE, IF NOT GIVEN
1117	if [[ $queue = none ]]
1118	then
1119	case $host in
1120	(ibmh) queue=cluster;;
1121	(ibmkisti) queue=class.32plus;;
1122	(lckiaps) queue=normal;;
1123	(lckyoto) queue=ph;;
1124	(lckyuh) queue=fx-single;;
1125	(lckyut) queue=cx-single;;
1126	(lctit) queue=S;;
1127	(nech) queue=none;;
1128	(unics) queue=unics;;
1129	esac
1130	fi
1131
1132
1133	# GENERATE FULL FILENAMES OF INPUT-FILES, INCLUDING THEIR PATH
1134	# CHECK, IF INPUT-FILES EXIST, AND DETERMINE HIGHEST CYCLE NUMBER (IF CYCLES EXIST)
1135	(( i = 0 ))
1136	while (( i < iin ))
1137	do
1138	(( i = i + 1 ))
1139	(( maxcycle = 0 ))
1140
1141	# GENERATE PATH AND FULL FILE NAME (then-BRANCH: FIXED FULL NAME IS GIVEN, I.E. THE
1142	# FILE IDENTIFIER IS NOT PART OF THE FILENAME))
1143	if [[ "${actionin[$i]}" = di ]]
1144	then
1145	remotepathin[$i]=${pathin[$i]}/${endin[$i]} # EVALUATE REMOTE-PATH ON THE REMOTE
1146	# HOST ONLY
1147	eval filename=${pathin[$i]}/${endin[$i]}
1148	else
1149	remotepathin[$i]=${pathin[$i]}/${afname}${endin[$i]} # EVALUATE REMOTE-PATH ON THE REMOTE
1150	# HOST ONLY
1151	eval filename=${pathin[$i]}/${afname}${endin[$i]}
1152	fi
1153	eval pathname=${pathin[$i]}
1154
1155	# SAVE INPUT FILE NAME FOR PARAMETER FILE CHECK
1156	if [[ ("${transin[$i]}" = job) && (-f $filename) ]]
1157	then
1158	filename_input=$filename
1159	fi
1160
1161	# CHECK IF FILE EXISTS
1162	if [[ $(ls $filename* 2>&1 \| grep -c "not found") = 1 \|\| \
1163	$(ls $filename* 2>&1 \| grep -c "nicht gefunden") = 1 \|\| \
1164	$(ls $filename* 2>&1 \| grep -c "No such file") = 1 \|\| \
1165	$(ls $filename* 2>&1 \| grep -c "does not exist") = 1 ]]
1166	then
1167
1168	# FILES WITH ATTRIBUTE locopt ARE OPTIONAL. NO ABORT, IF THEY DO NOT EXIST.
1169	if [[ "${transin[$i]}" != "locopt" ]]
1170	then
1171	printf "\n\n +++ INPUT-file: "
1172	if [[ "${extin[$i]}" = "" ]]
1173	then
1174	printf "\n $filename"
1175	else
1176	printf "\n $filename.${extin[$i]}"
1177	fi
1178	printf "\n does not exist\n"
1179	locat=input; exit
1180	else
1181	transin[$i]="unavailable"
1182	fi
1183	else
1184
1185	# DETERMINE THE FILE'S CYCLE NUMBER
1186	ls -1 -d $filename > filelist 2>/dev/null
1187	ls -1 -d $filename.* >> filelist 2>/dev/null
1188	while read zeile
1189	do
1190	cycle=$(echo $zeile \| cut -f2 -d".")
1191	if [[ "$cycle" = "$zeile" ]]
1192	then
1193	(( icycle = 0 ))
1194	elif [[ "$cycle" = "${extin[$i]}" ]]
1195	then
1196	(( icycle = 0 ))
1197	else
1198	(( icycle = $cycle ))
1199	fi
1200	if (( icycle > maxcycle ))
1201	then
1202	(( maxcycle = icycle ))
1203	file_to_be_used=$zeile
1204	fi
1205	done <filelist
1206	rm filelist
1207
1208	# APPEND CYCLE NUMBER TO FILENAME
1209	if (( maxcycle > 0 ))
1210	then
1211	if [[ "${extin[$i]}" != " " && "${extin[$i]}" != "" ]]
1212	then
1213	filename=${filename}.$maxcycle.${extin[$i]}
1214	else
1215	filename=${filename}.$maxcycle
1216	fi
1217	else
1218	if [[ "${extin[$i]}" != " " && "${extin[$i]}" != "" ]]
1219	then
1220	filename=${filename}.${extin[$i]}
1221	fi
1222	fi
1223
1224	# STORE FILENAME WITHOUT PATH BUT WITH CYCLE NUMBER,
1225	# BECAUSE IT MIGHT BE REQUIRED LATER TO RESTORE THE FILE FROM AN ARCHIVE-SYSTEM
1226	absnamein[$i]=$filename
1227	if (( maxcycle > 0 ))
1228	then
1229	if [[ "${actionin[$i]}" = di ]]
1230	then
1231	frelin[$i]=${endin[$i]}.$maxcycle
1232	else
1233	frelin[$i]=${afname}${endin[$i]}.$maxcycle
1234	fi
1235	else
1236	if [[ "${actionin[$i]}" = di ]]
1237	then
1238	frelin[$i]=${endin[$i]}
1239	else
1240	frelin[$i]=${afname}${endin[$i]}
1241	fi
1242	fi
1243
1244	fi
1245	done
1246
1247	# GENERATE FULL FILENAMES OF OUTPUT-FILES (WITHOUT $ OR ~),
1248	# CHECK, IF OUTPUT-FILES EXIST, AND DETERMINE HIGHEST CYCLE NUMBER (IF CYCLES EXIST),
1249	# OR, IN CASE THAT FILE DOES NOT EXIST, CHECK, IF IT CAN BE CREATED
1250	# THESE ACTIONS ARE NOT CARRIED OUT, IF FILES SHALL BE TRASFERRED FROM THE REMOTE TO
1251	# THE LOCAL HOST (BECAUSE THEIR IS NO DIRECT ACCESS TO THE LOCAL DIRECTORIES FROM THE
1252	# REMOTE HOST)
1253	(( i = 0 ))
1254	while (( i < iout ))
1255	do
1256	(( i = i + 1 ))
1257	if [[ ! ( $fromhost != $localhost && ( "${actionout[$i]}" = tr \|\| "${actionout[$i]}" = tra \|\| "${actionout[$i]}" = trpe ) ) ]]
1258	then
1259	if [[ "${actionout[$i]}" = tr ]]
1260	then
1261	actionout[$i]=""
1262	elif [[ "${actionout[$i]}" = trpe ]]
1263	then
1264	actionout[$i]=pe
1265	elif [[ "${actionout[$i]}" = tra ]]
1266	then
1267	actionout[$i]=a
1268	fi
1269	(( maxcycle = 0 ))
1270	eval filename=${pathout[$i]}/${fname}${endout[$i]}
1271	eval catalogname=${pathout[$i]}
1272	if [[ $(ls $filename* 2>&1 \| grep -c "not found") = 1 \|\| \
1273	$(ls $filename* 2>&1 \| grep -c "nicht gefunden") = 1 \|\| \
1274	$(ls $filename* 2>&1 \| grep -c "No such file") = 1 \|\| \
1275	$(ls $filename* 2>&1 \| grep -c "does not exist") = 1 ]]
1276	then
1277
1278	# IF OUTPUT-FILE DOES NOT EXIST CHECK, IF IT CAN BE CREATED
1279	if cat /dev/null > $filename
1280	then
1281	rm $filename
1282	else
1283
1284	# CHECK, IF THE DIRECTORY WHERE FILE SHALL BE COPIED TO EXISTS
1285	# IF IT DOES NOT EXIST, TRY TO CREATE IT
1286	if [[ ! -d $catalogname ]]
1287	then
1288	if mkdir -p $catalogname
1289	then
1290	printf "\n\n *** directory:"
1291	printf "\n $catalogname"
1292	printf "\n was created\n"
1293	else
1294	printf "\n\n +++ OUTPUT-file:"
1295	printf "\n $filename"
1296	printf "\n cannot be created, because directory does not exist"
1297	printf "\n and cannot be created either"
1298	printf "\n"
1299	locat=output ; exit
1300	fi 2>/dev/null
1301	else
1302	printf "\n\n +++ OUTPUT-file:"
1303	printf "\n $filename"
1304	printf "\n cannot be created, although directory exists"
1305	printf "\n"
1306	locat=output ; exit
1307	fi
1308	fi 2>/dev/null
1309	else
1310
1311	# DETERMINE THE CYCLE NUMBER
1312	ls -1 -d $filename > filelist 2>/dev/null
1313	ls -1 -d $filename.* >> filelist 2>/dev/null
1314	while read zeile
1315	do
1316	cycle=$(echo $zeile \| cut -f2 -d".")
1317	if [[ "$cycle" = "$zeile" \|\| "$cycle" = ${extout[$i]} ]]
1318	then
1319	(( icycle = 1 ))
1320	else
1321	(( icycle = $cycle + 1 ))
1322	fi
1323	if (( icycle > maxcycle ))
1324	then
1325	(( maxcycle = icycle ))
1326	fi
1327	done <filelist
1328	rm filelist
1329	fi
1330
1331	# APPEND CYCLE NUMBER TO FILENAME AND CHECK, IF FILE CAN BE CREATED
1332	# IN CASE OF FILE-APPEND, FILE MUST BE APPENDED TO THE ONE WITH HIGHEST CYCLE NUMBER
1333	if [[ "${actionout[$i]}" != a ]]
1334	then
1335	if (( maxcycle > 0 ))
1336	then
1337	filename=${filename}.$maxcycle
1338	if cat /dev/null > $filename
1339	then
1340	rm $filename
1341	else
1342	printf "\n +++ OUTPUT-file:"
1343	printf "\n $filename"
1344	printf "\n cannot be created"
1345	locat=output ; exit
1346	fi
1347	fi
1348	else
1349	(( maxcycle = maxcycle - 1 ))
1350	if (( maxcycle > 0 ))
1351	then
1352	filename=${filename}.$maxcycle
1353	fi
1354	fi
1355
1356	# STORE FILENAME WITHOUT PATH BUT WITH CYCLE NUMBER,
1357	# BECAUSE IT MIGHT BE REQUIRED LATER TO STORE THE FILE ON AN ARCHIVE-SYSTEM
1358	# OR TO PUT THIS FILENAME ON FILE OUTPUT_FILE_CONNECTIONS
1359	pathout[$i]=$filename
1360	if (( maxcycle > 0 ))
1361	then
1362	frelout[$i]=${fname}${endout[$i]}.$maxcycle
1363	else
1364	frelout[$i]=${fname}${endout[$i]}
1365	fi
1366
1367	fi
1368	done
1369
1370
1371	# THE DVR-PACKAGE REQUIRES ITS OWN LIBRARY
1372	if [[ $(echo $package_list \| grep -c dvrp_graphics) != 0 ]]
1373	then
1374	if [[ "$dvr_inc" = "" ]]
1375	then
1376	printf "\n\n +++ no value for \"dvr_inc\" given in configuration file"
1377	printf "\n This is required for the dvrp_graphics package.\n"
1378	locat=dvr; exit
1379	fi
1380	if [[ "$dvr_lib" = "" ]]
1381	then
1382	printf "\n\n +++ no value for \"dvr_lib\" given in configuration file"
1383	printf "\n This is required for the dvrp_graphics package.\n"
1384	locat=dvr; exit
1385	fi
1386	fi
1387
1388
1389	# CHECK, WETHER A MAIN PROGRAM OR AN EXECUTABLE HAVE BEEN DECLARED.
1390	# EXECUTABLES DO NOT NEED TO BE COMPILED.
1391	if [[ "$mainprog" = "" && "$executable" = "" ]]
1392	then
1393	printf "\n +++ neither main program nor executable defined"
1394	locat=source; exit
1395	elif [[ "$mainprog" != "" && "$executable" != "" ]]
1396	then
1397	printf "\n +++ main program as well as executable defined"
1398	locat=source; exit
1399	elif [[ "$mainprog" = "" && "$executable" != "" ]]
1400	then
1401	do_compile=false
1402	fi
1403
1404
1405	# CREATE SOURCE-DIRECTORY TO COLLECT ROUTINES TO BE COMPILED.
1406	# LATER THE MRUN-SCRIPT AND CONFIGURATION-FILE WILL ALSO BE COPIED TO THIS DIRECTORY.
1407	if [[ $restart_run != true && "$SOURCES_COMPLETE" = "" ]]
1408	then
1409	rm -rf SOURCES_FOR_RUN_$fname
1410	mkdir SOURCES_FOR_RUN_$fname
1411	fi
1412
1413
1414	# COLLECT ALL ROUTINES TO BE COMPILED
1415	# THIS IS NOT REQUIRED WITHIN BATCH-JOBS, BECAUSE ROUTINES HAVE ALREADY BEEN COLLECTED
1416	# BY THE MRUN-CALL WHICH CREATED THE BATCH-JOB.
1417	if [[ $do_compile = true && "$SOURCES_COMPLETE" = "" ]]
1418	then
1419
1420	[[ "$source_list" = LM ]] && source_list=LOCALLY_MODIFIED
1421
1422	if [[ "$source_list" = LOCALLY_MODIFIED ]]
1423	then
1424
1425	# DETERMINE MODIFIED FILES OF THE SVN WORKING COPY
1426	source_list=""
1427	cd $source_path
1428
1429
1430	# CHECK, IF DIRECTORY IS UNDER SVN CONTROL
1431	if [[ ! -d .svn ]]
1432	then
1433	printf "\n\n +++ source directory"
1434	printf "\n \"$source_path\" "
1435	printf "\n is not under control of \"subversion\"."
1436	printf "\n Please do not use mrun-option \"-s LOCALLY_MODIFIED\"\n"
1437	fi
1438
1439
1440	# LIST ALL MODIFIED SOURCE CODE FILES
1441	Filenames=""
1442	svn status > tmp_mrun
1443	while read line
1444	do
1445	firstc=`echo $line \| cut -c1`
1446	if [[ $firstc = M \|\| $firstc = "?" ]]
1447	then
1448	Name=`echo "$line" \| cut -c8-`
1449	extension=`echo $Name \| cut -d. -f2`
1450	if [[ "$extension" = f90 \|\| "$extension" = F90 \|\| "$extension" = f \|\| "$extension" = F \|\| "$extension" = c ]]
1451	then
1452	Filenames="$Filenames "$Name
1453	fi
1454	fi
1455	done < tmp_mrun
1456
1457
1458	# COPY FILES TO SOURCES_FOR_RUN_...
1459	for dateiname in $Filenames
1460	do
1461	cp $dateiname $working_directory/SOURCES_FOR_RUN_$fname
1462	source_list=$source_list"$dateiname "
1463	done
1464
1465	cd - > /dev/null
1466
1467
1468	# COPY FILES GIVEN BY OPTION -s TO DIRECTORY SOURCES_FOR_RUN_...
1469	# AUTOMATIC RESTART RUNS JUST ACCESS THE DIRECTORY CREATED BY THE INITIAL RUN
1470	elif [[ "$source_list" != "" && $restart_run != true ]]
1471	then
1472
1473	cd $source_path
1474
1475	for filename in $source_list
1476	do
1477
1478	# SOURCE CODE FILE IS NOT ALLOWED TO INCLUDE PATH
1479	if [[ $(echo $filename \| grep -c "/") != 0 ]]
1480	then
1481	printf "\n +++ source code file: $filename"
1482	printf "\n must not contain (\"/\") "
1483	locat=source; exit
1484	fi
1485
1486	if [[ ! -f $filename ]]
1487	then
1488	printf "\n +++ source code file: $filename"
1489	printf "\n does not exist"
1490	locat=source; exit
1491	else
1492	cp $filename $working_directory/SOURCES_FOR_RUN_$fname
1493	fi
1494
1495	done
1496
1497	cd - > /dev/null
1498
1499	fi
1500
1501
1502	# CHECK, IF A MAIN PROGRAM EXISTS AND IF IT IS PART OF THE LIST OF FILES
1503	# TO BE COMPILED. IF NOT, ADD IT TO THE LIST.
1504	if [[ $restart_run != true ]]
1505	then
1506
1507	if [[ ! -f "$source_path/$mainprog" ]]
1508	then
1509	printf "\n\n +++ main program: $mainprog"
1510	printf "\n does not exist in source directory"
1511	printf "\n \"$source_path\"\n"
1512	locat=source; exit
1513	else
1514	if [[ $(echo $source_list \| grep -c $mainprog) = 0 ]]
1515	then
1516	cp $source_path/$mainprog SOURCES_FOR_RUN_$fname
1517	source_list=${mainprog}" $source_list"
1518	fi
1519	fi
1520	fi
1521
1522
1523	# CHECK, IF MAKEFILE EXISTS AND COPY IT TO THE SOURCES_FOR_RUN... DIRECTORY
1524	# NOT REQUIRED FOR RESTART RUNS, SOURCES_FOR_RUN... HAS BEEN CREATED BY THE INITIAL RUN
1525	if [[ "$restart_run" != true ]]
1526	then
1527	[[ "$makefile" = "" ]] && makefile=$source_path/Makefile
1528	if [[ ! -f $makefile ]]
1529	then
1530	printf "\n +++ file \"$makefile\" does not exist"
1531	locat=make; exit
1532	else
1533	cp $makefile SOURCES_FOR_RUN_$fname/Makefile
1534	fi
1535	fi
1536
1537
1538	# COPY FILES FROM OPTIONAL SOURCE PATH GIVEN IN THE CONFIGURATION FILE
1539	if [[ $restart_run != true && "$add_source_path" != "" ]]
1540	then
1541
1542	# DOES THE DIRECTORY EXIST?
1543	if [[ ! -d $add_source_path ]]
1544	then
1545	printf "\n\n +++ WARNING: additional source code directory"
1546	printf "\n \"$add_source_path\" "
1547	printf "\n does not exist or is not a directory."
1548	printf "\n No source code will be used from this directory!\n"
1549	add_source_path=""
1550	if [[ $silent == false ]]
1551	then
1552	sleep 3
1553	fi
1554	else
1555
1556	cd $add_source_path
1557	found=false
1558
1559	Names=$(ls -1 *.f90 2>&1)
1560	[[ $(echo $Names \| grep -c '*.f90') = 0 ]] && AddFilenames="$Names"
1561	Names=$(ls -1 *.F90 2>&1)
1562	[[ $(echo $Names \| grep -c '*.F90') = 0 ]] && AddFilenames="$AddFilenames $Names"
1563	Names=$(ls -1 *.F 2>&1)
1564	[[ $(echo $Names \| grep -c '*.F') = 0 ]] && AddFilenames="$AddFilenames $Names"
1565	Names=$(ls -1 *.f 2>&1)
1566	[[ $(echo $Names \| grep -c '*.f') = 0 ]] && AddFilenames="$AddFilenames $Names"
1567	Names=$(ls -1 *.c 2>&1)
1568	[[ $(echo $Names \| grep -c '*.c') = 0 ]] && AddFilenames="$AddFilenames $Names"
1569
1570	cd - > /dev/null
1571	cd SOURCES_FOR_RUN_$fname
1572
1573	# COPY MAKEFILE IF EXISTING
1574	if [[ -f $add_source_path/Makefile ]]
1575	then
1576	printf "\n\n *** user Makefile from directory"
1577	printf "\n \"$add_source_path\" is used \n"
1578	if [[ $silent == false ]]
1579	then
1580	sleep 1
1581	fi
1582	cp $add_source_path/Makefile .
1583	fi
1584
1585	for dateiname in $AddFilenames
1586	do
1587	if [[ -f $dateiname ]]
1588	then
1589	printf "\n +++ source code file \"$dateiname\" found in additional"
1590	printf "\n source code directory \"$add_source_path\" "
1591	printf "\n but was also given with option \"-s\" which means that it should be taken"
1592	printf "\n from directory \"$source_path\"."
1593	locat=source; exit
1594	fi
1595
1596	cp $add_source_path/$dateiname .
1597	source_list="$source_list $dateiname"
1598
1599	# CHECK IF FILE IS CONTAINED IN MAKEFILE
1600	if [[ $(grep -c $dateiname Makefile) = 0 ]]
1601	then
1602	printf "\n\n +++ user file \"$dateiname\" "
1603	printf "\n is not listed in Makefile \n"
1604	locat=source; exit
1605	else
1606
1607	if [[ $found = false ]]
1608	then
1609	found=true
1610	printf "\n\n *** following user file(s) added to the"
1611	printf " files to be translated:\n "
1612	fi
1613	printf "$dateiname "
1614	if [[ $silent == false ]]
1615	then
1616	sleep 0.5
1617	fi
1618
1619	fi
1620	done
1621	[[ $found = true ]] && printf "\n"
1622	cd - > /dev/null
1623	fi
1624	fi
1625
1626
1627	# ADD ALL ROUTINES BELONGING TO SOFTWARE PACKAGES (GIVEN BY OPTION -p)
1628	# TO THE LIST OF FILES TO BE COMPILED
1629	if [[ $restart_run != true && -n $package_list ]]
1630	then
1631
1632	cd $source_path
1633
1634	for package in $package_list
1635	do
1636
1637	[[ $package = "dvrp_graphics+1PE" ]] && package=dvrp_graphics
1638
1639	# DETERMINE FILES BELONGING TO THE PACKAGE
1640	# ERROR MESSAGE ARE REDIRECTED TO /dev/null, BECAUSE WILDCARD (*) ALSO GIVES
1641	# THE NAME OF THE DIRECTORY
1642	package_source_list=`grep "defined( __$package " * 2>/dev/null \| cut -f1 -d:`
1643
1644
1645	# ADD THESE FILES TO THE LIST OF FILES TO BE COMPILED,
1646	# IF THEY ARE NOT ALREADY PART OF THE LIST
1647	for source_list_name in $package_source_list
1648	do
1649	if [[ $(echo $source_list \| grep -c $source_list_name) = 0 ]]
1650	then
1651
1652	# ONLY TAKE FILES WITH VALID SUFFIX
1653	ending=`echo $source_list_name \| cut -f2 -d.`
1654	if [[ "$ending" = f90 \|\| "$ending" = F90 \|\| "$ending" = f \|\| "$ending" = F \|\| "$ending" = c ]]
1655	then
1656	cp $source_list_name $working_directory/SOURCES_FOR_RUN_$fname
1657	source_list="$source_list $source_list_name"
1658	fi
1659	fi
1660	done
1661	done
1662
1663	cd - > /dev/null
1664	fi
1665
1666	fi # do_compile=true
1667
1668
1669	# IF SOURCE CODE IS TO BE COMPILED, DO SOME MORE CHECKS
1670	# AND SET PRE-PROCESSOR DIRECTIVES
1671	if [[ $do_compile = true \|\| $create_executable_for_batch = true ]]
1672	then
1673
1674	# SET PREPROCESSOR-DIRECTIVES TO SELECT OPERATING SYSTEM SPECIFIC CODE
1675	if [[ $(echo $localhost \| cut -c1-3) = ibm ]]
1676	then
1677	cpp_options="${cpp_options},-D__ibm=__ibm"
1678	elif [[ $(echo $localhost \| cut -c1-3) = nec ]]
1679	then
1680	cpp_options="$cpp_options -D__nec"
1681	elif [[ $(echo $localhost \| cut -c1-2) = lc ]]
1682	then
1683	cpp_options="$cpp_options -D__lc"
1684	else
1685	cpp_options="$cpp_options -D__$localhost"
1686	fi
1687
1688	# SET DIRECTIVES GIVEN BY OPTION -K (E.G. parallel)
1689	if [[ $(echo $localhost \| cut -c1-3) = ibm ]]
1690	then
1691	[[ -n $cond1 ]] && cpp_options="${cpp_options},-D__$cond1=__$cond1"
1692	[[ -n $cond2 ]] && cpp_options="${cpp_options},-D__$cond2=__$cond2"
1693	else
1694	[[ -n $cond1 ]] && cpp_options="$cpp_options -D__$cond1"
1695	[[ -n $cond2 ]] && cpp_options="$cpp_options -D__$cond2"
1696	fi
1697
1698	# SET DIRECTIVES FOR ACTIVATING SOFTWARE-PACKAGES (OPTION -p)
1699	if [[ -n $package_list ]]
1700	then
1701	for package in $package_list
1702	do
1703	if [[ $(echo $localhost \| cut -c1-3) = ibm ]]
1704	then
1705	if [[ $package != "dvrp_graphics+1PE" ]]
1706	then
1707	cpp_options="${cpp_options},-D__$package=__$package"
1708	else
1709	cpp_options="${cpp_options},-D__dvrp_graphics=__dvrp_graphics"
1710	export use_seperate_pe_for_dvrp_output=true
1711	fi
1712	else
1713	if [[ $package != "dvrp_graphics+1PE" ]]
1714	then
1715	cpp_options="$cpp_options -D__$package"
1716	else
1717	cpp_options="$cpp_options -D__dvrp_graphics"
1718	export use_seperate_pe_for_dvrp_output=true
1719	fi
1720	fi
1721	done
1722	fi
1723
1724	# SET DIRECTIVES GIVEN BY OPTION -D
1725	if [[ -n $cpp_opts ]]
1726	then
1727	for popts in $cpp_opts
1728	do
1729	if [[ $(echo $localhost \| cut -c1-3) = ibm ]]
1730	then
1731	cpp_options="${cpp_options},-D__$popts=__$popts"
1732	else
1733	cpp_options="$cpp_options -D__$popts"
1734	fi
1735	done
1736	fi
1737
1738	else
1739
1740
1741	# FOR LOCAL RUNS CHECK AGAIN, IF EXECUTABLE EXISTS
1742	if [[ $do_remote = false ]]
1743	then
1744	if [[ ! -f $executable ]]
1745	then
1746	printf "\n +++ executable file: $executable"
1747	printf "\n does not exist"
1748	locat=executable; exit
1749	fi
1750	fi
1751	fi
1752
1753
1754	# DETERMINE THE JOB MODE
1755	if [[ "$ENVIRONMENT" = BATCH ]]
1756	then
1757	jobmo=BATCH
1758	else
1759	jobmo=INTERACTIVE
1760	fi
1761
1762
1763	# NO INTERACTIVE RUNS ALLOWED ON LCTIT
1764	if [[ $host = lctit && $jobmo = INTERACTIVE && $do_batch = false ]]
1765	then
1766	printf "\n +++ no interactive runs allowed on host \"$host\" "
1767	printf "\n please submit batch job using mrun option \"-b\" \n"
1768	locat=normal; exit
1769	fi
1770
1771
1772	# CHECK, IF USER DEFINED A COMPILER
1773	if [[ "$compiler_name" = "" ]]
1774	then
1775
1776	printf "\n +++ no compiler specified for \"$host $cond1 $cond2\""
1777	locat=compiler_name; exit
1778
1779	fi
1780
1781
1782	# DETERMINE THE NAME OF MRUN'S TEMPORARY WORKING DIRECTORY
1783	kennung=$RANDOM
1784	if [[ "$tmp_user_catalog" = "" ]]
1785	then
1786	if [[ $localhost = ibmh ]]
1787	then
1788	tmp_user_catalog=$SCRATCH
1789	elif [[ $localhost = nech ]]
1790	then
1791	tmp_user_catalog=$WRKSHR
1792	else
1793	tmp_user_catalog=/tmp
1794	fi
1795	fi
1796	TEMPDIR=$tmp_user_catalog/${usern}.$kennung
1797
1798
1799	# DETERMINE THE NAME OF THE DIRECTORY WHICH IS USED TO TEMPORARILY STORE DATA FOR RESTART RUNS
1800	if [[ "$tmp_data_catalog" = "" ]]
1801	then
1802	if [[ $localhost = nech ]]
1803	then
1804	tmp_data_catalog=$WRKSHR/mrun_restart_data
1805	else
1806	tmp_data_catalog=/tmp/mrun_restart_data
1807	fi
1808	fi
1809
1810
1811	# IN CASE OF LOCAL RUNS REPLACE ENVIRONMENT VARIABLES BY THEIR VALUES
1812	if [[ $do_remote = false && $do_compile = true \|\| $create_executable_for_batch = true ]]
1813	then
1814	eval fopts=\"$fopts\"
1815	eval lopts=\"$lopts\"
1816	fi
1817
1818
1819
1820	# DETERMINE COMPILE- AND LINK-OPTIONS
1821	fopts="$fopts $netcdf_inc $fftw_inc $dvr_inc"
1822	lopts="$lopts $netcdf_lib $fftw_lib $dvr_lib"
1823	ROPTS="$ropts"
1824	# if [[ ( $(echo $host \| cut -c1-3) = nec \|\| $(echo $host \| cut -c1-3) = ibm \|\| $host = lckyoto \|\| $host = lctit \|\| $host = lcflow \|\| $host = lcxe6 \|\| $host = lcxt5m \|\| $host = lck \|\| $host = lckiaps \|\| $host = lckordi \|\| $host = lcsb \|\| $host ) && -n $numprocs ]]
1825	# then
1826	XOPT="-X $numprocs"
1827	# fi
1828
1829
1830
1831	# CHECK THE CPU-LIMIT. IT MUST BE GIVEN FOR BATCH-JOBS AND IS COMMUNICATED TO THE
1832	# EXECUTABLE VIA NAMELIST-PARAMETER cputime
1833	done=false
1834	while [[ $done = false ]]
1835	do
1836	cputime=$cpumax
1837	if (( $cputime == 0 ))
1838	then
1839	if [[ $do_batch = true ]]
1840	then
1841	printf "\n +++ cpu-time is undefined"
1842	printf "\n >>> Please type CPU-time in seconds as INTEGER:"
1843	printf "\n >>> "
1844	read cputime 1>/dev/null 2>&1
1845	else
1846	cputime=10000000 # NO CPU LIMIT FOR INTERACTIVE RUNS
1847	fi
1848	else
1849	done=true
1850	fi
1851	cpumax=$cputime
1852	done
1853
1854	(( minuten = cputime / 60 ))
1855	(( sekunden = cputime - minuten * 60 ))
1856
1857
1858	# CHECK THE MEMORY DEMAND
1859	if [[ $do_batch = true ]]
1860	then
1861	done=false
1862	while [[ $done = false ]]
1863	do
1864	if (( memory == 0 ))
1865	then
1866	printf "\n +++ memory demand is undefined"
1867	printf "\n >>> Please type memory in MByte per process as INTEGER:"
1868	printf "\n >>> "
1869	read memory 1>/dev/null 2>&1
1870	else
1871	done=true
1872	fi
1873	done
1874	fi
1875
1876
1877	# IN CASE OF REMOTE-JOBS CHECK, IF A USERNAME FOR THE REMOTE HOST IS GIVEN
1878	if [[ $do_remote = true && -z $remote_username ]]
1879	then
1880	while [[ -z $remote_username ]]
1881	do
1882	printf "\n +++ username on remote host \"$host\" is undefined"
1883	printf "\n >>> Please type username:"
1884	printf "\n >>> "
1885	read remote_username
1886	done
1887	mc="$mc -u$remote_username"
1888	fi
1889
1890	# CHECK FOR INITIAL COMMANDS AFTER LOGIN
1891	if [[ "$login_init_cmd" != "" ]]
1892	then
1893	export init_cmds="${login_init_cmd};"
1894	fi
1895
1896
1897	# SET THE MODULE-LOAD COMMAD AND EXPORT IT FOR subjob
1898	if [[ "$modules" != "" ]]
1899	then
1900	if [[ $host = lctit ]]
1901	then
1902	export module_calls=". $modules"
1903	else
1904	export module_calls="module load ${modules};"
1905	fi
1906	fi
1907
1908
1909
1910	# OUTPUT OF THE MRUN-HEADER
1911	calltime=$(date)
1912	printf "\n"
1913	printf "#------------------------------------------------------------------------# \n"
1914	printf "\| %-35s%35s \| \n" "$version" "$calltime"
1915	printf "\| \| \n"
1916	spalte1="called on:"; spalte2=$localhost_realname
1917	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
1918	if [[ $do_remote = true ]]
1919	then
1920	spalte1="execution on:"; spalte2="$host (username: $remote_username)"
1921	else
1922	spalte1="execution on:"; spalte2="$host ($localhost_realname)"
1923	fi
1924	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
1925	if [[ -n $numprocs ]]
1926	then
1927	if [[ $run_coupled_model = false ]]
1928	then
1929	spalte1="number of PEs:"; spalte2=$numprocs
1930	else
1931	spalte1="number of PEs:"; spalte2="$numprocs (atmosphere: $numprocs_atmos, ocean: $numprocs_ocean)"
1932	fi
1933	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
1934	fi
1935	if [[ -n $tasks_per_node ]]
1936	then
1937	spalte1="tasks per node:"; spalte2="$tasks_per_node (number of nodes: $nodes)"
1938	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
1939	if (( remaining_pes > 0 ))
1940	then
1941	spalte1=" "; spalte2="one of the nodes only filled with $remaining_pes tasks"
1942	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
1943	fi
1944	fi
1945	if [[ $maximum_parallel_io_streams != $numprocs ]]
1946	then
1947	spalte1="max par io streams:"; spalte2="$maximum_parallel_io_streams"
1948	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
1949	fi
1950	if [[ $use_openmp = true ]]
1951	then
1952	spalte1="threads per task:"; spalte2="$threads_per_task"
1953	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
1954	fi
1955	printf "\| \| \n"
1956	if [[ $do_compile = true ]]
1957	then
1958	if [[ "$mopts" != "" ]]
1959	then
1960	spalte1="make options:"; spalte2=$(echo "$mopts" \| cut -c-45)
1961	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
1962	zeile=$(echo "$mopts" \| cut -c46-)
1963	while [[ "$zeile" != "" ]]
1964	do
1965	spalte1=""
1966	spalte2=$(echo "$zeile" \| cut -c-45)
1967	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
1968	zeile=$(echo "$zeile" \| cut -c46-)
1969	done
1970	fi
1971
1972	spalte1="cpp directives:"; spalte2=$(echo "$cpp_options" \| cut -c-45)
1973	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
1974	zeile=$(echo "$cpp_options" \| cut -c46-)
1975	while [[ "$zeile" != "" ]]
1976	do
1977	spalte1=""
1978	spalte2=$(echo "$zeile" \| cut -c-45)
1979	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
1980	zeile=$(echo "$zeile" \| cut -c46-)
1981	done
1982
1983	spalte1="compiler options:"; spalte2=$(echo "$fopts" \| cut -c-45)
1984	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
1985	zeile=$(echo "$fopts" \| cut -c46-)
1986	while [[ "$zeile" != "" ]]
1987	do
1988	spalte1=""
1989	spalte2=$(echo "$zeile" \| cut -c-45)
1990	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
1991	zeile=$(echo "$zeile" \| cut -c46-)
1992	done
1993
1994	spalte1="linker options:"; spalte2=$(echo "$lopts" \| cut -c-45)
1995	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
1996	zeile=$(echo "$lopts" \| cut -c46-)
1997	while [[ "$zeile" != "" ]]
1998	do
1999	spalte1=""
2000	spalte2=$(echo "$zeile" \| cut -c-45)
2001	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2002	zeile=$(echo "$zeile" \| cut -c46-)
2003	done
2004
2005	spalte1="modules to be load:"; spalte2=$(echo "$modules" \| cut -c-45)
2006	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2007	zeile=$(echo "$modules" \| cut -c46-)
2008	while [[ "$zeile" != "" ]]
2009	do
2010	spalte1=""
2011	spalte2=$(echo "$zeile" \| cut -c-45)
2012	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2013	zeile=$(echo "$zeile" \| cut -c46-)
2014	done
2015
2016	spalte1="main program:"; spalte2=$mainprog
2017	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2018	else
2019	spalte1=executable:; spalte2=$executable
2020	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2021	fi
2022	printf "\| \| \n"
2023	spalte1="base name of files:"; spalte2=$fname
2024	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2025	if [[ $fname != $afname ]]
2026	then
2027	spalte1="base name of input files:"; spalte2=$afname
2028	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2029	fi
2030	spalte1="INPUT control list:"; spalte2=$(echo $input_list)
2031	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2032	spalte1="OUTPUT control list:"; spalte2=$(echo $output_list)
2033	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2034
2035	if [[ "$ocean_file_appendix" = true ]]
2036	then
2037	printf "\| %-35s%-35s \| \n" "suffix \"_O\" is added to local files" " "
2038	fi
2039
2040	if [[ $do_batch = true \|\| "$LOADLBATCH" = yes ]]
2041	then
2042	spalte1="memory demand / PE":; spalte2="$memory MB"
2043	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2044	spalte1=CPU-time:; spalte2="$minuten:$sekunden"
2045	printf "\| %-25s%-45s \| \n" "$spalte1" "$spalte2"
2046	fi
2047
2048	if [[ $do_compile = true ]]
2049	then
2050	printf "\| \| \n"
2051	printf "\| Files to be compiled: \| \n"
2052	zeile=$source_list
2053	while [[ "$zeile" != "" ]]
2054	do
2055	linestart=$(echo $zeile \| cut -c-70)
2056	printf "\| %-70s \| \n" "$linestart"
2057	zeile=$(echo "$zeile" \| cut -c71-)
2058	done
2059	fi
2060	printf "#------------------------------------------------------------------------#"
2061
2062
2063
2064	# OUTPUT OF FILE CONNECTIONS IN CASE OF TRACEBACK
2065	if [[ $do_trace = true ]]
2066	then
2067	(( i = 0 ))
2068	while (( i < iin ))
2069	do
2070	(( i = i + 1 ))
2071	if (( i == 1 ))
2072	then
2073	printf "\n\n >>> INPUT-file assignments:\n"
2074	fi
2075	printf "\n ${localin[$i]} : ${absnamein[$i]}"
2076	done
2077	(( i = 0 ))
2078	while (( i < iout ))
2079	do
2080	(( i = i + 1 ))
2081	if (( i == 1 ))
2082	then
2083	printf "\n\n >>> OUTPUT-file assignments:\n"
2084	fi
2085	printf "\n ${localout[$i]} : ${pathout[$i]}"
2086	done
2087	(( i = 0 ))
2088	while (( i < iic ))
2089	do
2090	(( i = i + 1 ))
2091	if (( i == 1 ))
2092	then
2093	printf "\n\n >>> INPUT-commands:\n"
2094	fi
2095	printf "\n ${in_command[$i]}"
2096	done
2097	(( i = 0 ))
2098	while (( i < ioc ))
2099	do
2100	(( i = i + 1 ))
2101	if (( i == 1 ))
2102	then
2103	printf "\n\n >>> OUTPUT-commands:\n"
2104	fi
2105	printf "\n ${out_command[$i]}"
2106	done
2107	fi
2108
2109
2110	# QUERY FOR CONTINUE (ON LOCAL MACHINES ONLY)
2111	if [[ $remotecall = false && $silent = false && $jobmo != BATCH ]]
2112	then
2113	antwort=dummy
2114	printf "\n\n"
2115	printf " >>> everything o.k. (y/n) ? "
2116	while read antwort
2117	do
2118	if [[ "$antwort" != y && "$antwort" != Y && "$antwort" != n && "$antwort" != N ]]
2119	then
2120	printf " >>> everything o.k. (y/n) ? "
2121	else
2122	break
2123	fi
2124	done
2125	if [[ $antwort = n \|\| $antwort = N ]]
2126	then
2127	locat=user_abort; (( iec = 0 )); exit
2128	fi
2129	if [[ $do_batch = true ]]
2130	then
2131	printf " >>> batch-job will be created and submitted"
2132	else
2133	printf " >>> MRUN will now continue to execute on this machine"
2134	fi
2135	fi
2136
2137
2138	# PERFORM PARAMETER FILE CHECK (COUPLED RUNS ARE NOT SUPPORTED YET)
2139	# DEFINE VARIABLES FOR FREQUENTLY USED DIRECTORIES
2140	check_depository="${working_directory}/trunk/UTIL"
2141	check_sources="${working_directory}/tmp_check_namelist_files"
2142	skip_check=false
2143
2144	# CHECK IF NAMELIST_FILE_CHECK HAS BEEN COMPILED SUCCESSFULLY
2145	if [[ ! -f $check_depository/check_namelist_files.tar ]]
2146	then
2147	skip_check=true
2148	reason="run on remote host or parameter file check has not been compiled."
2149	fi
2150	if [[ ! -f $PALM_BIN/check_namelist_files.x ]]
2151	then
2152	skip_check=true
2153	reason="parameter file check has not been compiled."
2154	fi
2155
2156	# CHECK FOR PARALLEL RUN; OTHERWISE SKIP CHECK
2157	if [[ "$cond1" != "parallel" && "$cond2" != "parallel" ]]
2158	then
2159	skip_check=true
2160	reason="serial run."
2161	fi
2162
2163	# ONLY PERFORM CHECK IF -z OPTION IS NOT SET, NO RESTART RUN IS CARRIED OUT
2164	# AND IF THE EXECUTION HOST IS THE LOCAL HOST
2165	# ATTENTION: THIS ROUTINE DOES NOT WORK IF THE COMPILER ON THE LOCAL HOST
2166	# DIFFERS FROM THE COMPILER ON THE REMOTE HOST
2167	if [[ $check_namelist_files == false ]]
2168	then
2169	skip_check=true
2170	reason="-z option set."
2171	fi
2172
2173	if [[ $fromhost != $localhost ]]
2174	then
2175	skip_check=true
2176	reason="submitting host is local host."
2177	fi
2178
2179	if [[ $run_coupled_model == true ]]
2180	then
2181	skip_check=true
2182	reason="coupled run."
2183	fi
2184
2185	if [[ $restart_run == true ]]
2186	then
2187	skip_check=true
2188	reason="restart run."
2189	fi
2190
2191	# SKIP CHECK IN CASE OF RESTART RUN: CHECK WHETHER THE LAST CHAR IS "f" IN PARIN
2192	(( last_char_int = `echo $filename_input \| wc -c` - 1 ))
2193	last_char=`echo $filename_input \| cut -c $last_char_int`
2194	if [[ "$last_char" == "f" ]]
2195	then
2196	skip_check=true
2197	reason="restart run."
2198	fi
2199
2200	if [[ $skip_check == false ]]
2201	then
2202	tmp_check=${working_directory}/tmp_check
2203
2204	# GET TOPOGRAPHY PARAMETER FILE SUFFIX (USUALLY "_TOPO"). THIS FILE IS
2205	# NOT NECESSARILY REQUIRED
2206	line=""
2207	found=false
2208	grep "TOPOGRAPHY_DATA" $config_file > $tmp_check
2209	while read line1
2210	do
2211	line="$line1"
2212	if [[ "$line" != "" && $(echo $line \| cut -c1) != "#" ]]
2213	then
2214	topo_suffix=`echo $line \| tr -s " " \| cut -d" " -s -f5`
2215	found=true
2216	fi
2217	done < $tmp_check
2218
2219	if [[ $found = false ]]
2220	then
2221	printf "\n +++ no TOPOGRAPHY list entry found in the configuration file."
2222	fi
2223
2224	# GET NUDGING PARAMETER FILE SUFFIX (USUALLY "_NUDGE"). THIS FILE IS
2225	# NOT NECESSARILY REQUIRED
2226	line=""
2227	found=false
2228	grep "NUDGING_DATA" $config_file > $tmp_check
2229	while read line1
2230	do
2231	line="$line1"
2232	if [[ "$line" != "" && $(echo $line \| cut -c1) != "#" ]]
2233	then
2234	nudge_suffix=`echo $line \| tr -s " " \| cut -d" " -s -f5`
2235	found=true
2236	fi
2237	done < $tmp_check
2238
2239	if [[ $found = false ]]
2240	then
2241	printf "\n +++ no NUDGING list entry found in the configuration file."
2242	fi
2243
2244	# GET LARGE SCALE FORCING PARAMETER FILE SUFFIX (USUALLY "_LSF"). THIS FILE IS
2245	# NOT NECESSARILY REQUIRED
2246	line=""
2247	found=false
2248	grep "LSF_DATA" $config_file > $tmp_check
2249	while read line1
2250	do
2251	line="$line1"
2252	if [[ "$line" != "" && $(echo $line \| cut -c1) != "#" ]]
2253	then
2254	lsf_suffix=`echo $line \| tr -s " " \| cut -d" " -s -f5`
2255	found=true
2256	fi
2257	done < $tmp_check
2258
2259	if [[ $found = false ]]
2260	then
2261	printf "\n +++ no LSF list entry found in the configuration file."
2262	fi
2263
2264	rm -rf ${working_directory}/tmp_check
2265
2266	# CHECK IF THE P3DF FILE MUST BE CHECKED
2267	for item in $input_list
2268	do
2269	if [[ "$item" == "restart" ]]
2270	then
2271	check_restart=1
2272	else
2273	check_restart=0
2274	fi
2275	done
2276
2277	# ERROR IF NO PARAMETER FILE WAS FOUND, OTHERWISE PROCEED
2278	if [[ (! -f $filename_input) && ( "$filename_input" != "" ) ]]
2279	then
2280	printf "\n\n +++ ERROR: parameter file ($filename_input) not found."
2281	locat=check_namelist; exit
2282	fi
2283
2284	# CHECK IF THE RESTART PARAMETER FILE EXISTS (IF NECESSARY)
2285	if [[ $check_restart == 1 ]]
2286	then
2287
2288	filenamef="${filename_input}f"
2289	if [[ ! -f $filenamef ]]
2290	then
2291	printf "\n\n +++ WARNING: restart parameter file ($filenamef) is missing."
2292	check_restart=0
2293	answer=dummy
2294	printf "\n\n"
2295
2296	if [[ $silent == false ]]
2297	then
2298	while [[ "$answer" != c && "$answer" != C && "$answer" != a && "$answer" != A ]]
2299	do
2300	printf " >>> continue anyway (c(ontinue)/a(bort)) ? "
2301	read answer
2302	done
2303	if [[ $answer = a \|\| $answer = A ]]
2304	then
2305	printf "\n +++ Aborting...."
2306	locat=normal; exit
2307	fi
2308	fi
2309	else
2310	check_restart=1
2311	fi
2312	fi
2313
2314	# CREATE TEMPORARY SOURCES_FOR_CHECK PATH
2315	mkdir $check_sources
2316	cd $check_sources
2317
2318
2319	# CHECK FOR USER CODE, OTHERWISE USE THE PRECOMPILED CHECK_NAMELIST_FILES.X
2320	if [[ -d $add_source_path ]]
2321	then
2322	printf "\n\n *** copying files from $check_depository"
2323	cp $check_depository/check_namelist_files.tar ./
2324
2325	printf "\n\n *** untar of makefile and source files in $check_sources"
2326	tar -xf check_namelist_files.tar > /dev/null 2>&1
2327
2328	printf "\n\n *** adding user code."
2329	cp $add_source_path/* ./
2330	touch check_namelist_files.f90
2331
2332	# GET COMPILER OPTIONS AND PERFORM MAKE
2333	printf "\n\n *** compiling code if necessary...\n"
2334
2335	# workaround for batch jobs on local machine (lcxe6)
2336	if [[ $do_batch == true && $do_remote == false ]]
2337	then
2338	eval $init_cmds
2339	fi
2340
2341	# GET CHECK OPTIONS
2342	line=""
2343	found=false
2344
2345	grep "$localhost" ${base_directory}/${config_file} \| grep "%cpp_options" > $tmp_check
2346
2347	while read line1
2348	do
2349
2350	if [[ $(echo $line1 \| cut -d" " -s -f3-) = "$localhost" ]]
2351	then
2352	line="$line1"
2353	fi
2354
2355	if [[ "$line" != "" && $(echo $line \| cut -c1) != "#" ]]
2356	then
2357	# REMOVE COLONS FROM OPTION-STRING, REMOVE ALL -D OPTIONS
2358	line="$line "
2359	copts_check=`echo $line \| cut -d" " -s -f2 \| sed 's/::/%DUM%/g' \| sed 's/:/ /g' \| sed 's/%DUM%/:/g' \| sed 's/-D[^ ]* //g' \| sed 's/ -D.*//g'`
2360	found=true
2361	fi
2362
2363	done < $tmp_check
2364	copts_check="$copts_check -D__check -D__parallel"
2365
2366	make -f Makefile_check F90=$compiler_name_ser COPT="$copts_check"
2367
2368	# GET MAKE OUTPUT
2369	if [[ $? != 0 ]]
2370	then
2371	printf "\n +++ error during make."
2372	answer=dummy
2373	printf "\n\n"
2374	if [[ $silent == false ]]
2375	then
2376	while [[ "$answer" != c && "$answer" != C && "$answer" != a && "$answer" != A ]]
2377	do
2378	printf " >>> continue anyway (c(ontinue)/a(bort)) ? "
2379	read answer
2380	done
2381	if [[ $answer = a \|\| $answer = A ]]
2382	then
2383	printf "\n +++ Aborting..."
2384	rm -rf $check_sources
2385	locat=normal; exit
2386	else
2387	skip_check=true
2388	fi
2389	else
2390	skip_check=true
2391	fi
2392	fi
2393	else
2394	cp $PALM_BIN/check_namelist_files.x ./
2395	fi
2396
2397	cp $filename_input ./PARIN
2398	if [[ $check_restart == 1 ]]
2399	then
2400	cp $filenamef ./PARINF
2401	fi
2402
2403	if [[ -f ${pathname}/${fname}${topo_suffix} && $skip_check == false ]]
2404	then
2405	printf "\n *** adding topography data"
2406	cp ${pathname}/${fname}${topo_suffix} ./TOPOGRAPHY_DATA
2407
2408	# IN CASE OF TOPOGRAPHY AND HIGH GRID POINT NUMBERS, THE STACK SIZE
2409	# MUST BE INCREASED. THIS IS DUE TO THE ARRAY nzb_local AND topo_height,
2410	# WHICH REQUIRE SUFFICIENT MEMORY
2411	ulimit -s unlimited
2412	fi
2413
2414	if [[ -f ${pathname}/${fname}${nudge_suffix} && $skip_check == false ]]
2415	then
2416	printf "\n *** adding nudging data"
2417	cp ${pathname}/${fname}${nudge_suffix} ./NUDGING_DATA
2418	fi
2419
2420	if [[ -f ${pathname}/${fname}${lsf_suffix} && $skip_check == false ]]
2421	then
2422	printf "\n *** adding large scale forcing data"
2423	cp ${pathname}/${fname}${lsf_suffix} ./LSF_DATA
2424	fi
2425
2426	# CREATE ENVPAR FILE, WHICH IS NEEDED BY CHECK_NAMELIST_FILES.X
2427	cat > ENVPAR << %%END%%
2428	&envpar run_identifier = '$fname', host = '$host',
2429	write_binary = '$write_binary', tasks_per_node = $tasks_per_node,
2430	maximum_parallel_io_streams = $maximum_parallel_io_streams,
2431	maximum_cpu_time_allowed = ${cpumax}.,
2432	revision = '$global_revision',
2433	local_dvrserver_running = $local_dvrserver_running /
2434
2435	%%END%%
2436
2437	# SAFETY CHECK: ONLY PROCEED IF THE PARAMETER CHECK PROGRAM WAS PROPERLY COMPILED
2438	if [[ ! -f check_namelist_files.x && $skip_check == false ]]
2439	then
2440	printf "\n +++ WARNING: check_namelist_files.x not found."
2441	answer=dummy
2442	printf "\n\n"
2443
2444	if [[ $silent == false ]]
2445	then
2446	while [[ "$answer" != c && "$answer" != C && "$answer" != a && "$answer" != A ]]
2447	do
2448	printf " >>> continue anyway (c(ontinue)/a(bort)) ? "
2449	read answer
2450	done
2451	if [[ $answer = a \|\| $answer = A ]]
2452	then
2453	printf "\n +++ Aborting..."
2454	rm -rf $check_sources
2455	locat=normal; exit
2456	else
2457	printf "\n *** skipping parameter file check."
2458	fi
2459	fi
2460
2461	elif [[ $skip_check == false ]]
2462	then
2463	# STARTING THE PARAMETER FILE CHECK
2464	printf "\n\n *** starting parameter file check..."
2465
2466	# CHECKING THE P3D FILE
2467	printf "\n\n (1) checking $filename_input"
2468	echo "$numprocs 0 0" > VARIN
2469	errors=`./check_namelist_files.x < VARIN 2>&1`
2470
2471	check_error=false
2472	if [[ "$errors" == "" ]]
2473	then
2474	printf " --> o.k."
2475	else
2476	printf " --> failed."
2477	check_error=true
2478	printf "\n\n $errors"
2479	fi
2480
2481	# CHECKING THE PD3F FILE IF NECESSARY
2482	if [[ $check_restart == 1 && $check_error == false ]]
2483	then
2484	printf "\n\n (2) checking $filenamef"
2485
2486	# FIRST CHECK IF INITIALIZING_ACTIONS="READ_RESTART_DATA" IS SET
2487	# IN &INIPAR LIST
2488	found=false
2489	cat PARINF \| while read line
2490	do
2491	line=$(echo $line\|sed 's/ //g')
2492	if [[ $line == "&inipar" ]]
2493	then
2494	start_search=true
2495	fi
2496
2497	if [[ $start_search == true ]]
2498	then
2499	if [[ $line == "initializing_actions='read_restart_data'" ]]
2500	then
2501	found=true
2502	break
2503	fi
2504	fi
2505
2506	if [[ $line == "/" ]]
2507	then
2508	start_search=false
2509	fi
2510
2511	done
2512
2513	if [[ $found = false ]]
2514	then
2515	printf "\n\n +++ ERROR: initializing_actions = 'read_restart_data' not found"
2516	printf "\n in &inipar list in $fname$p3df_suffix."
2517	rm -rf $check_sources
2518	locat=check_namelist; exit
2519	fi
2520
2521	# READ max_user_pr FROM FILES
2522	if [[ -f parin_for_check ]]
2523	then
2524	read max_pr_user < parin_for_check
2525	else
2526	max_user_pr=0
2527	fi
2528
2529	echo "$numprocs 1 $max_pr_user" > VARIN
2530	errors=`./check_namelist_files.x < VARIN 2>&1`
2531
2532	if [[ "$errors" == "" ]]
2533	then
2534	printf " --> o.k."
2535	else
2536	printf " --> failed."
2537	check_error=true
2538	printf "\n\n $errors"
2539	fi
2540	fi
2541
2542	# REPORT ERRORS AND CONTINUE/EXIT
2543	if [[ $check_error == true ]]
2544	then
2545	printf "\n +++ errors found in the parameter file!\n"
2546	answer=dummy
2547	printf "\n\n"
2548
2549	while [[ "$answer" != c && "$answer" != C && "$answer" != a && "$answer" != A ]]
2550	do
2551	printf " >>> continue anyway (c(ontinue)/a(bort)) ? "
2552	read answer
2553	done
2554
2555	if [[ $answer = a \|\| $answer = A ]]
2556	then
2557	printf "\n +++ Aborting..."
2558	rm -rf $check_sources
2559	locat=normal; exit
2560	fi
2561	else
2562	printf "\n\n *** parameter file(s) seem(s) to be o.k.\n"
2563	fi
2564	rm -rf $check_sources
2565	fi
2566	else
2567	printf "\n\n +++ skipping parameter file check due to following reason: $reason \n"
2568	fi
2569
2570	# DELETE TEMPORARY DIRECTORY AND FINISH NAMELIST FILE CHECK
2571	rm -rf $check_sources
2572	cd $working_directory
2573
2574
2575	# DETERMINE PATH FOR MAKE DEPOSITORY
2576	if [[ $do_batch = false \|\| $create_executable_for_batch = true ]]
2577	then
2578
2579	line=""
2580	grep "%depository_path" $config_file > tmp_mrun
2581	while read line
2582	do
2583	if [[ "$line" != "" && $(echo $line \| cut -c1) != "#" ]]
2584	then
2585	if [[ "$(echo $line \| cut -d" " -s -f3)" = "" ]]
2586	then
2587	global_depository_path=`echo $line \| cut -d" " -s -f2`
2588	fi
2589	fi
2590	done < tmp_mrun
2591
2592	line=""
2593	grep " $localhost" $config_file \| grep "%depository_path" > tmp_mrun
2594	while read line
2595	do
2596	if [[ "$line" != "" && $(echo $line \| cut -c1) != "#" ]]
2597	then
2598	if [[ "$(echo $line \| cut -d" " -s -f4)" = "$cond1" && "$(echo $line \| cut -d" " -s -f5)" = "$cond2" ]]
2599	then
2600	local_depository_path=`echo $line \| cut -d" " -s -f2`
2601	fi
2602	fi
2603	done < tmp_mrun
2604
2605	if [[ "$local_depository_path" = "" ]]
2606	then
2607	if [[ "$global_depository_path" != "" ]]
2608	then
2609	local_depository_path=$global_depository_path
2610	else
2611	printf "\n\n +++ no depository path found in configuration file"
2612	printf "\n for local host \"$localhost\" "
2613	printf "\n please set \"\%depository_path\" in configuration file\n"
2614	locat=config_file; exit
2615	fi
2616	fi
2617	eval local_depository_path=$local_depository_path
2618	[[ "$cond1" != "" ]] && local_depository_path=${local_depository_path}_$cond1
2619	[[ "$cond2" != "" ]] && local_depository_path=${local_depository_path}_$cond2
2620
2621
2622	basename=`echo $mainprog \| cut -f1 -d"."`
2623	eval make_depository=${local_depository_path}/${basename}_current_version.tar
2624	if [[ ! -f $make_depository ]]
2625	then
2626	printf "\n"
2627	printf "\n *** WARNING: make depository \"$make_depository\" not found"
2628	printf "\n \"make\" will fail, if the Makefile or other source files are missing\n"
2629	fi
2630
2631	fi
2632
2633
2634	# NOW PERFORM THOSE ACTIONS REQUIRED TO EXECUTE THE PROGRAM (PALM) ON THIS MACHINE
2635	# (COMPILING/LINKING, EXECUTING, COPYING I/O FILES)
2636	if [[ $do_batch = false ]]
2637	then
2638
2639
2640	# CREATE THE TEMPORARY WORKING DIRECTORY
2641	mkdir -p $TEMPDIR
2642	chmod go+rx $TEMPDIR
2643	tmpcreate=true
2644
2645
2646	# COPY EITHER THE COMPLETE SOURCE CODE FILES TO BE COMPILED OR THE EXECUTABLE
2647	# INTO THE TEMPORARY WORKING DIRECTORY
2648	if [[ $do_compile = true ]]
2649	then
2650
2651	# ON NEC, COMPILATION IS DONE ON HOST CROSS VIA CROSS COMPILING
2652	# CREATE A TEMPORARY DIRECTORY ON THAT MACHINE (HOME MOUNTED VIA NFS)
2653	if [[ $localhost = nech ]]
2654	then
2655	TEMPDIR_COMPILE=$HOME/work/${usern}.$kennung
2656	if mkdir -p $TEMPDIR_COMPILE
2657	then
2658	printf "\n *** \"$TEMPDIR_COMPILE\" "
2659	printf "\n is generated as temporary directory for cross compiling\n"
2660	else
2661	printf "\n +++ creating directory \"$TEMPDIR_COMPILE\" "
2662	printf "\n needed for cross compilation failed"
2663	locat=compile
2664	exit
2665	fi
2666	else
2667	TEMPDIR_COMPILE=$TEMPDIR
2668	fi
2669
2670	cp $make_depository $TEMPDIR_COMPILE
2671	cd $TEMPDIR_COMPILE
2672	tar -xf $make_depository > /dev/null 2>&1
2673	cd - > /dev/null
2674
2675	# DUE TO UNKNOWN REASONS, COPY WITH cp COMMAND CREATES CORRUPT
2676	# FILES ON CRAY XC30 SYSTEMS (HLRN III), rsync IS USED INSTEAD
2677	if [[ $(echo $host \| cut -c1-6) = lccray ]]
2678	then
2679	rsync -av -t SOURCES_FOR_RUN_$fname/* $TEMPDIR_COMPILE > /dev/null
2680	else
2681	cp SOURCES_FOR_RUN_$fname/* $TEMPDIR_COMPILE > /dev/null
2682	fi
2683
2684	else
2685
2686	cp $executable ${TEMPDIR}/a.out
2687
2688	fi
2689
2690
2691	# CHANGE TO THE TEMPORARY WORKING DIRECTORY
2692	cd $TEMPDIR
2693	printf "\n *** changed to temporary directory: $TEMPDIR"
2694
2695
2696	# THE FOLLOWING IS REQUIRED FPR AVS-OUTPUT WITH PALM ??? REMOVE ???
2697	# WRITE INFORMATIONS ABOUT THE OUTPUT-FILE CONNECTIONS TO A TEMPORARY FILE
2698	# THIS FILE CAN LATER BE READ FROM USER-DEFINED CODE TO DETERMINE THE
2699	# RELATION BETWEEN THE LOCAL TEMPORARY AND PERMANENT FILE NAMES
2700	(( i = 0 ))
2701	while (( i < iout ))
2702	do
2703	(( i = i + 1 ))
2704	if [[ "${actionout[$i]}" = tr \|\| "${actionout[$i]}" = tra \|\| "${actionout[$i]}" = trpe ]]
2705	then
2706	printf "${localout[$i]} ${actionout[$i]}\n${pathout[$i]}\n${localhost}_${fname}${endout[$i]}\n" >> OUTPUT_FILE_CONNECTIONS
2707	else
2708	printf "${localout[$i]} ${actionout[$i]}\n${pathout[$i]}\n${frelout[$i]}\n" >> OUTPUT_FILE_CONNECTIONS
2709	fi
2710	done
2711
2712
2713	# IF REQUIRED, START WITH COMPILING
2714	if [[ $do_compile = true ]]
2715	then
2716
2717
2718	if [[ -f a.out ]]
2719	then
2720
2721	# EXECUTABLE WAS CREATED DURING INTERACTIVE CALL OF MRUN
2722	printf "\n\n\n *** executable a.out found"
2723	printf "\n no compilation required \n"
2724
2725	else
2726
2727	# COMPILING WITH MAKE (ON NEC COMPILER IS CALLED ON HOST CROSS)
2728	printf "\n\n\n *** compilation starts \n$dashes\n"
2729	printf " *** compilation with make using following options:\n"
2730	printf " make depository: $make_depository"
2731	if [[ "$mopts" != "" ]]
2732	then
2733	printf " make options: $mopts\n"
2734	fi
2735	printf " compilername: $compiler_name\n"
2736	printf " compiler options: $fopts\n"
2737	printf " preprocessor directives: $cpp_options \n"
2738	printf " linker options: $lopts \n"
2739	if [[ "$modules" != "" ]]
2740	then
2741	printf " modules to be load: $modules \n"
2742	fi
2743	printf " source code files: $source_list \n"
2744
2745	if [[ $localhost = nech ]]
2746	then
2747	ssh $SSH_PORTOPT 136.172.44.192 -l $usern "$init_cmds $module_calls cd \$HOME/work/${usern}.$kennung; sxmake $mopts -f Makefile PROG=a.out F90=$compiler_name COPT=\"$cpp_options\" F90FLAGS=\"$fopts\" LDFLAGS=\"$lopts\" "
2748	cp $TEMPDIR_COMPILE/a.out .
2749	[[ $? != 0 ]] && compile_error=true
2750	rm -rf $TEMPDIR_COMPILE
2751	elif [[ $localhost = ibmh ]]
2752	then
2753	printf " compiler is called via ssh on \"plogin1\" \n"
2754	ssh $SSH_PORTOPT plogin1 -l $usern "$init_cmds export PATH=/sw/ibm/xlf/13.1.0.8/usr/bin:$PATH; $module_calls cd $TEMPDIR; make $mopts -f Makefile PROG=a.out F90=$compiler_name COPT=\"$cpp_options\" F90FLAGS=\"$fopts\" LDFLAGS=\"$lopts\" "
2755	[[ ! -f a.out ]] && compile_error=true
2756	continue # STATUS=1, IF a.out EXISTS
2757	elif [[ $localhost = lcflow ]]
2758	then
2759	printf " compiler is called via ssh on \"flow\" \n"
2760	/usr/bin/ssh $SSH_PORTOPT flow02.hpc.uni-oldenburg.de -l $usern "$init_cmds $module_calls cd $TEMPDIR; make $mopts -f Makefile PROG=a.out F90=$compiler_name COPT=\"$cpp_options\" F90FLAGS=\"$fopts\" LDFLAGS=\"$lopts\" "
2761	check_for_file=`/usr/bin/ssh $SSH_PORTOPT flow02.hpc.uni-oldenburg.de -l $usern "ls $TEMPDIR/a.out 2> /dev/null"`
2762	[[ "$check_for_file" = "" ]] && compile_error=true
2763	continue # STATUS=1, IF a.out EXISTS
2764	elif [[ $localhost = lccrayb \|\| $localhost = lccrayf \|\| $localhost = lccrayh ]]
2765	then
2766	make $mopts -f Makefile PROG=a.out F90=$compiler_name COPT="$cpp_options" F90FLAGS="$fopts" LDFLAGS="$lopts"
2767	else
2768	[[ "$init_cmds" != "" ]] && eval $init_cmds
2769	[[ "$module_calls" != "" ]] && eval $module_calls
2770	make $mopts -f Makefile PROG=a.out F90=$compiler_name COPT="$cpp_options" F90FLAGS="$fopts" LDFLAGS="$lopts"
2771	fi
2772
2773	if [[ $? != 0 \|\| "$compile_error" = true \|\| "$module_compile_error" = true ]]
2774	then
2775	printf "\n +++ error occured while compiling or linking"
2776	locat=compile
2777
2778	exit
2779	else
2780	printf "$dashes\n *** compilation finished \n"
2781	fi
2782
2783	fi
2784
2785	fi
2786
2787
2788	# PROVIDE THE INPUT FILES
2789	# LOOP OVER ALL ACTIVATED FILES (LISTED IN THE CONFIGURATION FILE)
2790	(( i = 0 ))
2791	while (( i < iin ))
2792	do
2793	(( i = i + 1 ))
2794	if (( i == 1 ))
2795	then
2796	printf "\n\n *** providing INPUT-files:\n$dashes"
2797	fi
2798
2799
2800	# SKIP OPTIONAL FILES, IF THEY DO NOT EXIST
2801	if [[ "${transin[$i]}" = unavailable ]]
2802	then
2803	if [[ "${extin[$i]}" = "" \|\| "${extin[$i]}" = " " ]]
2804	then
2805	printf "\n +++ WARNING: input file \"${pathin[$i]}/${afname}${endin[$i]}\" "
2806	printf "\n is not available!"
2807	else
2808	printf "\n +++ WARNING: input file \"${pathin[$i]}/${afname}${endin[$i]}.${extin[$i]}\" "
2809	printf "\n is not available!"
2810	fi
2811	continue
2812	fi
2813
2814	# CHECK FOR SINGLE FILE (SERIAL RUN) OR DIRECTORY (ONE FILE PER CORE FOR PARELLEL EXECUTION)
2815	files_for_pes=false; datentyp=file
2816	if [[ "${actionin[$i]}" = pe && -n $numprocs ]]
2817	then
2818	files_for_pes=true; datentyp=directory
2819	actionin[$i]=""
2820	elif [[ "${actionin[$i]}" = pe && ! -n $numprocs ]]
2821	then
2822	actionin[$i]=""
2823	elif [[ "${actionin[$i]}" = arpe && -n $numprocs ]]
2824	then
2825	files_for_pes=true; datentyp=directory
2826	actionin[$i]="ar"
2827	elif [[ "${actionin[$i]}" = arpe && ! -n $numprocs ]]
2828	then
2829	actionin[$i]="ar"
2830	elif [[ "${actionin[$i]}" = flpe && -n $numprocs ]]
2831	then
2832	files_for_pes=true; datentyp=directory
2833	actionin[$i]="fl"
2834	elif [[ "${actionin[$i]}" = flpe && ! -n $numprocs ]]
2835	then
2836	actionin[$i]="fl"
2837	fi
2838
2839	if [[ $files_for_pes = true ]]
2840	then
2841	printf "\n >>> INPUT: ${absnamein[$i]}/.... to ${localin[$i]}"
2842	else
2843	printf "\n >>> INPUT: ${absnamein[$i]} to ${localin[$i]}"
2844	fi
2845
2846	# INPUT-FILE FOR A RESTART RUN.
2847	# CHECK, IF THIS FILE STILL EXISTS ON THE TEMPORARY DATA DIRECTORY.
2848	# IF NOT, TRY TO GET IT FROM THE ARCHIVE SYSTEM
2849	if [[ "${actionin[$i]}" = fl ]]
2850	then
2851	printf "\n $datentyp will be fetched from temporary directory \"${tmp_data_catalog}\" !"
2852	if [[ $files_for_pes = false ]]
2853	then
2854	if [[ -f "$tmp_data_catalog/${frelin[$i]}" ]]
2855	then
2856	ln $tmp_data_catalog/${frelin[$i]} ${localin[$i]}
2857	got_tmp[$i]=true
2858	elif [[ -f "$WORK/${frelin[$i]}" && $ignore_archive_error = true ]]
2859	then
2860	printf "\n +++ $datentyp not found in \"$tmp_data_catalog\" !"
2861	printf "\n *** trying to use backup copy in \"$WORK\" "
2862	cp $WORK/${frelin[$i]} ${localin[$i]}
2863	else
2864	printf "\n +++ $datentyp not found in \"$tmp_data_catalog\" "
2865	printf "\n or \"$tmp_data_catalog\" does not exist!"
2866	printf "\n *** trying to get copy from archive"
2867	actionin[$i]=ar
2868	fi
2869	else
2870	if [[ -d "$tmp_data_catalog/${frelin[$i]}" ]]
2871	then
2872	mkdir ${localin[$i]}
2873	cd $tmp_data_catalog/${frelin[$i]}
2874	for file in $(ls *)
2875	do
2876	ln $file $TEMPDIR/${localin[$i]}
2877	done
2878	cd $TEMPDIR
2879	got_tmp[$i]=true
2880	elif [[ -d "$WORK/${frelin[$i]}" && $ignore_archive_error = true ]]
2881	then
2882	printf "\n +++ $datentyp not found in \"$tmp_data_catalog\" !"
2883	printf "\n *** trying to use backup copy in \"$WORK\" "
2884	cp -r $WORK/${frelin[$i]} ${localin[$i]}
2885	else
2886	printf "\n +++ $datentyp not found in \"$tmp_data_catalog\" "
2887	printf "\n or \"$tmp_data_catalog\" does not exist!"
2888	printf "\n *** trying to get copy from archive"
2889	actionin[$i]=ar
2890	fi
2891	fi
2892	fi
2893
2894
2895	# FILE IS STORED ON THE ARCHIVE SYSTEM
2896	if [[ "${actionin[$i]}" = ar ]]
2897	then
2898
2899	if [[ $files_for_pes = false ]]
2900	then
2901	printf "\n file will be restored from archive-system ($archive_system)!"
2902	else
2903	printf "\n directory will be restored from archive-system ($archive_system)!"
2904	fi
2905
2906	file_restored=false
2907
2908	if [[ $archive_system = asterix ]]
2909	then
2910	do_stagein=true
2911	(( stagein_anz = 0 ))
2912	while [[ $do_stagein = true ]]
2913	do
2914	if [[ $files_for_pes = false ]]
2915	then
2916	stagein -O ${frelin[$i]} > STAGEIN_OUTPUT
2917	else
2918	stagein -t -O ${frelin[$i]} > STAGEIN_OUTPUT
2919	fi
2920	cat STAGEIN_OUTPUT
2921	if [[ $(grep -c "st.msg:i24" STAGEIN_OUTPUT) != 0 ]]
2922	then
2923	file_restored=true
2924	do_stagein=false
2925	else
2926	(( stagein_anz = stagein_anz + 1 ))
2927	if (( stagein_anz == 10 ))
2928	then
2929	printf "\n +++ stagein stoped after 10 tries"
2930	locat=stage
2931	exit
2932	fi
2933	printf "\n +++ restoring from archive failed, trying again:"
2934	sleep 900
2935	fi
2936	done
2937	elif [[ $archive_system = DMF ]]
2938	then
2939	if [[ $files_for_pes = false ]]
2940	then
2941	printf "\n +++ restoring of single files impossible with $archive_system !\n"
2942	locat=DMF
2943	exit
2944	else
2945	find $ARCHIVE/${frelin[$i]} -type m -print \| dmget
2946	cp -r $ARCHIVE/${frelin[$i]} $PWD
2947	file_restored=true
2948	fi
2949	elif [[ $archive_system = tivoli ]]
2950	then
2951	if [[ $files_for_pes = false ]]
2952	then
2953	ssh $SSH_PORTOPT $usern@bicedata.hlrn.de "cp $PERM/${frelin[$i]} $PWD"
2954	else
2955	(( inode = 0 ))
2956	while (( inode < nodes ))
2957	do
2958	ssh $SSH_PORTOPT $usern@bicedata.hlrn.de "cd $PWD; tar xf $PERM/${frelin[$i]}/${frelin[$i]}.node_$inode.tar"
2959	(( inode = inode + 1 ))
2960	done
2961	fi
2962	file_restored=true
2963	elif [[ $archive_system = ut ]]
2964	then
2965	if [[ $files_for_pes = false ]]
2966	then
2967	cp $UT/${frelin[$i]} .
2968	else
2969	(( inode = 0 ))
2970	while (( inode < nodes ))
2971	do
2972	tar xf $UT/${frelin[$i]}/${frelin[$i]}.node_$inode.tar
2973	(( inode = inode + 1 ))
2974	done
2975	fi
2976	file_restored=true
2977	else
2978	printf "\n +++ archive_system=\"$archive_system\" restore impossible!"
2979	locat=rearchive
2980	exit
2981	fi
2982
2983	if [[ $file_restored = true ]]
2984	then
2985
2986	# PUT FILE ON THE TEMPORARY DATA DIRECTORY TOO.
2987	# THIS WILL MAKE FILE ACCESS EASIER FOR LATER RUNS
2988	[[ ! -d $tmp_data_catalog ]] && mkdir -p $tmp_data_catalog; chmod g+rx $tmp_data_catalog
2989	if [[ $files_for_pes = false ]]
2990	then
2991	ln -f ${frelin[$i]} $tmp_data_catalog/${frelin[$i]}
2992	else
2993	mkdir $tmp_data_catalog/${frelin[$i]}
2994	ln -f ${frelin[$i]}/* $tmp_data_catalog/${frelin[$i]}
2995	fi
2996	got_tmp[$i]=true
2997
2998	# PROVIDE THE FILE UNDER ITS LOCAL NAME
2999	mv ${frelin[$i]} ${localin[$i]}
3000
3001	fi
3002	fi
3003
3004
3005	# FILE IS STORED IN THE RESPECTIVE DIRECTORY GIVEN IN THE CONFIGURATION FILE
3006	if [[ "${actionin[$i]}" = "" \|\| "${actionin[$i]}" = "di" \|\| "${actionin[$i]}" = "npe" ]]
3007	then
3008
3009	if [[ "${actionin[$i]}" = "npe" && -n $numprocs ]]
3010	then
3011
3012	# FILE COPIES ARE PROVIDED FOR ALL CORES
3013	# EACH FILE GETS A UNIQUE FILENAME WITH A FOUR DIGIT NUMBER
3014	printf "\n file will be provided for $numprocs processors"
3015	mkdir ${localin[$i]}
3016	ival=$numprocs
3017	(( ii = 0 ))
3018	while (( ii <= ival-1 ))
3019	do
3020	if (( ii < 10 ))
3021	then
3022	cp ${absnamein[$i]} ${localin[$i]}/_000$ii
3023	elif (( ii < 100 ))
3024	then
3025	cp ${absnamein[$i]} ${localin[$i]}/_00$ii
3026	elif (( ii < 1000 ))
3027	then
3028	cp ${absnamein[$i]} ${localin[$i]}/_0$ii
3029	else
3030	cp ${absnamein[$i]} ${localin[$i]}/_$ii
3031	fi
3032	(( ii = ii + 1 ))
3033	done
3034
3035	else
3036
3037	if [[ $files_for_pes = true ]]
3038	then
3039
3040	# PROVIDE FILES FOR EACH CORE
3041	# FIRST CREATE THE LOCAL DIRECTORY, THEN COPY FILES
3042	# FROM THE PERMANENT DIRECTORY BY LINKING THEM TO THE LOCAL ONE
3043	printf "\n providing $numprocs files for the respective processors"
3044	mkdir ${localin[$i]}
3045	if [[ $link_local_input = true ]]
3046	then
3047	printf " using ln -f\n"
3048	cd ${absnamein[$i]}
3049	for file in $(ls *)
3050	do
3051	ln -f $file ${localin[$i]}
3052	done
3053	cd $TEMPDIR
3054	fi
3055
3056	# IF "ln -f" FAILED OR IF "$link_local_input = false" DO A NORMAL "cp -r"
3057	if [[ ! -f "${localin[$i]}/_0000" ]]
3058	then
3059	if [[ $link_local_input = true ]]
3060	then
3061	printf " ln failed for .../_0000, using cp...\n"
3062	fi
3063	cp -r ${absnamein[$i]}/* ${localin[$i]}
3064	fi
3065
3066	else
3067
3068	# PROVIDE FILE FOR RUNS ON A SINGLE CORE
3069	if [[ $link_local_input = true ]]
3070	then
3071	printf " using ln -f\n"
3072	ln -f ${absnamein[$i]} ${localin[$i]}
3073	fi
3074	# If "ln -f" fails of if "$link_local_input = false" do a normal "cp"
3075	if [[ ! -f "${localin[$i]}" ]]
3076	then
3077	if [[ $link_local_input = true ]]
3078	then
3079	printf " ln failed, using cp...\n"
3080	fi
3081	cp ${absnamein[$i]} ${localin[$i]}
3082	fi
3083	fi
3084	fi
3085	fi
3086
3087	done
3088	if (( i != 0 ))
3089	then
3090	printf "\n$dashes\n *** all INPUT-files provided \n"
3091	fi
3092
3093
3094	# EXECUTE INPUT-COMMANDS GIVEN IN THE CONFIGURATION FILE
3095	(( i = 0 ))
3096	while (( i < iic ))
3097	do
3098	(( i = i + 1 ))
3099	if (( i == 1 ))
3100	then
3101	printf "\n\n *** execution of INPUT-commands:\n$dashes"
3102	fi
3103	printf "\n >>> ${in_command[$i]}"
3104	eval ${in_command[$i]}
3105	if (( i == iic ))
3106	then
3107	printf "\n$dashes\n"
3108	fi
3109	done
3110
3111
3112	# SET THE REMAINING CPU-TIME
3113	cpurest=${cpumax}.
3114
3115
3116	# START DVR STREAMING SERVER
3117	if [[ $(echo $package_list \| grep -c dvrp_graphics) != 0 ]]
3118	then
3119	if [[ "$dvr_server" != "" ]]
3120	then
3121
3122	printf "\n\n *** preparing the dvr streaming server configuration file"
3123
3124	# CHECK, IF A DVR SERVER IS ALREADY RUNNING
3125	running_dvrserver_id=`echo $(ps -edaf \| grep .dvrserver.config \| grep -v grep) \| cut -d" " -f2`
3126	if [[ "$running_dvrserver_id" != "" ]]
3127	then
3128
3129	printf "\n\n +++ WARNING: A dvr server with id=$running_dvrserver_id is already running!"
3130	printf "\n This server is used instead starting a new one!"
3131	printf "\n If required, script \"process_dvr_output\" has to be run manually."
3132
3133	else
3134
3135	# COPY CONFIGURATION FILE FOR STREAMING SERVER FROM REPOSITORY TO HERE
3136	if [[ -f ${PALM_BIN}/.dvrserver.config ]]
3137	then
3138	cp ${PALM_BIN}/.dvrserver.config .
3139
3140	# ENTERING THE BASEDIR, UID AND GID INTO THIS FILE
3141	user_id=`id -u`
3142	group_id=`id -g`
3143
3144	# "&" IS REQUIRED AS A SEPERATOR, BECAUSE TEMPDIR CONTAINS "/"
3145	sed "s&<replace by dvr data directory>&${TEMPDIR}&g" .dvrserver.config > .dvrserver.1
3146	sed "s/<replace by user id>/$user_id/g" .dvrserver.1 > .dvrserver.2
3147	sed "s/<replace by group id>/$group_id/g" .dvrserver.2 > .dvrserver.3
3148	mv .dvrserver.3 .dvrserver.config
3149	rm .dvrserver.1 .dvrserver.2
3150
3151	# START DVR SERVER IN BACKGROUND, GET HIS ID AND PRINT ON TERMINAL
3152	$dvr_server .dvrserver.config >> DVR_LOGFILE 2>&1 &
3153	dvrserver_id=`echo $(ps -edaf \| grep .dvrserver.config) \| cut -d" " -f2`
3154	printf "\n *** streaming server with id=$dvrserver_id is started in background"
3155	local_dvrserver_running=.TRUE.
3156	else
3157	printf "\n +++ missing file \".dvrserver.config\" in directory:"
3158	printf "\n \"$PALM_BIN\" "
3159	locat=dvr
3160	exit
3161	fi
3162
3163	fi
3164
3165	else
3166	printf "\n\n --- INFORMATIVE: no dvr streaming server will be started"
3167	fi
3168	fi
3169
3170
3171	# CREATE THE NAMELIST-FILE WITH VALUES OF ENVIRONMENT-VARIABLES REQUIRED BY PALM
3172	# (FILE ENVPAR WILL BE READ BY PALM)
3173	cat > ENVPAR << %%END%%
3174	&envpar run_identifier = '$fname', host = '$localhost',
3175	write_binary = '$write_binary', tasks_per_node = $tasks_per_node,
3176	maximum_parallel_io_streams = $maximum_parallel_io_streams,
3177	maximum_cpu_time_allowed = ${cpumax}.,
3178	revision = '$global_revision',
3179	local_dvrserver_running = $local_dvrserver_running /
3180
3181	%%END%%
3182
3183
3184	# STARTING THE EXECUTABLE
3185	printf "\n\n *** execution starts in directory\n \"`pwd`\"\n$dashes\n"
3186	PATH=$PATH:$TEMPDIR
3187
3188	# MPI DEBUG OPTION (ARGUMENT CHECKING, SLOWS DOWN EXECUTION DUE TO INCREASED LATENCY)
3189	if [[ "$mpi_debug" = true ]]
3190	then
3191	export MPI_CHECK_ARGS=1
3192	printf "\n +++ MPI_CHECK_ARGS=$MPI_CHECK_ARGS"
3193	fi
3194
3195	if [[ "$totalview" = true ]]
3196	then
3197	printf "\n *** totalview debugger will be used"
3198	tv_opt="-tv"
3199	else
3200	tv_opt=""
3201	fi
3202
3203	if [[ "$cond1" = debug \|\| "$cond2" = debug ]]
3204	then
3205	if [[ "$ENVIRONMENT" = BATCH ]]
3206	then
3207	continue
3208	fi
3209	if [[ $localhost = ibmh ]]
3210	then
3211
3212	# SETUP THE IBM MPI ENVIRONMENT
3213	export MP_SHARED_MEMORY=yes
3214	export AIXTHREADS_SCOPE=S
3215	export OMP_NUM_THREADS=$threads_per_task
3216	export AUTHSTATE=files
3217	export XLFRTEOPTS="nlwidth=132:err_recovery=no" # RECORD-LENGTH OF NAMELIST-OUTPUT
3218
3219	# FOLLOWING OPTIONS ARE MANDATORY FOR TOTALVIEW
3220	export MP_ADAPTER_USE=shared
3221	export MP_CPU_USE=multiple
3222	export MP_TIMEOUT=1200
3223
3224	unset MP_TASK_AFFINITY
3225
3226	# SO FAR, TOTALVIEW NEEDS HOSTFILE MECHANISM FOR EXECUTION
3227	#(( ii = 1 ))
3228	#while (( ii <= $numprocs ))
3229	#do
3230	# echo $localhost_realname >> hostfile
3231	# (( ii = ii + 1 ))
3232	#done
3233	#export MP_HOSTFILE=hostfile
3234
3235	if [[ "$LOADLBATCH" = yes ]]
3236	then
3237	totalview poe a.out $ROPTS
3238	else
3239	echo totalview poe -a a.out -procs $numprocs -rmpool 0 -nodes 1 $ROPTS
3240	export TVDSVRLAUNCHCMD=ssh
3241	totalview poe -a a.out -procs $numprocs -rmpool 0 -nodes 1 $ROPTS
3242	fi
3243
3244	fi # END DEBUG MODE
3245
3246	else
3247
3248	# NORMAL EXECUTION
3249	if [[ -n $numprocs ]]
3250	then
3251
3252	# RUNNING THE PROGRAM ON PARALLEL MACHINES
3253	if [[ $(echo $host \| cut -c1-3) = ibm ]]
3254	then
3255
3256	# SETUP THE IBM MPI ENVIRONMENT
3257	if [[ $host != ibmh && $host != ibmkisti ]]
3258	then
3259	export MP_SHARED_MEMORY=yes
3260	export AIXTHREAD_SCOPE=S
3261	export OMP_NUM_THREADS=$threads_per_task
3262	export XLSMPOPTS="spins=0:yields=0:stack=20000000"
3263	export AUTHSTATE=files
3264	export XLFRTEOPTS="nlwidth=132:err_recovery=no" # RECORD-LENGTH OF NAMELIST-OUTPUT
3265	# export MP_PRINTENV=yes
3266
3267	# TUNING-VARIABLES TO IMPROVE COMMUNICATION SPEED
3268	# DO NOT SHOW SIGNIFICANT EFFECTS (SEP 04, FEDERATION-SWITCH)
3269	export MP_WAIT_MODE=poll
3270	[[ $node_usage = not_shared ]] && export MP_SINGLE_THREAD=yes
3271	fi
3272
3273	if [[ $host = ibmkisti ]]
3274	then
3275	export LANG=en_US
3276	export MP_SHARED_MEMORY=yes
3277	if [[ $threads_per_task = 1 ]]
3278	then
3279	export MP_SINGLE_THREAD=yes
3280	export MEMORY_AFFINITY=MCM
3281	else
3282	export OMP_NUM_THREADS=$threads_per_task
3283	fi
3284	fi
3285
3286	if [[ "$LOADLBATCH" = yes ]]
3287	then
3288	printf "\n--- Control: OMP_NUM_THREADS = \"$OMP_NUM_THREADS\" \n"
3289	if [[ "$cond1" = hpmcount \|\| "$cond2" = hpmcount ]]
3290	then
3291	/opt/optibm/HPM_2_4_1/bin/hpmcount a.out $ROPTS
3292	else
3293	if [[ $run_coupled_model = false ]]
3294	then
3295	if [[ "$ocean_file_appendix" = true ]]
3296	then
3297	echo "precursor_ocean" > runfile_atmos
3298	else
3299	echo "precursor_atmos" > runfile_atmos
3300	fi
3301	else
3302	(( iia = $numprocs_atmos / $threads_per_task ))
3303	(( iio = $numprocs_ocean / $threads_per_task ))
3304	printf "\n coupled run ($iia atmosphere, $iio ocean)"
3305	printf "\n using $coupled_mode coupling"
3306	printf "\n\n"
3307	echo "coupled_run $iia $iio" > runfile_atmos
3308	fi
3309	poe ./a.out $ROPTS < runfile_atmos
3310	fi
3311	else
3312	if [[ $localhost = ibmh \|\| $localhost = ibms ]]
3313	then
3314	poe a.out -procs $numprocs -nodes 1 -rmpool 0 $ROPTS
3315	elif [[ $localhost = ibmkisti \|\| $localhost = ibmku ]]
3316	then
3317	if [[ -f $hostfile ]]
3318	then
3319	cp $hostfile hostfile
3320	else
3321	(( ii = 1 ))
3322	while (( ii <= $numprocs ))
3323	do
3324	echo $localhost_realname >> hostfile
3325	(( ii = ii + 1 ))
3326	done
3327	fi
3328	export MP_HOSTFILE=hostfile
3329	if [[ $run_coupled_model = false ]]
3330	then
3331	if [[ "$ocean_file_appendix" = true ]]
3332	then
3333	echo "precursor_ocean" > runfile_atmos
3334	else
3335	echo "precursor_atmos" > runfile_atmos
3336	fi
3337	else
3338	(( iia = $numprocs_atmos / $threads_per_task ))
3339	(( iio = $numprocs_ocean / $threads_per_task ))
3340	printf "\n coupled run ($iia atmosphere, $iio ocean)"
3341	printf "\n using $coupled_mode coupling"
3342	printf "\n\n"
3343	echo "coupled_run $iia $iio" > runfile_atmos
3344	fi
3345
3346	poe ./a.out -procs $numprocs $ROPTS < runfile_atmos
3347
3348	else
3349	if [[ "$host_file" = "" ]]
3350	then
3351	printf "\n +++ no hostfile given in configuration file"
3352	locat=config_file
3353	exit
3354	else
3355	eval host_file=$host_file
3356	fi
3357	export MP_HOSTFILE=$host_file
3358	poe a.out -procs $numprocs -tasks_per_node $numprocs $ROPTS
3359	fi
3360	fi
3361	elif [[ $host = nech ]]
3362	then
3363	(( ii = nodes ))
3364	if [[ $ii = 1 ]]
3365	then
3366	export F_ERRCNT=0 # acceptable number of errors before program is stopped
3367	export MPIPROGINF=YES
3368	# export F_TRACE=YES\|FMT1\|FMT2 # output of ftrace informations to job protocol
3369	echo "*** execution on single node with mpirun"
3370	mpirun -np $numprocs ./a.out $ROPTS
3371	else
3372	(( i = 0 ))
3373	while (( i < ii ))
3374	do
3375	echo "-h $i -p $tasks_per_node -e ./mpi_exec_shell" >> multinode_config
3376	(( i = i + 1 ))
3377	done
3378
3379	echo "#!/bin/sh" > mpi_exec_shell
3380	echo " " >> mpi_exec_shell
3381	echo "set -u" >> mpi_exec_shell
3382	echo "F_ERRCNT=0" >> mpi_exec_shell
3383	echo "MPIPROGINV=YES" >> mpi_exec_shell
3384	echo "OMP_NUM_THREADS=$threads_per_task" >> mpi_exec_shell
3385	echo "cpurest=$cpurest" >> mpi_exec_shell
3386	echo "fname=$fname" >> mpi_exec_shell
3387	echo "localhost=$localhost" >> mpi_exec_shell
3388	echo "return_addres=$return_addres" >> mpi_exec_shell
3389	echo "return_username=$return_username" >> mpi_exec_shell
3390	echo "tasks_per_node=$tasks_per_node" >> mpi_exec_shell
3391	echo "write_binary=$write_binary" >> mpi_exec_shell
3392	echo "use_seperate_pe_for_dvrp_output=$use_seperate_pe_for_dvrp_output" >> mpi_exec_shell
3393	echo " " >> mpi_exec_shell
3394	echo "export F_ERRCNT" >> mpi_exec_shell
3395	echo "export MPIPROGINV" >> mpi_exec_shell
3396	echo "export OMP_NUM_THREADS" >> mpi_exec_shell
3397	echo "export cpurest" >> mpi_exec_shell
3398	echo "export fname" >> mpi_exec_shell
3399	echo "export localhost" >> mpi_exec_shell
3400	echo "export return_addres" >> mpi_exec_shell
3401	echo "export return_username" >> mpi_exec_shell
3402	echo "export tasks_per_node" >> mpi_exec_shell
3403	echo "export write_binary" >> mpi_exec_shell
3404	echo "export use_seperate_pe_for_dvrp_output" >> mpi_exec_shell
3405	echo " " >> mpi_exec_shell
3406	echo "exec ./a.out" >> mpi_exec_shell
3407
3408	chmod u+x mpi_exec_shell
3409	export MPIPROGINF=YES
3410	mpirun -f multinode_config &
3411	wait
3412
3413	fi
3414	elif [[ $(echo $host \| cut -c1-2) = lc && $host != lckyoto && $host != lctit ]]
3415	then
3416
3417	# COPY HOSTFILE FROM SOURCE DIRECTORY OR CREATE IT, IF IT
3418	# DOES NOT EXIST
3419	if [[ $host != lccrayb &&$host != lccrayf && $host != lccrayh && $host != lckyuh && $host != lckyut ]]
3420	then
3421	if [[ -f $hostfile ]]
3422	then
3423	cp $hostfile hostfile
3424	(( ii = $numprocs / $threads_per_task ))
3425	else
3426	(( ii = 1 ))
3427	while (( ii <= $numprocs / $threads_per_task ))
3428	do
3429	echo $localhost_realname >> hostfile
3430	(( ii = ii + 1 ))
3431	done
3432	fi
3433	eval zeile=\"`head -n $ii hostfile`\"
3434	printf "\n *** running on: $zeile"
3435	fi
3436
3437	(( ii = $numprocs / $threads_per_task ))
3438	export OMP_NUM_THREADS=$threads_per_task
3439
3440	if [[ $threads_per_task != 1 ]]
3441	then
3442	# INCREASE STACK SIZE TO UNLIMITED, BECAUSE OTHERWISE LARGE RUNS
3443	# MAY ABORT
3444	ulimit -s unlimited
3445	printf "\n threads per task: $threads_per_task stacksize: unlimited"
3446	fi
3447	if [[ $run_coupled_model = false ]]
3448	then
3449	if [[ "$ocean_file_appendix" = true ]]
3450	then
3451	echo "precursor_ocean" > runfile_atmos
3452	else
3453	echo "precursor_atmos" > runfile_atmos
3454	fi
3455	printf "\n\n"
3456
3457	if [[ $host = lccrayb \|\| $host = lccrayh ]]
3458	then
3459	echo "--- aprun -n $ii -N $tasks_per_node a.out $ROPTS < runfile_atmos"
3460	aprun -n $ii -N $tasks_per_node a.out $ROPTS < runfile_atmos
3461	# aprun -n $ii -ss -r2 -j1 a.out $ROPTS < runfile_atmos
3462	elif [[ $host = lccrayf ]]
3463	then
3464	aprun -n $ii -N $tasks_per_node -m ${memory}M a.out $ROPTS < runfile_atmos
3465	elif [[ $host = lcxe6 \|\| $host = lcxt5m ]]
3466	then
3467	aprun -n $ii -N $tasks_per_node a.out $ROPTS < runfile_atmos
3468	elif [[ $host = lcflow ]]
3469	then
3470	mpirun -bootstrap sge -np $ii -env I_MPI_FABRICS shm:ofa a.out < runfile_atmos $ROPTS
3471	elif [[ $host = lcsb ]]
3472	then
3473	mpirun_rsh -hostfile $PBS_NODEFILE -np `cat $PBS_NODEFILE \| wc -l` a.out < runfile_atmos $ROPTS
3474	elif [[ $host = lckiaps ]]
3475	then
3476	mpirun -np $ii -f $PBS_NODEFILE a.out < runfile_atmos $ROPTS
3477	elif [[ $host = lckyu* ]]
3478	then
3479	mpiexec -n $ii --stdin runfile_atmos ./a.out $ROPTS
3480	else
3481	mpiexec -machinefile hostfile -n $ii a.out < runfile_atmos $ROPTS
3482	fi
3483	else
3484
3485	# CURRENTLY THERE IS NO FULL MPI-2 SUPPORT ON ICE AND XT4
3486	(( iia = $numprocs_atmos / $threads_per_task ))
3487	(( iio = $numprocs_ocean / $threads_per_task ))
3488	printf "\n coupled run ($iia atmosphere, $iio ocean)"
3489	printf "\n using $coupled_mode coupling"
3490	printf "\n\n"
3491
3492	if [[ $coupled_mode = "mpi2" ]]
3493	then
3494	echo "atmosphere_to_ocean $iia $iio" > runfile_atmos
3495	echo "ocean_to_atmosphere $iia $iio" > runfile_ocean
3496
3497	if [[ $host = lccrayf \|\| $host = lcxe6 \|\| $host = lcxt5m ]]
3498	then
3499
3500	aprun -n $iia -N $tasks_per_node a.out < runfile_atmos $ROPTS &
3501	aprun -n $iio -N $tasks_per_node a.out < runfile_ocean $ROPTS &
3502
3503	else
3504	# WORKAROUND BECAUSE mpiexec WITH -env option IS NOT AVAILABLE ON SOME SYSTEMS
3505	mpiexec -machinefile hostfile -n $iia a.out $ROPTS < runfile_atmos &
3506	mpiexec -machinefile hostfile -n $iio a.out $ROPTS < runfile_ocean &
3507	# mpiexec -machinefile hostfile -n $iia -env coupling_mode atmosphere_to_ocean a.out $ROPTS &
3508	# mpiexec -machinefile hostfile -n $iio -env coupling_mode ocean_to_atmosphere a.out $ROPTS &
3509	fi
3510	wait
3511
3512	else
3513
3514	echo "coupled_run $iia $iio" > runfile_atmos
3515
3516	if [[ $host = lccrayf \|\| $host = lcxe6 \|\| $host = lcxt5m ]]
3517	then
3518
3519	aprun -n $ii -N $tasks_per_node a.out < runfile_atmos $ROPTS
3520
3521	elif [[ $host = lck \|\| $host = lckordi ]]
3522	then
3523
3524	mpiexec -n $ii ./a.out $ROPTS < runfile_atmos &
3525
3526	elif [[ $host = lckyu* ]]
3527	then
3528
3529	mpiexec -n $ii --stdin runfile_atmos ./a.out $ROPTS
3530
3531	fi
3532	wait
3533	fi
3534
3535	fi
3536
3537	elif [[ $host = lckyoto ]]
3538	then
3539	set -xv
3540	export P4_RSHCOMMAND=plesh
3541	echo " P4_RSHCOMMAND = $P4_RSHCOMMAND"
3542	if [[ "$ENVIRONMENT" = BATCH ]]
3543	then
3544	if [[ "$cond2" = fujitsu ]]
3545	then
3546	mpiexec -n $numprocs ./a.out $ROPTS # for fujitsu-compiler
3547	elif [[ "cond2" = pgi ]]
3548	then
3549	mpirun -np $numprocs -machinefile ${QSUB_NODEINF} ./a.out $ROPTS
3550	else
3551	mpirun_rsh -np $numprocs -hostfile ${QSUB_NODEINF} MV2_USE_SRQ=0 ./a.out ${ROPTS} \|\| /bin/true
3552	fi
3553	else
3554	if [[ "$cond2" = "" ]]
3555	then
3556	mpiruni_rsh -np $numprocs ./a.out $ROPTS # for intel
3557	else
3558	mpirun -np $numprocs ./a.out $ROPTS
3559	fi
3560	fi
3561	set +xv
3562
3563	elif [[ $host = lctit ]]
3564	then
3565	export OMP_NUM_THREADS=$threads_per_task
3566	echo "OMP_NUM_THREADS=$OMP_NUM_THREADS"
3567	if [[ "$threads_per_task" != 1 ]]
3568	then
3569	export MV2_ENABLE_AFFINITY=0
3570	fi
3571	echo "----- PBS_NODEFILE content:"
3572	cat $PBS_NODEFILE
3573	echo "-----"
3574	(( ii = $numprocs / $threads_per_task ))
3575	echo "mpirun -np $ii -hostfile $PBS_NODEFILE ./a.out $ROPTS"
3576	mpirun -np $ii -hostfile $PBS_NODEFILE ./a.out $ROPTS
3577	else
3578	mpprun -n $numprocs a.out $ROPTS
3579	fi
3580	else
3581	a.out $ROPTS
3582	fi
3583	fi
3584
3585	if [[ $? != 0 ]]
3586	then
3587
3588	# ABORT IN CASE OF RUNTIME ERRORS
3589	printf "\n +++ runtime error occured"
3590	locat=execution
3591	exit
3592
3593	else
3594
3595	printf "\n$dashes\n *** execution finished \n"
3596
3597	# STOP THE DVR STREAMING SERVER AND PROCESS THE DVR OUTPUT IN ORDER
3598	# TO CREAT DVRS- AND HTML-FILES CONTAINING ALL STREAMS
3599	if [[ "$dvrserver_id" != "" ]]
3600	then
3601	kill $dvrserver_id
3602	printf "\n *** dvr server with id=$dvrserver_id has been stopped"
3603
3604	# IF THERE IS A DIRECTORY, DATA HAVE BEEN OUTPUT BY THE
3605	# STREAMING SERVER. OTHERWISE, USER HAS CHOSEN dvrp_output=local
3606	if [[ -d DATA_DVR ]]
3607	then
3608
3609	# ADD THE CURRENT DVR CONFIGURATION FILE TO THE DVR OUTPUT
3610	# DIRECTORY
3611	cp .dvrserver.config DATA_DVR
3612
3613	# PROCESS THE DVR OUTPUT (OPTION -s FOR GENERATING
3614	# SEQUENCE MODE DATA TOO)
3615	process_dvr_output -d DATA_DVR -f $fname -s
3616
3617	else
3618
3619	# PROCESS THE LOCAL OUTPUT
3620	process_dvr_output -l -d DATA_DVR -f $fname
3621
3622	fi
3623
3624	elif [[ $(echo $package_list \| grep -c dvrp_graphics) != 0 ]]
3625	then
3626
3627	# PROCESS DVR OUTPUT GENERATD IN LOCAL MODE (dvrp_output=local)
3628	process_dvr_output -l -d DATA_DVR -f $fname
3629
3630	fi
3631	fi
3632
3633
3634
3635	# CALL OF combine_plot_fields IN ORDER TO MERGE SINGLE FILES WRITTEN
3636	# BY EACH CORE INTO ONE FILE
3637	if [[ ! -f ${PALM_BIN}/combine_plot_fields${block}.x ]]
3638	then
3639	printf "\n\n\n +++ WARNING: no combine_plot_fields found for given block \"$cond1 $cond2\""
3640	printf "\n 2d- and/or 3d-data may be incomplete!"
3641	printf "\n Run \"mbuild -u -h $localhost\" to generate utilities for this block.\n"
3642	elif [[ "$combine_plot_fields" == true ]]
3643	then
3644	if [[ $localhost = lccrayh \|\| $localhost = lccrayb ]]
3645	then
3646	printf "\n\n\n *** post-processing: now executing \"aprun -n 1 -N 1 combine_plot_fields${block}.x\" ..."
3647	aprun -n 1 -N 1 combine_plot_fields${block}.x #$ROPTS < runfile_atmos
3648	else
3649	printf "\n\n\n *** post-processing: now executing \"combine_plot_fields${block}.x\" ..."
3650	combine_plot_fields${block}.x
3651	fi
3652	else
3653
3654	# TEMPORARY SOLUTION TO SKIP combine_plot_fields. THIS IS REQUIRED IN CASE OF HUGE AMOUNT OF
3655	# DATA OUTPUT. TO DO: EXTEND THIS BRANCH BY CREATING A BATCH JOB for combine_plot_fields.
3656	# ??? koennen wir das streichen ???
3657	printf "\n\n\n *** post-processing: skipping combine_plot_fields (-Z option set) ..."
3658	fi
3659
3660
3661
3662	# EXECUTE OUTPUT-COMMANDS GIVEN IN THE CONFIGURATION FILE
3663	(( i = 0 ))
3664	while (( i < ioc ))
3665	do
3666	(( i = i + 1 ))
3667	if (( i == 1 ))
3668	then
3669	printf "\n\n *** execution of OUTPUT-commands:\n$dashes"
3670	fi
3671	printf "\n >>> ${out_command[$i]}"
3672	eval ${out_command[$i]}
3673	if (( i == ioc ))
3674	then
3675	printf "\n$dashes\n"
3676	fi
3677	done
3678
3679
3680	# IN TRACE-MODE PRINT CONTENTS OF THE CURRENT (TEMPORARY) WORKING DIRECTORY
3681	if [[ $do_trace = true ]]
3682	then
3683	printf "\n\n"
3684	ls -al
3685	fi
3686
3687
3688
3689	# COPY LOCAL OUTPUT-FILES TO THEIR PERMANENT DESTINATIONS
3690	(( i = 0 ))
3691	while (( i < iout ))
3692	do
3693	(( i = i + 1 ))
3694	if (( i == 1 ))
3695	then
3696	printf "\n\n *** saving OUTPUT-files:\n$dashes"
3697	fi
3698
3699	# CHECK FOR SINGLE FILE (SERIAL RUN) OR DIRECTORY (ONE FILE PER CORE FOR PARELLEL EXECUTION)
3700	files_for_pes=false; filetyp=file
3701	if [[ "${actionout[$i]}" = pe && -n $numprocs ]]
3702	then
3703	files_for_pes=true; filetyp=directory
3704	actionout[$i]=""
3705	elif [[ "${actionout[$i]}" = pe && ! -n $numprocs ]]
3706	then
3707	actionout[$i]=""
3708	elif [[ "${actionout[$i]}" = arpe && -n $numprocs ]]
3709	then
3710	files_for_pes=true; filetyp=directory
3711	actionout[$i]="ar"
3712	elif [[ "${actionout[$i]}" = arpe && ! -n $numprocs ]]
3713	then
3714	actionout[$i]="ar"
3715	elif [[ "${actionout[$i]}" = flpe && -n $numprocs ]]
3716	then
3717	files_for_pes=true; filetyp=directory
3718	actionout[$i]="fl"
3719	elif [[ "${actionout[$i]}" = flpe && ! -n $numprocs ]]
3720	then
3721	actionout[$i]="fl"
3722	elif [[ "${actionout[$i]}" = trpe && -n $numprocs ]]
3723	then
3724	files_for_pes=true; filetyp=directory
3725	actionout[$i]="tr"
3726	elif [[ "${actionout[$i]}" = trpe && ! -n $numprocs ]]
3727	then
3728	actionout[$i]="tr"
3729	fi
3730
3731	if [[ ! -f ${localout[$i]} && $files_for_pes = false ]]
3732	then
3733	printf "\n +++ temporary OUTPUT-file ${localout[$i]} does not exist\n"
3734	elif [[ ! -d ${localout[$i]} && $files_for_pes = true ]]
3735	then
3736	printf "\n +++ temporary OUTPUT-file ${localout[$i]}/.... does not exist\n"
3737	else
3738
3739
3740	# COPY VIA SCP TO LOCAL HOST (ALWAYS IN BINARY MODE USING batch_scp option -m)
3741	# IF TARGET DIRECTORY DOES NOT EXISTS, TRY TO CREATE IT
3742	if [[ "${actionout[$i]}" = tr ]]
3743	then
3744	if [[ $localhost != $fromhost ]]
3745	then
3746	if [[ $files_for_pes = false ]]
3747	then
3748	cps=""
3749	cst=""
3750	else
3751	cps=-c
3752	cst="/"
3753	fi
3754	if [[ $localhost = nech ]]
3755	then
3756
3757	# TRANSFER IN SEPERATE JOB
3758	# FIRST COPY FILE TO TEMPORY DATA DIRECTORY
3759	[[ ! -d $tmp_data_catalog/TRANSFER ]] && mkdir -p $tmp_data_catalog/TRANSFER
3760	file_to_transfer=${fname}_${localout[$i]}_to_transfer_$kennung
3761	if [[ $files_for_pes = false ]]
3762	then
3763	ln -f ${localout[$i]} $tmp_data_catalog/TRANSFER/$file_to_transfer
3764	else
3765	mkdir $tmp_data_catalog/TRANSFER/$file_to_transfer
3766	ln ${localout[$i]}/* $tmp_data_catalog/TRANSFER/$file_to_transfer
3767	fi
3768
3769	echo "set -x" > transfer_${localout[$i]}
3770	echo "cd $tmp_data_catalog/TRANSFER" >> transfer_${localout[$i]}
3771
3772	printf "\n >>> OUTPUT: ${localout[$i]}$cst by SCP in seperate job to"
3773	printf "\n ${pathout[$i]}/${localhost}_${fname}${endout[$i]}$cst"
3774	printf "\n or higher cycle\n"
3775	echo "batch_scp $PORTOPT $cps -b -m -u $return_username $return_addres $file_to_transfer \"${pathout[$i]}\" ${localhost}_${fname}${endout[$i]} ${extout[$i]}" >> transfer_${localout[$i]}
3776
3777	echo "[[ \$? = 0 ]] && rm $file_to_transfer" >> transfer_${localout[$i]}
3778
3779	if [[ $localhost = nech ]]
3780	then
3781	subjob -d -c /pf/b/$usern/job_queue -v -q pp -X 0 -m 1000 -t 900 $PORTOPT transfer_${localout[$i]}
3782	else
3783	if [[ "$LOGNAME" = b323013 ]]
3784	then
3785	subjob -v -q c1 -X 0 -m 1000 -t 900 -c $job_catalog $PORTOPT transfer_${localout[$i]}
3786	else
3787	subjob -d -v -q c1 -X 0 -m 1000 -t 900 -c $job_catalog $PORTOPT transfer_${localout[$i]}
3788	fi
3789	fi
3790
3791	else
3792
3793	# TRANSFER WITHIN THIS JOB
3794	transfer_failed=false
3795	printf "\n >>> OUTPUT: ${localout[$i]}$cst by SCP to"
3796	printf "\n ${pathout[$i]}/${localhost}_${fname}${endout[$i]}$cst\n"
3797	if [[ $localhost = lccrayb ]]
3798	then
3799	ssh $usern@blogin1 "cd $TEMPDIR; batch_scp $PORTOPT $cps -b -m -u $return_username $return_addres ${localout[$i]} \"${pathout[$i]}\" ${localhost}_${fname}${endout[$i]} ${extout[$i]}"
3800	elif [[ $localhost = lccrayh ]]
3801	then
3802	ssh $usern@hlogin1 "cd $TEMPDIR; batch_scp $PORTOPT $cps -b -m -u $return_username $return_addres ${localout[$i]} \"${pathout[$i]}\" ${localhost}_${fname}${endout[$i]} ${extout[$i]}"
3803	else
3804	batch_scp $PORTOPT $cps -b -m -u $return_username $return_addres ${localout[$i]} "${pathout[$i]}" ${localhost}_${fname}${endout[$i]} ${extout[$i]}
3805	fi
3806	[[ $? != 0 ]] && transfer_failed=true
3807
3808	# IF TRANSFER FAILED, CREATE BACKUP COPY ON THIS MACHINE
3809	if [[ $transfer_failed = true ]]
3810	then
3811	printf " +++ transfer failed. Trying to save a copy on this host under:\n"
3812	printf " ${pathout[$i]}/${localhost}_${fname}${endout[$i]}_$kennung\n"
3813
3814	# FIRST CHECK, IF DIRECTORY EXISTS, AND CREATE IT, IF NECESSARY
3815	eval local_catalog=${pathout[$i]}
3816	if [[ ! -d $local_catalog ]]
3817	then
3818	printf " *** local directory does not exist. Trying to create:\n"
3819	printf " $local_catalog \n"
3820	mkdir -p $local_catalog
3821	fi
3822	eval cp ${localout[$i]} ${pathout[$i]}/${localhost}_${fname}${endout[$i]}_$kennung
3823	transfer_problems=true
3824	fi
3825	fi
3826	else
3827
3828	# UNSET actionout. DUE TO THIS SETTING, FILE WILL LATER JUST BE COPIED ON THIS MACHINE
3829	actionout[$i]=""
3830	fi
3831	fi
3832
3833
3834	# APPEND VIA SCP TO LOCAL HOST (ALWAYS IN BINARY MODE USING batch_scp option -m)
3835	# IF TARGET DIRECTORY DOES NOT EXISTS, TRY TO CREATE IT
3836	if [[ "${actionout[$i]}" = tra ]]
3837	then
3838	if [[ $localhost != $fromhost ]]
3839	then
3840	if [[ $localhost = ibmh \|\| $localhost = nech ]]
3841	then
3842
3843	# TRANSFER IN SEPERATE JOB
3844	# FIRST COPY FILE TO TEMPORY DATA DIRECTORY
3845	[[ ! -d $tmp_data_catalog/TRANSFER ]] && mkdir -p $tmp_data_catalog/TRANSFER
3846	file_to_transfer=${fname}_${localout[$i]}_to_transfer_$kennung
3847	ln -f ${localout[$i]} $tmp_data_catalog/TRANSFER/$file_to_transfer
3848
3849	echo "set -x" > transfer_${localout[$i]}
3850	echo "cd $tmp_data_catalog/TRANSFER" >> transfer_${localout[$i]}
3851
3852	printf "\n >>> OUTPUT: ${localout[$i]} append by SCP in seperate job to"
3853	printf "\n ${pathout[$i]}/${localhost}_${fname}${endout[$i]}"
3854	printf "\n or higher cycle\n"
3855	echo "batch_scp $PORTOPT -A -b -m -u $return_username $return_addres $file_to_transfer \"${pathout[$i]}\" ${localhost}_${fname}${endout[$i]} ${extout[$i]}" >> transfer_${localout[$i]}
3856
3857	echo "[[ \$? = 0 ]] && rm $file_to_transfer" >> transfer_${localout[$i]}
3858
3859	if [[ $localhost = nech ]]
3860	then
3861	subjob -d -c /pf/b/$usern/job_queue -v -q pp -X 0 -m 1000 -t 900 $PORTOPT transfer_${localout[$i]}
3862	else
3863	if [[ $LOGNAME = b323013 ]]
3864	then
3865	subjob -v -q c1 -X 0 -m 1000 -t 900 -c $job_catalog $PORTOPT transfer_${localout[$i]}
3866	else
3867	subjob -d -v -q c1 -X 0 -m 1000 -t 900 -c $job_catalog $PORTOPT transfer_${localout[$i]}
3868	fi
3869	fi
3870
3871	else
3872
3873	# TRANSFER WITHIN THIS JOB
3874	transfer_failed=false
3875	printf "\n >>> OUTPUT: ${localout[$i]} append by SCP to"
3876	printf "\n ${pathout[$i]}/${localhost}_${fname}${endout[$i]}\n"
3877	if [[ $localhost = lccrayb ]]
3878	then
3879	ssh $usern@blogin1 "cd $TEMPDIR; batch_scp $PORTOPT -A -b -m -u $return_username $return_addres ${localout[$i]} \"${pathout[$i]}\" ${localhost}_${fname}${endout[$i]} ${extout[$i]}"
3880	elif [[ $localhost = lccrayh ]]
3881	then
3882	ssh $usern@hlogin1 "cd $TEMPDIR; batch_scp $PORTOPT -A -b -m -u $return_username $return_addres ${localout[$i]} \"${pathout[$i]}\" ${localhost}_${fname}${endout[$i]} ${extout[$i]}"
3883	else
3884	batch_scp $PORTOPT -A -b -m -u $return_username $return_addres ${localout[$i]} "${pathout[$i]}" ${localhost}_${fname}${endout[$i]} ${extout[$i]}
3885	fi
3886	[[ $? != 0 ]] && transfer_failed=true
3887
3888	# IF TRANSFER FAILED, CREATE BACKUP COPY ON THIS MACHINE
3889	if [[ $transfer_failed = true ]]
3890	then
3891	printf " +++ transfer failed. Trying to save a copy on this host under:\n"
3892	printf " ${pathout[$i]}/${localhost}_${fname}${endout[$i]}_$kennung\n"
3893
3894	# FIRST CHECK, IF DIRECTORY EXISTS, AND CREATE IT, IF NECESSARY
3895	eval local_catalog=${pathout[$i]}
3896	if [[ ! -d $local_catalog ]]
3897	then
3898	printf " *** local directory does not exist. Trying to create:\n"
3899	printf " $local_catalog \n"
3900	mkdir -p $local_catalog
3901	fi
3902
3903	eval cp ${localout[$i]} ${pathout[$i]}/${localhost}_${fname}${endout[$i]}_$kennung
3904	transfer_problems=true
3905	fi
3906	fi
3907	else
3908
3909	# RESET actionout. DUE TO THIS SETTING, FILE WILL LATER JUST BE APPENDED ON THIS MACHINE
3910	actionout[$i]=a
3911	fi
3912	fi
3913
3914
3915	# OUTPUT-FILE FOR A RESTART-RUN.
3916	# FILE IS MOVED TO THE TEMPORARY DATA DIRECTORY USING link COMMAND.
3917	# OPTION -f IS USED TO OVERWRITE EXISTING FILES.
3918	# AS LAST ACTION, ARCHIVING IS INITIATED.
3919	if [[ "${actionout[$i]}" = fl ]]
3920	then
3921	[[ ! -d $tmp_data_catalog ]] && mkdir -p $tmp_data_catalog
3922	chmod g+rx $tmp_data_catalog
3923	if [[ $files_for_pes = false ]]
3924	then
3925	printf "\n >>> OUTPUT: ${localout[$i]} to"
3926	printf "\n $tmp_data_catalog/${frelout[$i]} (temporary data catalog)\n"
3927	ln -f ${localout[$i]} $tmp_data_catalog/${frelout[$i]}
3928	else
3929	printf "\n >>> OUTPUT: ${localout[$i]}/.... to"
3930	printf "\n $tmp_data_catalog/${frelout[$i]} (temporary data catalog)\n"
3931	mkdir $tmp_data_catalog/${frelout[$i]}
3932	cd ${localout[$i]}
3933	for file in $(ls *)
3934	do
3935	ln -f $file $tmp_data_catalog/${frelout[$i]}
3936	done
3937	cd $TEMPDIR
3938	fi
3939
3940
3941	# CREATE AND SUBMIT ARCHIVING-JOB
3942	if [[ $store_on_archive_system = true ]]
3943	then
3944
3945	if [[ $archive_system = asterix ]]
3946	then
3947	echo "cd $tmp_data_catalog" >> archive_${frelout[$i]}
3948	if [[ $files_for_pes = false ]]
3949	then
3950	echo "stageout ${frelout[$i]} > STAGE_OUTPUT${i}_$kennung" >> archive_${frelout[$i]}
3951	else
3952	echo "stageout -t ${frelout[$i]} > STAGE_OUTPUT${i}_$kennung" >> archive_${frelout[$i]}
3953	fi
3954	echo "cat STAGE_OUTPUT${i}_$kennung" >> archive_${frelout[$i]}
3955	echo "if [[ \$(grep -c \"st.msg:150\" STAGE_OUTPUT${i}_$kennung) != 0 ]]" >> archive_${frelout[$i]}
3956	echo "then" >> archive_${frelout[$i]}
3957	echo " do_stageout=false" >> archive_${frelout[$i]}
3958	echo "else" >> archive_${frelout[$i]}
3959	echo " echo \" +++ $filetyp ${frelout[$i]} could not be stored on archive-system\" " >> archive_${frelout[$i]}
3960	echo " cat /dev/null > ~/job_queue/ARCHIVE_ERROR_$fname" >> archive_${frelout[$i]}
3961	echo " cat STAGE_OUTPUT${i}_$kennung > ~/job_queue/archive_${frelout[$i]}_error" >> archive_${frelout[$i]}
3962	echo " echo \" *** $filetyp ${frelout[$i]} will be copied to \$WORK as backup\" " >> archive_${frelout[$i]}
3963	if [[ $files_for_pes = false ]]
3964	then
3965	echo " cp ${frelout[$i]} \$WORK" >> archive_${frelout[$i]}
3966	else
3967	echo " cp -r ${frelout[$i]} \$WORK/${frelout[$i]}" >> archive_${frelout[$i]}
3968	fi
3969	echo " echo \" *** $filetyp ${frelout[$i]} saved\" " >> archive_${frelout[$i]}
3970	echo "fi" >> archive_${frelout[$i]}
3971	echo "rm STAGE_OUTPUT${i}_$kennung" >> archive_${frelout[$i]}
3972	elif [[ $archive_system = DMF ]]
3973	then
3974	echo "cd $tmp_data_catalog" >> archive_${frelout[$i]}
3975	if [[ $files_for_pes = false ]]
3976	then
3977	printf "\n +++ archiving of single files impossible with $archive_system !\n"
3978	locat=DMF
3979	exit
3980	else
3981	# ??? nicht benÃ¶tigte Archiv-Systeme entfernen ???
3982	# FUER RECHNER IN JUELICH. DORT KOENNTEN AUCH
3983	# EINZELNE DATEIEN GESICHERT WERDEN (SPAETER KORR.)
3984	echo "rm -rf \$ARCHIVE/${frelout[$i]}" >> archive_${frelout[$i]}
3985	echo "cp -r ${frelout[$i]} \$ARCHIVE" >> archive_${frelout[$i]}
3986	fi
3987	elif [[ $archive_system = tivoli ]]
3988	then
3989	echo "cd $tmp_data_catalog" >> archive_${frelout[$i]}
3990	if [[ $files_for_pes = false ]]
3991	then
3992	# REMOVE EXISTING OLD FILES FROM THE ARCHIVE
3993	echo "set -x" >> archive_${frelout[$i]}
3994	echo "rm -rf \$PERM/${frelout[$i]}" >> archive_${frelout[$i]}
3995	echo "cp ${frelout[$i]} \$PERM/${frelout[$i]}" >> archive_${frelout[$i]}
3996	else
3997
3998	echo "set -x" >> archive_${frelout[$i]}
3999	echo "rm -rf \$PERM/${frelout[$i]}/*" >> archive_${frelout[$i]}
4000	echo "[[ ! -d \$PERM/${frelout[$i]} ]] && mkdir $PERM/${frelout[$i]}" >> archive_${frelout[$i]}
4001	cd $tmp_data_catalog
4002	all_files=`ls -1 ${frelout[$i]}/*`
4003	cd - > /dev/null
4004	(( inode = 0 ))
4005	(( tp1 = tasks_per_node + 1 ))
4006	while (( inode < nodes ))
4007	do
4008	files=`echo $all_files \| cut -d" " -f1-$tasks_per_node`
4009	all_files=`echo $all_files \| cut -d" " -f$tp1-`
4010	echo "tar cvf \$PERM/${frelout[$i]}/${frelout[$i]}.node_$inode.tar $files" >> archive_${frelout[$i]}
4011	(( inode = inode + 1 ))
4012	done
4013	fi
4014	elif [[ $archive_system = ut ]]
4015	then
4016	echo "cd $tmp_data_catalog" >> archive_${frelout[$i]}
4017	if [[ $files_for_pes = false ]]
4018	then
4019	# REMOVE EXISTING OLD FILES FROM THE ARCHIVE
4020	echo "set -x" >> archive_${frelout[$i]}
4021	echo "rm -rf \$UT/${frelout[$i]}" >> archive_${frelout[$i]}
4022	echo "cp ${frelout[$i]} \$UT/${frelout[$i]}" >> archive_${frelout[$i]}
4023	else
4024
4025	echo "set -x" >> archive_${frelout[$i]}
4026	echo "rm -rf \$UT/${frelout[$i]}/*" >> archive_${frelout[$i]}
4027	echo "[[ ! -d \$UT/${frelout[$i]} ]] && mkdir $UT/${frelout[$i]}" >> archive_${frelout[$i]}
4028	cd $tmp_data_catalog
4029	all_files=`ls -1 ${frelout[$i]}/*`
4030	cd - > /dev/null
4031	(( inode = 0 ))
4032	(( tp1 = tasks_per_node + 1 ))
4033	while (( inode < nodes ))
4034	do
4035	files=`echo $all_files \| cut -d" " -f1-$tasks_per_node`
4036	all_files=`echo $all_files \| cut -d" " -f$tp1-`
4037	echo "tar cvf \$UT/${frelout[$i]}/${frelout[$i]}.node_$inode.tar $files" >> archive_${frelout[$i]}
4038	(( inode = inode + 1 ))
4039	done
4040	fi
4041	elif [[ $archive_system = none ]]
4042	then
4043	printf " +++ archiving on $localhost not available!\n"
4044	fi
4045
4046	if [[ $archive_system != none ]]
4047	then
4048	if [[ $localhost = nech ]]
4049	then
4050	subjob -d -c /pf/b/$usern/job_queue -v -q pp -X 0 -m 1000 -t 7200 $PORTOPT archive_${frelout[$i]}
4051	fi
4052	printf " Archiving of $tmp_data_catalog/${frelout[$i]} initiated (batch job submitted)\n"
4053	fi
4054	else
4055	printf " +++ caution: option -A is switched off. No archiving on $archive_system!\n"
4056	fi
4057
4058
4059	# CREATE AN EMPTY DIRECTORY IN THE USERS PERMANENT DIRECTORY,
4060	# IN ORDER TO NOTE THE RESPECTIVE CYCLE NUMBER ON THE ARCHIVE SYSTEM
4061	# RESTART-JOBS ARE USING THESE EMPTY-DIRECTORIES TO GET THE CURRENT HIGHEST
4062	# CYCLE NUMBER ON THE ARCHIVE SYSTEM (IN CASE THAT INFORMATIONS ARE DIFFICULT TO
4063	# TO ACCESS FROM THE ARCHIVE-SYSTEM DIRECTLY))
4064	if [[ $files_for_pes = false ]]
4065	then
4066	cat /dev/null > ${pathout[$i]}
4067	else
4068	mkdir -p ${pathout[$i]}
4069	fi
4070
4071	fi
4072
4073
4074	# COPY FROM THIS HOST TO THE ARCHIVE-SYSTEM
4075	# IF ANY ARCHIVING FAILS, AN ERROR-FLAG-FILE IS SET
4076	# THIS FILE REMAINS TO BE SET, EVEN IF ARCHIVING OF FURTHER FILES IS SUCCESSFULL
4077	if [[ "${actionout[$i]}" = ar ]]
4078	then
4079	if [[ $files_for_pes = false ]]
4080	then
4081	printf "\n >>> OUTPUT: ${localout[$i]} to"
4082	printf "\n ${pathout[$i]}"
4083	printf "\n File will be copied to archive-system ($archive_system) !\n"
4084	else
4085	printf "\n >>> OUTPUT: ${localout[$i]}/_.... to"
4086	printf "\n ${pathout[$i]}"
4087	printf "\n Directory will be copied to archive-system ($archive_system) !\n"
4088	fi
4089	mv ${localout[$i]} ${frelout[$i]}
4090
4091	file_saved=false
4092
4093	if [[ $archive_system = asterix ]]
4094	then
4095	do_stageout=true
4096	(( stageout_anz = 0 ))
4097	while [[ $do_stageout = true ]]
4098	do
4099	if [[ $files_for_pes = false ]]
4100	then
4101	stageout ${frelout[$i]} > STAGE_OUTPUT
4102	else
4103	stageout -t ${frelout[$i]} > STAGE_OUTPUT
4104	fi
4105	cat STAGE_OUTPUT
4106	if [[ $(grep -c "st.msg:150" STAGE_OUTPUT) != 0 ]]
4107	then
4108	file_saved=true
4109	do_stageout=false
4110	else
4111	if [[ $files_for_pes = false ]]
4112	then
4113	printf "\n +++ file ${frelout[$i]} could not be saved on archive-system"
4114	else
4115	printf "\n +++ directory ${frelout[$i]} could not be saved on archive-system"
4116	fi
4117	(( stageout_anz = stageout_anz + 1 ))
4118	if (( stageout_anz == 10 ))
4119	then
4120	printf "\n +++ stoped after 10 unsuccessful tries!"
4121	archive_save=false
4122	do_stageout=false
4123	else
4124	printf "\n *** new try to store on archive after 15 min:"
4125	sleep 900
4126	fi
4127	fi
4128	done
4129	elif [[ $archive_system = DMF ]]
4130	then
4131	if [[ $files_for_pes = false ]]
4132	then
4133	printf "\n +++ archiving of single files impossible on $archive_system!\n"
4134	locat=DMF
4135	exit
4136	else
4137	rm -rf $ARCHIVE/${frelout[$i]}
4138	cp -r ${frelout[$i]} $ARCHIVE
4139	fi
4140	file_saved=true
4141	elif [[ $archive_system = tivoli ]]
4142	then
4143	# ARCHIVING ONLY POSSIBLE VIA BATCH-JOB
4144	# IN THE MEANTIME, FILE IS STORED IN THE TEMPORARY DATA DIRECTORY,
4145	# BECAUSE MRUN'S CURRENT TEMPORARY WORKING DIRECTORY MAY ALREADY BE DELETED
4146	# WHEN THE ARCHIVE-JOB IS EXECUTED
4147	[[ ! -d $tmp_data_catalog ]] && mkdir -p $tmp_data_catalog
4148	chmod g+rx $tmp_data_catalog
4149	if [[ $files_for_pes = false ]]
4150	then
4151	ln -f ${frelout[$i]} $tmp_data_catalog/${frelout[$i]}
4152	else
4153	mkdir $tmp_data_catalog/${frelout[$i]}
4154	ln -f ${frelout[$i]}/* $tmp_data_catalog/${frelout[$i]}
4155	fi
4156
4157	# GENERATE AND SUBMIT BATCH-JOB
4158	# FILE HAS TO BE DELETED FROM THE TEMPORARY DATA DIRECTORY
4159	# DELETE OLD EXISTING FILES FROM THE ARCHIVE
4160	echo "cd $tmp_data_catalog" > archive_${frelout[$i]}
4161	if [[ $files_for_pes = false ]]
4162	then
4163	echo "rm -rf \$PERM/${frelout[$i]}" >> archive_${frelout[$i]}
4164	echo "cp ${frelout[$i]} \$PERM/${frelout[$i]}" >> archive_${frelout[$i]}
4165	echo "rm -rf ${frelout[$i]}" >> archive_${frelout[$i]}
4166	else
4167	echo "rm -rf \$PERM/${frelout[$i]}.tar" >> archive_${frelout[$i]}
4168	echo "tar cvf \$PERM/${frelout[$i]}.tar ${frelout[$i]}" >> archive_${frelout[$i]}
4169	echo "rm -rf ${frelout[$i]}" >> archive_${frelout[$i]}
4170	fi
4171
4172	subjob -v -d -q cdata -X 0 -m 1000 -t 43200 -c $job_catalog $PORTOPT archive_${frelout[$i]}
4173	printf " Archiving of $tmp_data_catalog/${frelout[$i]} initiated (batch job submitted)\n"
4174	file_saved=true
4175
4176	elif [[ $archive_system = ut ]]
4177	then
4178	# ARCHIVING ONLY POSSIBLE VIA BATCH-JOB
4179	# IN THE MEANTIME, FILE IS STORED IN THE TEMPORARY DATA DIRECTORY,
4180	# BECAUSE MRUN'S CURRENT TEMPORARY WORKING DIRECTORY MAY ALREADY BE DELETED
4181	# WHEN THE ARCHIVE-JOB IS EXECUTED
4182	[[ ! -d $tmp_data_catalog ]] && mkdir -p $tmp_data_catalog
4183	chmod g+rx $tmp_data_catalog
4184	if [[ $files_for_pes = false ]]
4185	then
4186	ln -f ${frelout[$i]} $tmp_data_catalog/${frelout[$i]}
4187	else
4188	mkdir $tmp_data_catalog/${frelout[$i]}
4189	ln -f ${frelout[$i]}/* $tmp_data_catalog/${frelout[$i]}
4190	fi
4191
4192	# GENERATE AND SUBMIT BATCH-JOB
4193	# FILE HAS TO BE DELETED FROM THE TEMPORARY DATA DIRECTORY
4194	# DELETE OLD EXISTING FILES FROM THE ARCHIVE
4195	echo "cd $tmp_data_catalog" > archive_${frelout[$i]}
4196	if [[ $files_for_pes = false ]]
4197	then
4198	echo "rm -rf \$UT/${frelout[$i]}" >> archive_${frelout[$i]}
4199	echo "cp ${frelout[$i]} \$UT/${frelout[$i]}" >> archive_${frelout[$i]}
4200	echo "rm -rf ${frelout[$i]}" >> archive_${frelout[$i]}
4201	else
4202	echo "rm -rf \$UT/${frelout[$i]}.tar" >> archive_${frelout[$i]}
4203	echo "tar cvf \$UT/${frelout[$i]}.tar ${frelout[$i]}" >> archive_${frelout[$i]}
4204	echo "rm -rf ${frelout[$i]}" >> archive_${frelout[$i]}
4205	fi
4206
4207	subjob -v -c /pf/b/$usern/job_queue -d -q pp -X 0 -m 1000 -t 7200 $PORTOPT archive_${frelout[$i]}
4208	printf " Archiving of $tmp_data_catalog/${frelout[$i]} initiated (batch job submitted)\n"
4209	file_saved=true
4210
4211	else
4212	printf "\n +++ archive_system=\"$archive_system\" archiving impossible!"
4213	archive_save=false
4214	fi
4215
4216	# CREATE EMPTY FLAG-FILE OR -DIRECTORY
4217	# IN ORDER TO NOTE THE RESPECTIVE CYCLE NUMBER ON THE ARCHIVE SYSTEM
4218	# RESTART-JOBS ARE USING THESE EMPTY-DIRECTORIES TO GET THE CURRENT HIGHEST
4219	# CYCLE NUMBER ON THE ARCHIVE SYSTEM (IN CASE THAT INFORMATIONS ARE DIFFICULT TO
4220	# TO ACCESS FROM THE ARCHIVE-SYSTEM DIRECTLY))
4221	if [[ $file_saved = true ]]
4222	then
4223	if [[ $files_for_pes = false ]]
4224	then
4225	cat /dev/null > ${pathout[$i]}
4226	else
4227	mkdir -p ${pathout[$i]}
4228	fi
4229	fi
4230
4231	fi
4232
4233	# APPEND ON THIS MACHINE
4234	if [[ "${actionout[$i]}" = "a" ]]
4235	then
4236	if [[ "${extout[$i]}" != " " && "${extout[$i]}" != "" ]]
4237	then
4238	printf "\n >>> OUTPUT: ${localout[$i]} append to ${pathout[$i]}.${extout[$i]}\n"
4239	cat ${localout[$i]} >> ${pathout[$i]}.${extout[$i]}
4240	else
4241	printf "\n >>> OUTPUT: ${localout[$i]} append to ${pathout[$i]}\n"
4242	cat ${localout[$i]} >> ${pathout[$i]}
4243	fi
4244	fi
4245
4246	# COPY ON THIS MACHINE
4247	# COPY HAS TO BE USED, BECAUSE MOVE DOES NOT WORK IF FILE-ORIGIN AND TARGET ARE
4248	# ON DIFFERENT FILE-SYSTEMS
4249	if [[ "${actionout[$i]}" = "" && $files_for_pes = false ]]
4250	then
4251
4252	# COPY IN CASE OF RUNS ON SINGLE CORES
4253	if [[ "${extout[$i]}" != " " && "${extout[$i]}" != "" ]]
4254	then
4255	printf "\n >>> OUTPUT: ${localout[$i]} to ${pathout[$i]}.${extout[$i]}\n"
4256	if [[ $link_local_output = true ]]
4257	then
4258	printf " using ln -f\n"
4259	ln -f ${localout[$i]} ${pathout[$i]}.${extout[$i]}
4260	fi
4261	# If "ln -f" fails of if "$link_local_output = false" do a normal "cp"
4262	if [[ ! -f "${pathout[$i]}.${extout[$i]}" ]]
4263	then
4264	if [[ $link_local_output = true ]]
4265	then
4266	printf " ln failed, using cp...\n"
4267	fi
4268	cp ${localout[$i]} ${pathout[$i]}.${extout[$i]}
4269	fi
4270	else
4271	printf "\n >>> OUTPUT: ${localout[$i]} to ${pathout[$i]}\n"
4272	if [[ $link_local_output = true ]]
4273	then
4274	printf " using ln -f\n"
4275	ln -f ${localout[$i]} ${pathout[$i]}
4276	fi
4277	# If "ln -f" fails of if "$link_local_output = false" do a normal "cp"
4278	if [[ ! -f "${pathout[$i]}" ]]
4279	then
4280	if [[ $link_local_output = true ]]
4281	then
4282	printf " ln failed, using cp...\n"
4283	fi
4284	cp ${localout[$i]} ${pathout[$i]}
4285	fi
4286	fi
4287
4288	elif [[ "${actionout[$i]}" = "" && $files_for_pes = true ]]
4289	then
4290
4291	# FILES FROM THE DIFFERENT CORES ARE MOVED WITH ln-COMMAND TO THE PERMANENT DIRECTORY
4292	# AS A FIRST STEP, THE PERMANENT DIRECTORY IS CREATED
4293	printf "\n >>> OUTPUT: ${localout[$i]}/_.... to ${pathout[$i]}\n"
4294	if [[ $link_local_output = true ]]
4295	then
4296	printf " using ln -f\n"
4297	mkdir ${pathout[$i]}
4298	cd ${localout[$i]}
4299	for file in $(ls *)
4300	do
4301	ln -f $file ${pathout[$i]}
4302	done
4303	cd $TEMPDIR
4304	fi
4305
4306	# IF "ln -f" HAS FAILED OR IF "$link_local_output = false" DO A NORMAL COPY "cp -r"
4307	if [[ ! -f "${pathout[$i]}/_0000" ]]
4308	then
4309	if [[ $link_local_output = true ]]
4310	then
4311	printf " ln failed for .../_0000, using cp...\n"
4312	fi
4313	cp -r ${localout[$i]} ${pathout[$i]}
4314	fi
4315
4316	fi
4317	fi
4318	done
4319	if (( i != 0 ))
4320	then
4321	if [[ $transfer_problems = true ]]
4322	then
4323	printf "\n$dashes\n *** OUTPUT-files saved"
4324	printf "\n +++ WARNING: some data transfers failed! \n"
4325	else
4326	printf "\n$dashes\n *** all OUTPUT-files saved \n"
4327	fi
4328	fi
4329
4330
4331	# IF REQUIRED, START A RESTART-JOB
4332	# FILE CONTINUE_RUN MUST HAVE BEEN CREATED BY THE EXECUTABLE (PALM)
4333	if [[ -f CONTINUE_RUN ]]
4334	then
4335
4336	if [[ $archive_save = true ]]
4337	then
4338
4339	# ADD RESTART-OPTIONS TO THE MRUN-CALL (IF THEY ARE NOT USED ALREADY):
4340	# -C TELLS MRUN THAT IT IS A RESTART-RUN
4341	# -v SILENT MODE WITHOUT INTERACTIVE QUERIES
4342	# -n BATCH-MODE (IMPORTANT ONLY IN CASE OF BATCH JOBS ON THE LOCAL MACHINE)
4343	[[ $(echo $mc \| grep -c "\-C") = 0 ]] && mc="$mc -C"
4344	[[ $(echo $mc \| grep -c "\-v") = 0 ]] && mc="$mc -v"
4345	[[ $(echo $mc \| grep -c "\-b") = 0 ]] && mc="$mc -b"
4346	if [[ $(echo $mc \| grep -c "#") != 0 ]]
4347	then
4348	mc=`echo $mc \| sed 's/#/f/g'`
4349	fi
4350
4351
4352	# START THE RESTART-JOB
4353	printf "\n\n *** initiating restart-run on \"$return_addres\" using command:\n"
4354	echo " $mc"
4355	printf "\n$dashes\n"
4356	if [[ $localhost != $fromhost ]]
4357	then
4358
4359	if [[ $localhost = lccrayh \|\| $localhost = nech \|\| $localhost = ibmh \|\| $localhost = ibmkisti \|\| $localhost = ibmku \|\| $localhost = ibms \|\| $localhost = lcflow \|\| $localhost = lckyu* ]]
4360	then
4361	echo "*** ssh will be used to initiate restart-runs!"
4362	echo " return_addres=\"$return_addres\" "
4363	echo " return_username=\"$return_username\" "
4364	if [[ $(echo $return_addres \| grep -c "130.75.105") = 1 ]]
4365	then
4366	if [[ $localhost = ibmh ]]
4367	then
4368	ssh $SSH_PORTOPT $usern@136.172.40.15 "ssh $SSH_PORTOPT $return_addres -l $return_username \"PATH=\\\$PATH:$LOCAL_MRUN_PATH;. /muksoft/packages/intel/bin/compilervars.sh intel64;export PALM_BIN=$LOCAL_MRUN_PATH;cd $LOCAL_PWD; $mc\" "
4369	elif [[ $localhost = lccrayb ]]
4370	then
4371	ssh $usern@blogin1 "ssh $SSH_PORTOPT $return_addres -l $return_username \"PATH=\\\$PATH:$LOCAL_MRUN_PATH;. /muksoft/packages/intel/bin/compilervars.sh intel64;export PALM_BIN=$LOCAL_MRUN_PATH;cd $LOCAL_PWD; $mc\" "
4372	elif [[ $localhost = lccrayh ]]
4373	then
4374	ssh $usern@hlogin1 "ssh $SSH_PORTOPT $return_addres -l $return_username \"PATH=\\\$PATH:$LOCAL_MRUN_PATH;. /muksoft/packages/intel/bin/compilervars.sh intel64;export PALM_BIN=$LOCAL_MRUN_PATH;cd $LOCAL_PWD; $mc\" "
4375
4376	else
4377	ssh $SSH_PORTOPT $return_addres -l $return_username "PATH=\$PATH:$LOCAL_MRUN_PATH;. /muksoft/packages/intel/bin/compilervars.sh intel64;export PALM_BIN=$LOCAL_MRUN_PATH;cd $LOCAL_PWD; $mc "
4378	fi
4379	else
4380	if [[ $localhost = ibmkisti ]]
4381	then
4382	ssh $SSH_PORTOPT $usern@gaiad "ssh $SSH_PORTOPT $return_addres -l $return_username \"PATH=\\\$PATH:$LOCAL_MRUN_PATH;export PALM_BIN=$LOCAL_MRUN_PATH;cd $LOCAL_PWD; $mc\" "
4383	elif [[ $localhost = lcflow ]]
4384	then
4385	/usr/bin/ssh $SSH_PORTOPT $return_addres -l $return_username "PATH=\$PATH:$LOCAL_MRUN_PATH;export PALM_BIN=$LOCAL_MRUN_PATH;cd $LOCAL_PWD; $mc "
4386	elif [[ $localhost = lccrayb ]]
4387	then
4388	ssh $usern@blogin1 "ssh $SSH_PORTOPT $return_addres -l $return_username \"PATH=\\\$PATH:$LOCAL_MRUN_PATH;export PALM_BIN=$LOCAL_MRUN_PATH;cd $LOCAL_PWD; $mc\" "
4389	elif [[ $localhost = lccrayh ]]
4390	then
4391	ssh $usern@hlogin1 "ssh $SSH_PORTOPT $return_addres -l $return_username \"PATH=\\\$PATH:$LOCAL_MRUN_PATH;export PALM_BIN=$LOCAL_MRUN_PATH;cd $LOCAL_PWD; $mc\" "
4392	else
4393	ssh $SSH_PORTOPT $return_addres -l $return_username "PATH=\$PATH:$LOCAL_MRUN_PATH;export PALM_BIN=$LOCAL_MRUN_PATH;cd $LOCAL_PWD; $mc "
4394	fi
4395	fi
4396	else
4397	printf "\n +++ no restart mechanism available for host \"$localhost\" "
4398	locat=restart; exit
4399	fi
4400
4401	# WAIT TO ALLOW THE RESTART-JOB TO BE QUEUED, BEFORE THE CURRENT JOB IS FINISHED
4402	if [[ $queue = special1q ]]
4403	then
4404	sleep 120
4405	else
4406	sleep 30
4407	fi
4408
4409	else
4410
4411	# JOBS RUNNING (AND STARTED) ON THE LOCAL MACHINE CAN DIRECTLY CALL MRUN (WITHOUT
4412	# USING SSH)
4413	cd $LOCAL_PWD
4414	if [[ $localhost = lckyuh ]]
4415	then
4416	printf "\n +++ WARNING: no restart mechanism available for host \"$localhost\" "
4417	printf "\n please restart job manually using command:\n"
4418	printf "\n \"$mc\" "
4419	else
4420	eval $mc # THE ' MUST BE EVALUATED
4421	fi
4422	cd - > /dev/null
4423	fi
4424	if [[ $localhost = lckyuh ]]
4425	then
4426	printf "\n$dashes\n *** restart-run to be initiated manually\n"
4427	else
4428	printf "\n$dashes\n *** restart-run initiated \n"
4429	fi
4430
4431
4432	# DELETE INPUT-(RESTART)FILES, WHICH HAVE BEEN FETCHED FROM THE TEMPORARY DATA
4433	# DIRECTORY, BACAUSE THEY ARE NOT REQUIRED BY THE RESTART-JOB.
4434	# THIS IS DONE IN ORDER TO AVOID EXCEEDING DISC QUOTAS OR DISC SPACE (RESTART-FILES
4435	# MAY BE VERY HUGE)
4436	(( i = 0 ))
4437	while (( i < iin ))
4438	do
4439	(( i = i + 1 ))
4440	if [[ "${got_tmp[$i]}" = true && $keep_data_from_previous_run = false ]]
4441	then
4442	rm -r $tmp_data_catalog/${frelin[$i]}
4443	fi
4444	done
4445
4446	else
4447
4448	printf "\n +++ no restart-run possible, since errors occured"
4449	printf "\n during the archive process"
4450	fi
4451
4452	fi
4453
4454
4455
4456	# SEND EMAIL NOTIFICATION ABOUT THE FINISHED RUN
4457	if [[ "$email_notification" != "none" ]]
4458	then
4459
4460	if [[ $localhost != $fromhost ]]
4461	then
4462	if [[ -f CONTINUE_RUN ]]
4463	then
4464	echo "PALM restart run necessary" > email_text
4465	echo "description header of actual run:" >> email_text
4466	cat CONTINUE_RUN >> email_text
4467	echo "mrun-command to restart:" >> email_text
4468	echo "$mc" >> email_text
4469	else
4470	echo "PALM run with base filename \"$fname\" on host \"$localhost\" finished" > email_text
4471	fi
4472	mail $email_notification < email_text
4473	printf "\n *** email notification sent to \"$email_notification\" "
4474	fi
4475	fi
4476
4477
4478
4479	# ALL ACTIONS FINISHED, TEMPORARY WORKING-DIRECTORY CAN BE DELETED
4480	cd $HOME
4481	[[ $delete_temporary_catalog = true ]] && rm -rf $TEMPDIR
4482
4483	else
4484
4485
4486	# PREPARING ACTIONS,
4487	# IF A BATCH-JOB IS TO BE GENERATED AND STARTED ON A REMOTE-MACHINE GERECHNET
4488
4489	# BUILD THE MRUN-COMMAND TO BE CALLED IN THE BATCH-JOB ON THE REMOTE-MACHINE
4490	mrun_com="$mrun_script_name -a $afname -c $config_file -d $fname -h $host -H $fromhost -m $memory -t $cpumax -q $queue -R $return_addres -U $return_username -u $remote_username"
4491	[[ "$cpp_opts" != "" ]] && mrun_com=${mrun_com}" -D \"$cpp_opts\""
4492	[[ "$global_revision" != "" ]] && mrun_com=${mrun_com}" -G \"$global_revision\""
4493	[[ $group_number != none ]] && mrun_com=${mrun_com}" -g $group_number"
4494	[[ $do_compile = true ]] && mrun_com=${mrun_com}" -s \"$source_list\""
4495	[[ "$input_list" != "" ]] && mrun_com=${mrun_com}" -i \"$input_list\""
4496	[[ $ignore_archive_error = true ]] && mrun_com=${mrun_com}" -I"
4497	[[ $keep_data_from_previous_run = true ]] && mrun_com=${mrun_com}" -k"
4498	[[ "$additional_conditions" != "" ]] && mrun_com=${mrun_com}" -K \"$additional_conditions\""
4499	[[ "$output_list" != "" ]] && mrun_com=${mrun_com}" -o \"$output_list\""
4500	[[ "$read_from_config" = false ]] && mrun_com=${mrun_com}" -S"
4501	[[ $do_trace = true ]] && mrun_com=${mrun_com}" -x"
4502	[[ "$numprocs" != "" ]] && mrun_com=${mrun_com}" -X $numprocs"
4503	if [[ $use_openmp = true ]]
4504	then
4505	mrun_com=${mrun_com}" -O $threads_per_task"
4506	fi
4507	[[ "$tasks_per_node" != "" ]] && mrun_com=${mrun_com}" -T $tasks_per_node"
4508	[[ $store_on_archive_system = true ]] && mrun_com=${mrun_com}" -A"
4509	[[ $package_list != "" ]] && mrun_com=${mrun_com}" -p \"$package_list\""
4510	[[ $return_password != "" ]] && mrun_com=${mrun_com}" -P $return_password"
4511	[[ $delete_temporary_catalog = false ]] && mrun_com=${mrun_com}" -B"
4512	[[ $node_usage != default && "$(echo $node_usage \| cut -c1-3)" != "sla" && $node_usage != novice ]] && mrun_com=${mrun_com}" -n $node_usage"
4513	[[ "$ocean_file_appendix" = true ]] && mrun_com=${mrun_com}" -y"
4514	[[ $run_coupled_model = true ]] && mrun_com=${mrun_com}" -Y \"$coupled_dist\""
4515	[[ "$check_namelist_files" = false ]] && mrun_com=${mrun_com}" -z"
4516	[[ "$combine_plot_fields" = false ]] && mrun_com=${mrun_com}" -Z"
4517	[[ "$max_par_io_str" != "" ]] && mrun_com=${mrun_com}" -w $max_par_io_str"
4518	if [[ $do_remote = true ]]
4519	then
4520	printf "\n>>>> MRUN-command on execution host:\n>>>> $mrun_com \n"
4521	fi
4522
4523
4524	# CREATE EXECUTABLE FOR BATCH JOB
4525	if [[ $create_executable_for_batch = true && $restart_run != true ]]
4526	then
4527
4528	printf "\n *** creating the executable for batch job\n"
4529
4530	# METHOD ONLY WORKS FOR BATCH JOBS ON LOCAL HOSTS
4531	if [[ $host != $localhost ]]
4532	then
4533	printf "\n +++ creation of executables is only allowed for batch jobs on local hosts."
4534	printf "\n Please set create_executable_for_batch = false in the config-file.\n"
4535	locat=create_executable; exit
4536	fi
4537
4538	mkdir $working_directory/SOURCES_FOR_RUN_${fname}/TMPDIR_FOR_CREATING_EXECUTABLE
4539	cd $working_directory/SOURCES_FOR_RUN_${fname}/TMPDIR_FOR_CREATING_EXECUTABLE
4540
4541	cp $make_depository .
4542	tar -xf $make_depository > /dev/null 2>&1
4543	cp ../* . > /dev/null 2>&1
4544
4545	make $mopts -f Makefile PROG=a.out F90=$compiler_name COPT="$cpp_options" F90FLAGS="$fopts" LDFLAGS="$lopts"
4546
4547	if [[ $? != 0 \|\| "$compile_error" = true \|\| "$module_compile_error" = true ]]
4548	then
4549	printf "\n +++ error occured while compiling or linking"
4550	locat=compile
4551	exit
4552	fi
4553
4554	mv a.out ..
4555	cd - > /dev/null 2>&1
4556	rm -rf $working_directory/SOURCES_FOR_RUN_${fname}/TMPDIR_FOR_CREATING_EXECUTABLE
4557
4558	printf " *** executable created in \"$working_directory/SOURCES_FOR_RUN_${fname}\"\n "
4559
4560	fi
4561
4562
4563	# BUILD THE JOB-SCRIPTS ON FILE jobfile
4564	jobfile=jobfile.$RANDOM
4565
4566
4567	# CREATE TEMPORARY DIRECTORY AND SWITCH TO IT
4568	echo "mkdir $TEMPDIR" >> $jobfile
4569	echo "cd $TEMPDIR" >> $jobfile
4570
4571
4572	# ACTIVATE ERROR-TRACEBACK
4573	if [[ $do_trace = true ]]
4574	then
4575	echo "set -x" >> $jobfile
4576	else
4577	echo "set +vx" >> $jobfile
4578	fi
4579
4580
4581	# INITIALIZE THE ENVIRONMENT AND LOAD MODULES
4582	if [[ "$init_cmds" != "" ]]
4583	then
4584	echo "$init_cmds" >> $jobfile
4585	fi
4586	if [[ "$module_calls" != "" ]]
4587	then
4588	echo "$module_calls" >> $jobfile
4589	fi
4590
4591
4592	# PROVIDE SOURCE-CODE FILES, MRUN-SCRIPT AND CONFIGURATION-FILE FOR THE JOB
4593	# then-CLAUSE: FILES ARE COLLECTED IN THE SOURCES_FOR_RUN_... DIRECTORY ON THE LOCAL HOST,
4594	# WHICH IS THEN FETCHED FROM THE BATCH-JOB BY USING cp/scp
4595	# THE SOURCE-CODE FILES ALREADY EXIST IN THIS DIRECTORY
4596	# else-CLAUSE: FILE-CONTENTS ARE PUT INTO THE JOB-FILE AS HERE-DOCUMENTS
4597	# THIS MAY CREATE A QUITE LARGE JOB-FILE, WHICH CAN CAUSE PROBLEMS WITH SOME
4598	# QUEUEING-SYSTEMS
4599	if [[ $host = ibmkisti \|\| $host = lccrayb \|\| $host = lccrayf \|\| $host = lccrayh ]]
4600	then
4601
4602	# COPY CONFIGURATION-FILE AND MRUN-SCRIPT INTO THE SOURCES_FOR_RUN... DIRECTORY
4603	if [[ $restart_run != true ]]
4604	then
4605	cp $config_file $working_directory/SOURCES_FOR_RUN_$fname
4606	cp ${PALM_BIN}/$mrun_script_name $working_directory/SOURCES_FOR_RUN_$fname
4607	fi
4608
4609	# COPY THE SOURCES_FOR_RUN... DIRECTORY FROM THE LOCAL HOST TO THE JOB VIA scp
4610	# (then-CLAUSE: JOBS ON THE LOCAL HOST CAN JUST USE cp)
4611	echo "set -x" >> $jobfile
4612	if [[ $host = $localhost ]]
4613	then
4614
4615	# DUE TO UNKNOWN REASONS, COPY WITH cp COMMAND CREATES CORRUPT
4616	# FILES ON CRAY XC30 SYSTEMS (CSC HELSINKI), rsync IS USED INSTEAD
4617	if [[ $(echo $host \| cut -c1-6) = lccray ]]
4618	then
4619	echo "rsync -av -t $working_directory/SOURCES_FOR_RUN_$fname ." >> $jobfile
4620	else
4621	echo "cp -r $working_directory/SOURCES_FOR_RUN_$fname ." >> $jobfile
4622	fi
4623
4624	else
4625	if [[ $host = ibmkisti ]]
4626	then
4627	# ON KISTI'S IBM FIREWALL IS ONLY OPENED ON INTERACTIVE NODE
4628	echo "localdir=\`pwd\`" >> $jobfile
4629	echo "ssh $SSH_PORTOPT $remote_username@gaiad \"cd \$localdir; scp $PORTOPT -r $return_username@$return_addres:$working_directory/SOURCES_FOR_RUN_$fname .\" " >> $jobfile
4630	elif [[ $host = lccrayb ]]
4631	then
4632	echo "localdir=\`pwd\`" >> $jobfile
4633	echo "ssh $SSH_PORTOPT $remote_username@blogin1 \"cd \$localdir; scp $PORTOPT -r $return_username@$return_addres:$working_directory/SOURCES_FOR_RUN_$fname .\" " >> $jobfile
4634	elif [[ $host = lccrayh ]]
4635	then
4636	echo "localdir=\`pwd\`" >> $jobfile
4637	echo "ssh $SSH_PORTOPT $remote_username@hlogin1 \"cd \$localdir; scp $PORTOPT -r $return_username@$return_addres:$working_directory/SOURCES_FOR_RUN_$fname .\" " >> $jobfile
4638
4639	else
4640	echo "scp $PORTOPT -r $return_username@$return_addres:$working_directory/SOURCES_FOR_RUN_$fname ." >> $jobfile
4641	fi
4642	fi
4643	echo "export SOURCES_COMPLETE=true" >> $jobfile
4644
4645	# MOVE MRUN-SCRIPT AND CONFIGURATION-FILE FROM THE SOURCES_FOR_RUN... DIRECTORY TO THE
4646	# WORKING DIRECTORY OF THE JOB
4647	echo "mv SOURCES_FOR_RUN_$fname/$config_file . " >> $jobfile
4648	echo "mv SOURCES_FOR_RUN_$fname/$mrun_script_name . " >> $jobfile
4649	echo "chmod u+rwx $mrun_script_name" >> $jobfile
4650	echo "execute_mrun=true" >> $jobfile
4651	echo " " >> $jobfile
4652
4653	else
4654
4655	# PROVIDE SOURCE-CODE FILES AND MAKEFILE AS HERE DOCUMENT
4656	if [[ $do_compile = true ]]
4657	then
4658
4659	source_catalog=SOURCES_FOR_RUN_$fname
4660
4661	# CREATE SOURCES_FOR_RUN... DIRECTORY TO STORE THE SOURCE CODE FILES AND THE MAKEFILE
4662	echo "mkdir SOURCES_FOR_RUN_$fname" >> $jobfile
4663	echo "export SOURCES_COMPLETE=true" >> $jobfile
4664	echo "cd SOURCES_FOR_RUN_$fname" >> $jobfile
4665
4666	for filename in $source_list
4667	do
4668	# BACKSLASH IS USED FOR MASKING
4669	echo "cat > $filename << \"%END%\"" >> $jobfile
4670	cat $source_catalog/$filename >> $jobfile
4671	echo " " >> $jobfile
4672	echo "%END%" >> $jobfile
4673	echo " " >> $jobfile
4674	done
4675
4676	# BACKSLASH IS USED FOR MASKING
4677	echo "cat > Makefile << \"%END%\"" >> $jobfile
4678	cat $source_catalog/Makefile >> $jobfile
4679	echo " " >> $jobfile
4680	echo "%END%" >> $jobfile
4681	echo " " >> $jobfile
4682
4683	echo "cd - > /dev/null" >> $jobfile
4684
4685	fi
4686
4687
4688	# PROVIDE THE CONFIGURATION-FILE AS HERE-DOCUMENT
4689	# BACKSLASH IS USED FOR MASKING
4690	# LINES WITH #$ IN THE CONFIGURATION-FILE, COMING FROM THE SVN KEYWORD SUBSTITUTION,
4691	# ARE REMOVED FROM THE FILE IN ORDER TO AVOID PROBLEMS WITH THE SGE BATCH SYSTEM
4692	echo "cat > $config_file << \"%END%\"" >> $jobfile
4693	if [[ $host = lckyuh ]]
4694	then
4695	# NO CROSS-COMPILER ON COMPUTE NODE
4696	sed 's/frtpx/frt/g' $config_file >> $jobfile
4697	else
4698	sed 's/#$.*//g' $config_file >> $jobfile
4699	fi
4700	echo "%END%" >> $jobfile
4701	echo " " >> $jobfile
4702
4703
4704	# PROVIDE THE MRUN-SCRIPTS AS HERE-DOCUMENT
4705	# BACKSLASH IS USED FOR MASKING
4706	echo "cat > $mrun_script_name <<\"%END%\"" >> $jobfile
4707	if [[ $host = lckyuh ]]
4708	then
4709	sed 's/\/bin\/ksh/\/bin\/bash/g' ${PALM_BIN}/$mrun_script_name >> $jobfile
4710	else
4711	cat ${PALM_BIN}/$mrun_script_name >> $jobfile
4712	fi
4713	echo "%END%" >> $jobfile
4714	echo "chmod u+x $mrun_script_name" >> $jobfile
4715	echo "execute_mrun=true" >> $jobfile
4716	echo " " >> $jobfile
4717
4718	fi
4719
4720
4721	# GET REQUIRED INPUT-FILES BY SCP OR BY SENDING THEM WITH THE JOB AS HERE-DOCUMENT
4722	# PUT THESE FILES INTO THE USER'S RESPECTIVE PERMANENT DIRECTORIES ON THE REMOTE-HOST
4723	# IF THE DIRECTORIES DO NOT EXIST, TRY TO CREATE THEM
4724	if [[ $do_remote = true ]]
4725	then
4726	(( i = 0 ))
4727	while (( i < iin ))
4728	do
4729	(( i = i + 1 ))
4730	echo "[[ ! -d ${pathin[$i]} ]] && mkdir -p ${pathin[$i]}" >> $jobfile
4731	if [[ "${transin[$i]}" = job ]]
4732	then
4733	echo "cat > ${remotepathin[$i]} <<\"%END%\"" >> $jobfile
4734	eval cat ${pathin[$i]}/${frelin[$i]} >> $jobfile
4735	echo " " >> $jobfile
4736	echo "%END%" >> $jobfile
4737	else
4738	echo "batch_scp $PORTOPT -b -o -g -s -u $return_username $return_addres ${remotepathin[$i]} \"${pathin[$i]}\" ${frelin[$i]}" >> $jobfile
4739	fi
4740
4741	# CHECK, IF FILE COULD BE CREATED
4742	echo "if [[ \$? = 1 ]]" >> $jobfile
4743	echo "then" >> $jobfile
4744	echo " echo \" \" " >> $jobfile
4745	echo " echo \"+++ file ${remotepathin[$i]} could not be created\" " >> $jobfile
4746	echo " echo \" please check, if directory exists on $host!\" " >> $jobfile
4747	echo " echo \"+++ MRUN will not be continued\" " >> $jobfile
4748	echo " execute_mrun=false" >> $jobfile
4749	echo "fi" >> $jobfile
4750	done
4751	fi
4752
4753	# PROVIDE NAME OF THE CURRENT WORKING-DIRECTORY ON THE LOCAL MACHINE (FROM WHERE THE JOB IS
4754	# STARTED) BY SETTING AN ENVIRONMENT-VARIABLE. THIS INFORMATION IS USED IN THE JOB BY MRUN
4755	# IN CASE THAT RESTART-RUNS HAVE TO BE GENERATED
4756	echo "LOCAL_PWD=$working_directory" >> $jobfile
4757	echo "export LOCAL_PWD" >> $jobfile
4758
4759	# PROVIDE THE PATH OF THE LOCAL MRUN-SCRIPT FOR THE SAME REASON
4760	echo "LOCAL_MRUN_PATH=$PALM_BIN" >> $jobfile
4761	echo "export LOCAL_MRUN_PATH" >> $jobfile
4762
4763	# lcflow ALSO REQUIRES TO PROVIDE PATH FOR THE PALM-SCRIPTS
4764	if [[ $host = lcflow \|\| $localhost = lcflow ]]
4765	then
4766	echo "export PALM_BIN=$PALM_BIN" \| sed -e 's:'$HOME':$HOME:' >> $jobfile
4767	echo "export PATH=\$PATH:\$PALM_BIN" >> $jobfile
4768	fi
4769
4770	# CALL MRUN WITHIN THE JOB (SETTING QUEUE IS A WORKAROUND FOR ibmkisti)
4771	# AS FINAL ACTION, REMOVE THE TEMPORARY DIRECTORY CREATED AT THE BEGINNING OF THE JOB
4772	echo "set -x" >> $jobfile
4773	echo "queue=$queue" >> $jobfile
4774	echo "[[ \$execute_mrun = true ]] && ./$mrun_com" >> $jobfile
4775	echo 'ls -al; echo `pwd`' >> $jobfile
4776	echo "cd \$HOME" >> $jobfile
4777	echo "rm -rf $TEMPDIR" >> $jobfile
4778
4779
4780
4781
4782	# START THE JOB USING SUBJOB-COMMAND
4783	if [[ $silent = false ]]
4784	then
4785	printf "\n "
4786	else
4787	printf "\n\n"
4788	fi
4789
4790	subjob $job_on_file -h $host -u $remote_username -g $group_number -q $queue -m $memory -N $node_usage -t $cpumax $XOPT $TOPT $OOPT -n $fname -v -c $job_catalog -e $email_notification $PORTOPT $jobfile
4791	rm -rf $jobfile
4792
4793
4794	fi # END OF REMOTE-PART

Note: See TracBrowser for help on using the repository browser.

Download in other formats:

| Impressum | ©Leibniz Universität Hannover |