Home

Context Navigation

source: palm/trunk/SCRIPTS/mrun @ 749

Last change on this file since 749 was 746, checked in by letzel, 13 years ago
'wiki' inserted into weblink path to error message database (message) support for Linux OS with German locale (mrun) properly report hosts on general Linux clusters (mrun)
Property svn:keywords set to `Id Rev`
File size: 175.9 KB

Line
1	#!/bin/ksh
2	# mrun - script for running PALM jobs
3	# $Id: mrun 746 2011-08-18 21:14:48Z raasch $
4
5	# Procedure for interactive/batch PALM runs (initial runs and automatic
6	# restart runs)
7
8	# Last changes:
9	# 03/03/94 - Siggi - Entwicklungsbeginn
10	# 21/03/94 - Siggi - Entwicklungsabschluss (Version 1.0)
11	# 09/12/94 - Siggi - allexport unterbunden; statt dessen werden jetzt ein-
12	# zelne Variable exportiert. Dies ist noetig, weil an-
13	# sonsten irgendein Speicher ueberlaeuft und von mrun
14	# aufgerufene Programme (z.B. stageout) mit Fehlern
15	# abbrechen (too many arguments)
16	# 20/06/95 - Siggi - noclobber muss explizit abgeschaltet, da jetzt
17	# defaultmaessig im RRZN-Profile eingeschaltet
18	# 10/06/97 - Siggi - Zusaetzliche Option -C bei cpp, damit // nicht
19	# wegfallen
20	# 25/04/00 - Siggi - Version 1.5
21	# remote_addres und remote_user in return_addres bzw.
22	# return_username umbenannt, return_username wird dem
23	# mrun-Aufruf im Job ueber Option -U mitgegeben.
24	# Neue Variable remote_username, die beim interaktiven
25	# mrun-Aufruf zwingend ueber die neue Option -u oder
26	# in der Konfigurationsdatei angegeben werden muss.
27	# 08/02/01 - Siggi - Alle mrun-Meldungen ins englische uebersetzt,
28	# Version 1.7
29	# 05/05/02 - Siggi - Uebersetzungen mittels make-Mechanismus moeglich
30	# 21/11/02 - Siggi - ENV-Variable XLFRTEOPTS wird vor Ausfuehrung auf
31	# IBM gesetzt, um Record-Laenge der NAMELIST-Dateien,
32	# zu bestimmen; Praeprozessoroptionen haben nun alle
33	# die Form -Dtext=text, weil auf IBM sonst text durch
34	# die leere Zeichenkette ersetzt wird
35	# 01/08/03 - Siggi - Test of implementing dvrp+1PE on ibm
36	# 07/01/04 - Siggi - additional preprocessor directive for ibm included
37	# (-D$OMP=OMP) in order to avoid problems with
38	# OMP_NUM_THREADS
39	# 04/01/05 - Siggi - archiving on hanni and berni is done in separate
40	# tar files, one for each node used
41	# 07/01/04 - Siggi - old code for t3e, vpp and hpcs eliminated
42	# 09/01/05 - Siggi - workaround for getting the IP address on gfdl3
43	# 17/01/05 - Siggi - job for data transfer to local machine now within
44	# job class c1 (HLRN)
45	# 27/01/05 - Siggi - IP address workaround for gfdl3 removed
46	# 28/01/05 - Siggi - tar-filenames on $PERM (HLRN) now include the
47	# base filename and cycle number
48	# 16/02/05 - Gerald - hababai validated
49	# 14/03/05 - Siggi - abort on NEC after first runtime error (F_ERRCNT)
50	# 29/03/05 - Marcus - berni*-en0 validated
51	# 21/04/05 - Siggi - transfer-job-protocol for avs-data is no more stored
52	# 24/04/05 - Siggi - netcdf support on lcmuk
53	# 25/04/05 - Siggi - netcdf support on gfld3 (decalpha)
54	# 11/05/05 - Siggi - files with non-numeric extensions (.xxx after cycle
55	# number) can now be used within interactive runs
56	# 12/05/05 - Siggi - netcdf support on ibm
57	# 13/05/05 - Siggi - error in tar-filenames on $PERM (HLRN) removed
58	# 18/05/05 - Siggi - netcdf support on nec
59	# 24/05/05 - Siggi - netcdf support on ibms
60	# 14/06/05 - Siggi - sleep interval after submit of restart job increased
61	# to 30 seconds
62	# 28/06/05 - Siggi - bora adjusted to new intel compiler (LD_LIBRARY_
63	# PATH is temporarily set)
64	# 07/09/05 - Siggi - setting of MP_EAGER_LIMIT switched off because it
65	# caused warnings in the job protocol
66	# 20/10/05 - Siggi - update of netcdf-version on decalpha (gfdl3)
67	# 25/10/05 - Siggi - error in listing files for getting the cycle number
68	# removed
69	# 26/10/05 - Siggi - new paths for dvrp-library, transfer of catalogs
70	# realized by file attribute trpe
71	# 28/10/05 - Siggi - if existing, the file extension is output in case
72	# of missing input file - no job abort in case of
73	# missing input files with file extensions
74	# (preliminary solution)
75	# 31/10/05 - Siggi - data transfer from nech now within seperate job
76	# 04/11/05 - Siggi - netcdf 3.6.0-p1 on ibmh/ibmb
77	# 07/12/05 - Siggi - gallego and elephanta admitted
78	# 30/12/05 - Siggi - gfdl5 (ibmy) admitted
79	# 10/01/06 - Siggi - cpp directive for NetCDF 64bit support
80	# 20/01/06 - Siggi - cpp directive for ibmy
81	# 09/02/06 - Siggi - ibmy admitted for batch mode
82	# 13/04/06 - Siggi - ostria admitted
83	# 18/04/06 - Siggi - usage of OpenMP implemented (additional option -O)
84	# 10/05/06 - Siggi - environment variable XLSMPOPTS set on IBM for
85	# optimization of OpenMP runs (option stacksize is
86	# necessary for large gridpoint numbers because
87	# otherwise a segmentation fault occurs)
88	# 23/05/05 - Siggi - lctit (SUN Fire X4600) admitted
89	# 21/08/06 - Siggi - standard mrun path on scirocco is as on bora
90	# 23/08/06 - Siggi - netcdf support for scirocco (notebook)
91	# 20/09/06 - Marcus - add DVRP library on nech
92	# 24/10/06 - Siggi - Environment variables for PALM steering are written
93	# on local NAMELIST file ENVPAR
94	# 24/11/06 - Siggi - levanto admitted, maestro switched to ifc 9.1
95	# 28/11/06 - Siggi - call to interpret_config.x_levanto because of SuSe 10
96	# 06/02/07 - Siggi - add DVRP library on lcmuk
97	# 07/02/07 - Siggi - revision of source code handling, source code is
98	# now expected in directory given by source_path and
99	# not in the current working directory, source code
100	# files provided in add_source_path are allways
101	# compiled, source code to be compiled is allways
102	# sampled in SOURCES_FOR_RUN_$fname which is a
103	# subdirectory of the current working directory,
104	# -s TEST changed to -s WRITE_PERMIT or -s WP,
105	# adapted for RIAM (neck)
106	# bugfix for cycle number of output file (extout)
107	# all hpmuk-related code removed
108	# 14/02/07 - Siggi - dvrp support for neck added
109	# 28/02/07 - Siggi - empty lines in configuration file are accepted
110	# mrun_path replaced by PALM_BIN,
111	# all machines are calling interpret_config.x
112	# 14/03/07 - Siggi - fimm admitted, revision number added to terminal
113	# output
114	# 16/03/07 - Siggi - adjustments for lctit
115	# 29/03/07 - Siggi - global revision transfered to batch job by new
116	# option -G
117	# 30/03/07 - Siggi - compilation "by hand" removed, cpp-directives/options
118	# + netcdf/dvrp-options are read from configuration
119	# file, host identifier (local_host) is read from
120	# config file, code related to ftp filetransfer
121	# removed (incl. option -f)
122	# 19/06/07 - Siggi - time limit for cdata jobs increased
123	# 25/07/07 - Siggi - two sets of executables can be started on lcmuk,
124	# if new option -Y (coupling) is given,
125	# output of executables to aout_output removed,
126	# messages are immediately written to stdout instead
127	# 03/08/07 - Marcus - add XOPT="-X $numprocs" for lcfimm
128	# 09/08/07 - Marcus - workaround on lcfimm to propagate environment
129	# variables out to the nodes in coupled mode -disabled-
130	# 13/08/07 - Marcus - start local restart jobs per ssh on lcfimm
131	# 28/08/07 - Marcus - completely remove workaround on lcfimm to propagate
132	# environment variables out to the nodes in coupled mode
133	# 15/10/07 - Siggi - Preliminary adjustments for lctit, based on Jin's
134	# suggestions
135	# 19/10/07 - Marcus - further adjustments for lctit: add new optional
136	# argument -g group_number, admit all sla* node_usage
137	# queues
138	# 30/10/07 - Marcus - further adjustments for queues on lctit
139	# 14/02/08 - Bjoern - link of restart files with long file name possible
140	# 28/02/08 - Marcus - If environment variable link_local_output is set to
141	# true, mrun tries "ln -f" on local output and resorts
142	# to "cp" or "cp -r" on error
143	# 15/04/08 - Siggi - argument -c introduced to most of the subjob calls,
144	# which allows the user to choose his own job catalog
145	# by setting job_catalog in the configuration file
146	# (default is ~/job_queue),
147	# workaround for mpiexec with -env option,
148	# adjustments for lcxt4 (Bergen Center for Computational
149	# Science)
150	# 22/05/08 - Marcus - If environment variable link_local_input is set to
151	# true, mrun tries "ln -f" on local input and resorts
152	# to "cp" or "cp -r" on error
153	# 27/05/08 - Siggi - PATH is set to PALM_BIN everywhere (missing so far)
154	# 14/07/08 - Siggi - adjustments for lcsgih
155	# 08/08/08 - Marcus - typo removed in lcxt4 branch
156	# 17/09/08 - Siggi - restart mechanism adjusted for lcsgi
157	# 02/10/08 - BjornM - argument "-Y" modified, adjustments for coupled runs
158	# 21/10/08 - Siggi - bugfix for the case that -K has more than one
159	# argument
160	# 08/11/08 - Siggi - bugfix for depository path settings in configuration
161	# file
162	# 11/11/08 - Siggi - rules for using user code files changed: user
163	# code splitted into one file per subroutine,
164	# user can provide his/her own makefile, only
165	# default source-code filenames are allowed
166	# 14/11/08 - Siggi - cond1 and cond2 are part of the depository name
167	# 08/01/09 - Siggi - mpi module on sgi machines to be used can be set
168	# with environment variable mpilib
169	# 13/01/09 - Siggi - totalview on hice/bice can be used in debug mode
170	# using environment variable totalview
171	# 25/02/09 - Siggi - cputime-option (-t) is allowed to be omitted for
172	# interactive runs
173	# 05/03/09 - Siggi - adjustments for new NEC-SX9 at RIAM (necriam)
174	# 16/03/09 - Siggi - dvrp_lib, dvrp_inc replaced by dvr_lib, dvr_inc,
175	# dvr streaming server is automatically started using
176	# new variable dvr_server and a configuration file
177	# .dvrserver.config,
178	# processing of dvr output files using new script
179	# process_dvr_output implemented, in order to make
180	# dvr application more user friendly
181	# 20/03/09 - Marcus - update of n1ge command for lctit
182	# 16/04/09 - Siggi - new option -y for precursor (uncoupled) ocean runs
183	# to be followed by a coupled atmosphere-ocean run
184	# 21/04/09 - Siggi - adjustments for new IBM at DKRZ, which is now ibmh
185	# 24/06/09 - BjornM - adjustments for coupled/precursor runs on ibmy
186	# 08/07/09 - Siggi - default value for email_notification is none,
187	# option -e added to subjob
188	# 19/08/09 - Marcus - MPI debug option
189	# 21/08/09 - Marcus - totalview on hice/bice also available for mvapich2
190	# 25/08/09 - BjornM - adpated for lck
191	# 26/08/09 - Siggi - additional environment variables for mpt on ice2
192	# 26/08/09 - Marcus - adjustment for ice2 on lcsgib
193	# 29/09/09 - Siggi - default value for archiving is false, option "-A"
194	# switches on archiving, archiving on SGI-ICE enabled
195	# 16/10/09 - Carolin- adjustments for archiving on SGI-ICE of binary files;
196	# adjustment for special1q
197	# 18/12/09 - Carolin- modification of archiving (permq)
198	# 01/02/10 - Siggi - adapted for lcxt5m and lckyoto (Fujitsu HX600)
199	# 03/02/10 - Siggi - make options (mopts) to be set by configuration file
200	# implemented
201	# 08/02/10 - Siggi - loading of modules now controlled via configuration
202	# file (modules)
203	# 02/03/10 - Siggi - for each block in the configuration file, separate
204	# utility programs are used, combine_plot_fields is
205	# directly called and must not be given by an output
206	# command in the configuration file any more
207	# 15/06/10 - Rieke - ssh from compute nodes to login nodes on hosts
208	# lcsgih/lcsgib is done using the names
209	# hicegate0/bicegate0 instead of the IP addresses
210	# 17/08/10 - BjornM - adjustments for interactive runs on lcxt4
211	# 07/09/10 - Siggi - bugfix for wrong netcdf/3.6.3 module on lcsgi
212	# 08/12/10 - Siggi - new handling of openmp/hybrid runs, option -O
213	# has now argument threads_per_task
214	# adjustments for Kyushu Univ. (lcrte, ibmku)
215	# 14/12/10 - Siggi - adjustments for new Tsubame system at Tokyo
216	# institute of technology (lctit)
217	# 23/12/10 - Micha - different number of processors in ocean and atmosphere
218	# is now allowed
219	# 02/02/10 - Siggi - further adjustments on Tsubame and concerning openMP
220	# usage
221	# 09/03/10 - Siggi - adjustments for ibmkisti, mpt bugfix for netCDF4
222	# usage, totalview usage extended
223	# 17/03/11 - Siggi - adjustments for openmp usage on ibmkisti
224	# 03/04/11 - Micha - added lckordi
225	# 06/04/11 - BjornM - bugfix for runs with mpt on lcsgi
226	# 17/08/11 - Siggi - extensions for impi library
227	# 18/08/11 - Siggi - bugfix for local append of output files with suffix (.nc)
228	# 18/08/11 - Marcus - support for Linux OS with German locale
229	# - properly report hosts on general Linux clusters
230
231
232
233	# VARIABLENVEREINBARUNGEN + DEFAULTWERTE
234
235	set +o allexport # SICHERHEITSHALBER UNTERBINDEN, DA SONST EVTL. STAGEOUT
236	# NICHT LAUEFT (TOO MANY ARGUMENTS - PROBLEM)
237	set +o noclobber # EXISTIERENDE DATEIEN DUERFEN UEBERSCHRIEBEN WERDEN
238
239	AddFilenames=""
240	additional_conditions=""
241	add_source_path=""
242	afname=""
243	archive_save=true
244	archive_system=none
245	compiler_name=""
246	cond1=""
247	cond2=""
248	config_file=.mrun.config
249	coupled_dist=""
250	coupled_mode="mpi1"
251	cpp_opts=""
252	cpp_options=""
253	cpumax=0
254	cpurest=0
255	delete_temporary_catalog=true
256	do_batch=false
257	do_compile=true
258	do_remote=false
259	do_stagein=true
260	do_stageout=true
261	do_trace=false
262	email_notification="none"
263	exclude=""
264	executable=""
265	execution_error=false
266	fimm=false
267	fname=test
268	fromhost=""
269	global_revision=""
270	group_number=none
271	host=""
272	host_file=""
273	hp=""
274	ignore_archive_error=false
275	input_list=""
276	interpreted_config_file=""
277	job_catalog="~/job_queue"
278	job_on_file=""
279	keep_data_from_previous_run=false
280	link_local_input=false
281	link_local_output=false
282	localhost_realname=$(hostname)
283	local_compile=false
284	local_dvrserver_running=.FALSE.
285	locat=normal
286	mainprog=""
287	makefile=""
288	mc=$0
289	while [[ $(echo $mc \| grep -c "/") != 0 ]]
290	do
291	mc=`echo $mc \| cut -f2- -d"/"`
292	done
293	module_calls=""
294	mrun_script_name=$mc
295	netcdf_inc=""
296	netcdf_lib=""
297	netcdf_support=false
298	node_usage=default
299	numprocs=""
300	numprocs_atmos=0
301	numprocs_ocean=0
302	OOPT=""
303	openmp=false
304	output_list=""
305	package_list=""
306	punkte="..........................................................."
307	queue=none
308	read_from_config=""
309	restart_run=false
310	if [[ `hostname` = rte10 ]]
311	then
312	return_addres=133.5.185.60
313	echo "+++ WARNING: fixed return_addres = $return_addres is used !!!!!"
314	elif [[ `hostname` = climate0 ]]
315	then
316	return_addres=165.132.26.68
317	echo "+++ WARNING: fixed return_addres = $return_addres is used !!!!!"
318	else
319	return_addres=$(nslookup `hostname` 2>&1 \| grep "Address:" \| tail -1 \| awk '{print $2}')
320	fi
321	return_password=""
322	return_username=$LOGNAME
323	remotecall=false
324	remote_username=""
325	run_coupled_model=false
326	run_mode=""
327	scirocco=false
328	store_on_archive_system=false
329	striche=" ----------------------------------------------------------------------------"
330	silent=false
331	source_list=""
332	source_path=SOURCE
333	tasks_per_node=""
334	threads_per_task=1
335	tmpcreate=false
336	tmp_data_catalog=""
337	transfer_problems=false
338	usern=$LOGNAME
339	use_openmp=false
340	working_directory=`pwd`
341	TOPT=""
342	XOPT=""
343	zeit=$( date \| cut -c 12-19 )
344
345	typeset -i iec=0 iic=0 iin=0 ioc=0 iout=0 memory=0 stagein_anz=0 stageout_anz=0
346	typeset -i cputime i ii iia iii iio icycle inode ival jobges jobsek maxcycle minuten nodes pes sekunden tp1
347
348	typeset -R30 calltime
349	typeset -L20 spalte1
350	typeset -L40 spalte2
351	typeset -L60 spalte3
352	typeset -L35 string1=`date`
353	typeset -L12 string2=$usern
354	typeset -L12 string3=$localhost_realname
355	typeset -L12 string4
356	typeset -L12 string5
357	typeset -L30 version="MRUN 2.0 Rev$Rev: 746 $"
358
359
360
361	# EINZELNE VARIABLE FUER HAUPTPROGRAMM EXPORTIEREN
362	export cpurest fname host localhost return_addres return_username remotecall tasks_per_node
363
364	# FOLGENDE VARIABLEN MUESSEN FUER DIE INTERPRETATION DER KONFIGURATIONSDATEI
365	# EXPORTIERT WERDEN
366	export afname config_file cpp_opts cpumax do_batch do_trace fname fromhost
367	export group_number input_list memory numprocs output_list queue run_mode
368
369
370
371	# FEHLERBEHANDLUNG
372	# BEI EXIT:
373	trap 'rm -rf $working_directory/tmp_mrun
374	if [[ $locat != localhost ]]
375	then
376	# if [[ ! -f ${mrun_path}/statistik/mrun_statistik ]]
377	# then
378	# cat > ${mrun_path}/statistik/mrun_statistik << %STATEND%
379	#MRUN-calls on $localhost
380	#
381	#date and time user localhost remotehost termination mrun-command
382	#--------------------------------------------------------------------------------------------------------------------
383	#%STATEND%
384	# chmod 666 ${mrun_path}/statistik/mrun_statistik
385	# fi
386	#
387	# # EINTRAG IN DIE STATISTIK-DATEI
388	# string4=$host
389	# string5=$locat
390	# if [[ "$job_on_file" = "" && $locat != control_c && $locat != user_abort ]]
391	# then
392	# if [[ $do_batch = true ]]
393	# then
394	# printf "$string1$string2$string3$string4$string5$mrun_com \n" >> ${mrun_path}/statistik/mrun_statistik
395	# else
396	# printf "$string1$string2$string3$string4$string5$mc \n" >> ${mrun_path}/statistik/mrun_statistik
397	# fi
398	# fi
399	echo " " > /dev/null
400	fi
401
402	if [[ $locat != normal && $locat != control_c && $locat != local_compile ]]
403	then
404
405	# EVENTUELLE ERROR-KOMMANDOS ABARBEITEN
406	(( i = 0 ))
407	while (( i < iec ))
408	do
409	(( i = i + 1 ))
410	printf "\n *** Execution of ERROR-command:\n"
411	printf " >>> ${err_command[$i]}\n"
412	eval ${err_command[$i]}
413	done
414	if [[ -n $interpreted_config_file ]] then
415	rm -rf $interpreted_config_file
416	fi
417	if [[ -n .mrun_environment ]] then
418	rm -rf .mrun_environment
419	fi
420	if [[ $tmpcreate = true ]]
421	then
422	printf "\n *** Contents of \"$TEMPDIR\":\n"
423	ls -al; cd
424	[[ $delete_temporary_catalog = true ]] && rm -rf $TEMPDIR
425	fi
426	if [[ "$dvrserver_id" != "" ]]
427	then
428	echo "+++ killing dvrserver_id=$dvrserver_id"
429	kill $dvrserver_id
430	fi
431	if [[ -f ~/job_queue/JOBINFO.$QSUB_REQID ]]
432	then
433	rm -rf ~/job_queue/JOBINFO.$QSUB_REQID
434	fi
435	printf "\n\n+++ MRUN killed \n\n"
436	elif [[ $locat != control_c ]]
437	then
438	printf "\n\n --> all actions finished\n\n"
439	printf " Bye, bye $usern !!\n\n"
440	fi' exit
441
442
443	# BEI TERMINAL-BREAK:
444	trap 'rm -rf $working_directory/tmp_mrun
445	[[ $tmpcreate = true ]] && (cd; rm -rf $TEMPDIR)
446	if [[ -f ~/job_queue/JOBINFO.$QSUB_REQID ]]
447	then
448	rm -rf ~/job_queue/JOBINFO.$QSUB_REQID
449	fi
450	if [[ "$dvrserver_id" != "" ]]
451	then
452	echo "+++ killing dvrserver_id=$dvrserver_id"
453	kill $dvrserver_id
454	fi
455	printf "\n+++ MRUN killed by \"^C\" \n\n"
456	locat=control_c
457	exit
458	' 2
459
460
461	# CHECK IF THE PATH FOR THE PALM BINARIES (SCRIPTS+UTILITY-PROGRAMS) HAS
462	# BEEN SET
463	if [[ "$PALM_BIN" = "" ]]
464	then
465	printf "\n +++ environment variable PALM_BIN has not been set"
466	printf "\n please set it to the directory where the PALM scripts are located"
467	locat=palm_bin; exit
468	fi
469	export PATH=$PALM_BIN:$PATH
470
471
472
473	# SHELLSCRIPT-OPTIONEN EINLESEN UND KOMMANDO NEU ZUSAMMENSETZEN, FALLS ES
474	# FUER FOLGEJOBS BENOETIGT WIRD
475	while getopts :a:AbBc:Cd:D:Fg:G:h:H:i:IkK:m:M:n:o:O:p:P:q:r:R:s:St:T:u:U:vxX:yY: option
476	do
477	case $option in
478	(a) afname=$OPTARG;;
479	(A) store_on_archive_system=true; mc="$mc -A";;
480	(b) do_batch=true; mc="$mc -b";;
481	(B) delete_temporary_catalog=false; mc="$mc -B";;
482	(c) config_file=$OPTARG; mc="$mc -c$OPTARG";;
483	(C) restart_run=true; mc="$mc -C";;
484	(d) fname=$OPTARG; mc="$mc -d$OPTARG";;
485	(D) cpp_opts="$cpp_opts $OPTARG"; mc="$mc -D'$OPTARG'";;
486	(F) job_on_file="-D"; mc="$mc -F";;
487	(g) group_number=$OPTARG; mc="$mc -g$OPTARG";;
488	(G) global_revision=$OPTARG; mc="$mc -G'$OPTARG'";;
489	(h) host=$OPTARG; mc="$mc -h$OPTARG";;
490	(H) fromhost=$OPTARG; mc="$mc -H$OPTARG";;
491	(i) input_list=$OPTARG; mc="$mc -i'$OPTARG'";;
492	(I) ignore_archive_error=true; mc="$mc -I";;
493	(k) keep_data_from_previous_run=true; mc="$mc -k";;
494	(K) additional_conditions="$OPTARG"; mc="$mc -K'$OPTARG'";;
495	(m) memory=$OPTARG; mc="$mc -m$OPTARG";;
496	(M) makefile=$OPTARG; mc="$mc -M$OPTARG";;
497	(n) node_usage=$OPTARG; mc="$mc -n$OPTARG";;
498	(o) output_list=$OPTARG; mc="$mc -o'$OPTARG'";;
499	(O) use_openmp=true; threads_per_task=$OPTARG; mc="$mc -O$OPTARG";;
500	(p) package_list=$OPTARG; mc="$mc -p'$OPTARG'";;
501	(P) return_password=$OPTARG; mc="$mc -P$OPTARG";;
502	(q) queue=$OPTARG; mc="$mc -q$OPTARG";;
503	(r) run_mode=$OPTARG; mc="$mc -r'$OPTARG'";;
504	(R) remotecall=true;return_addres=$OPTARG; mc="$mc -R$OPTARG";;
505	(s) source_list=$OPTARG; mc="$mc -s'$OPTARG'";;
506	(S) read_from_config=false; mc="$mc -S";;
507	(t) cpumax=$OPTARG; mc="$mc -t$OPTARG";;
508	(T) tasks_per_node=$OPTARG; mc="$mc -T$OPTARG";;
509	(u) remote_username=$OPTARG; mc="$mc -u$OPTARG";;
510	(U) return_username=$OPTARG; mc="$mc -U$OPTARG";;
511	(v) silent=true; mc="$mc -v";;
512	(x) do_trace=true;set -x; mc="$mc -x";;
513	(X) numprocs=$OPTARG; mc="$mc -X$OPTARG";;
514	(y) ocean_file_appendix=true; mc="$mc -y";;
515	(Y) run_coupled_model=true; coupled_dist=$OPTARG; mc="$mc -Y'$OPTARG'";;
516	(\?) printf "\n +++ unknown option $OPTARG \n"
517	printf "\n --> type \"$0 ?\" for available options \n"
518	locat=parameter;exit;;
519	esac
520	done
521
522
523	# EVTL. POSITIONSPARAMETER EINLESEN
524	# ZUR ZEIT GIBT ES NUR DEN PARAMETER ? (=KURZINFO)
525	shift OPTIND-1
526
527
528	# KURZE AUFRUFBESCHREIBUNG WIRD HIER AUSGEGEBEN
529	if [[ "$1" = "?" ]]
530	then
531	(printf "\n *** mrun can be called as follows:\n"
532	printf "\n $mrun_script_name -b -c.. -d.. -D.. -f.. -F -h.. -i.. -I -K.. -m.. -o.. -p.. -r.. -R -s.. -t.. -T.. -v -x -X.. -y -Y.. <modus>\n"
533	printf "\n Description of available options:\n"
534	printf "\n Option Description Default-Value"
535	printf "\n -a base name of input files equiv. -d"
536	printf "\n -A archiving when using file-attribute fl"
537	printf "\n -b batch-job on local machine ---"
538	printf "\n -B do not delete temporary directory at end ---"
539	printf "\n -c configuration file .mrun.config"
540	printf "\n -d base name of files attached to program test"
541	printf "\n -D preprocessor(cpp)-directives \"\" "
542	printf "\n -F create remote job file only ---"
543	printf "\n -h execution host $localhost_realname"
544	printf "\n -i INPUT control list \"\" "
545	printf "\n -I archiving errors of previous batch-jobs"
546	printf "\n will be ignored"
547	printf "\n -k keep data from previous run"
548	printf "\n -K additional conditions for controling"
549	printf "\n usage of conditional code and"
550	printf "\n env-variables in configuration file \"\" "
551	printf "\n -m memory demand in MB (batch-jobs) 0 MB"
552	printf "\n -M Makefile name Makefile"
553	printf "\n -n node usage (shared/not_shared) depending on -h"
554	printf "\n -o OUTPUT control list \"\" "
555	printf "\n -O use OpenMP ---"
556	printf "\n -p software package list \"\" "
557	printf "\n -q queue \"$queue\" "
558	printf "\n -r run control list (combines -i -o) \"\" "
559	printf "\n -s filenames of routines to be compiled \"\" "
560	printf "\n must end with .f, .f90, .F, or .c !"
561	printf "\n use \"..\" for more than one file and wildcards"
562	printf "\n -s TEST compiles all files with w-permit"
563	printf "\n -S config file interpreted by shellscript ---"
564	printf "\n -t allowed cpu-time in seconds (batch) 0"
565	printf "\n -T tasks per node depending on -h"
566	printf "\n -u username on remote machine \"\" "
567	printf "\n -v no prompt for confirmation ---"
568	printf "\n -x tracing of mrun for debug purposes ---"
569	printf "\n -X # of processors (on parallel machines) 1"
570	printf "\n -y add appendix \"_O\" to all local output"
571	printf "\n files (ocean precursor runs followed by"
572	printf "\n coupled atmosphere-ocean runs) ---"
573	printf "\n -Y run coupled model, \"#1 #2\" with"
574	printf "\n #1 atmosphere and #2 ocean processors \"#/2 #/2\" depending on -X"
575	printf "\n "
576	printf "\n Possible values of positional parameter <modus>:"
577	printf "\n \"?\" - this outline \n\n") \| more
578	exit
579	elif [[ "$1" != "" ]]
580	then
581	printf "\n +++ positional parameter $1 unknown \n"
582	locat=parameter; exit
583	fi
584
585
586
587	# KURZE STARTMELDUNG
588	printf "\n*** $version "
589	printf "\n will be executed. Please wait ..."
590
591
592
593	# PRUEFEN, OB KONFIGURATIONS-DATEI VORHANDEN
594	if [[ ! -f $config_file ]]
595	then
596	printf "\n\n +++ configuration file: "
597	printf "\n $config_file"
598	printf "\n does not exist"
599	locat=connect; exit
600	fi
601
602
603	# HOST-IDENTIFIER (local_host) AUS KONFIGURATIONSDATEI BESTIMMEN
604	line=""
605	grep "%host_identifier" $config_file > tmp_mrun
606	while read line
607	do
608	if [[ "$line" != "" && $(echo $line \| cut -c1) != "#" ]]
609	then
610	HOSTNAME=`echo $line \| cut -d" " -s -f2`
611	host_identifier=`echo $line \| cut -d" " -s -f3`
612	if [[ $localhost_realname = $HOSTNAME ]]
613	then
614	localhost=$host_identifier
615	break
616	fi
617	fi
618	done < tmp_mrun
619
620	if [[ "$localhost" = "" ]]
621	then
622	printf "\n\n +++ no host identifier found in configuration file \"$config_file\""
623	printf "\n for local host \"$localhost_realname\"."
624	printf "\n Please add line"
625	printf "\n \"\%host_identifier $localhost_realname <identifier>\""
626	printf "\n to the configuration file."
627	locat=localhost; exit
628	fi
629
630
631	# HOSTSPEZIFISCHE VARIABLEN SETZEN
632	case $localhost_realname in
633	(r1\|r2\|h01\|b01) archive_system=tivoli;;
634	(cs*) archive_system=ut;;
635	(fimm.bccs.uib.no) fimm=true;;
636	(gate\|n-sx) PATH=$PALM_BIN:$PATH:/usr/bin/nqsII;;
637	(scirocco) scirocco=true;;
638	esac
639
640
641
642	# BASISNAME DER INPUT-DATEIEN GLEICH ALLGEMEINEM BASISNAMEN SETZEN,
643	# WENN NICHT VOM BENUTZER ANDERS BESTIMMT
644	[[ "$afname" = "" ]] && afname=$fname
645
646
647	# EVTL. RUN-MODUS DEN I/O-LISTEN HINZUFUEGEN
648	if [[ "$run_mode" != "" ]]
649	then
650	input_list="$input_list $run_mode"
651	output_list="$output_list $run_mode"
652	fi
653
654
655	# RECHNERNAMEN ABSPEICHERN, VON DEM AUS JOB GESTARTET WIRD,
656	# ALLERDINGS NUR DANN, WENN NICHT PER OPTION -H BEREITS EIN WERT
657	# ZUGEWIESEN WURDE (MRUN MACHT DIES IMMER, WENN ES SELBST BATCH-JOBS
658	# STARTET)
659	if [[ "$fromhost" = "" ]]
660	then
661	fromhost=$localhost
662	fi
663
664
665	# PRUEFEN, OB MRUN NUR TESTWEISE AUF DER LOKALEN MASCHINE KOMPILIEREN SOLL
666	if [[ "$source_list" = LOCAL_COMPILE_TEST ]]
667	then
668	source_list=TEST
669	local_compile=true
670	host=$localhost
671	fi
672
673
674	# PRUEFEN, OB AUF REMOTE-MASCHINE GERECHNET WERDEN SOLL
675	# WERT VON do_remote WIRD FUER DATEIVERBINDUNGEN BENOETIGT.
676	# WENN AUF REMOTE-MASCHINE GERECHNET WIRD, IST GLEICHZEITIG KLAR,
677	# DASS EIN BATCH-JOB GESTARTET WERDEN MUSS
678	if [[ -n $host && "$host" != $localhost ]]
679	then
680	do_batch=true
681	do_remote=true
682	case $host in
683	(ibm\|ibmh\|ibmkisti\|ibmku\|ibms\|ibmy\|nech\|necriam\|lckyoto\|lcsgib\|lcsgih\|unics\|lcxt4\|lcxt5m\|lck\|lckordi) true;;
684	(*) printf "\n"
685	printf "\n +++ sorry: execution of batch jobs on remote host \"$host\""
686	printf "\n is not available"
687	locat=nqs; (( iec = 0 )); exit;;
688	esac
689	else
690	host=$localhost
691	fi
692
693
694	# ZUSATZBEDINGUNGEN (OPTION -K) AUSWERTEN
695	if [[ -n $additional_conditions ]]
696	then
697	# echo $additional_conditions \| cut -d" " -f1-3 \| read cond1 cond2 dummy
698	cond1=`echo $additional_conditions \| cut -d" " -f1`
699	cond2=`echo $additional_conditions \| cut -d" " -s -f2`
700	dummy=`echo $additional_conditions \| cut -d" " -s -f3`
701	if [[ -n $dummy ]]
702	then
703	printf "\n +++ more than 2 additional conditions given for Option \"-K\""
704	locat=options; exit
705	fi
706	block=_$cond1
707	[[ -n $cond2 ]] && block=${block}_$cond2
708	fi
709
710
711	# KOPPLUNGSEIGENSCHAFTEN (-Y) AUSWERTEN UND coupled_mode BESTIMMEN
712	if [[ $run_coupled_model = true ]]
713	then
714
715	if [[ -n $coupled_dist ]]
716	then
717
718	numprocs_atmos=`echo $coupled_dist \| cut -d" " -s -f1`
719	numprocs_ocean=`echo $coupled_dist \| cut -d" " -s -f2`
720
721	if (( $numprocs_ocean + $numprocs_atmos != $numprocs ))
722	then
723
724	printf "\n +++ number of processors does not fit to specification by \"-Y\"."
725	printf "\n PEs (total) : $numprocs"
726	printf "\n PEs (atmosphere): $numprocs_atmos"
727	printf "\n PEs (ocean) : $numprocs_ocean"
728	locat=coupling; exit
729
730	fi
731
732	else
733
734	(( numprocs_ocean = $numprocs / 2 ))
735	(( numprocs_atmos = $numprocs / 2 ))
736
737	fi
738	coupled_dist=`echo "$numprocs_atmos $numprocs_ocean"`
739
740	# GET coupled_mode FROM THE CONFIG FILE
741	line=""
742	grep "%cpp_options.-D__mpi2.$host" $config_file > tmp_mrun
743	while read line
744	do
745	if [[ "$line" != "" && $(echo $line \| cut -c1) != "#" && ( $(echo $line \| cut -d" " -s -f4) = $cond1 \|\| $(echo $line \| cut -d" " -s -f4) = $cond2 ) ]]
746	then
747	coupled_mode="mpi2"
748	fi
749	done < tmp_mrun
750
751	fi
752
753
754	# PRUEFEN, OB EVTL. BEI VORHERGEHENDEM LAUF (KETTENJOB) EINE
755	# ARCHIVIERUNG FEHLGESCHLAGEN IST
756	if [[ -f ~/job_queue/ARCHIVE_ERROR_$fname ]]
757	then
758	if [[ $ignore_archive_error = false ]]
759	then
760	printf "\n +++ data archiving of previous run failed"
761	printf "\n see directory \~/job_queue on remote machine"
762	locat=archive; exit
763	else
764	printf "\n +++ warning: data archiving in a previous run failed"
765	printf "\n MRUN continues, trying to get backup copy"
766	fi
767	fi
768
769
770
771	# LESEN UND INTERPRETIEREN DER KONFIGURATIONS-DATEI VOM SHELLSCRIPT AUS
772	# VORUEBERGEHEND ZWINGEND AUF LINUX-RECHNERN
773	if [[ "$read_from_config" = false ]]
774	then
775
776	[[ $silent = false ]] && printf "\n Reading the configuration file... "
777	while read zeile
778	do
779	[[ $silent = false ]] && printf "."
780
781
782	# ZUERST EVENTUELL VORKOMMENDE ENVIRONMENT-VARIABLEN DURCH IHRE WERTE
783	# ERSETZEN
784	eval zeile=\"$zeile\"
785
786
787	# INTERPRETATION DER ZEILE
788	if [[ "$(echo $zeile)" = "" ]]
789	then
790	# LEERZEILE, KEINE AKTION
791	continue
792
793	elif [[ "$(echo $zeile \| cut -c1)" = "#" ]]
794	then
795
796	# ZEILE IST KOMMENTARZEILE
797	true
798
799	elif [[ "$(echo $zeile \| cut -c1)" = "%" ]]
800	then
801
802	# ZEILE DEFINIERT ENVIRONMENT-VARIABLE
803	zeile=$(echo $zeile \| cut -c2-)
804	# echo $zeile \| cut -d" " -f1-5 \| read var value for_host for_cond1 for_cond2
805	var=`echo $zeile \| cut -d" " -f1`
806	value=`echo $zeile \| cut -d" " -s -f2`
807	for_host=`echo $zeile \| cut -d" " -s -f3`
808	for_cond1=`echo $zeile \| cut -d" " -s -f4`
809	for_cond2=`echo $zeile \| cut -d" " -s -f5`
810
811	if [[ "$for_host" = "" \|\| ( "$for_host" = $host && "$for_cond1" = "$cond1" && "$for_cond2" = "$cond2" ) \|\| $(echo "$input_list$output_list"\|grep -c "$for_host") != 0 ]]
812	then
813
814	# BEI COMPILER- CPP- ODER LINKEROPTIONEN EVTL ":" DURCH " "
815	# ERSETZEN. "::" WIRD DURCH ":" ERSETZT.
816	value=`echo $value \| sed 's/::/%DUM%/g' \| sed 's/:/ /g' \| sed 's/%DUM%/:/g'`
817
818
819	# ENVIRONMENT-VARIABLE WIRD WERT AUS KONFIGURATIONSDATEI
820	# ZUGEWIESEN, WENN SIE SELBST NOCH KEINEN WERT UEBER DIE
821	# ENTSPRECHENDE SCRIPT-OPTION ERHALTEN HAT. SOLCHE
822	# VARIABLEN HAETTEN DANN DEN WERT "" ODER IM INTEGER-FALL DEN
823	# WERT 0. ALLGEMEINE REGEL ALSO: SCRIPT-OPTION GEHT UEBER
824	# KONFIGURATIONSDATEI
825	if [[ "$(eval echo \$$var)" = "" \|\| "$(eval echo \$$var)" = "0" ]]
826	then
827	eval $var=\$value
828
829	# EVTL. BILDSCHIRMAUSGABEN DER ENVIRONMENT-VARIABLEN
830	if [[ $do_trace = true ]]
831	then
832	printf "\n*** ENVIRONMENT-VARIABLE $var = $value"
833	fi
834	fi
835
836	# WENN ENVIRONMENT-VARIABLE HOST VEREINBART, DANN SOFORT AUSWERTEN
837	# WERT VON do-remote WIRD BEI DATEIVERBINDUNGEN BENOETIGT
838	# WENN AUF REMOTE-MASCHINE GERECHNET WIRD, IST GLEICHZEITIG KLAR,
839	# DASS EIN BATCH-JOB GESTARTET WERDEN MUSS
840	if [[ $var = host ]]
841	then
842	if [[ -n $host && "$host" != $localhost ]]
843	then
844	do_batch=true
845	do_remote=true
846	case $host in
847	(ibm\|ibmh\|ibmkisti\|ibmku\|ibms\|ibmy\|lckyoto\|lcsgib\|lcsgih\|nech\|necriam\|unics\|lcxt4\|lcxt5m\|lck\|lckordi) true;;
848	(*) printf "\n +++ sorry: execution of batch jobs on remote host \"$host\""
849	printf "\n is not available"
850	locat=nqs; exit;;
851	esac
852	else
853	host=$localhost
854	fi
855	fi
856
857	# VOM BENUTZER DEFINIERTE ENVIRONMENT VARIABLEN MUESSEN PRINZIPIELL
858	# EXPORTIERT WERDEN, DA SIE VIELLEICHT IN WEITER UNTEN AUFZURUFEN-
859	# DEN PROGRAMMEN BENOETIGT WERDEN
860	export $var
861	fi
862
863
864
865
866	elif [[ "$(echo $zeile \| cut -c1-3)" = "EC:" ]]
867	then
868
869	# ZEILE DEFINIERT ERROR-KOMMANDO
870	(( iec = iec + 1 ))
871	zeile=$(echo $zeile \| cut -c4-)
872	err_command[$iec]="$zeile"
873
874	elif [[ "$(echo $zeile \| cut -c1-3)" = "IC:" ]]
875	then
876
877	# ZEILE DEFINIERT INPUT-KOMMANDO
878	(( iic = iic + 1 ))
879	zeile=$(echo $zeile \| cut -c4-)
880	in_command[$iic]="$zeile"
881
882	elif [[ "$(echo $zeile \| cut -c1-3)" = "OC:" ]]
883	then
884
885	# ZEILE DEFINIERT OUTPUT-KOMMANDO
886	(( ioc = ioc + 1 ))
887	zeile=$(echo $zeile \| cut -c4-)
888	out_command[$ioc]="$zeile"
889
890	else
891
892	# ZEILE DEFINIERT DATEIVERBINDUNG. EINLESEN DER DATEIEIGENSCHAFTEN
893	# s2a: in/out - Feld
894	# s2b: loc - Feld (optional)
895	# s2c: tr/ar - Feld (optional)
896	# echo $zeile \| cut -d" " -f1-2 \| read s1 s2
897	s1=`echo $zeile \| cut -d" " -f1`
898	s2=`echo $zeile \| cut -d" " -s -f2`
899	s2a=$(echo $s2 \| cut -d":" -f1)
900	if [[ $(echo $s2 \| grep -c ":") = 0 ]]
901	then
902	s2b=""
903	s2c=""
904	else
905	# echo $s2 \| cut -d":" -f2-3 \| sed 's/:/ /g' \| read s2b s2c
906	s2b=`echo $s2 \| cut -d":" -f2 \| sed 's/:/ /g'`
907	s2c=`echo $s2 \| cut -d":" -s -f3 \| sed 's/:/ /g'`
908	fi
909	# echo $zeile \| cut -d" " -f3-6 \| read s3 s4 s5 s6
910	s3=`echo $zeile \| cut -d" " -f3`
911	s4=`echo $zeile \| cut -d" " -s -f4`
912	s5=`echo $zeile \| cut -d" " -s -f5`
913	s6=`echo $zeile \| cut -d" " -s -f6`
914
915
916	# ABSPEICHERN DER DATEIVERBINDUNG, FALLS IN INPUT- ODER OUTPUT-LIST
917	# VERMERKT. VARIABLE S3 KANN AUCH LISTE ENTHALTEN (FELDTRENNER ":")
918	# DATEIVERBINDUNG WIRD DANN NICHT ABGESPEICHERT UND GEPRUEFT, WENN
919	# PROGRAMMLAUF AUF REMOTE-MASCHINE ERFOLGT UND DATEI NUR LOKAL VOR-
920	# HANDEN SEIN MUSS (D.H. s2b = loc)
921	IFSALT="$IFS"; IFS="$IFS:"
922	if [[ "$s2a" = in && ! ( $do_remote = true && ( "$s2b" = loc \|\| "$s2b" = locopt ) ) ]]
923	then
924	found=false
925	for actual in $input_list
926	do
927	for formal in $s3
928	do
929	[[ $actual = $formal \|\| "$formal" = "-" ]] && found=true
930	done
931	done
932	if [[ $found = true ]]
933	then
934	(( iin = iin + 1 ))
935	localin[$iin]=$s1; transin[$iin]=$s2b; actionin[$iin]=$s2c;
936	typein[$iin]=$s3; pathin[$iin]=$s4; endin[$iin]=$s5;
937	extin[$iin]=$s6
938	fi
939	elif [[ "$s2a" = out && ! ( $do_remote = true && "$s2b" = loc ) ]]
940	then
941	found=false
942	for actual in $output_list
943	do
944	for formal in $s3
945	do
946	[[ $actual = $formal \|\| "$formal" = "-" ]] && found=true
947	done
948	done
949	if [[ $found = true ]]
950	then
951	(( iout = iout + 1 ))
952	localout[$iout]=$s1; actionout[$iout]=$s2c; typeout[$iout]=$s3;
953	pathout[$iout]=$s4; endout[$iout]=$s5; extout[$iout]=$s6
954	fi
955	elif [[ "$s2a" != in && "$s2a" != out ]]
956	then
957	printf "\n +++ I/O-attribute in configuration file $config_file has the invalid"
958	printf "\n value \"$s2\". Only \"in\" and \"out\" are allowed!"
959	locat=connect; exit
960	fi
961	IFS="$IFSALT"
962	fi
963	done < $config_file
964
965	else
966
967
968	# INTERPRETATION DER KONFIGURATIONSDATEI MITTELS FORTRAN 90 - PROGRAMM
969	[[ $silent = false ]] && printf "..."
970	export cond1 cond2 config_file do_remote do_trace input_list localhost output_list
971	export interpreted_config_file=.icf.$RANDOM
972
973
974
975	# ENVIRONMENT-VARIABLEN FUER INTERPRET_CONFIG UEBER NAMELIST_DATEI ZUR
976	# VERFUEGUNG STELLEN
977	cat > .mrun_environment << %%END%%
978	&mrun_environment cond1 = '$cond1', cond2 = '$cond2',
979	config_file = '$config_file', do_remote = '$do_remote',
980	do_trace = '$do_trace', host = '$host',
981	input_list = '$input_list', icf = '$interpreted_config_file',
982	localhost = '$localhost', output_list = '$output_list' /
983
984	%%END%%
985
986	# WERTE VON MRUN-OPTIONEN SICHERN UND DAMIT GEGEBENENFALLS SPAETER DIE
987	# IN DER KONFIGURAIONSDATEI ANGEGEBENEN WERTE UEBERSTEUERN
988	mrun_memory=$memory
989	mrun_group_number=$group_number
990	mrun_cpumax=$cpumax
991	mrun_numprocs=$numprocs
992
993	if [[ $localhost_realname = "sx-fep" ]]
994	then
995	/home/COAR/NC/raasch/pub/interpret_config_necriam.x
996	else
997
998	if [[ "$host" != $localhost ]]
999	then
1000
1001	# REMOTE JOB FROM LOCAL HOST: JUST TAKE THE FIRST EXECUTABLE FOUND
1002	interpret_config_executable=`ls -1 ${PALM_BIN}/interpret_config*.x 2>/dev/null`
1003	if [[ $? != 0 ]]
1004	then
1005	printf "\n\n +++ no interpret_config found"
1006	printf "\n run \"mbuild -u -h ...\" to generate utilities for this host"
1007	locat=interpret_config; exit
1008	fi
1009	interpret_config_executable=`echo $interpret_config_executable \| cut -d" " -f1`
1010	$interpret_config_executable
1011
1012	else
1013
1014	# CHECK, IF THERE IS AN EXECUTABLE FOR THE BLOCK
1015	if [[ ! -f ${PALM_BIN}/interpret_config${block}.x ]]
1016	then
1017	printf "\n\n +++ no interpret_config found for given block \"$cond1 $cond2\""
1018	printf "\n run \"mbuild -u -h ...\" to generate utilities for this block"
1019	locat=interpret_config; exit
1020	else
1021	interpret_config${block}.x
1022	fi
1023
1024	fi
1025	fi
1026	rm .mrun_environment
1027
1028
1029	# AUSFUEHRUNG DER GENERIERTEN SHELL-KOMMANDOS IN DIESER SHELL
1030	chmod u+x $interpreted_config_file
1031	export PATH=$PATH:.
1032	. $interpreted_config_file
1033	rm $interpreted_config_file
1034
1035
1036	# OPTIONSWERTE UEBERSTEUERN KONFIGURATIONSDATEI
1037	[[ $mrun_memory != 0 ]] && memory=$mrun_memory
1038	[[ "$mrun_group_number" != "none" ]] && group_number=$mrun_group_number
1039	[[ $mrun_cpumax != 0 ]] && cpumax=$mrun_cpumax
1040	[[ "$mrun_numprocs" != "" ]] && numprocs=$mrun_numprocs
1041
1042	fi
1043
1044
1045	# QUELLTEXTVERZEICHNIS AUF LOKALER MASCHINE AUS KONFIGURATIONSDATEI
1046	# BESTIMMEN (WUERDE SONST EVTL. DAS VERZEICHNIS DES JEWEILS UNTER -h
1047	# ANGEGEBENEN REMOTE-RECHNERS SEIN)
1048	# BEI BATCH-JOBS SIND DIE ZU UEBERSETZENDEN PROGRAMMTEILE SCHON KOMPLETT
1049	if [[ "$SOURCES_COMPLETE" = "" ]]
1050	then
1051
1052	# ZUERST PRUEFEN, OB EIN GLOBALER QUELLTEXTPFAD FUER ALLE RECHNER
1053	# VEREINBART WURDE
1054	source_path=""
1055	line=""
1056	grep "%source_path" $config_file > tmp_mrun
1057	while read line
1058	do
1059	if [[ "$line" != "" && $(echo $line \| cut -c1) != "#" ]]
1060	then
1061	if [[ "$(echo $line \| cut -d" " -f3)" = "" ]]
1062	then
1063	global_source_path=`echo $line \| cut -d" " -f2`
1064	fi
1065	fi
1066	done < tmp_mrun
1067
1068	line=""
1069	found=false
1070	grep " $localhost" $config_file \| grep "%source_path" > tmp_mrun
1071	while read line
1072	do
1073	if [[ "$line" != "" && $(echo $line \| cut -c1) != "#" ]]
1074	then
1075	if [[ $found = true ]]
1076	then
1077	printf "\n\n +++ more than one source path found in configuration file"
1078	printf "\n for local host \"$localhost\" "
1079	locat=source_path; exit
1080	fi
1081	source_path=`echo $line \| cut -d" " -f2`
1082	found=true
1083	fi
1084	done < tmp_mrun
1085	rm tmp_mrun
1086
1087	if [[ "$source_path" = "" ]]
1088	then
1089	if [[ "$global_source_path" != "" ]]
1090	then
1091	source_path=$global_source_path
1092	else
1093	printf "\n\n +++ no source path found in configuration file"
1094	printf "\n for local host \"$localhost\" "
1095	locat=source_path; exit
1096	fi
1097	fi
1098	eval source_path=$source_path
1099
1100	if [[ ! -d $source_path ]]
1101	then
1102	printf "\n\n +++ source path \"$source_path\" on local host"
1103	printf "\n \"$localhost\" does not exist"
1104	locat=source_path; exit
1105	fi
1106
1107	fi
1108
1109
1110	# GLOBALE REVISIONSNUMMER ERMITTELN (FORTSETZUNGSLAEUFEN WIRD DIESE
1111	# DURCH OPTION -G MITGETEILT)
1112	if [[ "$global_revision" = "" && $host != "ibmkisti" ]]
1113	then
1114	global_revision=`svnversion $source_path 2>/dev/null`
1115	global_revision="Rev: $global_revision"
1116	fi
1117
1118
1119	# NOCHMAL PRUEFEN, OB AUF REMOTE-MASCHINE GERECHNET WERDEN SOLL
1120	# (HOST KANN IN KONFIGURATIONSDATEI ANDERS FESTGELEGT WORDEN SEIN)
1121	# WERT VON do_remote WIRD FUER DATEIVERBINDUNGEN BENOETIGT.
1122	# WENN AUF REMOTE-MASCHINE GERECHNET WIRD, IST GLEICHZEITIG KLAR,
1123	# DASS EIN BATCH-JOB GESTARTET WERDEN MUSS
1124	if [[ -n $host && "$host" != $localhost ]]
1125	then
1126	do_batch=true
1127	do_remote=true
1128	case $host in
1129	(ibm\|ibmh\|ibmkisti\|ibmku\|ibms\|ibmy\|lckyoto\|lcsgib\|lcsgih\|nech\|necriam\|unics\|lcxt4\|lcxt5m\|lck\|lckordi) true;;
1130	(*) printf "\n"
1131	printf "\n +++ sorry: execution of batch jobs on remote host \"$host\""
1132	printf "\n is not available"
1133	locat=nqs; (( iec = 0 )); exit;;
1134	esac
1135	else
1136	host=$localhost
1137	fi
1138
1139
1140	# PRUEFUNG EINIGER PROZESSORZAHLANGABEN BEI RECHNUNGEN AUF PARALLELRECHNERN
1141	if [[ "$cond1" = parallel \|\| "$cond2" = parallel ]]
1142	then
1143
1144	# PRUEFEN, OB DIE ANZAHL DER ZU VERWENDENDEN PES ANGEGEBEN WURDE
1145	if [[ ! -n $numprocs ]]
1146	then
1147	printf "\n"
1148	printf "\n +++ option \"-K parallel\" requires additional specification"
1149	printf "\n of the number of processors to be used by"
1150	printf "\n mrun-option \"-X\" or by environment-variable"
1151	printf "\n \"numprocs\" in the configuration file"
1152	locat=numprocs; (( iec = 0 )); exit
1153	fi
1154
1155	# PRUEFEN, OB DIE PROZESSORANZAHL PRO KNOTEN ANGEGEBEN WURDE (GGF.
1156	# DEFAULT-WERT SETZEN) UND OB SIE EIN GANZZAHLIGER TEILER DER
1157	# GESAMTPROZESSORANZAHL IST
1158	if [[ $host = nech \|\| $host = necriam \|\| $host = ibmh \|\| $host = ibmkisti \|\| $host = ibms ]]
1159	then
1160	[[ "$tasks_per_node" = "" ]] && tasks_per_node=6
1161	(( ival = $tasks_per_node ))
1162	(( pes = numprocs ))
1163	# if [[ $(echo $package_list \| grep -c dvrp_graphics+1PE) = 1 ]]
1164	# then
1165	# (( pes = pes - 1 ))
1166	# fi
1167	(( ii = pes / ival ))
1168	if (( pes - ii * ival > 0 ))
1169	then
1170	printf "\n"
1171	printf "\n +++ tasks per node (option \"-T\") must be an integral"
1172	printf "\n divisor of the total number of processors (option \"-X\")"
1173	printf "\n values of this mrun-call: \"-T $tasks_per_node\" \"-X $numprocs\""
1174	locat=tasks_per_node; (( iec = 0 )); exit
1175	fi
1176	fi
1177
1178	# IBMY HAT NUR EINEN KNOTEN
1179	if [[ $host = ibmy ]]
1180	then
1181	if [[ "$tasks_per_node" != "" && "$tasks_per_node" != "$numprocs" ]]
1182	then
1183	printf "\n"
1184	printf "\n +++ on ibmy, tasks per node (option \"-T\") must be equal to the"
1185	printf "\n total number of processors (option \"-X\")"
1186	printf "\n values of this mrun-call: \"-T $tasks_per_node\" \"-X $numprocs\""
1187	locat=tasks_per_node; (( iec = 0 )); exit
1188	fi
1189	fi
1190
1191	# FALLS OPENMP PARALLELISIERUNG VERWENDET WERDEN SOLL, ANZAHL VON THREADS
1192	# SETZEN UND ZAHL DER TASKS PRO KNOTEN AUF 1 SETZEN
1193	# if [[ $use_openmp = true ]]
1194	# then
1195	# threads_per_task=$tasks_per_node
1196	# tasks_per_node=1
1197	# fi
1198
1199	# SETTINGS FOR SUBJOB-COMMAND
1200	TOPT="-T $tasks_per_node"
1201	OOPT="-O $threads_per_task"
1202
1203	# GESAMTZAHL DER KNOTEN BESTIMMEN
1204	if [[ "$tasks_per_node" != "" ]]
1205	then
1206	(( nodes = numprocs / ( tasks_per_node * threads_per_task ) ))
1207	fi
1208
1209	# PRUEFEN, OB NODE USAGE EINEN ERLAUBTEN WERT HAT BZW. DEN DEFAULT
1210	# SETZEN
1211	if [[ $node_usage = default ]]
1212	then
1213	if [[ $host = ibms ]]
1214	then
1215	node_usage=shared
1216	elif [[ $(echo $host \| cut -c1-5) = lcsgi ]]
1217	then
1218	node_usage=singlejob
1219	else
1220	node_usage=not_shared
1221	fi
1222	fi
1223
1224	if [[ $node_usage != shared && $node_usage != not_shared && $node_usage != singlejob && "$(echo $node_usage \| cut -c1-3)" != "sla" ]]
1225	then
1226	printf "\n"
1227	printf "\n +++ node usage (option \"-n\") is only allowed to be set"
1228	printf "\n \"shared\" or \"not_shared\""
1229	locat=tasks_per_node; (( iec = 0 )); exit
1230	fi
1231
1232	fi
1233
1234	# PRUEFEN, OB HOSTFILE EXISTIERT
1235	if [[ -n $hostfile ]]
1236	then
1237	if [[ ! -f $hostfile ]]
1238	then
1239	printf "\n"
1240	printf "\n +++ hostfile \"$hostfile\" does not exist"
1241	locat=hostfile; exit
1242	fi
1243	fi
1244
1245	# PRUEFEN, OB RHOSTS DATEI EXISTIERT. GEGEBENENFALLS ANLEGEN BZW. ERWEITERN
1246	if [[ $host = ibmy && $do_remote = false ]]
1247	then
1248	if [[ ! -f $HOME/.rhosts ]]
1249	then
1250	echo "gfdl5.yonsei.ac.kr" > $HOME/.rhosts
1251	printf "\n\n *** file:"
1252	printf "\n $HOME/.rhosts"
1253	printf "\n was created\n"
1254	fi
1255	if [[ $(grep -c gfdl5.yonsei.ac.kr $HOME/.rhosts) = 0 ]]
1256	then
1257	echo "gfdl5.yonsei.ac.kr" >> $HOME/.rhosts
1258	printf "\n\n *** file:"
1259	printf "\n $HOME/.rhosts"
1260	printf "\n was extended by the name of the current host\n"
1261	fi
1262	fi
1263
1264
1265
1266	# FALLS NICHT VORGEGEBEN, DEFAULT-QUEUE AUF DER ZIELMASCHINE FESTLEGEN
1267	if [[ $queue = none ]]
1268	then
1269	case $host in
1270	(ibmh) queue=no_class;;
1271	(ibmkisti) queue=class.32plus;;
1272	(ibmy) queue=parallel;;
1273	(lckyoto) queue=ph;;
1274	(lcsgib) queue=bigq;;
1275	(lcsgih) queue=bigq;;
1276	(lctit) queue=S;;
1277	(nech) queue=none;;
1278	(necriam) queue=SP;;
1279	(unics) queue=unics;;
1280	esac
1281	fi
1282
1283
1284	# VOLLSTAENDIGE DATEINAMEN DER INPUT-FILES BILDEN,
1285	# INPUT-DATEIEN AUF VORHANDENSEIN PRUEFEN UND EVTL. HOECHSTE ZYKLUSNUMMER
1286	# ERMITTELN
1287	(( i = 0 ))
1288	while (( i < iin ))
1289	do
1290	(( i = i + 1 ))
1291	(( maxcycle = 0 ))
1292
1293	# NAMENSBILDUNG (EVTL. IST FESTER DATEINAME VORGEGEBEN)
1294	if [[ "${actionin[$i]}" = di ]]
1295	then
1296	remotepathin[$i]=${pathin[$i]}/${endin[$i]} # REMOTE-PFAD ERST AUF
1297	# REM-MASCHINE AUSWERTEN
1298	eval filename=${pathin[$i]}/${endin[$i]}
1299	else
1300	remotepathin[$i]=${pathin[$i]}/${afname}${endin[$i]} # REMOTE-PFAD ERST AUF
1301	# REM-MASCHINE AUSWERTEN
1302	eval filename=${pathin[$i]}/${afname}${endin[$i]}
1303	fi
1304
1305	# PRUEFEN AUF VORHANDENSEIN
1306	if [[ $(ls $filename* 2>&1 \| grep -c "not found") = 1 \|\| \
1307	$(ls $filename* 2>&1 \| grep -c "nicht gefunden") = 1 \|\| \
1308	$(ls $filename* 2>&1 \| grep -c "No such file") = 1 \|\| \
1309	$(ls $filename* 2>&1 \| grep -c "does not exist") = 1 ]]
1310	then
1311	# DATEIEN MIT EXTENSION (Z.B. NC) MUESSEN NICHT ZWINGEND VORHANDEN
1312	# SEIN, DESHALB IN SOLCHEN FAELLEN KEIN ABBRUCH. DIES IST NUR EINE
1313	# VORUEBERGEHENDE LOESUNG (OKT 05). NICHT ZWINGEND BENOETIGTE
1314	# EINGABEDATEIEN SOLLTEN EINE SPEZIELLE OPTION IN DER DATEI-
1315	# VERBINDUNGSANWEISUNG BEKOMMEN (Z.B. inopt?)
1316	if [[ "${transin[$i]}" != "locopt" ]]
1317	then
1318	printf "\n\n +++ INPUT-file: "
1319	if [[ "${extin[$i]}" = "" ]]
1320	then
1321	printf "\n $filename"
1322	else
1323	printf "\n $filename.${extin[$i]}"
1324	fi
1325	printf "\n does not exist\n"
1326	locat=input; exit
1327	else
1328	transin[$i]="unavailable"
1329	fi
1330	else
1331
1332	# ZYKLUSNUMMER FESTSTELLEN
1333	ls -1 -d $filename > filelist 2>/dev/null
1334	ls -1 -d $filename.* >> filelist 2>/dev/null
1335	while read zeile
1336	do
1337	cycle=$(print $zeile \| cut -f2 -d".")
1338	if [[ "$cycle" = "$zeile" ]]
1339	then
1340	(( icycle = 0 ))
1341	elif [[ "$cycle" = "${extin[$i]}" ]]
1342	then
1343	(( icycle = 0 ))
1344	else
1345	(( icycle = $cycle ))
1346	fi
1347	if (( icycle > maxcycle ))
1348	then
1349	(( maxcycle = icycle ))
1350	file_to_be_used=$zeile
1351	fi
1352	done <filelist
1353	rm filelist
1354
1355	# EVTL. ZYKLUSNUMMER AN DATEINAMEN ANFUEGEN
1356	if (( maxcycle > 0 ))
1357	then
1358	if [[ "${extin[$i]}" != " " && "${extin[$i]}" != "" ]]
1359	then
1360	filename=${filename}.$maxcycle.${extin[$i]}
1361	else
1362	filename=${filename}.$maxcycle
1363	fi
1364	else
1365	if [[ "${extin[$i]}" != " " && "${extin[$i]}" != "" ]]
1366	then
1367	filename=${filename}.${extin[$i]}
1368	fi
1369	fi
1370
1371	# DATEINAMEN OHNE PFAD ABER MIT ZYKLUSNUMMER ABSPEICHERN,
1372	# DA SPAETER BEI RUECKSPEICHERN VOM ARCHIVSYSTEM EVTL. BENOETIGT
1373	absnamein[$i]=$filename
1374	if (( maxcycle > 0 ))
1375	then
1376	if [[ "${actionin[$i]}" = di ]]
1377	then
1378	frelin[$i]=${endin[$i]}.$maxcycle
1379	else
1380	frelin[$i]=${afname}${endin[$i]}.$maxcycle
1381	fi
1382	else
1383	if [[ "${actionin[$i]}" = di ]]
1384	then
1385	frelin[$i]=${endin[$i]}
1386	else
1387	frelin[$i]=${afname}${endin[$i]}
1388	fi
1389	fi
1390
1391	fi
1392	done
1393
1394
1395	# VOLLSTAENDIGE DATEINAMEN (OHNE $ ODER ~) DER OUTPUT-FILES BILDEN,
1396	# OUTPUT-DATEIEN AUF VORHANDENSEIN PRUEFEN UND EVTL. HOECHSTE ZYKLUSNUMMER
1397	# ERMITTELN ODER, FALLS NICHT VORHANDEN, PRUEFEN, OB SIE SICH ANLEGEN LASSEN
1398	# DIESE AKTIONEN FINDEN NICHT STATT, WENN PROGRAMM AUF REMOTE-MASCHINE
1399	# RECHNET UND DATEI ANSCHLIESSEND TRANSFERIERT WERDEN SOLL!
1400	(( i = 0 ))
1401	while (( i < iout ))
1402	do
1403	(( i = i + 1 ))
1404	if [[ ! ( $fromhost != $localhost && ( "${actionout[$i]}" = tr \|\| "${actionout[$i]}" = tra \|\| "${actionout[$i]}" = trpe ) ) ]]
1405	then
1406	if [[ "${actionout[$i]}" = tr ]]
1407	then
1408	actionout[$i]=""
1409	elif [[ "${actionout[$i]}" = trpe ]]
1410	then
1411	actionout[$i]=pe
1412	elif [[ "${actionout[$i]}" = tra ]]
1413	then
1414	actionout[$i]=a
1415	fi
1416	(( maxcycle = 0 ))
1417	eval filename=${pathout[$i]}/${fname}${endout[$i]}
1418	eval catalogname=${pathout[$i]}
1419	if [[ $(ls $filename* 2>&1 \| grep -c "not found") = 1 \|\| \
1420	$(ls $filename* 2>&1 \| grep -c "nicht gefunden") = 1 \|\| \
1421	$(ls $filename* 2>&1 \| grep -c "No such file") = 1 \|\| \
1422	$(ls $filename* 2>&1 \| grep -c "does not exist") = 1 ]]
1423	then
1424
1425	# OUTPUT-DATEI NICHT VORHANDEN. PRUEFEN, OB ANLEGEN MOEGLICH.
1426	if cat /dev/null > $filename
1427	then
1428	rm $filename
1429	else
1430
1431	# PRUEFEN, OB KATALOG VORHANDEN UND EVTL. DIESEN ANLEGEN
1432	if [[ ! -d $catalogname ]]
1433	then
1434	if mkdir -p $catalogname
1435	then
1436	printf "\n\n *** directory:"
1437	printf "\n $catalogname"
1438	printf "\n was created\n"
1439	else
1440	printf "\n\n +++ OUTPUT-file:"
1441	printf "\n $filename"
1442	printf "\n cannot be created, because directory does not exist"
1443	printf "\n and cannot be created either"
1444	printf "\n"
1445	locat=output ; exit
1446	fi 2>/dev/null
1447	else
1448	printf "\n\n +++ OUTPUT-file:"
1449	printf "\n $filename"
1450	printf "\n cannot be created, although directory exists"
1451	printf "\n"
1452	locat=output ; exit
1453	fi
1454	fi 2>/dev/null
1455	else
1456
1457	# ZYKLUSNUMMER FESTSTELLEN
1458	ls -1 -d $filename > filelist 2>/dev/null
1459	ls -1 -d $filename.* >> filelist 2>/dev/null
1460	while read zeile
1461	do
1462	cycle=$(print $zeile \| cut -f2 -d".")
1463	if [[ "$cycle" = "$zeile" \|\| "$cycle" = ${extout[$i]} ]]
1464	then
1465	(( icycle = 1 ))
1466	else
1467	(( icycle = $cycle + 1 ))
1468	fi
1469	if (( icycle > maxcycle ))
1470	then
1471	(( maxcycle = icycle ))
1472	fi
1473	done <filelist
1474	rm filelist
1475	fi
1476
1477	# EVTL. ZYKLUSNUMMER AN DATEINAMEN ANFUEGEN UND PRUEFEN, OB SICH
1478	# DATEI AUCH WIRKLICH ANLEGEN LAESST. IM APPEND-FALL MUSS AN DEN
1479	# HOECHSTEN VORHANDENEN ZYKLUS-ANGEHAENGT WERDEN. ANLAGEPRUEFUNG
1480	# IST DANN NATUERLICH NICHT ZULAESSIG BZW. MOEGLICH
1481	if [[ "${actionout[$i]}" != a ]]
1482	then
1483	if (( maxcycle > 0 ))
1484	then
1485	filename=${filename}.$maxcycle
1486	if cat /dev/null > $filename
1487	then
1488	rm $filename
1489	else
1490	printf "\n +++ OUTPUT-file:"
1491	printf "\n $filename"
1492	printf "\n cannot be created"
1493	locat=output ; exit
1494	fi
1495	fi
1496	else
1497	(( maxcycle = maxcycle - 1 ))
1498	if (( maxcycle > 0 ))
1499	then
1500	filename=${filename}.$maxcycle
1501	fi
1502	fi
1503
1504	# DATEINAMEN OHNE PFAD ABER MIT ZYKLUSNUMMER ABSPEICHERN,
1505	# DA SPAETER BEI ABLAGE AUF ARCHIVSYSTEM BZW. FUER
1506	# DATEI OUTPUT_FILE_CONNECTIONS EVTL. BENOETIGT
1507	pathout[$i]=$filename
1508	if (( maxcycle > 0 ))
1509	then
1510	frelout[$i]=${fname}${endout[$i]}.$maxcycle
1511	else
1512	frelout[$i]=${fname}${endout[$i]}
1513	fi
1514
1515	fi
1516	done
1517
1518
1519	# DAS DVR-PAKET ERFORDERT EINE ENTSPRECHENDE BIBLIOTHEK
1520	if [[ $(echo $package_list \| grep -c dvrp_graphics) != 0 ]]
1521	then
1522	if [[ "$dvr_inc" = "" ]]
1523	then
1524	printf "\n\n +++ no value for \"dvr_inc\" given in configuration file"
1525	printf "\n This is required for the dvrp_graphics package.\n"
1526	locat=dvr; exit
1527	fi
1528	if [[ "$dvr_lib" = "" ]]
1529	then
1530	printf "\n\n +++ no value for \"dvr_lib\" given in configuration file"
1531	printf "\n This is required for the dvrp_graphics package.\n"
1532	locat=dvr; exit
1533	fi
1534	fi
1535
1536
1537	# PRUEFEN, OB ENTWEDER HAUPTPROGRAMM ODER NUR EIN AUSFUEHRBARES
1538	# PROGRAMM VEREINBART WURDE (IN DIESEM FALL BRAUCHT IM WEITEREN NICHT
1539	# UEBERSETZT ZU WERDEN)
1540	if [[ "$mainprog" = "" && "$executable" = "" ]]
1541	then
1542	printf "\n +++ neither main program nor executable defined"
1543	locat=source; exit
1544	elif [[ "$mainprog" != "" && "$executable" != "" ]]
1545	then
1546	printf "\n +++ main program as well as executable defined"
1547	locat=source; exit
1548	elif [[ "$mainprog" = "" && "$executable" != "" ]]
1549	then
1550	do_compile=false
1551	fi
1552
1553
1554	# SOURCE-VERZEICHNIS ZUM AUFSAMMELN DER ZU UEBERSETZENDEN PROGRAMMTEILE
1555	# ERZEUGEN. HIERHIN WERDEN SPAETER IM FALL VON BATCH-JOBS AUCH DAS MRUN-
1556	# SCRIPTS SOWIE DIE KONFIGURATIONSDATEI KOPIERT
1557	if [[ $restart_run != true && "$SOURCES_COMPLETE" = "" ]]
1558	then
1559	rm -rf SOURCES_FOR_RUN_$fname
1560	mkdir SOURCES_FOR_RUN_$fname
1561	fi
1562
1563
1564	# ALLE ZU UEBERSETZENDEN PROGRAMMTEILE ZUSAMMENSAMMELN
1565	# BEI BATCH-JOBS IST DIES NICHT NOETIG, WEIL DIE PROGRAMMTEILE BEREITS DURCH
1566	# DEN MRUN-AUFRUF ERMITTELT SIND, DER DEN BATCH-JOB GENERIERT HAT, UND
1567	# IM VERZEICHNIS SOURCES_FOR_RUN_... ABGELEGT SIND
1568	if [[ $do_compile = true && "$SOURCES_COMPLETE" = "" ]]
1569	then
1570
1571	[[ "$source_list" = LM ]] && source_list=LOCALLY_MODIFIED
1572	[[ "$source_list" = WP ]] && source_list=WRITE_PERMIT
1573
1574
1575	# EVTL. ZU UEBERSETZENDE PROGRAMMTEILE DADURCH BESTIMMEN, DASS GEPRUEFT
1576	# WIRD, WELCHE DATEIEN WRITE-PERMIT BESITZEN (DATEIEN SOLLTEN DESHALB
1577	# UNTER RCS-KONTROLLE STEHEN). MIT DER VARIABLEN EXCLUDE KOENNEN
1578	# BESTIMMTE DATEIEN GRUNDSAETZLICH VON DER UEBERSETZUNG AUSGESCHLOSSEN
1579	# WERDEN. DIES IST NUR BEI MANUELLEM START VON MRUN DURCH DEN BENUTZER
1580	# MOEGLICH, DA BEI AUTOMATISCHEN RESTARTS TEST DURCH DIE TATSAECHLICH
1581	# ZU UBERSETZENDEN DATEIEN ERSETZT IST.
1582	# ALLE ERMITTELTEN DATEIEN WERDEN IM VERZEICHNIS SOURCES_FOR_RUN_...
1583	# GESAMMELT
1584	if [[ "$source_list" = WRITE_PERMIT ]]
1585	then
1586
1587	source_list=""
1588	cd $source_path
1589
1590
1591	# ALLE MOEGLICHEN QUELLCODEDATEIEN AUFLISTEN
1592	Names=$(ls -1 *.f90 2>&1)
1593	[[ $(echo $Names \| grep -c '*.f90') = 0 ]] && Filenames="$Names"
1594	Names=$(ls -1 *.F90 2>&1)
1595	[[ $(echo $Names \| grep -c '*.F90') = 0 ]] && Filenames="$Filenames $Names"
1596	Names=$(ls -1 *.F 2>&1)
1597	[[ $(echo $Names \| grep -c '*.F') = 0 ]] && Filenames="$Filenames $Names"
1598	Names=$(ls -1 *.f 2>&1)
1599	[[ $(echo $Names \| grep -c '*.f') = 0 ]] && Filenames="$Filenames $Names"
1600	Names=$(ls -1 *.c 2>&1)
1601	[[ $(echo $Names \| grep -c '*.c') = 0 ]] && Filenames="$Filenames $Names"
1602
1603
1604	# DATEIEN MIT WRITE-PERMIT NACH SOURCES_FOR_RUN_... KOPIEREN
1605	for dateiname in $Filenames
1606	do
1607	if [[ -w $dateiname ]]
1608	then
1609	if [[ "$exclude" = "" \|\| $(echo $exclude \| grep -c $dateiname) = 0 ]]
1610	then
1611	cp $dateiname $working_directory/SOURCES_FOR_RUN_$fname
1612	source_list=$source_list"$dateiname "
1613	fi
1614	fi
1615	done
1616
1617	cd - > /dev/null
1618
1619
1620	elif [[ "$source_list" = LOCALLY_MODIFIED ]]
1621	then
1622
1623	# MODIFIZIERTE DATEIEN DER SVN-ARBEITSKOPIE BESTIMMEN
1624	source_list=""
1625	cd $source_path
1626
1627
1628	# PRUEFEN, OB VERZEICHNIS UEBERHAUPT UNTER SVN-KONTROLLE STEHT
1629	if [[ ! -d .svn ]]
1630	then
1631	printf "\n\n +++ source directory"
1632	printf "\n \"$source_path\" "
1633	printf "\n is not under control of \"subversion\"."
1634	printf "\n Please do not use mrun-option \"-s LOCALLY_MODIFIED\"\n"
1635	fi
1636
1637
1638	# ALLE MODIFIZIERTEN QUELLCODEDATEIEN AUFLISTEN
1639	Filenames=""
1640	svn status > tmp_mrun
1641	while read line
1642	do
1643	firstc=`echo $line \| cut -c1`
1644	if [[ $firstc = M \|\| $firstc = "?" ]]
1645	then
1646	Name=`echo "$line" \| cut -c8-`
1647	extension=`echo $Name \| cut -d. -f2`
1648	if [[ "$extension" = f90 \|\| "$extension" = F90 \|\| "$extension" = f \|\| "$extension" = F \|\| "$extension" = c ]]
1649	then
1650	Filenames="$Filenames "$Name
1651	fi
1652	fi
1653	done < tmp_mrun
1654
1655
1656	# DATEIEN NACH SOURCES_FOR_RUN_... KOPIEREN
1657	for dateiname in $Filenames
1658	do
1659	cp $dateiname $working_directory/SOURCES_FOR_RUN_$fname
1660	source_list=$source_list"$dateiname "
1661	done
1662
1663	cd - > /dev/null
1664
1665
1666	# MITTELS OPTION -s ANGEGEBENE DATEIEN NACH SOURCES_FOR_RUN_... KOPIEREN
1667	# BEI AUTOMATISCHEN FORTSETZUNGSLAEUFEN SIND DORT SCHON ALLE DATEIEN
1668	# VORHANDEN
1669	elif [[ "$source_list" != "" && $restart_run != true ]]
1670	then
1671
1672	cd $source_path
1673
1674	for filename in $source_list
1675	do
1676
1677	# QUELLTEXT-DATEI DARF KEINE PFADE BEINHALTEN
1678	if [[ $(print $filename \| grep -c "/") != 0 ]]
1679	then
1680	printf "\n +++ source code file: $filename"
1681	printf "\n must not contain (\"/\") "
1682	locat=source; exit
1683	fi
1684
1685	if [[ ! -f $filename ]]
1686	then
1687	printf "\n +++ source code file: $filename"
1688	printf "\n does not exist"
1689	locat=source; exit
1690	else
1691	cp $filename $working_directory/SOURCES_FOR_RUN_$fname
1692	fi
1693
1694	done
1695
1696	cd - > /dev/null
1697
1698	fi
1699
1700
1701	# PRUEFEN, OB ENTWEDER HAUPTPROGRAMM VORHANDEN UND ES EVTL. IN DER
1702	# LISTE DER ZU UEBERSETZENDEN PROGRAMMTEILE MIT ENTHALTEN IST (WENN
1703	# NICHT, WIRD ES DIESER LISTE HINZUGEFUEGT)
1704	if [[ $restart_run != true ]]
1705	then
1706
1707	if [[ ! -f "$source_path/$mainprog" ]]
1708	then
1709	printf "\n\n +++ main program: $mainprog"
1710	printf "\n does not exist in source directory"
1711	printf "\n \"$source_path\"\n"
1712	locat=source; exit
1713	else
1714	if [[ $(echo $source_list \| grep -c $mainprog) = 0 ]]
1715	then
1716	cp $source_path/$mainprog SOURCES_FOR_RUN_$fname
1717	source_list=${mainprog}" $source_list"
1718	fi
1719	fi
1720	fi
1721
1722
1723	# MAKEFILE AUF VORHANDENSEIN PRUEFEN UND KOPIEREN
1724	# BEI RESTART-LAEUFEN LIEGT ES SCHON IM VERZEICHNIS SOURCES_FOR_RUN...
1725	if [[ "$restart_run" != true ]]
1726	then
1727	[[ "$makefile" = "" ]] && makefile=$source_path/Makefile
1728	if [[ ! -f $makefile ]]
1729	then
1730	printf "\n +++ file \"$makefile\" does not exist"
1731	locat=make; exit
1732	else
1733	cp $makefile SOURCES_FOR_RUN_$fname/Makefile
1734	fi
1735	fi
1736
1737
1738	# DATEIEN AUS ZUSAETZLICHEM QUELLVERZEICHNIS HINZUFUEGEN
1739	if [[ $restart_run != true && "$add_source_path" != "" ]]
1740	then
1741
1742	# GIBT ES DAS VERZEICHNIS UEBERHAUPT?
1743	if [[ ! -d $add_source_path ]]
1744	then
1745	printf "\n\n +++ WARNING: additional source code directory"
1746	printf "\n \"$add_source_path\" "
1747	printf "\n does not exist or is not a directory."
1748	printf "\n No source code will be used from this directory!\n"
1749	add_source_path=""
1750	sleep 3
1751	else
1752
1753	cd $add_source_path
1754	found=false
1755
1756	Names=$(ls -1 *.f90 2>&1)
1757	[[ $(echo $Names \| grep -c '*.f90') = 0 ]] && AddFilenames="$Names"
1758	Names=$(ls -1 *.F90 2>&1)
1759	[[ $(echo $Names \| grep -c '*.F90') = 0 ]] && AddFilenames="$AddFilenames $Names"
1760	Names=$(ls -1 *.F 2>&1)
1761	[[ $(echo $Names \| grep -c '*.F') = 0 ]] && AddFilenames="$AddFilenames $Names"
1762	Names=$(ls -1 *.f 2>&1)
1763	[[ $(echo $Names \| grep -c '*.f') = 0 ]] && AddFilenames="$AddFilenames $Names"
1764	Names=$(ls -1 *.c 2>&1)
1765	[[ $(echo $Names \| grep -c '*.c') = 0 ]] && AddFilenames="$AddFilenames $Names"
1766
1767	cd - > /dev/null
1768	cd SOURCES_FOR_RUN_$fname
1769
1770	# COPY MAKEFILE IF EXISTING
1771	if [[ -f $add_source_path/Makefile ]]
1772	then
1773	printf "\n\n *** user Makefile from directory"
1774	printf "\n \"$add_source_path\" is used \n"
1775	sleep 1
1776	cp $add_source_path/Makefile .
1777	fi
1778
1779	for dateiname in $AddFilenames
1780	do
1781	if [[ -f $dateiname ]]
1782	then
1783	printf "\n +++ source code file \"$dateiname\" found in additional"
1784	printf "\n source code directory \"$add_source_path\" "
1785	printf "\n but was also given with option \"-s\" which means that it should be taken"
1786	printf "\n from directory \"$source_path\"."
1787	locat=source; exit
1788	fi
1789
1790	cp $add_source_path/$dateiname .
1791	source_list="$source_list $dateiname"
1792
1793	# CHECK IF FILE IS CONTAINED IN MAKEFILE
1794	if [[ $(grep -c $dateiname Makefile) = 0 ]]
1795	then
1796	printf "\n\n +++ user file \"$dateiname\" "
1797	printf "\n is not listed in Makefile \n"
1798	locat=source; exit
1799	else
1800
1801	# Default User-Interface von der Liste entfernen, falls Datei
1802	# ein User-Interface enthaelt
1803	# if [[ $( cat $dateiname \| grep -c "END SUBROUTINE user_parin" ) != 0 ]]
1804	# then
1805	# if [[ $dateiname != user_interface.f90 && -f user_interface.f90 ]]
1806	# then
1807	# rm -rf user_interface.f90
1808	# source_list=`echo $source_list \| sed -e 's/user_interface.f90//'`
1809	# printf "\n\n *** default \"user_interface.f90\" removed from the files to be translated"
1810	# printf "\n since a user-interface is found in file"
1811	# printf "\n \"$add_source_path/$dateiname\" \n"
1812	# sleep 3
1813	# else
1814	if [[ $found = false ]]
1815	then
1816	found=true
1817	printf "\n\n *** following user file(s) added to the"
1818	printf " files to be translated:\n "
1819	fi
1820	printf "$dateiname "
1821	sleep 0.5
1822	fi
1823	done
1824	[[ $found = true ]] && printf "\n"
1825	cd - > /dev/null
1826	fi
1827	fi
1828
1829
1830	# ALLE UNTERPROGRAMME, DIE ZU VEREINBARTEN SOFTWAREPAKETEN GEHOEREN,
1831	# DER LISTE DER ZU UEBERSETZENDEN DATEIEN HINZUFUEGEN
1832	if [[ $restart_run != true && -n $package_list ]]
1833	then
1834
1835	cd $source_path
1836
1837	for package in $package_list
1838	do
1839
1840	[[ $package = "dvrp_graphics+1PE" ]] && package=dvrp_graphics
1841
1842	# ERMITTELE ALLE DATEIEN, DIE ZUM PAKET GEHOEREN
1843	# FEHLERMELDUNGEN WERDEN ABGEFANGEN, DA * AUCH VERZEICHNISSNAMEN
1844	# LIEFERT
1845	package_source_list=`grep "defined( __$package " * 2>/dev/null \| cut -f1 -d:`
1846
1847
1848	# FUEGE DIESE DATEIEN DER LISTE DER ZU UEBERSETZENDEN DATEIEN
1849	# HINZU, FALLS SIE NOCH NICHT DAZUGEHOEREN
1850	for source_list_name in $package_source_list
1851	do
1852	if [[ $(echo $source_list \| grep -c $source_list_name) = 0 ]]
1853	then
1854
1855	# NUR DATEIEN MIT GUELTIGEN ENDUNGEN VERWENDEN
1856	ending=`echo $source_list_name \| cut -f2 -d.`
1857	if [[ "$ending" = f90 \|\| "$ending" = F90 \|\| "$ending" = f \|\| "$ending" = F \|\| "$ending" = c ]]
1858	then
1859	cp $source_list_name $working_directory/SOURCES_FOR_RUN_$fname
1860	source_list="$source_list $source_list_name"
1861	fi
1862	fi
1863	done
1864	done
1865
1866	cd - > /dev/null
1867	fi
1868
1869
1870	# MAKEFILE AUF VORHANDENSEIN PRUEFEN UND KOPIEREN
1871	# BEI RESTART-LAEUFEN LIEGT ES SCHON IM VERZEICHNIS SOURCES_FOR_RUN...
1872	# if [[ "$restart_run" != true ]]
1873	# then
1874	# [[ "$makefile" = "" ]] && makefile=$source_path/Makefile
1875	# if [[ ! -f $makefile ]]
1876	# then
1877	# printf "\n +++ file \"$makefile\" does not exist"
1878	# locat=make; exit
1879	# else
1880	# cp $makefile SOURCES_FOR_RUN_$fname/Makefile
1881	# fi
1882	# fi
1883
1884	fi # do_compile=true
1885
1886
1887	# FALLS PROGRAMMTEILE UEBERSETZT WERDEN SOLLEN, FOLGEN JETZT EINIGE
1888	# UEBERPRUEFUNGEN UND DAS SETZEN DER PRAEPROZESSOR-DIREKTIVEN
1889	if [[ $do_compile = true ]]
1890	then
1891
1892	# PRAEPROZESSOR-DIREKTIVEN ZUM SELEKTIVEN AUSWAEHLEN VON CODETEILEN
1893	# ZUSAMMENSETZEN
1894	# DIREKTIVEN ZUM AKTIVIEREN VON RECHNERSPEZIFISCHEM CODE
1895	if [[ $(echo $localhost \| cut -c1-3) = ibm ]]
1896	then
1897	cpp_options="${cpp_options},-D__ibm=__ibm"
1898	elif [[ $(echo $localhost \| cut -c1-3) = nec ]]
1899	then
1900	cpp_options="$cpp_options -D__nec"
1901	elif [[ $(echo $localhost \| cut -c1-2) = lc ]]
1902	then
1903	cpp_options="$cpp_options -D__lc"
1904	else
1905	cpp_options="$cpp_options -D__$localhost"
1906	fi
1907
1908	# DIREKTIVEN DIE DURCH OPTION -K BESTIMMT WERDEN (Z.B. PARALLEL)
1909	if [[ $(echo $localhost \| cut -c1-3) = ibm ]]
1910	then
1911	[[ -n $cond1 ]] && cpp_options="${cpp_options},-D__$cond1=__$cond1"
1912	[[ -n $cond2 ]] && cpp_options="${cpp_options},-D__$cond2=__$cond2"
1913	else
1914	[[ -n $cond1 ]] && cpp_options="$cpp_options -D__$cond1"
1915	[[ -n $cond2 ]] && cpp_options="$cpp_options -D__$cond2"
1916	fi
1917
1918	# DIREKTIVEN DIE SOFTWAREPAKETE AKTIVIEREN (OPTION -p)
1919	if [[ -n $package_list ]]
1920	then
1921	for package in $package_list
1922	do
1923	if [[ $(echo $localhost \| cut -c1-3) = ibm ]]
1924	then
1925	if [[ $package != "dvrp_graphics+1PE" ]]
1926	then
1927	cpp_options="${cpp_options},-D__$package=__$package"
1928	else
1929	cpp_options="${cpp_options},-D__dvrp_graphics=__dvrp_graphics"
1930	export use_seperate_pe_for_dvrp_output=true
1931	fi
1932	else
1933	if [[ $package != "dvrp_graphics+1PE" ]]
1934	then
1935	cpp_options="$cpp_options -D__$package"
1936	else
1937	cpp_options="$cpp_options -D__dvrp_graphics"
1938	export use_seperate_pe_for_dvrp_output=true
1939	fi
1940	fi
1941	done
1942	fi
1943
1944	# DIREKTIVEN DIE DURCH OPTION -D FESTGELEGT SIND
1945	if [[ -n $cpp_opts ]]
1946	then
1947	for popts in $cpp_opts
1948	do
1949	if [[ $(echo $localhost \| cut -c1-3) = ibm ]]
1950	then
1951	cpp_options="${cpp_options},-D__$popts=__$popts"
1952	else
1953	cpp_options="$cpp_options -D__$popts"
1954	fi
1955	done
1956	fi
1957
1958	else
1959
1960
1961	# BEI LOKALEN RECHNUNGEN PRUEFEN, OB EXECUTABLE VORHANDEN
1962	if [[ $do_remote = false ]]
1963	then
1964	if [[ ! -f $executable ]]
1965	then
1966	printf "\n +++ executable file: $executable"
1967	printf "\n does not exist"
1968	locat=executable; exit
1969	fi
1970	fi
1971	fi
1972
1973
1974	# JOBMODUS FESTSTELLEN
1975	if [[ "$ENVIRONMENT" = BATCH ]]
1976	then
1977	jobmo=BATCH
1978	else
1979	jobmo=INTERACTIVE
1980	fi
1981
1982
1983	# no interactive runs on lctit
1984	if [[ $host = lctit && $jobmo = INTERACTIVE && $do_batch = false ]]
1985	then
1986	printf "\n +++ no interactive runs allowed on host \"$host\" "
1987	printf "\n please submit batch job using mrun option \"-b\" \n"
1988	locat=normal; exit
1989	fi
1990
1991
1992	# HOSTSPEZIFISCHE DEFAULT-COMPILER SETZEN, FALLS NICHT BEREITS
1993	# DURCH BENUTZER ANDERWEITIG VEREINBART
1994	if [[ "$compiler_name" = "" ]]
1995	then
1996
1997	printf "\n +++ no compiler specified for \"$host $cond1 $cond2\""
1998	locat=compiler_name; exit
1999
2000	fi
2001
2002
2003	# COMPILER AUF RIAMS NEC UEBERSCHREIBEN
2004	[[ $localhost = necriam ]] && compiler_name=mpif90
2005
2006
2007
2008	# TEMPORAEREN KATALOGNAMEN BESTIMMEN
2009	kennung=$RANDOM
2010	if [[ "$tmp_user_catalog" = "" ]]
2011	then
2012	if [[ $localhost = ibmh ]]
2013	then
2014	tmp_user_catalog=$SCRATCH
2015	elif [[ $localhost = nech ]]
2016	then
2017	tmp_user_catalog=$WRKSHR
2018	else
2019	tmp_user_catalog=/tmp
2020	fi
2021	fi
2022	TEMPDIR=$tmp_user_catalog/${usern}.$kennung
2023
2024
2025	# KATALOGNAMEN FUER ZWISCHENSPEICHERUNG VON FORTSETZUNGSLAUFDATEIEN
2026	# BESTIMMEN
2027	if [[ "$tmp_data_catalog" = "" ]]
2028	then
2029	if [[ $localhost = nech ]]
2030	then
2031	tmp_data_catalog=$WRKSHR/mrun_restart_data
2032	else
2033	tmp_data_catalog=/tmp/mrun_restart_data
2034	fi
2035	fi
2036
2037
2038	# EVENTUELL BEI LOKALEN RECHNUNGEN $-ZEICHEN IN ENVIRONMENT-VARIABLEN
2039	# ERSETZEN
2040	if [[ $do_remote = false && $do_compile = true ]]
2041	then
2042	eval fopts=\"$fopts\"
2043	eval lopts=\"$lopts\"
2044	fi
2045
2046
2047
2048	# COMPILE- UND LINK-OPTIONEN BESTIMMEN
2049	fopts="$fopts $netcdf_inc $dvr_inc"
2050	lopts="$lopts $netcdf_lib $dvr_lib"
2051	ROPTS="$ropts"
2052	if [[ ( $(echo $host \| cut -c1-3) = nec \|\| $(echo $host \| cut -c1-3) = ibm \|\| $host = lckyoto \|\| $host = lcsgih \|\| $host = lcsgib \|\| $host = lctit \|\| $host = lcfimm \|\| $host = lcxt4 \|\| $host = lcxt5m \|\| $host = lck \|\| $host = lckordi) && -n $numprocs ]]
2053	then
2054	XOPT="-X $numprocs"
2055	fi
2056
2057
2058
2059	# PRUEFEN DER CPU-ZEIT. (CPUMAX WIRD ALS ENV-VARIABLE VOM HAUTPRO-
2060	# GRAMM BENOETIGT
2061	done=false
2062	while [[ $done = false ]]
2063	do
2064	cputime=$cpumax
2065	if (( $cputime == 0 ))
2066	then
2067	if [[ $do_batch = true ]]
2068	then
2069	printf "\n +++ cpu-time is undefined"
2070	printf "\n >>> Please type CPU-time in seconds as INTEGER:"
2071	printf "\n >>> "
2072	read cputime 1>/dev/null 2>&1
2073	else
2074	cputime=10000000 # NO CPU LIMIT FOR INTERACTIVE RUNS
2075	fi
2076	else
2077	done=true
2078	fi
2079	cpumax=$cputime
2080	done
2081
2082	(( minuten = cputime / 60 ))
2083	(( sekunden = cputime - minuten * 60 ))
2084
2085
2086	# PRUEFEN DER KERNSPEICHERANFORDERUNG
2087	if [[ $do_batch = true ]]
2088	then
2089	done=false
2090	while [[ $done = false ]]
2091	do
2092	if (( memory == 0 ))
2093	then
2094	printf "\n +++ memory demand is undefined"
2095	printf "\n >>> Please type memory in MByte per process as INTEGER:"
2096	printf "\n >>> "
2097	read memory 1>/dev/null 2>&1
2098	else
2099	done=true
2100	fi
2101	done
2102	fi
2103
2104
2105	# PRUEFEN, OB FUER REMOTE-RECHNUNGEN EIN BENUTZERNAME ANGEGEBEN WURDE
2106	if [[ $do_remote = true && -z $remote_username ]]
2107	then
2108	while [[ -z $remote_username ]]
2109	do
2110	printf "\n +++ username on remote host \"$host\" is undefined"
2111	printf "\n >>> Please type username:"
2112	printf "\n >>> "
2113	read remote_username
2114	done
2115	mc="$mc -u$remote_username"
2116	fi
2117
2118
2119	# set module load command and export for subjob
2120	if [[ "$modules" != "" ]]
2121	then
2122	if [[ $host = lctit ]]
2123	then
2124	export module_calls=". $modules"
2125	else
2126	export module_calls="module load ${modules};"
2127	fi
2128	fi
2129
2130	# bugfix for wrong netcdf module and for netCDF4 usage in case of mpt
2131	if [[ $host = lcsgib \|\| $host = lcsgih ]]
2132	then
2133	if [[ $(echo $module_calls \| grep -c netcdf/3.6.3-intel) != 0 ]]
2134	then
2135	export module_calls="$module_calls export LD_LIBRARY_PATH=/sw/dataformats/netcdf/3.6.3-intel/lib:\$LD_LIBRARY_PATH;"
2136	fi
2137	# if [[ $(echo $module_calls \| grep -c mpt) != 0 ]]
2138	# then
2139	# export module_calls="$module_calls export LD_LIBRARY_PATH=/sw/sgi/mpt/2011-02-07/lib:\$LD_LIBRARY_PATH;"
2140	# echo "*** module_calls = $module_calls"
2141	# fi
2142	fi
2143
2144
2145	# SET DEFAULT VALUE FOR MPI MODULE TO BE USED ON SGI-ALTIX
2146	if [[ $host = lcsgib \|\| $host = lcsgih ]]
2147	then
2148	if [[ $(echo $modules \| grep -c mpt ) != 0 ]]
2149	then
2150	mpilib=mpt
2151	elif [[ $(echo $modules \| grep -c mvapich ) != 0 ]]
2152	then
2153	mpilib=mvapich
2154	elif [[ $(echo $modules \| grep -c impi ) != 0 ]]
2155	then
2156	mpilib=impi
2157	fi
2158	fi
2159
2160
2161
2162	###########################################################################
2163	# HEADER-AUSGABE
2164	###########################################################################
2165
2166
2167	calltime=$(date)
2168	printf "\n"
2169	# [[ $silent = false ]] && clear
2170	printf "#--------------------------------------------------------------# \n"
2171	printf "\| $version$calltime \| \n"
2172	printf "\| \| \n"
2173	spalte1="called on:"; spalte2=$localhost_realname
2174	printf "\| $spalte1$spalte2 \| \n"
2175	if [[ $local_compile = false ]]
2176	then
2177	if [[ $do_remote = true ]]
2178	then
2179	spalte1="execution on:"; spalte2="$host (username: $remote_username)"
2180	else
2181	spalte1="execution on:"; spalte2="$host ($localhost_realname)"
2182	fi
2183	else
2184	spalte1="compiling test only!"; spalte2=""
2185	fi
2186	printf "\| $spalte1$spalte2 \| \n"
2187	if [[ -n $numprocs ]]
2188	then
2189	spalte1="number of PEs:"; spalte2=$numprocs
2190	printf "\| $spalte1$spalte2 \| \n"
2191	fi
2192	if [[ -n $tasks_per_node ]]
2193	then
2194	spalte1="tasks per node:"; spalte2="$tasks_per_node (number of nodes: $nodes)"
2195	printf "\| $spalte1$spalte2 \| \n"
2196	fi
2197	if [[ $use_openmp = true ]]
2198	then
2199	spalte1="threads per task:"; spalte2="$threads_per_task"
2200	printf "\| $spalte1$spalte2 \| \n"
2201	fi
2202	printf "\| \| \n"
2203	if [[ $do_compile = true ]]
2204	then
2205	if [[ "$mopts" != "" ]]
2206	then
2207	spalte1="make options:"; spalte2=$mopts
2208	printf "\| $spalte1$spalte2 \| \n"
2209	zeile=$(echo "$mopts" \| cut -c41-)
2210	while [[ "$zeile" != "" ]]
2211	do
2212	spalte1=""
2213	spalte2=$zeile
2214	printf "\| $spalte1$spalte2 \| \n"
2215	zeile=$(echo "$zeile" \| cut -c41-)
2216	done
2217	fi
2218
2219	spalte1="cpp directives:"; spalte2=$cpp_options
2220	printf "\| $spalte1$spalte2 \| \n"
2221	zeile=$(echo "$cpp_options" \| cut -c41-)
2222	while [[ "$zeile" != "" ]]
2223	do
2224	spalte1=""
2225	spalte2=$zeile
2226	printf "\| $spalte1$spalte2 \| \n"
2227	zeile=$(echo "$zeile" \| cut -c41-)
2228	done
2229
2230	spalte1="compiler options:"; spalte2="$fopts"
2231	printf "\| $spalte1$spalte2 \| \n"
2232	zeile=$(echo "$fopts" \| cut -c41-)
2233	while [[ "$zeile" != "" ]]
2234	do
2235	spalte1=""
2236	spalte2=$zeile
2237	printf "\| $spalte1$spalte2 \| \n"
2238	zeile=$(echo "$zeile" \| cut -c41-)
2239	done
2240
2241	spalte1="linker options:"; spalte2=$lopts
2242	printf "\| $spalte1$spalte2 \| \n"
2243	zeile=$(echo "$lopts" \| cut -c41-)
2244	while [[ "$zeile" != "" ]]
2245	do
2246	spalte1=""
2247	spalte2=$zeile
2248	printf "\| $spalte1$spalte2 \| \n"
2249	zeile=$(echo "$zeile" \| cut -c41-)
2250	done
2251
2252	spalte1="modules to be load:"; spalte2=$modules
2253	printf "\| $spalte1$spalte2 \| \n"
2254	zeile=$(echo "$modules" \| cut -c41-)
2255	while [[ "$zeile" != "" ]]
2256	do
2257	spalte1=""
2258	spalte2=$zeile
2259	printf "\| $spalte1$spalte2 \| \n"
2260	zeile=$(echo "$zeile" \| cut -c41-)
2261	done
2262
2263	spalte1="main program":; spalte2=$mainprog
2264	printf "\| $spalte1$spalte2 \| \n"
2265	else
2266	spalte1=executable:; spalte2=$executable
2267	printf "\| $spalte1$spalte2 \| \n"
2268	fi
2269	printf "\| \| \n"
2270	spalte1="base name of files":; spalte2=$fname
2271	printf "\| $spalte1$spalte2 \| \n"
2272	if [[ $fname != $afname ]]
2273	then
2274	spalte1="base name of input files":; spalte2=$afname
2275	printf "\| $spalte1$spalte2 \| \n"
2276	fi
2277	spalte1="INPUT control list":; spalte2=$input_list
2278	printf "\| $spalte1$spalte2 \| \n"
2279	spalte1="OUTPUT control list":; spalte2=$output_list
2280	printf "\| $spalte1$spalte2 \| \n"
2281
2282	if [[ $do_batch = true \|\| "$LOADLBATCH" = yes ]]
2283	then
2284	spalte1="memory demand / PE":; spalte2="$memory MB"
2285	printf "\| $spalte1$spalte2 \| \n"
2286	spalte1=CPU-time:; spalte2="$minuten:$sekunden"
2287	printf "\| $spalte1$spalte2 \| \n"
2288	fi
2289
2290	if [[ $do_compile = true ]]
2291	then
2292	printf "\| \| \n"
2293	printf "\| Files to be compiled: \| \n"
2294	zeile=$source_list
2295	while [[ "$zeile" != "" ]]
2296	do
2297	spalte3=$zeile
2298	printf "\| $spalte3 \| \n"
2299	zeile=$(echo "$zeile" \| cut -c61-)
2300	done
2301	fi
2302	printf "#--------------------------------------------------------------#"
2303
2304
2305
2306	# BEDINGTE AUSGABE DER DATEIVERBINDUNGEN
2307	if [[ $do_trace = true ]]
2308	then
2309	(( i = 0 ))
2310	while (( i < iin ))
2311	do
2312	(( i = i + 1 ))
2313	if (( i == 1 ))
2314	then
2315	printf "\n\n >>> INPUT-file assignments:\n"
2316	fi
2317	printf "\n ${localin[$i]} : ${absnamein[$i]}"
2318	done
2319	(( i = 0 ))
2320	while (( i < iout ))
2321	do
2322	(( i = i + 1 ))
2323	if (( i == 1 ))
2324	then
2325	printf "\n\n >>> OUTPUT-file assignments:\n"
2326	fi
2327	printf "\n ${localout[$i]} : ${pathout[$i]}"
2328	done
2329	(( i = 0 ))
2330	while (( i < iic ))
2331	do
2332	(( i = i + 1 ))
2333	if (( i == 1 ))
2334	then
2335	printf "\n\n >>> INPUT-commands:\n"
2336	fi
2337	printf "\n ${in_command[$i]}"
2338	done
2339	(( i = 0 ))
2340	while (( i < ioc ))
2341	do
2342	(( i = i + 1 ))
2343	if (( i == 1 ))
2344	then
2345	printf "\n\n >>> OUTPUT-commands:\n"
2346	fi
2347	printf "\n ${out_command[$i]}"
2348	done
2349	fi
2350
2351
2352	# ABFRAGEN BEI AUFRUF AUF LOKALER MASCHINE
2353	if [[ $remotecall = false && $silent = false && $jobmo != BATCH ]]
2354	then
2355	antwort=dummy
2356	printf "\n\n"
2357	while [[ "$antwort" != y && "$antwort" != Y && "$antwort" != n && "$antwort" != N ]]
2358	do
2359	printf " >>> everything o.k. (y/n) ? "
2360	read antwort
2361	done
2362	if [[ $antwort = n \|\| $antwort = N ]]
2363	then
2364	locat=user_abort; (( iec = 0 )); exit
2365	fi
2366	if [[ $do_batch = true ]]
2367	then
2368	printf " >>> batch-job will be created and submitted"
2369	else
2370	if [[ $local_compile = false ]]
2371	then
2372	printf " >>> MRUN will now continue to execute on this machine"
2373	else
2374	printf " >>> a test compilation will now be carried out on this machine"
2375	fi
2376	fi
2377	fi
2378
2379
2380
2381
2382	# FALLS AUF DIESER MASCHINE GERECHNET WERDEN SOLL, WERDEN JETZT ENTSPRE-
2383	# CHENDE AKTIONEN DURCHGEFUEHRT
2384	if [[ $do_batch = false ]]
2385	then
2386
2387
2388	# TEMPORAEREN KATALOG ERZEUGEN
2389	mkdir -p $TEMPDIR
2390	chmod go+rx $TEMPDIR
2391	tmpcreate=true
2392
2393	# set striping on lustre file system
2394	# if [[ $localhost = lcsgih ]]
2395	# then
2396	# lfs setstripe -s 8192k -c 16 $TEMPDIR
2397	# lfs getstripe $TEMPDIR
2398	# fi
2399
2400
2401	# SAEMTLICHE QUELLTEXT-DATEIEN BZW. AUSFUEHRBARES PROGRAMM IN
2402	# TEMPORAERES VERZEICHNIS KOPIEREN
2403	if [[ $do_compile = true ]]
2404	then
2405
2406	# ON NEC, COMPILATION IS DONE ON HOST CROSS VIA CROSS COMPILING
2407	# CREATE A TEMPORARY DIRECTORY ON THAT MACHINE (HOME MOUNTED VIA NFS)
2408	if [[ $localhost = nech ]]
2409	then
2410	TEMPDIR_COMPILE=$HOME/work/${usern}.$kennung
2411	if mkdir -p $TEMPDIR_COMPILE
2412	then
2413	printf "\n *** \"$TEMPDIR_COMPILE\" "
2414	printf "\n is generated as temporary directory for cross compiling\n"
2415	else
2416	printf "\n +++ creating directory \"$TEMPDIR_COMPILE\" "
2417	printf "\n needed for cross compilation failed"
2418	locat=compile
2419	exit
2420	fi
2421	else
2422	TEMPDIR_COMPILE=$TEMPDIR
2423	fi
2424
2425
2426	# PFADNAMEN FUER DAS MAKE-DEPOSITORY ERMITTELN
2427	line=""
2428	grep "%depository_path" $config_file > tmp_mrun
2429	while read line
2430	do
2431	if [[ "$line" != "" && $(echo $line \| cut -c1) != "#" ]]
2432	then
2433	if [[ "$(echo $line \| cut -d" " -s -f3)" = "" ]]
2434	then
2435	global_depository_path=`echo $line \| cut -d" " -s -f2`
2436	fi
2437	fi
2438	done < tmp_mrun
2439
2440	line=""
2441	grep " $localhost" $config_file \| grep "%depository_path" > tmp_mrun
2442	while read line
2443	do
2444	if [[ "$line" != "" && $(echo $line \| cut -c1) != "#" ]]
2445	then
2446	if [[ "$(echo $line \| cut -d" " -s -f4)" = "$cond1" && "$(echo $line \| cut -d" " -s -f5)" = "$cond2" ]]
2447	then
2448	local_depository_path=`echo $line \| cut -d" " -s -f2`
2449	fi
2450	fi
2451	done < tmp_mrun
2452
2453	if [[ "$local_depository_path" = "" ]]
2454	then
2455	if [[ "$global_depository_path" != "" ]]
2456	then
2457	local_depository_path=$global_depository_path
2458	else
2459	printf "\n\n +++ no depository path found in configuration file"
2460	printf "\n for local host \"$localhost\" "
2461	printf "\n please set \"\%depository_path\" in configuration file\n"
2462	locat=config_file; exit
2463	fi
2464	fi
2465	eval local_depository_path=$local_depository_path
2466	[[ "$cond1" != "" ]] && local_depository_path=${local_depository_path}_$cond1
2467	[[ "$cond2" != "" ]] && local_depository_path=${local_depository_path}_$cond2
2468
2469
2470	basename=`print $mainprog \| cut -f1 -d"."`
2471	eval make_depository=${local_depository_path}/${basename}_current_version.tar
2472	if [[ ! -f $make_depository ]]
2473	then
2474	printf "\n"
2475	printf "\n *** WARNING: make depository \"$make_depository\" not found"
2476	printf "\n \"make\" will fail, if the Makefile or other source files are missing\n"
2477	else
2478	cp $make_depository $TEMPDIR_COMPILE
2479	cd $TEMPDIR_COMPILE
2480	tar -xf $make_depository > /dev/null 2>&1
2481	cd - > /dev/null
2482	fi
2483
2484	cp SOURCES_FOR_RUN_$fname/* $TEMPDIR_COMPILE
2485
2486	else
2487
2488	cp $executable ${TEMPDIR}/a.out
2489
2490	fi
2491
2492
2493	# WECHSEL IN TEMPORAEREN KATALOG
2494	cd $TEMPDIR
2495	printf "\n *** changed to temporary directory: $TEMPDIR"
2496
2497
2498	# OUTPUT-DATEI-VERBINDUNGEN AUF TEMPORAERER DATEI ABLEGEN
2499	# DIESE DATEI KANN VON SPAETER AUFZURUFENDEN BENUTZERPROZEDUREN GELESEN
2500	# WERDEN, UM ZU LOKALEN DATEINAMEN GEHOERENDE PERMANENTE NAMEN ZU
2501	# ERMITTELN
2502	(( i = 0 ))
2503	while (( i < iout ))
2504	do
2505	(( i = i + 1 ))
2506	if [[ "${actionout[$i]}" = tr \|\| "${actionout[$i]}" = tra \|\| "${actionout[$i]}" = trpe ]]
2507	then
2508	printf "${localout[$i]} ${actionout[$i]}\n${pathout[$i]}\n${localhost}_${fname}${endout[$i]}\n" >> OUTPUT_FILE_CONNECTIONS
2509	else
2510	printf "${localout[$i]} ${actionout[$i]}\n${pathout[$i]}\n${frelout[$i]}\n" >> OUTPUT_FILE_CONNECTIONS
2511	fi
2512	done
2513
2514
2515	# EVTL. UEBERSETZUNGSAKTIONEN STARTEN
2516	if [[ $do_compile = true ]]
2517	then
2518
2519
2520	# COMPILING WITH MAKE (ON NEC COMPILER IS CALLED ON HOST CROSS)
2521	printf "\n\n\n *** compilation starts \n$striche\n"
2522	printf " *** compilation with make using following options:\n"
2523	printf " make depository: $make_depository"
2524	if [[ "$mopts" != "" ]]
2525	then
2526	printf " make options: $mopts\n"
2527	fi
2528	printf " compilername: $compiler_name\n"
2529	printf " compiler options: $fopts\n"
2530	printf " preprocessor directives: $cpp_options \n"
2531	printf " linker options: $lopts \n"
2532	if [[ "$modules" != "" ]]
2533	then
2534	printf " modules to be load: $modules \n"
2535	fi
2536	printf " source code files: $source_list \n"
2537
2538	if [[ $localhost = nech ]]
2539	then
2540	ssh 136.172.44.192 -l $usern ". /SX/opt/etc/initsx.sh; $module_calls cd \$HOME/work/${usern}.$kennung; sxmake $mopts -f Makefile PROG=a.out F90=$compiler_name COPT=\"$cpp_options\" F90FLAGS=\"$fopts\" LDFLAGS=\"$lopts\" "
2541	cp $TEMPDIR_COMPILE/a.out .
2542	[[ $? != 0 ]] && compile_error=true
2543	rm -rf $TEMPDIR_COMPILE
2544	elif [[ $localhost = ibmh ]]
2545	then
2546	printf " compiler is called via ssh on \"plogin1\" \n"
2547	ssh plogin1 -l $usern "export PATH=/sw/ibm/xlf/12.1.0.3/usr/bin:$PATH; $module_calls cd $TEMPDIR; make $mopts -f Makefile PROG=a.out F90=$compiler_name COPT=\"$cpp_options\" F90FLAGS=\"$fopts\" LDFLAGS=\"$lopts\" "
2548	[[ ! -f a.out ]] && compile_error=true
2549	continue # ANDERENFALLS IST STATUS=1, FALLS A.OUT VORHANDEN
2550	elif [[ $localhost = lcsgib ]]
2551	then
2552	printf " compiler is called via ssh on \"bicegate0\" \n"
2553	ssh bicegate0 -l $usern ". /usr/share/modules/init/bash; $module_calls cd $TEMPDIR; make $mopts -f Makefile PROG=a.out F90=$compiler_name COPT=\"$cpp_options\" F90FLAGS=\"$fopts\" LDFLAGS=\"$lopts\" "
2554	[[ ! -f a.out ]] && compile_error=true
2555	continue # ANDERENFALLS IST STATUS=1, FALLS A.OUT VORHANDEN
2556	elif [[ $localhost = lcsgih ]]
2557	then
2558	printf " compiler is called via ssh on \"hicegate0\" \n"
2559	ssh hicegate0 -l $usern ". /usr/share/modules/init/bash; $module_calls cd $TEMPDIR; make $mopts -f Makefile PROG=a.out F90=$compiler_name COPT=\"$cpp_options\" F90FLAGS=\"$fopts\" LDFLAGS=\"$lopts\" 2>&1 "
2560	[[ ! -f a.out ]] && compile_error=true
2561	continue # ANDERENFALLS IST STATUS=1, FALLS A.OUT VORHANDEN
2562	else
2563	[[ $localhost = lcxt4 ]] && . /opt/modules/default/init/ksh
2564	[[ "$module_calls" != "" ]] && eval $module_calls
2565	make $mopts -f Makefile PROG=a.out F90=$compiler_name COPT="$cpp_options" F90FLAGS="$fopts" LDFLAGS="$lopts"
2566	fi
2567
2568	if [[ $? != 0 \|\| "$compile_error" = true \|\| "$module_compile_error" = true ]]
2569	then
2570	printf "\n +++ error occured while compiling or linking"
2571	locat=compile
2572
2573	# WORKAROUND: REMOVE IF CONSTRUCT LATER, BUT KEEP THE EXIT!
2574	# if [[ $localhost != lcsgib && $localhost != lcsgih ]]
2575	# then
2576	exit
2577	# else
2578	# locat=normal
2579	# fi
2580	else
2581	printf "$striche\n *** compilation finished \n"
2582	fi
2583	fi
2584
2585
2586	# FALLS NUR TESTWEISE KOMPILIERT WERDEN SOLLTE, IST MRUN JETZT FERTIG
2587	if [[ $local_compile = true ]]
2588	then
2589	cd $HOME
2590	rm -rf $TEMPDIR
2591	locat=local_compile; exit
2592	fi
2593
2594
2595	# BEREITSTELLEN DER INPUT-DATEIEN
2596	# SCHLEIFE UEBER ALLE VOM BENUTZER ANGEGEBENEN DATEIEN
2597	(( i = 0 ))
2598	while (( i < iin ))
2599	do
2600	(( i = i + 1 ))
2601	if (( i == 1 ))
2602	then
2603	printf "\n\n *** providing INPUT-files:\n$striche"
2604	fi
2605
2606
2607	# OPTIONALE DATEIEN BEI NICHTVORHANDENSEIN UEBERGEHEN
2608	if [[ "${transin[$i]}" = unavailable ]]
2609	then
2610	if [[ "${extin[$i]}" = "" \|\| "${extin[$i]}" = " " ]]
2611	then
2612	printf "\n +++ WARNING: input file \"${pathin[$i]}/${afname}${endin[$i]}\" "
2613	printf "\n is not available!"
2614	else
2615	printf "\n +++ WARNING: input file \"${pathin[$i]}/${afname}${endin[$i]}.${extin[$i]}\" "
2616	printf "\n is not available!"
2617	fi
2618	continue
2619	fi
2620
2621	# PRUEFEN, OB EINZELDATEI ODER DATEI PRO PROZESSOR
2622	files_for_pes=false; datentyp=file
2623	if [[ "${actionin[$i]}" = pe && -n $numprocs ]]
2624	then
2625	files_for_pes=true; datentyp=directory
2626	actionin[$i]=""
2627	elif [[ "${actionin[$i]}" = pe && ! -n $numprocs ]]
2628	then
2629	actionin[$i]=""
2630	elif [[ "${actionin[$i]}" = arpe && -n $numprocs ]]
2631	then
2632	files_for_pes=true; datentyp=directory
2633	actionin[$i]="ar"
2634	elif [[ "${actionin[$i]}" = arpe && ! -n $numprocs ]]
2635	then
2636	actionin[$i]="ar"
2637	elif [[ "${actionin[$i]}" = flpe && -n $numprocs ]]
2638	then
2639	files_for_pes=true; datentyp=directory
2640	actionin[$i]="fl"
2641	elif [[ "${actionin[$i]}" = flpe && ! -n $numprocs ]]
2642	then
2643	actionin[$i]="fl"
2644	fi
2645
2646	if [[ $files_for_pes = true ]]
2647	then
2648	printf "\n >>> INPUT: ${absnamein[$i]}/.... to ${localin[$i]}"
2649	else
2650	printf "\n >>> INPUT: ${absnamein[$i]} to ${localin[$i]}"
2651	fi
2652
2653	# INPUT-DATEI FUER EINEN FORTSETZUNGSLAUF. ES WIRD GEPRUEFT,
2654	# OB DIESE DATEI NOCH AUF DEM TEMPORAEREN DATENKATALOG VORHANDEN
2655	# IST. FALLS NICHT, WIRD VERSUCHT, SIE ANSCHLIESSEND VOM ARCHIV-
2656	# SERVER ZU HOLEN
2657	if [[ "${actionin[$i]}" = fl ]]
2658	then
2659	printf "\n $datentyp will be fetched from temporary directory \"${tmp_data_catalog}\" !"
2660	if [[ $files_for_pes = false ]]
2661	then
2662	if [[ -f "$tmp_data_catalog/${frelin[$i]}" ]]
2663	then
2664	ln $tmp_data_catalog/${frelin[$i]} ${localin[$i]}
2665	got_tmp[$i]=true
2666	elif [[ -f "$WORK/${frelin[$i]}" && $ignore_archive_error = true ]]
2667	then
2668	printf "\n +++ $datentyp not found in \"$tmp_data_catalog\" !"
2669	printf "\n *** trying to use backup copy in \"$WORK\" "
2670	cp $WORK/${frelin[$i]} ${localin[$i]}
2671	else
2672	printf "\n +++ $datentyp not found in \"$tmp_data_catalog\" "
2673	printf "\n or \"$tmp_data_catalog\" does not exist!"
2674	printf "\n *** trying to get copy from archive"
2675	actionin[$i]=ar
2676	fi
2677	else
2678	if [[ -d "$tmp_data_catalog/${frelin[$i]}" ]]
2679	then
2680	mkdir ${localin[$i]}
2681	cd $tmp_data_catalog/${frelin[$i]}
2682	for file in $(ls *)
2683	do
2684	ln $file $TEMPDIR/${localin[$i]}
2685	done
2686	cd $TEMPDIR
2687	got_tmp[$i]=true
2688	elif [[ -d "$WORK/${frelin[$i]}" && $ignore_archive_error = true ]]
2689	then
2690	printf "\n +++ $datentyp not found in \"$tmp_data_catalog\" !"
2691	printf "\n *** trying to use backup copy in \"$WORK\" "
2692	cp -r $WORK/${frelin[$i]} ${localin[$i]}
2693	else
2694	printf "\n +++ $datentyp not found in \"$tmp_data_catalog\" "
2695	printf "\n or \"$tmp_data_catalog\" does not exist!"
2696	printf "\n *** trying to get copy from archive"
2697	actionin[$i]=ar
2698	fi
2699	fi
2700	fi
2701
2702
2703	# DATEI LIEGT AUF ARCHIV-SERVER
2704	if [[ "${actionin[$i]}" = ar ]]
2705	then
2706
2707	if [[ $files_for_pes = false ]]
2708	then
2709	printf "\n file will be restored from archive-system ($archive_system)!"
2710	else
2711	printf "\n directory will be restored from archive-system ($archive_system)!"
2712	fi
2713
2714	file_restored=false
2715
2716	if [[ $archive_system = asterix ]]
2717	then
2718	do_stagein=true
2719	(( stagein_anz = 0 ))
2720	while [[ $do_stagein = true ]]
2721	do
2722	if [[ $files_for_pes = false ]]
2723	then
2724	stagein -O ${frelin[$i]} > STAGEIN_OUTPUT
2725	else
2726	stagein -t -O ${frelin[$i]} > STAGEIN_OUTPUT
2727	fi
2728	cat STAGEIN_OUTPUT
2729	if [[ $(grep -c "st.msg:i24" STAGEIN_OUTPUT) != 0 ]]
2730	then
2731	file_restored=true
2732	do_stagein=false
2733	else
2734	(( stagein_anz = stagein_anz + 1 ))
2735	if (( stagein_anz == 10 ))
2736	then
2737	printf "\n +++ stagein stoped after 10 tries"
2738	locat=stage
2739	exit
2740	fi
2741	printf "\n +++ restoring from archive failed, trying again:"
2742	sleep 900
2743	fi
2744	done
2745	elif [[ $archive_system = DMF ]]
2746	then
2747	if [[ $files_for_pes = false ]]
2748	then
2749	printf "\n +++ restoring of single files impossible with $archive_system !\n"
2750	locat=DMF
2751	exit
2752	else
2753	find $ARCHIVE/${frelin[$i]} -type m -print \| dmget
2754	cp -r $ARCHIVE/${frelin[$i]} $PWD
2755	file_restored=true
2756	fi
2757	elif [[ $archive_system = tivoli ]]
2758	then
2759	if [[ $files_for_pes = false ]]
2760	then
2761	if [[ $localhost = lcsgih ]]
2762	then
2763	ssh $usern@hicedata.hlrn.de "cp $PERM/${frelin[$i]} $PWD"
2764	else
2765	ssh $usern@bicedata.hlrn.de "cp $PERM/${frelin[$i]} $PWD"
2766	fi
2767	else
2768	(( inode = 0 ))
2769	while (( inode < nodes ))
2770	do
2771	if [[ $localhost = lcsgih ]]
2772	then
2773	ssh $usern@hicedata.hlrn.de "cd $PWD; tar xf $PERM/${frelin[$i]}/${frelin[$i]}.node_$inode.tar"
2774	else
2775	ssh $usern@bicedata.hlrn.de "cd $PWD; tar xf $PERM/${frelin[$i]}/${frelin[$i]}.node_$inode.tar"
2776	fi
2777	(( inode = inode + 1 ))
2778	done
2779	fi
2780	file_restored=true
2781	elif [[ $archive_system = ut ]]
2782	then
2783	if [[ $files_for_pes = false ]]
2784	then
2785	cp $UT/${frelin[$i]} .
2786	else
2787	(( inode = 0 ))
2788	while (( inode < nodes ))
2789	do
2790	tar xf $UT/${frelin[$i]}/${frelin[$i]}.node_$inode.tar
2791	(( inode = inode + 1 ))
2792	done
2793	fi
2794	file_restored=true
2795	else
2796	printf "\n +++ archive_system=\"$archive_system\" restore impossible!"
2797	locat=rearchive
2798	exit
2799	fi
2800
2801	if [[ $file_restored = true ]]
2802	then
2803
2804	# DATEI AUCH AUF TEMPORAERES DATENVERZEICHNIS LEGEN, DAMIT
2805	# SIE BEI WEITEREN ZUGRIFFEN NOCH VORHANDEN IST
2806	[[ ! -d $tmp_data_catalog ]] && mkdir -p $tmp_data_catalog; chmod g+rx $tmp_data_catalog
2807	if [[ $files_for_pes = false ]]
2808	then
2809	ln -f ${frelin[$i]} $tmp_data_catalog/${frelin[$i]}
2810	else
2811	mkdir $tmp_data_catalog/${frelin[$i]}
2812	ln -f ${frelin[$i]}/* $tmp_data_catalog/${frelin[$i]}
2813	fi
2814	got_tmp[$i]=true
2815
2816	# DATEI UNTER LOKALEM NAMEN ZUR VERFUEGUNG STELLEN
2817	mv ${frelin[$i]} ${localin[$i]}
2818
2819	fi
2820	fi
2821
2822
2823	# DATEI LIEGT IM VOM BENUTZER ANGEGEBENEN VERZEICHNIS
2824	if [[ "${actionin[$i]}" = "" \|\| "${actionin[$i]}" = "di" \|\| "${actionin[$i]}" = "npe" ]]
2825	then
2826
2827	if [[ "${actionin[$i]}" = "npe" && -n $numprocs ]]
2828	then
2829
2830	# DATEI WIRD FUER DIE PROZESSOREN EINES PARALLERECHNERS BEREITGESTELLT
2831	printf "\n file will be provided for $numprocs processors"
2832	mkdir ${localin[$i]}
2833	ival=$numprocs
2834	(( ii = 0 ))
2835	while (( ii <= ival-1 ))
2836	do
2837	if (( ii < 10 ))
2838	then
2839	cp ${absnamein[$i]} ${localin[$i]}/_000$ii
2840	elif (( ii < 100 ))
2841	then
2842	cp ${absnamein[$i]} ${localin[$i]}/_00$ii
2843	elif (( ii < 1000 ))
2844	then
2845	cp ${absnamein[$i]} ${localin[$i]}/_0$ii
2846	else
2847	cp ${absnamein[$i]} ${localin[$i]}/_$ii
2848	fi
2849	(( ii = ii + 1 ))
2850	done
2851
2852	else
2853
2854	if [[ $files_for_pes = true ]]
2855	then
2856
2857	# DIE DEN PROZESSOREN EINES PARALLELRECHNERS ZUGEHOERIGEN
2858	# DATEIEN WERDEN BEREITGESTELLT, INDEM ZUERST DER GESAMTE
2859	# KATALOGINHALT KOPIERT UND DANN DIE EINZELNEN DATEIEN
2860	# PER MOVE UMBENANNT WERDEN
2861	printf "\n providing $numprocs files for the respective processors"
2862	mkdir ${localin[$i]}
2863	if [[ $link_local_input = true ]]
2864	then
2865	printf " using ln -f\n"
2866	cd ${absnamein[$i]}
2867	for file in $(ls *)
2868	do
2869	ln -f $file ${localin[$i]}
2870	done
2871	cd $TEMPDIR
2872	fi
2873	# If "ln -f" fails of if "$link_local_input = false" do a normal "cp -r"
2874	if [[ ! -f "${localin[$i]}/_0000" ]]
2875	then
2876	if [[ $link_local_input = true ]]
2877	then
2878	printf " ln failed for .../_0000, using cp...\n"
2879	fi
2880	cp -r ${absnamein[$i]}/* ${localin[$i]}
2881	fi
2882
2883	else
2884	# BEREITSTELLUNG AUF EINPROZESSORRECHNERN
2885	if [[ $link_local_input = true ]]
2886	then
2887	printf " using ln -f\n"
2888	ln -f ${absnamein[$i]} ${localin[$i]}
2889	fi
2890	# If "ln -f" fails of if "$link_local_input = false" do a normal "cp"
2891	if [[ ! -f "${localin[$i]}" ]]
2892	then
2893	if [[ $link_local_input = true ]]
2894	then
2895	printf " ln failed, using cp...\n"
2896	fi
2897	cp ${absnamein[$i]} ${localin[$i]}
2898	fi
2899	fi
2900	fi
2901	fi
2902
2903	done
2904	if (( i != 0 ))
2905	then
2906	printf "\n$striche\n *** all INPUT-files provided \n"
2907	fi
2908
2909
2910	# EVENTUELLE INPUT-KOMMANDOS ABARBEITEN
2911	(( i = 0 ))
2912	while (( i < iic ))
2913	do
2914	(( i = i + 1 ))
2915	if (( i == 1 ))
2916	then
2917	printf "\n\n *** execution of INPUT-commands:\n$striche"
2918	fi
2919	printf "\n >>> ${in_command[$i]}"
2920	eval ${in_command[$i]}
2921	if (( i == iic ))
2922	then
2923	printf "\n$striche\n"
2924	fi
2925	done
2926
2927
2928	# VERBLEIBENDE CPU-ZEIT BERECHNEN
2929	cpurest=${cpumax}.
2930
2931
2932	# START DVR STREAMING SERVER
2933	if [[ $(echo $package_list \| grep -c dvrp_graphics) != 0 ]]
2934	then
2935	if [[ "$dvr_server" != "" ]]
2936	then
2937
2938	printf "\n\n *** preparing the dvr streaming server configuration file"
2939
2940	# Check, if a dvr server is already running
2941	running_dvrserver_id=`echo $(ps -edaf \| grep .dvrserver.config \| grep -v grep) \| cut -d" " -f2`
2942	if [[ "$running_dvrserver_id" != "" ]]
2943	then
2944
2945	printf "\n\n +++ WARNING: A dvr server with id=$running_dvrserver_id is already running!"
2946	printf "\n This server is used instead starting a new one!"
2947	printf "\n If required, script \"process_dvr_output\" has to be run manually."
2948
2949	else
2950
2951	# COPY CONFIGURATION FILE FOR STREAMING SERVER FROM REPOSITORY TO HERE
2952	if [[ -f ${PALM_BIN}/.dvrserver.config ]]
2953	then
2954	cp ${PALM_BIN}/.dvrserver.config .
2955
2956	# Entering the BASEDIR, UID and GID into this file
2957	user_id=`id -u`
2958	group_id=`id -g`
2959	# & is needed as seperator, because TEMPDIR contains /
2960	sed "s&<replace by dvr data directory>&${TEMPDIR}&g" .dvrserver.config > .dvrserver.1
2961	sed "s/<replace by user id>/$user_id/g" .dvrserver.1 > .dvrserver.2
2962	sed "s/<replace by group id>/$group_id/g" .dvrserver.2 > .dvrserver.3
2963	mv .dvrserver.3 .dvrserver.config
2964	rm .dvrserver.1 .dvrserver.2
2965
2966	# Start dvr server in background, get his id and print on terminal
2967	$dvr_server .dvrserver.config >> DVR_LOGFILE 2>&1 &
2968	dvrserver_id=`echo $(ps -edaf \| grep .dvrserver.config) \| cut -d" " -f2`
2969	printf "\n *** streaming server with id=$dvrserver_id is started in background"
2970	local_dvrserver_running=.TRUE.
2971	else
2972	printf "\n +++ missing file \".dvrserver.config\" in directory:"
2973	printf "\n \"$PALM_BIN\" "
2974	locat=dvr
2975	exit
2976	fi
2977
2978	fi
2979
2980	else
2981	printf "\n\n --- INFORMATIVE: no dvr streaming server will be started"
2982	fi
2983	fi
2984
2985
2986	# NAMELIST-DATEI MIT WERTEN VON ENVIRONMENT-VARIABLEN ERZEUGEN (ZU
2987	# LESEN VON PALM)
2988	cat > ENVPAR << %%END%%
2989	&envpar run_identifier = '$fname', host = '$localhost',
2990	write_binary = '$write_binary', tasks_per_node = $tasks_per_node,
2991	maximum_cpu_time_allowed = ${cpumax}.,
2992	revision = '$global_revision',
2993	local_dvrserver_running = $local_dvrserver_running /
2994
2995	%%END%%
2996
2997
2998	# PROGRAMMSTART
2999	printf "\n\n *** execution starts in directory\n \"`pwd`\"\n$striche\n"
3000	PATH=$PATH:$TEMPDIR
3001
3002	# MPI debug option (argument checking, slows down execution due to increased latency)
3003	if [[ "$mpi_debug" = true ]]
3004	then
3005	export MPI_CHECK_ARGS=1
3006	printf "\n +++ MPI_CHECK_ARGS=$MPI_CHECK_ARGS"
3007	fi
3008
3009	if [[ "$totalview" = true ]]
3010	then
3011	printf "\n *** totalview debugger will be used"
3012	tv_opt="-tv"
3013	else
3014	tv_opt=""
3015	fi
3016
3017	if [[ "$cond1" = debug \|\| "$cond2" = debug ]]
3018	then
3019	if [[ "$ENVIRONMENT" = BATCH ]]
3020	then
3021	if [[ $(echo $localhost \| cut -c1-5) != lcsgi ]]
3022	then
3023	printf "\n +++ debug is allowed in interactive mode only"
3024	locat=debug
3025	exit
3026	fi
3027	fi
3028	if [[ $localhost = decalpha ]]
3029	then
3030	dxladebug a.out
3031	elif [[ $localhost = ibmh ]]
3032	then
3033
3034	# SETUP THE IBM MPI ENVIRONMENT
3035	export MP_SHARED_MEMORY=yes
3036	export AIXTHREADS_SCOPE=S
3037	export OMP_NUM_THREADS=$threads_per_task
3038	export AUTHSTATE=files
3039	export XLFRTEOPTS="nlwidth=132:err_recovery=no" # RECORD-LENGTH OF NAMELIST-OUTPUT
3040
3041	# FOLLOWING OPTIONS ARE MANDATORY FOR TOTALVIEW
3042	export MP_ADAPTER_USE=shared
3043	export MP_CPU_USE=multiple
3044	export MP_TIMEOUT=1200
3045
3046	unset MP_TASK_AFFINITY
3047
3048	# SO FAR, TOTALVIEW NEEDS HOSTFILE MECHANISM FOR EXECUTION
3049	#(( ii = 1 ))
3050	#while (( ii <= $numprocs ))
3051	#do
3052	# echo $localhost_realname >> hostfile
3053	# (( ii = ii + 1 ))
3054	#done
3055	#export MP_HOSTFILE=hostfile
3056
3057	if [[ "$LOADLBATCH" = yes ]]
3058	then
3059	totalview poe a.out $ROPTS
3060	else
3061	echo totalview poe -a a.out -procs $numprocs -rmpool 0 -nodes 1 $ROPTS
3062	export TVDSVRLAUNCHCMD=ssh
3063	totalview poe -a a.out -procs $numprocs -rmpool 0 -nodes 1 $ROPTS
3064	fi
3065	elif [[ $(echo $localhost \| cut -c1-5) = lcsgi ]]
3066	then
3067	# CURRENTLY NO DEBUGGER ON LCSGI
3068	if [[ $run_coupled_model = true ]]
3069	then
3070	printf "\n +++ no debug in coupled mode available on \"$localhost\" "
3071	locat=debug
3072	exit
3073	else
3074	echo "no_coupling" > runfile_atmos
3075	fi
3076	(( ii = $numprocs / $threads_per_task ))
3077	export OMP_NUM_THREADS=$threads_per_task
3078	echo "OMP_NUM_THREADS=$OMP_NUM_THREADS"
3079	if [[ $threads_per_task != 1 ]]
3080	then
3081	printf "\n threads per task: $threads_per_task"
3082	fi
3083	printf "\n\n"
3084	if [[ $( echo $mpilib \| cut -c1-3 ) = mpt ]]
3085	then
3086	# export MPI_LAUNCH_TIMEOUT=360
3087	if [[ "$totalview" = true ]]
3088	then
3089	printf "\n running totalview debugger"
3090	mpiexec_mpt $tv_opt -n $ii ./a.out $ROPTS < runfile_atmos
3091	else
3092	mpiexec_mpt -np $ii ./a.out $ROPTS < runfile_atmos
3093	fi
3094	elif [[ $( echo $mpilib \| cut -c1-3 ) = mva ]]
3095	then
3096	# ulimit -s 300000 # A too large stack size causes problems
3097	# export MV2_NUM_PORTS=2
3098	# export MV2_CPU_MAPPING=0:1:2:3
3099	if [[ "$totalview" = true ]]
3100	then
3101	printf "\n running totalview debugger"
3102	mpiexec $tv_opt ./a.out $ROPTS < runfile_atmos
3103	else
3104	mpiexec ./a.out $ROPTS < runfile_atmos
3105	fi
3106	fi
3107	else
3108	printf "\n +++ no debug available on \"$localhost\" "
3109	printf "\n or not implemented in mrun so far"
3110	locat=debug
3111	exit
3112	fi
3113
3114	# end debug mode
3115	else
3116
3117	# normal execution
3118	if [[ -n $numprocs ]]
3119	then
3120	# RUNNING THE PROGRAM ON PARALLEL MACHINES
3121	if [[ $(echo $host \| cut -c1-3) = ibm ]]
3122	then
3123	# SETUP THE IBM MPI ENVIRONMENT
3124	if [[ $host != ibmh && $host != ibmkisti ]]
3125	then
3126	export MP_SHARED_MEMORY=yes
3127	export AIXTHREAD_SCOPE=S
3128	export OMP_NUM_THREADS=$threads_per_task
3129	export XLSMPOPTS="spins=0:yields=0:stack=20000000"
3130	export AUTHSTATE=files
3131	export XLFRTEOPTS="nlwidth=132:err_recovery=no" # RECORD-LENGTH OF NAMELIST-OUTPUT
3132	# export MP_PRINTENV=yes
3133
3134	# TUNING-VARIABLEN ZUR VERBESSERUNG DER KOMMUNIKATION
3135	# ZEIGEN ABER DERZEIT (SEP 04, FEDERATION) KAUM WIRKUNG
3136	export MP_WAIT_MODE=poll
3137	[[ $node_usage = not_shared ]] && export MP_SINGLE_THREAD=yes
3138	fi
3139
3140	if [[ $host = ibmkisti ]]
3141	then
3142	export LANG=en_US
3143	export MP_SHARED_MEMORY=yes
3144	if [[ $threads_per_task = 1 ]]
3145	then
3146	export MP_SINGLE_THREAD=yes
3147	export MEMORY_AFFINITY=MCM
3148	else
3149	export OMP_NUM_THREADS=$threads_per_task
3150	fi
3151	fi
3152
3153	if [[ "$LOADLBATCH" = yes ]]
3154	then
3155	printf "\n--- Control: OMP_NUM_THREADS = \"$OMP_NUM_THREADS\" \n"
3156	if [[ "$cond1" = hpmcount \|\| "$cond2" = hpmcount ]]
3157	then
3158	/opt/optibm/HPM_2_4_1/bin/hpmcount a.out $ROPTS
3159	else
3160	if [[ $run_coupled_model = false ]]
3161	then
3162	if [[ "$ocean_file_appendix" = true ]]
3163	then
3164	echo "precursor_ocean" > runfile_atmos
3165	else
3166	echo "precursor_atmos" > runfile_atmos
3167	fi
3168	else
3169	(( iia = $numprocs_atmos / $threads_per_task ))
3170	(( iio = $numprocs_ocean / $threads_per_task ))
3171	printf "\n coupled run ($iia atmosphere, $iio ocean)"
3172	printf "\n using $coupled_mode coupling"
3173	printf "\n\n"
3174	echo "coupled_run $iia $iio" > runfile_atmos
3175	fi
3176	poe ./a.out $ROPTS < runfile_atmos
3177	fi
3178	else
3179	if [[ $localhost = ibmh \|\| $localhost = ibms ]]
3180	then
3181	poe a.out -procs $numprocs -nodes 1 -rmpool 0 $ROPTS
3182	elif [[ $localhost = ibmkisti \|\| $localhost = ibmku \|\| $localhost = ibmy ]]
3183	then
3184	if [[ -f $hostfile ]]
3185	then
3186	cp $hostfile hostfile
3187	else
3188	(( ii = 1 ))
3189	while (( ii <= $numprocs ))
3190	do
3191	echo $localhost_realname >> hostfile
3192	(( ii = ii + 1 ))
3193	done
3194	fi
3195	export MP_HOSTFILE=hostfile
3196	if [[ $run_coupled_model = false ]]
3197	then
3198	if [[ "$ocean_file_appendix" = true ]]
3199	then
3200	echo "precursor_ocean" > runfile_atmos
3201	else
3202	echo "precursor_atmos" > runfile_atmos
3203	fi
3204	else
3205	(( iia = $numprocs_atmos / $threads_per_task ))
3206	(( iio = $numprocs_ocean / $threads_per_task ))
3207	printf "\n coupled run ($iia atmosphere, $iio ocean)"
3208	printf "\n using $coupled_mode coupling"
3209	printf "\n\n"
3210	echo "coupled_run $iia $iio" > runfile_atmos
3211	fi
3212	if [[ $localhost = ibmy ]]
3213	then
3214	./a.out -procs $tasks_per_node $ROPTS < runfile_atmos
3215	else
3216	poe ./a.out -procs $numprocs $ROPTS < runfile_atmos
3217	fi
3218
3219	else
3220	if [[ "$host_file" = "" ]]
3221	then
3222	printf "\n +++ no hostfile given in configuration file"
3223	locat=config_file
3224	exit
3225	else
3226	eval host_file=$host_file
3227	fi
3228	export MP_HOSTFILE=$host_file
3229	poe a.out -procs $numprocs -tasks_per_node $numprocs $ROPTS
3230	fi
3231	fi
3232	elif [[ $host = nech \|\| $host = necriam ]]
3233	then
3234	(( ii = nodes ))
3235	if [[ $ii = 1 ]]
3236	then
3237	export F_ERRCNT=0 # acceptable number of errors before program is stopped
3238	export MPIPROGINF=YES
3239	# export F_TRACE=YES\|FMT1\|FMT2 # output of ftrace informations to job protocol
3240	echo "*** execution on single node with mpirun"
3241	mpirun -np $numprocs ./a.out $ROPTS
3242	else
3243	(( i = 0 ))
3244	while (( i < ii ))
3245	do
3246	echo "-h $i -p $tasks_per_node -e ./mpi_exec_shell" >> multinode_config
3247	(( i = i + 1 ))
3248	done
3249
3250	echo "#!/bin/sh" > mpi_exec_shell
3251	echo " " >> mpi_exec_shell
3252	echo "set -u" >> mpi_exec_shell
3253	echo "F_ERRCNT=0" >> mpi_exec_shell
3254	echo "MPIPROGINV=YES" >> mpi_exec_shell
3255	echo "OMP_NUM_THREADS=$threads_per_task" >> mpi_exec_shell
3256	echo "cpurest=$cpurest" >> mpi_exec_shell
3257	echo "fname=$fname" >> mpi_exec_shell
3258	echo "localhost=$localhost" >> mpi_exec_shell
3259	echo "return_addres=$return_addres" >> mpi_exec_shell
3260	echo "return_username=$return_username" >> mpi_exec_shell
3261	echo "tasks_per_node=$tasks_per_node" >> mpi_exec_shell
3262	echo "write_binary=$write_binary" >> mpi_exec_shell
3263	echo "use_seperate_pe_for_dvrp_output=$use_seperate_pe_for_dvrp_output" >> mpi_exec_shell
3264	echo " " >> mpi_exec_shell
3265	echo "export F_ERRCNT" >> mpi_exec_shell
3266	echo "export MPIPROGINV" >> mpi_exec_shell
3267	echo "export OMP_NUM_THREADS" >> mpi_exec_shell
3268	echo "export cpurest" >> mpi_exec_shell
3269	echo "export fname" >> mpi_exec_shell
3270	echo "export localhost" >> mpi_exec_shell
3271	echo "export return_addres" >> mpi_exec_shell
3272	echo "export return_username" >> mpi_exec_shell
3273	echo "export tasks_per_node" >> mpi_exec_shell
3274	echo "export write_binary" >> mpi_exec_shell
3275	echo "export use_seperate_pe_for_dvrp_output" >> mpi_exec_shell
3276	echo " " >> mpi_exec_shell
3277	echo "exec ./a.out" >> mpi_exec_shell
3278
3279	chmod u+x mpi_exec_shell
3280	export MPIPROGINF=YES
3281	mpirun -f multinode_config &
3282	wait
3283
3284	fi
3285	elif [[ $(echo $host \| cut -c1-2) = lc && $host != lckyoto && $host != lctit ]]
3286	then
3287
3288	# COPY HOSTFILE FROM SOURCE DIRECTORY OR CREATE IT, IF IT
3289	# DOES NOT EXIST
3290	if [[ $host != lcsgih && $host != lcsgib ]]
3291	then
3292	if [[ -f $hostfile ]]
3293	then
3294	cp $hostfile hostfile
3295	(( ii = $numprocs / $threads_per_task ))
3296	else
3297	(( ii = 1 ))
3298	while (( ii <= $numprocs / $threads_per_task ))
3299	do
3300	echo $localhost_realname >> hostfile
3301	(( ii = ii + 1 ))
3302	done
3303	fi
3304	eval zeile=\"`head -n $ii hostfile`\"
3305	printf "\n *** running on: $zeile"
3306	fi
3307
3308	(( ii = $numprocs / $threads_per_task ))
3309	export OMP_NUM_THREADS=$threads_per_task
3310	# echo "*** OMP_NUM_THREADS=$OMP_NUM_THREADS"
3311	if [[ $threads_per_task != 1 ]]
3312	then
3313	# increase stack size to unlimited, because large runs
3314	# may abort otherwise
3315	ulimit -Ss unlimited
3316	printf "\n threads per task: $threads_per_task stacksize: unlimited"
3317	fi
3318	if [[ $run_coupled_model = false ]]
3319	then
3320	if [[ "$ocean_file_appendix" = true ]]
3321	then
3322	echo "precursor_ocean" > runfile_atmos
3323	else
3324	echo "precursor_atmos" > runfile_atmos
3325	fi
3326	printf "\n\n"
3327	if [[ $host = lcsgih \|\| $host = lcsgib ]]
3328	then
3329	if [[ $( echo $mpilib \| cut -c1-3 ) = mpt ]]
3330	then
3331	# MPI_DSM_DISTRIBUTE not necessary when MPI_DSM_CPULIST is set
3332	# export MPI_DSM_DISTRIBUTE=1
3333	# MPI_DSM_CPULIST: pin MPI processes to cores
3334	if [[ $use_openmp = false ]]
3335	then
3336	if [[ "$sgi_feature" = ice2 ]]
3337	then
3338	export MPI_DSM_CPULIST="0,4,1,5,2,6,3,7:allhosts"
3339	else
3340	export MPI_DSM_CPULIST="0,1,4,5,2,3,6,7:allhosts"
3341	fi
3342	else
3343	unset MPI_DSM_CPULIST
3344	fi
3345	# MPI_IB_RAILS: use both IB rails on ICE2
3346	export MPI_BUFS_PER_HOST=512
3347	export MPI_IB_RAILS=2
3348	# NECESSARY, IF MORE THAN 4096 PEs ARE USED
3349	export MPI_CONNECTIONS_THRESHOLD=8192
3350	# echo "*** MPI_DSM_CPULIST=$MPI_DSM_CPULIST"
3351	export MPI_TYPE_DEPTH=20
3352	# echo "*** MPI_TYPE_DEPTH=$MPI_TYPE_DEPTH"
3353	export MPI_GROUP_MAX=64
3354	# echo "*** MPI_GROUP_MAX=$MPI_GROUP_MAX"
3355	if [[ $use_openmp = true ]]
3356	then
3357	echo " mpiexec -npernode $tasks_per_node ./a.out $ROPTS < runfile_atmos"
3358	mpiexec -npernode $tasks_per_node $tv_opt ./a.out $ROPTS < runfile_atmos
3359	else
3360	mpiexec_mpt -np $ii $tv_opt ./a.out $ROPTS < runfile_atmos
3361	fi
3362
3363	# next is test for openmp usage
3364	# echo "mpiexec -npernode $tasks_per_node ./a.out $ROPTS < runfile_atmos"
3365	# mpiexec -npernode $tasks_per_node ./a.out $ROPTS < runfile_atmos
3366	elif [[ $( echo $mpilib \| cut -c1-3 ) = mva ]]
3367	then
3368	export MV2_NUM_PORTS=2
3369	# The default setting of MV2_CPU_MAPPING gives best results
3370	# export MV2_ENABLE_AFFINITY=1
3371	# export MV2_CPU_MAPPING=0,1,4,5,2,3,6,7
3372	# if [[ "$sgi_feature" = ice2 ]]
3373	# then
3374	# export MV2_CPU_MAPPING=0,4,1,5,2,6,3,7
3375	# else
3376	# export MV2_CPU_MAPPING=0,1,4,5,2,3,6,7
3377	# fi
3378	if [[ $use_openmp = true ]]
3379	then
3380	unset MV2_CPU_MAPPING
3381	export MV2_ENABLE_AFFINITY=0
3382	fi
3383	echo "*** MV2_CPU_MAPPING=$MV2_CPU_MAPPING"
3384	echo "*** MV2_ENABLE_AFFINITY=$MV2_ENABLE_AFFINITY"
3385	if [[ $use_openmp = true ]]
3386	then
3387	echo " mpiexec -npernode $tasks_per_node ./a.out $ROPTS < runfile_atmos"
3388	mpiexec -npernode $tasks_per_node $tv_opt ./a.out $ROPTS < runfile_atmos
3389	else
3390	mpiexec -np $ii $tv_opt ./a.out $ROPTS < runfile_atmos
3391	fi
3392	elif [[ "$mpilib" = impi ]]
3393	then
3394	echo "mpirun -np $tasks_per_node inspxe-cl -r result -collect mi3 -- ./a.out < runfile_atmos"
3395	mpirun -np $tasks_per_node inspxe-cl -r result -collect mi3 -- ./a.out < runfile_atmos
3396	fi
3397	elif [[ $host = lcxt4 \|\| $host = lcxt5m ]]
3398	then
3399	aprun -n $ii -N $tasks_per_node a.out $ROPTS < runfile_atmos
3400	else
3401	mpiexec -machinefile hostfile -n $ii a.out < runfile_atmos $ROPTS
3402	fi
3403	else
3404
3405	# currently there is no full MPI-2 support on ICE and XT4
3406	(( iia = $numprocs_atmos / $threads_per_task ))
3407	(( iio = $numprocs_ocean / $threads_per_task ))
3408	printf "\n coupled run ($iia atmosphere, $iio ocean)"
3409	printf "\n using $coupled_mode coupling"
3410	printf "\n\n"
3411
3412	if [[ $coupled_mode = "mpi2" ]]
3413	then
3414	echo "atmosphere_to_ocean $iia $iio" > runfile_atmos
3415	echo "ocean_to_atmosphere $iia $iio" > runfile_ocean
3416	if [[ $host = lcsgih \|\| $host = lcsgib ]]
3417	then
3418
3419	if [[ $( echo $mpilib \| cut -c1-3 ) = mpt ]]
3420	then
3421	# export MPI_LAUNCH_TIMEOUT=360
3422	mpiexec_mpt -np $iia ./a.out $ROPTS < runfile_atmos &
3423	mpiexec_mpt -np $iio ./a.out $ROPTS < runfile_ocean &
3424	elif [[ $( echo $mpilib \| cut -c1-3 ) = mva ]]
3425	then
3426	# ulimit -s 300000 # A too large stack size causes problems
3427	# export MV2_NUM_PORTS=2
3428	# export MV2_CPU_MAPPING=0:1:2:3
3429	mpiexec -n $iia ./a.out $ROPTS < runfile_atmos &
3430	mpiexec -n $iio ./a.out $ROPTS < runfile_ocean &
3431	fi
3432
3433	elif [[ $host = lcxt4 \|\| $host = lcxt5m ]]
3434	then
3435
3436	aprun -n $iia -N $tasks_per_node a.out < runfile_atmos $ROPTS &
3437	aprun -n $iio -N $tasks_per_node a.out < runfile_ocean $ROPTS &
3438
3439	else
3440	# WORKAROUND BECAUSE mpiexec WITH -env option IS NOT AVAILABLE ON SOME SYSTEMS
3441	mpiexec -machinefile hostfile -n $iia a.out $ROPTS < runfile_atmos &
3442	mpiexec -machinefile hostfile -n $iio a.out $ROPTS < runfile_ocean &
3443	# mpiexec -machinefile hostfile -n $iia -env coupling_mode atmosphere_to_ocean a.out $ROPTS &
3444	# mpiexec -machinefile hostfile -n $iio -env coupling_mode ocean_to_atmosphere a.out $ROPTS &
3445	fi
3446	wait
3447
3448	else
3449
3450	echo "coupled_run $iia $iio" > runfile_atmos
3451	if [[ $host = lcsgih \|\| $host = lcsgib ]]
3452	then
3453
3454	if [[ $( echo $mpilib \| cut -c1-3 ) = mpt ]]
3455	then
3456	# export MPI_LAUNCH_TIMEOUT=360
3457	mpiexec_mpt -np $ii ./a.out $ROPTS < runfile_atmos
3458	elif [[ $( echo $mpilib \| cut -c1-3 ) = mva ]]
3459	then
3460	# ulimit -s 300000 # A too large stack size causes problems
3461	# export MV2_NUM_PORTS=2
3462	# export MV2_CPU_MAPPING=0:1:2:3
3463	mpiexec ./a.out $ROPTS < runfile_atmos
3464	fi
3465
3466	elif [[ $host = lcxt4 \|\| $host = lcxt5m ]]
3467	then
3468
3469	aprun -n $ii -N $tasks_per_node a.out < runfile_atmos $ROPTS
3470
3471	elif [[ $host = lck \|\| $host = lckordi ]]
3472	then
3473	mpiexec -n $ii ./a.out $ROPTS < runfile_atmos &
3474	fi
3475	wait
3476	fi
3477
3478	fi
3479	elif [[ $host = decalpha ]]
3480	then
3481	dmpirun -np $numprocs a.out $ROPTS
3482	elif [[ $host = lckyoto ]]
3483	then
3484	set -xv
3485	export P4_RSHCOMMAND=plesh
3486	echo " P4_RSHCOMMAND = $P4_RSHCOMMAND"
3487	if [[ "$ENVIRONMENT" = BATCH ]]
3488	then
3489	if [[ "$cond2" = fujitsu ]]
3490	then
3491	mpiexec -n $numprocs ./a.out $ROPTS # for fujitsu-compiler
3492	elif [[ "cond2" = pgi ]]
3493	then
3494	mpirun -np $numprocs -machinefile ${QSUB_NODEINF} ./a.out $ROPTS
3495	else
3496	mpirun_rsh -np $numprocs -hostfile ${QSUB_NODEINF} MV2_USE_SRQ=0 ./a.out ${ROPTS} \|\| /bin/true
3497	fi
3498	else
3499	if [[ "$cond2" = "" ]]
3500	then
3501	mpiruni_rsh -np $numprocs ./a.out $ROPTS # for intel
3502	else
3503	mpirun -np $numprocs ./a.out $ROPTS
3504	fi
3505	fi
3506	set +xv
3507	elif [[ $host = lctit ]]
3508	then
3509	export OMP_NUM_THREADS=$threads_per_task
3510	echo "OMP_NUM_THREADS=$OMP_NUM_THREADS"
3511	if [[ "$threads_per_task" != 1 ]]
3512	then
3513	export MV2_ENABLE_AFFINITY=0
3514	fi
3515	echo "----- PBS_NODEFILE content:"
3516	cat $PBS_NODEFILE
3517	echo "-----"
3518	(( ii = $numprocs / $threads_per_task ))
3519	echo "mpirun -np $ii -hostfile $PBS_NODEFILE ./a.out $ROPTS"
3520	mpirun -np $ii -hostfile $PBS_NODEFILE ./a.out $ROPTS
3521	else
3522	mpprun -n $numprocs a.out $ROPTS
3523	fi
3524	[[ $? != 0 ]] && execution_error=true
3525
3526
3527	# PERFORMANCE-AUSWERTUNG MIT APPRENTICE
3528	if [[ "$cond1" = apprentice \|\| "$cond2" = apprentice ]]
3529	then
3530	apprentice
3531	fi
3532	else
3533	a.out $ROPTS
3534	fi
3535	fi
3536	if [[ $? != 0 \|\| $execution_error = true ]]
3537	then
3538
3539	# ABBRUCH BEI LAUFZEITFEHLER
3540	# [[ ! ( "$cond1" = debug \|\| "$cond2" = debug ) ]] && cat aout_output*
3541	printf "\n +++ runtime error occured"
3542	locat=execution
3543	exit
3544	else
3545	# [[ ! ( "$cond1" = debug \|\| "$cond2" = debug ) ]] && cat aout_output*
3546	printf "\n$striche\n *** execution finished \n"
3547
3548	# Stop the dvr streaming server and process the dvr output in order
3549	# to create dvrs- and html-files containing all streams
3550	if [[ "$dvrserver_id" != "" ]]
3551	then
3552	kill $dvrserver_id
3553	printf "\n *** dvr server with id=$dvrserver_id has been stopped"
3554
3555	# If there is a directory, data have been output by the
3556	# streaming server. Otherwise, user has chosen dvrp_output=local
3557	if [[ -d DATA_DVR ]]
3558	then
3559
3560	# Add the current dvr configuration file to the dvr output
3561	# directory
3562	cp .dvrserver.config DATA_DVR
3563
3564	# Process the dvr output (option -s for also generating
3565	# sequence mode data)
3566	process_dvr_output -d DATA_DVR -f $fname -s
3567
3568	else
3569
3570	# Process the local output
3571	process_dvr_output -l -d DATA_DVR -f $fname
3572
3573	fi
3574
3575	elif [[ $(echo $package_list \| grep -c dvrp_graphics) != 0 ]]
3576	then
3577
3578	# Process dvr output generated in local mode (dvrp_output=local)
3579	process_dvr_output -l -d DATA_DVR -f $fname
3580
3581	fi
3582	fi
3583
3584
3585
3586	# Call of combine_plot_fields in order to merge single files written
3587	# by each PE into one file.
3588	if [[ ! -f ${PALM_BIN}/combine_plot_fields${block}.x ]]
3589	then
3590	printf "\n\n\n +++ WARNING: no combine_plot_fields found for given block \"$cond1 $cond2\""
3591	printf "\n 2d- and/or 3d-data may be incomplete!"
3592	printf "\n Run \"mbuild -u -h $localhost\" to generate utilities for this block.\n"
3593	else
3594	printf "\n\n\n *** post-processing: now executing \"combine_plot_fields${block}.x\" ..."
3595	combine_plot_fields${block}.x
3596	fi
3597
3598
3599
3600	# EVENTUELLE OUTPUT-KOMMANDOS ABARBEITEN
3601	(( i = 0 ))
3602	while (( i < ioc ))
3603	do
3604	(( i = i + 1 ))
3605	if (( i == 1 ))
3606	then
3607	printf "\n\n *** execution of OUTPUT-commands:\n$striche"
3608	fi
3609	printf "\n >>> ${out_command[$i]}"
3610	eval ${out_command[$i]}
3611	if (( i == ioc ))
3612	then
3613	printf "\n$striche\n"
3614	fi
3615	done
3616
3617
3618	# EVTL. INHALT DES AKTUELLEN VERZEICHNISSES AUSGEBEN
3619	if [[ $do_trace = true ]]
3620	then
3621	printf "\n\n"
3622	ls -al
3623	fi
3624
3625
3626
3627	# OUTPUT-DATEIEN AN IHRE ZIELORTE KOPIEREN
3628	(( i = 0 ))
3629	while (( i < iout ))
3630	do
3631	(( i = i + 1 ))
3632	if (( i == 1 ))
3633	then
3634	printf "\n\n *** saving OUTPUT-files:\n$striche"
3635	fi
3636
3637	# PRUEFEN, OB EINZELDATEI ODER DATEI PRO PROZESSOR
3638	files_for_pes=false; filetyp=file
3639	if [[ "${actionout[$i]}" = pe && -n $numprocs ]]
3640	then
3641	files_for_pes=true; filetyp=directory
3642	actionout[$i]=""
3643	elif [[ "${actionout[$i]}" = pe && ! -n $numprocs ]]
3644	then
3645	actionout[$i]=""
3646	elif [[ "${actionout[$i]}" = arpe && -n $numprocs ]]
3647	then
3648	files_for_pes=true; filetyp=directory
3649	actionout[$i]="ar"
3650	elif [[ "${actionout[$i]}" = arpe && ! -n $numprocs ]]
3651	then
3652	actionout[$i]="ar"
3653	elif [[ "${actionout[$i]}" = flpe && -n $numprocs ]]
3654	then
3655	files_for_pes=true; filetyp=directory
3656	actionout[$i]="fl"
3657	elif [[ "${actionout[$i]}" = flpe && ! -n $numprocs ]]
3658	then
3659	actionout[$i]="fl"
3660	elif [[ "${actionout[$i]}" = trpe && -n $numprocs ]]
3661	then
3662	files_for_pes=true; filetyp=directory
3663	actionout[$i]="tr"
3664	elif [[ "${actionout[$i]}" = trpe && ! -n $numprocs ]]
3665	then
3666	actionout[$i]="tr"
3667	fi
3668
3669	if [[ ! -f ${localout[$i]} && $files_for_pes = false ]]
3670	then
3671	printf "\n +++ temporary OUTPUT-file ${localout[$i]} does not exist\n"
3672	elif [[ ! -d ${localout[$i]} && $files_for_pes = true ]]
3673	then
3674	printf "\n +++ temporary OUTPUT-file ${localout[$i]}/.... does not exist\n"
3675	else
3676
3677
3678	# KOPIEREN PER FTP/SCP (IMMER IM BINAERMODUS, -M: FALLS ZIELKATALOG
3679	# NICHT VORHANDEN, WIRD VERSUCHT, IHN ANZULEGEN), ABER NUR BEI
3680	# REMOTE-RECHNUNGEN
3681	if [[ "${actionout[$i]}" = tr ]]
3682	then
3683	if [[ $localhost != $fromhost ]]
3684	then
3685	if [[ $files_for_pes = false ]]
3686	then
3687	cps=""
3688	cst=""
3689	else
3690	cps=-c
3691	cst="/"
3692	fi
3693	if [[ $localhost = nech ]]
3694	then
3695
3696	# TRANSFER IN EIGENSTAENDIGEM JOB
3697	# ZUERST KOPIE DER DATEI INS TEMPORAERE DATENVERZEICHNIS
3698	[[ ! -d $tmp_data_catalog/TRANSFER ]] && mkdir -p $tmp_data_catalog/TRANSFER
3699	file_to_transfer=${fname}_${localout[$i]}_to_transfer_$kennung
3700	if [[ $files_for_pes = false ]]
3701	then
3702	ln -f ${localout[$i]} $tmp_data_catalog/TRANSFER/$file_to_transfer
3703	else
3704	mkdir $tmp_data_catalog/TRANSFER/$file_to_transfer
3705	ln ${localout[$i]}/* $tmp_data_catalog/TRANSFER/$file_to_transfer
3706	fi
3707
3708	echo "set -x" > transfer_${localout[$i]}
3709	echo "cd $tmp_data_catalog/TRANSFER" >> transfer_${localout[$i]}
3710
3711	printf "\n >>> OUTPUT: ${localout[$i]}$cst by SCP in seperate job to"
3712	printf "\n ${pathout[$i]}/${localhost}_${fname}${endout[$i]}$cst"
3713	printf "\n or higher cycle\n"
3714	echo "batch_scp $cps -b -m -u $return_username $return_addres $file_to_transfer \"${pathout[$i]}\" ${localhost}_${fname}${endout[$i]} ${extout[$i]}" >> transfer_${localout[$i]}
3715
3716	echo "[[ \$? = 0 ]] && rm $file_to_transfer" >> transfer_${localout[$i]}
3717
3718	if [[ $localhost = nech ]]
3719	then
3720	subjob -d -c /pf/b/$usern/job_queue -v -q pp -X 0 -m 1000 -t 900 transfer_${localout[$i]}
3721	else
3722	if [[ "$LOGNAME" = b323013 ]]
3723	then
3724	subjob -v -q c1 -X 0 -m 1000 -t 900 -c $job_catalog transfer_${localout[$i]}
3725	else
3726	subjob -d -v -q c1 -X 0 -m 1000 -t 900 -c $job_catalog transfer_${localout[$i]}
3727	fi
3728	fi
3729
3730	else
3731
3732	# TRANSFER INNERHALB DIESES JOBS
3733	transfer_failed=false
3734	printf "\n >>> OUTPUT: ${localout[$i]}$cst by SCP to"
3735	printf "\n ${pathout[$i]}/${localhost}_${fname}${endout[$i]}$cst\n"
3736	batch_scp $cps -b -m -u $return_username $return_addres ${localout[$i]} "${pathout[$i]}" ${localhost}_${fname}${endout[$i]} ${extout[$i]}
3737	[[ $? != 0 ]] && transfer_failed=true
3738
3739	# BEI FEHLGESCHLAGENEM TRANSFER SICHERUNGSKOPIE AUF
3740	# LOKALER MASCHINE ANLEGEN
3741	if [[ $transfer_failed = true ]]
3742	then
3743	printf " +++ transfer failed. Trying to save a copy on the local host under:\n"
3744	printf " ${pathout[$i]}/${localhost}_${fname}${endout[$i]}_$kennung\n"
3745
3746	# ERSTMAL PRUEFEN, OB VERZEICHNIS EXISTIERT. GEGEBENENFALLS
3747	# ANLEGEN.
3748	eval local_catalog=${pathout[$i]}
3749	if [[ ! -d $local_catalog ]]
3750	then
3751	printf " *** local directory does not exist. Trying to create:\n"
3752	printf " $local_catalog \n"
3753	mkdir -p $local_catalog
3754	fi
3755	eval cp ${localout[$i]} ${pathout[$i]}/${localhost}_${fname}${endout[$i]}_$kennung
3756	transfer_problems=true
3757	fi
3758	fi
3759	else
3760
3761	# WERTZUWEISUNG, SO DASS WEITER UNTEN NUR KOPIERT WIRD
3762	actionout[$i]=""
3763	fi
3764	fi
3765
3766
3767	# APPEND PER FTP/SCP (IMMER IM BINAERMODUS, -M: FALLS ZIELKATALOG
3768	# NICHT VORHANDEN, WIRD VERSUCHT, IHN ANZULEGEN), ABER NUR BEI
3769	# REMOTE-RECHNUNGEN
3770	if [[ "${actionout[$i]}" = tra ]]
3771	then
3772	if [[ $localhost != $fromhost ]]
3773	then
3774	if [[ $localhost = ibmh \|\| $localhost = nech ]]
3775	then
3776
3777	# TRANSFER IN EIGENSTAENDIGEM JOB
3778	# ZUERST KOPIE DER DATEI INS TEMPORAERE DATENVERZEICHNIS
3779	[[ ! -d $tmp_data_catalog/TRANSFER ]] && mkdir -p $tmp_data_catalog/TRANSFER
3780	file_to_transfer=${fname}_${localout[$i]}_to_transfer_$kennung
3781	ln -f ${localout[$i]} $tmp_data_catalog/TRANSFER/$file_to_transfer
3782
3783	echo "set -x" > transfer_${localout[$i]}
3784	echo "cd $tmp_data_catalog/TRANSFER" >> transfer_${localout[$i]}
3785
3786	printf "\n >>> OUTPUT: ${localout[$i]} append by SCP in seperate job to"
3787	printf "\n ${pathout[$i]}/${localhost}_${fname}${endout[$i]}"
3788	printf "\n or higher cycle\n"
3789	echo "batch_scp -A -b -m -u $return_username $return_addres $file_to_transfer \"${pathout[$i]}\" ${localhost}_${fname}${endout[$i]} ${extout[$i]}" >> transfer_${localout[$i]}
3790
3791	echo "[[ \$? = 0 ]] && rm $file_to_transfer" >> transfer_${localout[$i]}
3792
3793	if [[ $localhost = nech ]]
3794	then
3795	subjob -d -c /pf/b/$usern/job_queue -v -q pp -X 0 -m 1000 -t 900 transfer_${localout[$i]}
3796	else
3797	if [[ $LOGNAME = b323013 ]]
3798	then
3799	subjob -v -q c1 -X 0 -m 1000 -t 900 -c $job_catalog transfer_${localout[$i]}
3800	else
3801	subjob -d -v -q c1 -X 0 -m 1000 -t 900 -c $job_catalog transfer_${localout[$i]}
3802	fi
3803	fi
3804
3805	else
3806
3807	# TRANSFER INNERHALB DIESES JOBS
3808	transfer_failed=false
3809	printf "\n >>> OUTPUT: ${localout[$i]} append by SCP to"
3810	printf "\n ${pathout[$i]}/${localhost}_${fname}${endout[$i]}\n"
3811	batch_scp -A -b -m -u $return_username $return_addres ${localout[$i]} "${pathout[$i]}" ${localhost}_${fname}${endout[$i]} ${extout[$i]}
3812	[[ $? != 0 ]] && transfer_failed=true
3813
3814	# BEI FEHLGESCHLAGENEM TRANSFER SICHERUNGSKOPIE AUF
3815	# LOKALER MASCHINE ANLEGEN
3816	if [[ $transfer_failed = true ]]
3817	then
3818	printf " +++ transfer failed. Trying to save a copy on the local host under:\n"
3819	printf " ${pathout[$i]}/${localhost}_${fname}${endout[$i]}_$kennung\n"
3820
3821	# ERSTMAL PRUEFEN, OB VERZEICHNIS EXISTIERT. GEGEBENENFALLS
3822	# ANLEGEN
3823	eval local_catalog=${pathout[$i]}
3824	if [[ ! -d $local_catalog ]]
3825	then
3826	printf " *** local directory does not exist. Trying to create:\n"
3827	printf " $local_catalog \n"
3828	mkdir -p $local_catalog
3829	fi
3830
3831	eval cp ${localout[$i]} ${pathout[$i]}/${localhost}_${fname}${endout[$i]}_$kennung
3832	transfer_problems=true
3833	fi
3834	fi
3835	else
3836
3837	# WERTZUWEISUNG, SO DASS WEITER UNTEN NUR APPEND AUF
3838	# LOKALER MASCHINE DURCHGEFUEHRT WIRD
3839	actionout[$i]=a
3840	fi
3841	fi
3842
3843
3844	# OUTPUT-DATEI FUER EINEN FORTSETZUNGSLAUF. DATEI WIRD PER
3845	# LINK AUF DEN TEMPORAEREN DATENKATALOG GELEGT. OPTION -f WIRD
3846	# VERWENDET, FALLS DORT NOCH EINE DATEI GLEICHEN NAMENS VORHANDEN
3847	# SEIN SOLLTE. ANSCHLIESSEND WIRD SEINE ARCHIVIERUNG ANGESTOSSEN
3848	if [[ "${actionout[$i]}" = fl ]]
3849	then
3850	[[ ! -d $tmp_data_catalog ]] && mkdir -p $tmp_data_catalog
3851	chmod g+rx $tmp_data_catalog
3852	if [[ $files_for_pes = false ]]
3853	then
3854	printf "\n >>> OUTPUT: ${localout[$i]} to"
3855	printf "\n $tmp_data_catalog/${frelout[$i]} (temporary data catalog)\n"
3856	ln -f ${localout[$i]} $tmp_data_catalog/${frelout[$i]}
3857	else
3858	printf "\n >>> OUTPUT: ${localout[$i]}/.... to"
3859	printf "\n $tmp_data_catalog/${frelout[$i]} (temporary data catalog)\n"
3860	mkdir $tmp_data_catalog/${frelout[$i]}
3861	cd ${localout[$i]}
3862	for file in $(ls *)
3863	do
3864	ln -f $file $tmp_data_catalog/${frelout[$i]}
3865	done
3866	cd $TEMPDIR
3867	fi
3868
3869
3870	# ARCHIVIERUNGSJOB WIRD ERZEUGT UND ABGESCHICKT
3871	if [[ $store_on_archive_system = true ]]
3872	then
3873
3874	if [[ $archive_system = asterix ]]
3875	then
3876	echo "cd $tmp_data_catalog" >> archive_${frelout[$i]}
3877	if [[ $files_for_pes = false ]]
3878	then
3879	echo "stageout ${frelout[$i]} > STAGE_OUTPUT${i}_$kennung" >> archive_${frelout[$i]}
3880	else
3881	echo "stageout -t ${frelout[$i]} > STAGE_OUTPUT${i}_$kennung" >> archive_${frelout[$i]}
3882	fi
3883	echo "cat STAGE_OUTPUT${i}_$kennung" >> archive_${frelout[$i]}
3884	echo "if [[ \$(grep -c \"st.msg:150\" STAGE_OUTPUT${i}_$kennung) != 0 ]]" >> archive_${frelout[$i]}
3885	echo "then" >> archive_${frelout[$i]}
3886	echo " do_stageout=false" >> archive_${frelout[$i]}
3887	echo "else" >> archive_${frelout[$i]}
3888	echo " echo \" +++ $filetyp ${frelout[$i]} could not be stored on archive-system\" " >> archive_${frelout[$i]}
3889	echo " cat /dev/null > ~/job_queue/ARCHIVE_ERROR_$fname" >> archive_${frelout[$i]}
3890	echo " cat STAGE_OUTPUT${i}_$kennung > ~/job_queue/archive_${frelout[$i]}_error" >> archive_${frelout[$i]}
3891	echo " echo \" *** $filetyp ${frelout[$i]} will be copied to \$WORK as backup\" " >> archive_${frelout[$i]}
3892	if [[ $files_for_pes = false ]]
3893	then
3894	echo " cp ${frelout[$i]} \$WORK" >> archive_${frelout[$i]}
3895	else
3896	echo " cp -r ${frelout[$i]} \$WORK/${frelout[$i]}" >> archive_${frelout[$i]}
3897	fi
3898	echo " echo \" *** $filetyp ${frelout[$i]} saved\" " >> archive_${frelout[$i]}
3899	echo "fi" >> archive_${frelout[$i]}
3900	echo "rm STAGE_OUTPUT${i}_$kennung" >> archive_${frelout[$i]}
3901	elif [[ $archive_system = DMF ]]
3902	then
3903	echo "cd $tmp_data_catalog" >> archive_${frelout[$i]}
3904	if [[ $files_for_pes = false ]]
3905	then
3906	printf "\n +++ archiving of single files impossible with $archive_system !\n"
3907	locat=DMF
3908	exit
3909	else
3910	# FUER RECHNER IN JUELICH. DORT KOENNTEN AUCH
3911	# EINZELNE DATEIEN GESICHERT WERDEN (SPAETER KORR.)
3912	echo "rm -rf \$ARCHIVE/${frelout[$i]}" >> archive_${frelout[$i]}
3913	echo "cp -r ${frelout[$i]} \$ARCHIVE" >> archive_${frelout[$i]}
3914	fi
3915	elif [[ $archive_system = tivoli ]]
3916	then
3917	echo "cd $tmp_data_catalog" >> archive_${frelout[$i]}
3918	if [[ $files_for_pes = false ]]
3919	then
3920	# EVENTUELL NOCH VORHANDENE DATEI IM ARCHIV LOSCHEN
3921	echo "set -x" >> archive_${frelout[$i]}
3922	echo "rm -rf \$PERM/${frelout[$i]}" >> archive_${frelout[$i]}
3923	echo "cp ${frelout[$i]} \$PERM/${frelout[$i]}" >> archive_${frelout[$i]}
3924	else
3925
3926	echo "set -x" >> archive_${frelout[$i]}
3927	echo "rm -rf \$PERM/${frelout[$i]}/*" >> archive_${frelout[$i]}
3928	echo "[[ ! -d \$PERM/${frelout[$i]} ]] && mkdir $PERM/${frelout[$i]}" >> archive_${frelout[$i]}
3929	cd $tmp_data_catalog
3930	all_files=`ls -1 ${frelout[$i]}/*`
3931	cd - > /dev/null
3932	(( inode = 0 ))
3933	(( tp1 = tasks_per_node + 1 ))
3934	while (( inode < nodes ))
3935	do
3936	# echo "*** all_files = \"$all_files\" "
3937	files=`echo $all_files \| cut -d" " -f1-$tasks_per_node`
3938	all_files=`echo $all_files \| cut -d" " -f$tp1-`
3939	# echo "*** tasks_per_node = \"$tasks_per_node\" "
3940	# echo "*** files = \"$files\" "
3941	# echo "*** all_files = \"$all_files\" "
3942	echo "tar cvf \$PERM/${frelout[$i]}/${frelout[$i]}.node_$inode.tar $files" >> archive_${frelout[$i]}
3943	### echo "tar cvf ${frelout[$i]}.node_$inode.tar $files" >> archive_${frelout[$i]}
3944	### echo "cp ${frelout[$i]}.node_$inode.tar \$PERM/${frelout[$i]}/" >> archive_${frelout[$i]}
3945	###echo "rm ${frelout[$i]}.node_$inode.tar" >> archive_${frelout[$i]}
3946	# echo "dsmc incremental \$PERM/${frelout[$i]}/${frelout[$i]}.node_$inode.tar" >> archive_${frelout[$i]}
3947	# echo "dsmmigrate \$PERM/${frelout[$i]}/${frelout[$i]}.node_$inode.tar" >> archive_${frelout[$i]}
3948	(( inode = inode + 1 ))
3949	done
3950	# echo "rm -rf \$PERM/${frelout[$i]}.tar" >> archive_${frelout[$i]}
3951	# echo "tar cvf \$PERM/${frelout[$i]}.tar ${frelout[$i]}" >> archive_${frelout[$i]}
3952	fi
3953	elif [[ $archive_system = ut ]]
3954	then
3955	echo "cd $tmp_data_catalog" >> archive_${frelout[$i]}
3956	if [[ $files_for_pes = false ]]
3957	then
3958	# EVENTUELL NOCH VORHANDENE DATEI IM ARCHIV LOSCHEN
3959	echo "set -x" >> archive_${frelout[$i]}
3960	echo "rm -rf \$UT/${frelout[$i]}" >> archive_${frelout[$i]}
3961	echo "cp ${frelout[$i]} \$UT/${frelout[$i]}" >> archive_${frelout[$i]}
3962	else
3963
3964	echo "set -x" >> archive_${frelout[$i]}
3965	echo "rm -rf \$UT/${frelout[$i]}/*" >> archive_${frelout[$i]}
3966	echo "[[ ! -d \$UT/${frelout[$i]} ]] && mkdir $UT/${frelout[$i]}" >> archive_${frelout[$i]}
3967	cd $tmp_data_catalog
3968	all_files=`ls -1 ${frelout[$i]}/*`
3969	cd - > /dev/null
3970	(( inode = 0 ))
3971	(( tp1 = tasks_per_node + 1 ))
3972	while (( inode < nodes ))
3973	do
3974	files=`echo $all_files \| cut -d" " -f1-$tasks_per_node`
3975	all_files=`echo $all_files \| cut -d" " -f$tp1-`
3976	echo "tar cvf \$UT/${frelout[$i]}/${frelout[$i]}.node_$inode.tar $files" >> archive_${frelout[$i]}
3977	(( inode = inode + 1 ))
3978	done
3979	fi
3980	elif [[ $archive_system = none ]]
3981	then
3982	printf " +++ archiving on $localhost not available!\n"
3983	fi
3984
3985	if [[ $archive_system != none ]]
3986	then
3987	if [[ $localhost = lcsgih \|\| $localhost = lcsgib ]]
3988	then
3989	# subjob -d -v -q cdata -X 0 -m 1000 -t 43200 -c $job_catalog archive_${frelout[$i]}
3990	subjob -v -q permq -X 1 -T 1 -m 1000 -t 172800 -c $job_catalog archive_${frelout[$i]}
3991	elif [[ $localhost = nech ]]
3992	then
3993	subjob -d -c /pf/b/$usern/job_queue -v -q pp -X 0 -m 1000 -t 7200 archive_${frelout[$i]}
3994	fi
3995	printf " Archiving of $tmp_data_catalog/${frelout[$i]} initiated (batch job submitted)\n"
3996	fi
3997	else
3998	printf " +++ caution: option -A is switched off. No archiving on $archive_system!\n"
3999	fi
4000
4001
4002	# LEERDATEI IM BENUTZERVERZEICHNIS ANLEGEN, DAMIT BEKANNT IST,
4003	# WIE DIE HOECHSTE ZYKLUSNUMMER AUF DEM ARCHIV-SYSTEM LAUTET
4004	if [[ $files_for_pes = false ]]
4005	then
4006	cat /dev/null > ${pathout[$i]}
4007	else
4008	mkdir -p ${pathout[$i]}
4009	fi
4010
4011	fi
4012
4013
4014	# KOPIEREN AUF LOKALER MASCHINE ZUM ARCHIV-SYSTEM
4015	# AUSSERDEM MUSS LEERDATEI ANGELEGT WERDEN, DAMIT BEKANNT IST,
4016	# WIE DIE HOECHSTE ZYKLUSNUMMER AUF DEM ARCHIV-SYSTEM LAUTET
4017	# FALLS IRGENDEINE (VON MEHREREN) ARCHIVIERUNGEN SCHIEF GEHT,
4018	# WIRD FLAG GESETZT UND BLEIBT AUCH BEI WEITEREN ORDNUNGS-
4019	# GEMAESSEN ARCHIVIERUNGEN GESETZT
4020	if [[ "${actionout[$i]}" = ar ]]
4021	then
4022	if [[ $files_for_pes = false ]]
4023	then
4024	printf "\n >>> OUTPUT: ${localout[$i]} to"
4025	printf "\n ${pathout[$i]}"
4026	printf "\n File will be copied to archive-system ($archive_system) !\n"
4027	else
4028	printf "\n >>> OUTPUT: ${localout[$i]}/_.... to"
4029	printf "\n ${pathout[$i]}"
4030	printf "\n Directory will be copied to archive-system ($archive_system) !\n"
4031	fi
4032	mv ${localout[$i]} ${frelout[$i]}
4033
4034	file_saved=false
4035
4036	if [[ $archive_system = asterix ]]
4037	then
4038	do_stageout=true
4039	(( stageout_anz = 0 ))
4040	while [[ $do_stageout = true ]]
4041	do
4042	if [[ $files_for_pes = false ]]
4043	then
4044	stageout ${frelout[$i]} > STAGE_OUTPUT
4045	else
4046	stageout -t ${frelout[$i]} > STAGE_OUTPUT
4047	fi
4048	cat STAGE_OUTPUT
4049	if [[ $(grep -c "st.msg:150" STAGE_OUTPUT) != 0 ]]
4050	then
4051	file_saved=true
4052	do_stageout=false
4053	else
4054	if [[ $files_for_pes = false ]]
4055	then
4056	printf "\n +++ file ${frelout[$i]} could not be saved on archive-system"
4057	else
4058	printf "\n +++ directory ${frelout[$i]} could not be saved on archive-system"
4059	fi
4060	(( stageout_anz = stageout_anz + 1 ))
4061	if (( stageout_anz == 10 ))
4062	then
4063	printf "\n +++ stoped after 10 unsuccessful tries!"
4064	archive_save=false
4065	do_stageout=false
4066	else
4067	printf "\n *** new try to store on archive after 15 min:"
4068	sleep 900
4069	fi
4070	fi
4071	done
4072	elif [[ $archive_system = DMF ]]
4073	then
4074	if [[ $files_for_pes = false ]]
4075	then
4076	printf "\n +++ archiving of single files impossible on $archive_system!\n"
4077	locat=DMF
4078	exit
4079	else
4080	rm -rf $ARCHIVE/${frelout[$i]}
4081	cp -r ${frelout[$i]} $ARCHIVE
4082	fi
4083	file_saved=true
4084	elif [[ $archive_system = tivoli ]]
4085	then
4086	# ARCHIVIERUNG NUR PER BATCH-JOB MOEGLICH
4087	# DATEI MUSS ZWISCHENZEITLICH INS TEMPORAERE DATENVERZEICHNIS
4088	# GELEGT WERDEN
4089	[[ ! -d $tmp_data_catalog ]] && mkdir -p $tmp_data_catalog
4090	chmod g+rx $tmp_data_catalog
4091	if [[ $files_for_pes = false ]]
4092	then
4093	ln -f ${frelout[$i]} $tmp_data_catalog/${frelout[$i]}
4094	else
4095	mkdir $tmp_data_catalog/${frelout[$i]}
4096	ln -f ${frelout[$i]}/* $tmp_data_catalog/${frelout[$i]}
4097	fi
4098
4099	# BATCH JOB GENERIEREN UND ABSCHICKEN; DATEI MUSS WIEDER
4100	# AUS TEMPORAEREM DATENVERZEICHNIS ENTFERNT WERDEN
4101	echo "cd $tmp_data_catalog" > archive_${frelout[$i]}
4102	if [[ $files_for_pes = false ]]
4103	then
4104	# EVENTUELL NOCH VORHANDENE DATEI IM ARCHIV LOSCHEN
4105	echo "rm -rf \$PERM/${frelout[$i]}" >> archive_${frelout[$i]}
4106	echo "cp ${frelout[$i]} \$PERM/${frelout[$i]}" >> archive_${frelout[$i]}
4107	echo "rm -rf ${frelout[$i]}" >> archive_${frelout[$i]}
4108	else
4109	echo "rm -rf \$PERM/${frelout[$i]}.tar" >> archive_${frelout[$i]}
4110	echo "tar cvf \$PERM/${frelout[$i]}.tar ${frelout[$i]}" >> archive_${frelout[$i]}
4111	echo "rm -rf ${frelout[$i]}" >> archive_${frelout[$i]}
4112	fi
4113
4114	subjob -v -d -q cdata -X 0 -m 1000 -t 43200 -c $job_catalog archive_${frelout[$i]}
4115	printf " Archiving of $tmp_data_catalog/${frelout[$i]} initiated (batch job submitted)\n"
4116	file_saved=true
4117
4118	elif [[ $archive_system = ut ]]
4119	then
4120	# ARCHIVIERUNG NUR PER BATCH-JOB MOEGLICH
4121	# DATEI MUSS ZWISCHENZEITLICH INS TEMPORAERE DATENVERZEICHNIS
4122	# GELEGT WERDEN
4123	[[ ! -d $tmp_data_catalog ]] && mkdir -p $tmp_data_catalog
4124	chmod g+rx $tmp_data_catalog
4125	if [[ $files_for_pes = false ]]
4126	then
4127	ln -f ${frelout[$i]} $tmp_data_catalog/${frelout[$i]}
4128	else
4129	mkdir $tmp_data_catalog/${frelout[$i]}
4130	ln -f ${frelout[$i]}/* $tmp_data_catalog/${frelout[$i]}
4131	fi
4132
4133	# BATCH JOB GENERIEREN UND ABSCHICKEN; DATEI MUSS WIEDER
4134	# AUS TEMPORAEREM DATENVERZEICHNIS ENTFERNT WERDEN
4135	echo "cd $tmp_data_catalog" > archive_${frelout[$i]}
4136	if [[ $files_for_pes = false ]]
4137	then
4138	# EVENTUELL NOCH VORHANDENE DATEI IM ARCHIV LOSCHEN
4139	echo "rm -rf \$UT/${frelout[$i]}" >> archive_${frelout[$i]}
4140	echo "cp ${frelout[$i]} \$UT/${frelout[$i]}" >> archive_${frelout[$i]}
4141	echo "rm -rf ${frelout[$i]}" >> archive_${frelout[$i]}
4142	else
4143	echo "rm -rf \$UT/${frelout[$i]}.tar" >> archive_${frelout[$i]}
4144	echo "tar cvf \$UT/${frelout[$i]}.tar ${frelout[$i]}" >> archive_${frelout[$i]}
4145	echo "rm -rf ${frelout[$i]}" >> archive_${frelout[$i]}
4146	fi
4147
4148	subjob -v -c /pf/b/$usern/job_queue -d -q pp -X 0 -m 1000 -t 7200 archive_${frelout[$i]}
4149	printf " Archiving of $tmp_data_catalog/${frelout[$i]} initiated (batch job submitted)\n"
4150	file_saved=true
4151
4152	else
4153	printf "\n +++ archive_system=\"$archive_system\" archiving impossible!"
4154	archive_save=false
4155	fi
4156	if [[ $file_saved = true ]]
4157	then
4158	if [[ $files_for_pes = false ]]
4159	then
4160	cat /dev/null > ${pathout[$i]}
4161	else
4162	mkdir -p ${pathout[$i]}
4163	fi
4164	fi
4165	fi
4166
4167	# APPEND AUF LOKALER MASCHINE
4168	if [[ "${actionout[$i]}" = "a" ]]
4169	then
4170	if [[ "${extout[$i]}" != " " && "${extout[$i]}" != "" ]]
4171	then
4172	printf "\n >>> OUTPUT: ${localout[$i]} append to ${pathout[$i]}.${extout[$i]}\n"
4173	cat ${localout[$i]} >> ${pathout[$i]}.${extout[$i]}
4174	else
4175	printf "\n >>> OUTPUT: ${localout[$i]} append to ${pathout[$i]}\n"
4176	cat ${localout[$i]} >> ${pathout[$i]}
4177	fi
4178	fi
4179
4180	# KOPIEREN AUF LOKALER MASCHINE
4181	# ES MUSS KOPIERT WERDEN, DA MOVE NICHT UEBER FILESYSTEM HINAUS MOEGLICH
4182	if [[ "${actionout[$i]}" = "" && $files_for_pes = false ]]
4183	then
4184
4185	# KOPIEREN AUF EINPROZESSORMASCHINE
4186	if [[ "${extout[$i]}" != " " && "${extout[$i]}" != "" ]]
4187	then
4188	printf "\n >>> OUTPUT: ${localout[$i]} to ${pathout[$i]}.${extout[$i]}\n"
4189	if [[ $link_local_output = true ]]
4190	then
4191	printf " using ln -f\n"
4192	ln -f ${localout[$i]} ${pathout[$i]}.${extout[$i]}
4193	fi
4194	# If "ln -f" fails of if "$link_local_output = false" do a normal "cp"
4195	if [[ ! -f "${pathout[$i]}.${extout[$i]}" ]]
4196	then
4197	if [[ $link_local_output = true ]]
4198	then
4199	printf " ln failed, using cp...\n"
4200	fi
4201	cp ${localout[$i]} ${pathout[$i]}.${extout[$i]}
4202	fi
4203	else
4204	printf "\n >>> OUTPUT: ${localout[$i]} to ${pathout[$i]}\n"
4205	if [[ $link_local_output = true ]]
4206	then
4207	printf " using ln -f\n"
4208	ln -f ${localout[$i]} ${pathout[$i]}
4209	fi
4210	# If "ln -f" fails of if "$link_local_output = false" do a normal "cp"
4211	if [[ ! -f "${pathout[$i]}" ]]
4212	then
4213	if [[ $link_local_output = true ]]
4214	then
4215	printf " ln failed, using cp...\n"
4216	fi
4217	cp ${localout[$i]} ${pathout[$i]}
4218	fi
4219	fi
4220
4221	elif [[ "${actionout[$i]}" = "" && $files_for_pes = true ]]
4222	then
4223
4224	# DIE DEN PROZESSOREN EINES PARALLELRECHNERS ZUGEHOERIGEN
4225	# DATEIEN WERDEN ERST IN EINEM KATALOG GESAMMELT UND DIESER
4226	# WIRD DANN KOPIERT
4227	# PER MOVE UMBENANNT WERDEN
4228	printf "\n >>> OUTPUT: ${localout[$i]}/_.... to ${pathout[$i]}\n"
4229	if [[ $link_local_output = true ]]
4230	then
4231	printf " using ln -f\n"
4232	mkdir ${pathout[$i]}
4233	cd ${localout[$i]}
4234	for file in $(ls *)
4235	do
4236	ln -f $file ${pathout[$i]}
4237	done
4238	cd $TEMPDIR
4239	fi
4240	# If "ln -f" fails of if "$link_local_output = false" do a normal "cp -r"
4241	if [[ ! -f "${pathout[$i]}/_0000" ]]
4242	then
4243	if [[ $link_local_output = true ]]
4244	then
4245	printf " ln failed for .../_0000, using cp...\n"
4246	fi
4247	cp -r ${localout[$i]} ${pathout[$i]}
4248	fi
4249
4250	fi
4251	fi
4252	done
4253	if (( i != 0 ))
4254	then
4255	if [[ $transfer_problems = true ]]
4256	then
4257	printf "\n$striche\n *** OUTPUT-files saved"
4258	printf "\n +++ WARNING: some data transfers failed! \n"
4259	else
4260	printf "\n$striche\n *** all OUTPUT-files saved \n"
4261	fi
4262	fi
4263
4264
4265	# EVENTUELL FOLGEJOB STARTEN
4266	# DATEI CONTINUE_RUN MUSS VOM BENUTZERPROGRAMM AUS ERZEUGT WERDEN
4267	if [[ -f CONTINUE_RUN ]]
4268	then
4269
4270	if [[ $archive_save = true ]]
4271	then
4272
4273	# ZUERST IN MRUN-AUFRUF OPTIONEN FUER FORTSETZUNGSLAUF, FUER
4274	# STILLES ABARBEITEN (OHNE INTERAKTIVE RUECKFAGEN) UND FUER
4275	# BATCH-BETRIEB (NUR WICHTIG, FALLS BATCH AUF LOKALER MASCHINE
4276	# DURCHGEFUEHRT WERDEN SOLL) EINFUEGEN, FALLS NICHT BEREITS VOR-
4277	# HANDEN
4278	[[ $(echo $mc \| grep -c "\-C") = 0 ]] && mc="$mc -C"
4279	[[ $(echo $mc \| grep -c "\-v") = 0 ]] && mc="$mc -v"
4280	[[ $(echo $mc \| grep -c "\-b") = 0 ]] && mc="$mc -b"
4281	if [[ $(echo $mc \| grep -c "#") != 0 ]]
4282	then
4283	mc=`echo $mc \| sed 's/#/f/g'`
4284	fi
4285
4286
4287	# JOB STARTEN
4288	printf "\n\n *** initiating restart-run on \"$return_addres\" using command:\n"
4289	echo " $mc"
4290	printf "\n$striche\n"
4291	if [[ $localhost != $fromhost ]]
4292	then
4293
4294	if [[ $localhost = lcsgih \|\| $localhost = lcsgib \|\| $localhost = nech \|\| $localhost = ibmh \|\| $localhost = ibmkisti \|\| $localhost = ibmku \|\| $localhost = ibms ]]
4295	then
4296	echo "*** ssh will be used to initiate restart-runs!"
4297	echo " return_addres=\"$return_addres\" "
4298	echo " return_username=\"$return_username\" "
4299	if [[ $(echo $return_addres \| grep -c "130.75.105") = 1 ]]
4300	then
4301	ssh $return_addres -l $return_username "PATH=\$PATH:$LOCAL_MRUN_PATH;export LD_LIBRARY_PATH=\$LD_LIBRARY_PATH:/muksoft/packages/intel/Compiler/11.0/083/lib/intel64/;export PALM_BIN=$LOCAL_MRUN_PATH;cd $LOCAL_PWD; $mc "
4302	else
4303	if [[ $localhost = ibmkisti ]]
4304	then
4305	ssh $usern@gaiad "ssh $return_addres -l $return_username \"PATH=\\\$PATH:$LOCAL_MRUN_PATH;export PALM_BIN=$LOCAL_MRUN_PATH;cd $LOCAL_PWD; $mc\" "
4306	else
4307	ssh $return_addres -l $return_username "PATH=\$PATH:$LOCAL_MRUN_PATH;export PALM_BIN=$LOCAL_MRUN_PATH;cd $LOCAL_PWD; $mc "
4308	fi
4309	fi
4310	else
4311	printf "\n +++ no restart mechanism available for host \"$localhost\" "
4312	locat=restart; exit
4313	fi
4314
4315	# WARTEN, DAMIT SICH RESTART JOB IN QUEUE EINREIHEN KANN, BEVOR
4316	# DER AKTUELLE JOB ENDET
4317	if [[ $queue = special1q ]]
4318	then
4319	sleep 120
4320	else
4321	sleep 30
4322	fi
4323
4324	else
4325
4326	# BEI RECHNUNGEN AUF LOKALER MASCHINE KANN MRUN DIREKT AUFGE-
4327	# RUFEN WERDEN, AUSSER AUF lcfimm
4328	cd $LOCAL_PWD
4329	if [[ $localhost = lcfimm ]]
4330	then
4331	ssh $return_addres -l $return_username "PATH=\$PATH:$LOCAL_MRUN_PATH;export PALM_BIN=$LOCAL_MRUN_PATH;cd $LOCAL_PWD; $mc "
4332	else
4333	eval $mc # ' MUESSEN AUSGEWERTET WERDEN
4334	fi
4335	cd - > /dev/null
4336	fi
4337	printf "\n$striche\n *** restart-run initiated \n"
4338
4339
4340	# EVENTUELL INPUT-DATEIEN, DIE VON TEMPORAEREM DATENVERZEICHNIS
4341	# GEHOLT WORDEN SIND, LOESCHEN
4342	(( i = 0 ))
4343	while (( i < iin ))
4344	do
4345	(( i = i + 1 ))
4346	if [[ "${got_tmp[$i]}" = true && $keep_data_from_previous_run = false ]]
4347	then
4348	rm -r $tmp_data_catalog/${frelin[$i]}
4349	fi
4350	done
4351
4352	else
4353
4354	printf "\n +++ no restart-run possible, since errors occured"
4355	printf "\n during the archive process"
4356	fi
4357
4358	fi
4359
4360
4361
4362	# EVTL. EMAIL-BENACHRICHTIGUNG UEBER ABGESCHLOSSENEN LAUF
4363	if [[ "$email_notification" != "none" ]]
4364	then
4365
4366	if [[ $localhost != $fromhost ]]
4367	then
4368	if [[ -f CONTINUE_RUN ]]
4369	then
4370	echo "PALM restart run necessary" > email_text
4371	echo "description header of actual run:" >> email_text
4372	cat CONTINUE_RUN >> email_text
4373	echo "mrun-command to restart:" >> email_text
4374	echo "$mc" >> email_text
4375	else
4376	echo "PALM run with base filename \"$fname\" on host \"$localhost\" finished" > email_text
4377	fi
4378	mail $email_notification < email_text
4379	printf "\n *** email notification sent to \"$email_notification\" "
4380	fi
4381	fi
4382
4383
4384
4385	# ALLE ARBEITEN BEENDET. TEMPORAERER KATALOG KANN GELOESCHT WERDEN
4386	cd $HOME
4387	[[ $delete_temporary_catalog = true ]] && rm -rf $TEMPDIR
4388
4389	else
4390
4391
4392	# FALLS AUF REMOTE-MASCHINE GERECHNET WERDEN SOLL, WERDEN JETZT ENTSPRE-
4393	# CHENDE AKTIONEN DURCHGEFUEHRT
4394
4395	# MRUN-BEFEHL FUER REMOTE-MASCHINE ZUSAMMENSTELLEN
4396	mrun_com="$mrun_script_name -a $afname -c $config_file -d $fname -h $host -H $fromhost -m $memory -t $cpumax -q \$queue -R $return_addres -U $return_username -u $remote_username"
4397	[[ "$cpp_opts" != "" ]] && mrun_com=${mrun_com}" -D \"$cpp_opts\""
4398	[[ "$global_revision" != "" ]] && mrun_com=${mrun_com}" -G \"$global_revision\""
4399	[[ $group_number != none ]] && mrun_com=${mrun_com}" -g $group_number"
4400	[[ $do_compile = true ]] && mrun_com=${mrun_com}" -s \"$source_list\""
4401	[[ "$input_list" != "" ]] && mrun_com=${mrun_com}" -i \"$input_list\""
4402	[[ $ignore_archive_error = true ]] && mrun_com=${mrun_com}" -I"
4403	[[ $keep_data_from_previous_run = true ]] && mrun_com=${mrun_com}" -k"
4404	[[ "$additional_conditions" != "" ]] && mrun_com=${mrun_com}" -K \"$additional_conditions\""
4405	# [[ "$makefile" != "$source_path/Makefile" ]] && mrun_com=${mrun_com}" -M \"$makefile\""
4406	[[ "$output_list" != "" ]] && mrun_com=${mrun_com}" -o \"$output_list\""
4407	[[ "$read_from_config" = false ]] && mrun_com=${mrun_com}" -S"
4408	[[ $do_trace = true ]] && mrun_com=${mrun_com}" -x"
4409	[[ "$numprocs" != "" ]] && mrun_com=${mrun_com}" -X $numprocs"
4410	if [[ $use_openmp = true ]]
4411	then
4412	mrun_com=${mrun_com}" -O $threads_per_task"
4413	fi
4414	[[ "$tasks_per_node" != "" ]] && mrun_com=${mrun_com}" -T $tasks_per_node"
4415	[[ $store_on_archive_system = true ]] && mrun_com=${mrun_com}" -A"
4416	[[ $package_list != "" ]] && mrun_com=${mrun_com}" -p \"$package_list\""
4417	[[ $return_password != "" ]] && mrun_com=${mrun_com}" -P $return_password"
4418	[[ $delete_temporary_catalog = false ]] && mrun_com=${mrun_com}" -B"
4419	[[ $node_usage != default && "$(echo $node_usage \| cut -c1-3)" != "sla" && $node_usage != novice ]] && mrun_com=${mrun_com}" -n $node_usage"
4420	[[ "$ocean_file_appendix" = true ]] && mrun_com=${mrun_com}" -y"
4421	[[ $run_coupled_model = true ]] && mrun_com=${mrun_com}" -Y \"$coupled_dist\""
4422	if [[ $do_remote = true ]]
4423	then
4424	printf "\n>>>> MRUN-command on execution host:\n>>>> $mrun_com \n"
4425	fi
4426
4427
4428	# ZUSAMMENSTELLUNG DES JOBSCRIPTS AUF DATEI jobfile
4429	jobfile=jobfile.$RANDOM
4430
4431
4432	# TEMPORAERES VERZEICHNIS GENERIEREN UND NACH DORT WECHSELN
4433	echo "mkdir $TEMPDIR" >> $jobfile
4434	echo "cd $TEMPDIR" >> $jobfile
4435
4436
4437	# EVENTUELL FEHLERVERFOLGUNG AKTIVIEREN
4438	if [[ $do_trace = true ]]
4439	then
4440	echo "set -x" >> $jobfile
4441	else
4442	echo "set +vx" >> $jobfile
4443	fi
4444
4445
4446	# BEREITSTELLUNG VON QUELLTEXTEN, MRUN-SCRIPTS UND KONFIGURATIONS-
4447	# DATEI FUER DEN JOB
4448	if [[ $( echo $host \| cut -c1-5 ) = lcsgi \|\| $host = ibmkisti ]]
4449	then
4450
4451	# KONFIGURATIONSDATEI UND MRUN_SCRIPT IN DAS SOURCE-VERZEICHNIS
4452	# KOPIEREN
4453	if [[ $restart_run != true ]]
4454	then
4455	cp $config_file $working_directory/SOURCES_FOR_RUN_$fname
4456	cp ${PALM_BIN}/$mrun_script_name $working_directory/SOURCES_FOR_RUN_$fname
4457	fi
4458
4459	# SOURCE-VERZEICHNIS VOM LOKALEN RECHNER PER SCP TRANSFERIEREN
4460	# (BEI JOBS AUF LOKALEM RECHNER REICHT CP)
4461	echo "set -x" >> $jobfile
4462	if [[ $host = $localhost ]]
4463	then
4464	echo "cp -r $working_directory/SOURCES_FOR_RUN_$fname ." >> $jobfile
4465	else
4466	if [[ $host != ibmkisti ]]
4467	then
4468	echo "scp -r $return_username@$return_addres:$working_directory/SOURCES_FOR_RUN_$fname ." >> $jobfile
4469	else
4470	# on KISTI's IBM firewall is only opened on interactive node
4471	echo "localdir=\`pwd\`" >> $jobfile
4472	echo "ssh $remote_username@gaiad \"cd \$localdir; scp -r $return_username@$return_addres:$working_directory/SOURCES_FOR_RUN_$fname .\" " >> $jobfile
4473	fi
4474	fi
4475	echo "export SOURCES_COMPLETE=true" >> $jobfile
4476
4477	# QUELLTEXTE, MRUN-SCRIPT UND KONFIGURATIONSDATEI IN DAS AKTUELLE
4478	# ARBEITSVERZEICHNIS VERSCHIEBEN
4479	echo "mv SOURCES_FOR_RUN_$fname/$config_file . " >> $jobfile
4480	echo "mv SOURCES_FOR_RUN_$fname/$mrun_script_name . " >> $jobfile
4481	echo "execute_mrun=true" >> $jobfile
4482	echo " " >> $jobfile
4483
4484	else
4485
4486	# ABSPEICHERN DER QUELLTEXTE (NUR FALLS UEBERSETZT WERDEN SOLL)
4487	# SOWIE GEGEBENENFALLS DES MAKEFILES
4488	if [[ $do_compile = true ]]
4489	then
4490
4491	source_catalog=SOURCES_FOR_RUN_$fname
4492
4493	# UNTERVERZEICHNIS FUER QUELLTEXTE UND MAKEFILE ANLEGEN
4494	# MRUN WIRD DIESES VRZEICHNIS UEBER ENVIRONMENT VARIABLE
4495	# MITGETEILT (UEBERSTEUERT ANGABE IN KONFIGURATIONSDATEI)
4496	echo "mkdir SOURCES_FOR_RUN_$fname" >> $jobfile
4497	echo "export SOURCES_COMPLETE=true" >> $jobfile
4498	echo "cd SOURCES_FOR_RUN_$fname" >> $jobfile
4499
4500	for filename in $source_list
4501	do
4502	# ABDECKZEICHEN VERHINDERN, DASS ERSETZUNGEN ERFOLGEN
4503	echo "cat > $filename << \"%END%\"" >> $jobfile
4504	cat $source_catalog/$filename >> $jobfile
4505	echo " " >> $jobfile
4506	echo "%END%" >> $jobfile
4507	echo " " >> $jobfile
4508	done
4509
4510	# ABDECKZEICHEN VERHINDERN, DASS ERSETZUNGEN ERFOLGEN
4511	echo "cat > Makefile << \"%END%\"" >> $jobfile
4512	cat $source_catalog/Makefile >> $jobfile
4513	echo " " >> $jobfile
4514	echo "%END%" >> $jobfile
4515	echo " " >> $jobfile
4516
4517	echo "cd - > /dev/null" >> $jobfile
4518
4519	fi
4520
4521
4522	# ABSPEICHERN DER KONFIGURATIONSDATEI
4523	# ABDECKZEICHEN VERHINDERN, DASS ERSETZUNGEN ERFOLGEN
4524	echo "cat > $config_file << \"%END%\"" >> $jobfile
4525	cat $config_file >> $jobfile
4526	echo "%END%" >> $jobfile
4527	echo " " >> $jobfile
4528
4529
4530	# ABSPEICHERN DER AKTUELLEN MRUN-VERSION
4531	# ABDECKZEICHEN VERHINDERN, DASS ERSETZUNGEN ERFOLGEN
4532	echo "cat > $mrun_script_name <<\"%END%\"" >> $jobfile
4533	cat ${PALM_BIN}/$mrun_script_name >> $jobfile
4534	echo "%END%" >> $jobfile
4535	echo "chmod u+x $mrun_script_name" >> $jobfile
4536	echo "execute_mrun=true" >> $jobfile
4537	echo " " >> $jobfile
4538
4539	fi
4540
4541
4542	# EVTL. BENOETIGTE INPUT-DATEIEN PER FTP HOLEN ODER DEM JOB DIREKT
4543	# MITGEBEN UND AUF DEM REMOTE-RECHNER IM BENUTZERVERZEICHNIS ABLEGEN
4544	# FALLS DIESES NICHT EXISTIERT, WIRD VERSUCHT, DAS JEWEILS LETZTE
4545	# UNTERVERZEICHNIS DES PFADNAMENS ANZULEGEN
4546	if [[ $do_remote = true ]]
4547	then
4548	(( i = 0 ))
4549	while (( i < iin ))
4550	do
4551	(( i = i + 1 ))
4552	echo "[[ ! -d ${pathin[$i]} ]] && mkdir -p ${pathin[$i]}" >> $jobfile
4553	if [[ "${transin[$i]}" = job ]]
4554	then
4555	echo "cat > ${remotepathin[$i]} <<\"%END%\"" >> $jobfile
4556	eval cat ${pathin[$i]}/${frelin[$i]} >> $jobfile
4557	echo " " >> $jobfile
4558	echo "%END%" >> $jobfile
4559	else
4560	echo "batch_scp -b -o -g -s -u $return_username $return_addres ${remotepathin[$i]} \"${pathin[$i]}\" ${frelin[$i]}" >> $jobfile
4561	fi
4562
4563	# UEBERPRUEFEN, OB DATEI ANGELEGT WERDEN KONNTE
4564	echo "if [[ \$? = 1 ]]" >> $jobfile
4565	echo "then" >> $jobfile
4566	echo " echo \" \" " >> $jobfile
4567	echo " echo \"+++ file ${remotepathin[$i]} could not be created\" " >> $jobfile
4568	echo " echo \" please check, if directory exists on $host!\" " >> $jobfile
4569	echo " echo \"+++ MRUN will not be continued\" " >> $jobfile
4570	echo " execute_mrun=false" >> $jobfile
4571	echo "fi" >> $jobfile
4572	done
4573	fi
4574
4575	# ARBEITSKATALOG AUF DER LOKALEN MASCHINE FUER EVENTUELLE
4576	# FORTSETZUNGSLAUEFE PER ENV-VARIABLE UEBERGEBEN
4577	echo "LOCAL_PWD=$working_directory" >> $jobfile
4578	echo "export LOCAL_PWD" >> $jobfile
4579
4580	# EBENSO LOKALEN MRUN-KATALOG UEBERGEBEN
4581	echo "LOCAL_MRUN_PATH=$PALM_BIN" >> $jobfile
4582	echo "export LOCAL_MRUN_PATH" >> $jobfile
4583
4584	# WORKAROUND FUER RIAM-NEC-JOBS WEGEN PROFILE-SCHWIERIGKEITEN
4585	if [[ $localhost_realname = "gate" \|\| $localhost = lctit ]]
4586	then
4587	echo "export PALM_BIN=$PALM_BIN" >> $jobfile
4588	fi
4589
4590	# MRUN AUF ZIELRECHNER AUFRUFEN (queue is workaround for ibmkisti)
4591	echo "set -x" >> $jobfile
4592	echo "queue=$queue" >> $jobfile
4593	echo "[[ \$execute_mrun = true ]] && ./$mrun_com" >> $jobfile
4594
4595	echo 'ls -al; echo `pwd`' >> $jobfile
4596	echo "cd \$HOME" >> $jobfile
4597	echo "rm -rf $TEMPDIR" >> $jobfile
4598
4599
4600
4601
4602	# JOB PER SUBJOB STARTEN
4603	if [[ $silent = false ]]
4604	then
4605	printf "\n "
4606	else
4607	printf "\n\n"
4608	fi
4609
4610	subjob $job_on_file -h $host -u $remote_username -g $group_number -q $queue -m $memory -N $node_usage -t $cpumax $XOPT $TOPT $OOPT -n $fname -v -c $job_catalog -e $email_notification $jobfile
4611	rm -rf $jobfile
4612
4613
4614	fi # ENDE REMOTE-TEIL
4615
4616
4617
4618	# ENDE DER PROZEDUR

Note: See TracBrowser for help on using the repository browser.

Download in other formats:

| Impressum | ©Leibniz Universität Hannover |