Home

Context Navigation

source: palm/trunk/SCRIPTS/mrun @ 821

Last change on this file since 821 was 821, checked in by maronga, 13 years ago
bugfix in mrun (queue was not properly set, bigq not available on ice1)
Property svn:keywords set to `Id Rev`
File size: 188.9 KB

Rev	Line
[71]	1	#!/bin/ksh
[503]	2	# mrun - script for running PALM jobs
[169]	3	# $Id: mrun 821 2012-02-14 10:51:28Z maronga $
[1]	4
[503]	5	# Procedure for interactive/batch PALM runs (initial runs and automatic
	6	# restart runs)
[1]	7
[503]	8	# Last changes:
[1]	9	# 03/03/94 - Siggi - Entwicklungsbeginn
	10	# 21/03/94 - Siggi - Entwicklungsabschluss (Version 1.0)
	11	# 09/12/94 - Siggi - allexport unterbunden; statt dessen werden jetzt ein-
	12	# zelne Variable exportiert. Dies ist noetig, weil an-
	13	# sonsten irgendein Speicher ueberlaeuft und von mrun
	14	# aufgerufene Programme (z.B. stageout) mit Fehlern
	15	# abbrechen (too many arguments)
	16	# 20/06/95 - Siggi - noclobber muss explizit abgeschaltet, da jetzt
	17	# defaultmaessig im RRZN-Profile eingeschaltet
	18	# 10/06/97 - Siggi - Zusaetzliche Option -C bei cpp, damit // nicht
	19	# wegfallen
	20	# 25/04/00 - Siggi - Version 1.5
	21	# remote_addres und remote_user in return_addres bzw.
	22	# return_username umbenannt, return_username wird dem
	23	# mrun-Aufruf im Job ueber Option -U mitgegeben.
	24	# Neue Variable remote_username, die beim interaktiven
	25	# mrun-Aufruf zwingend ueber die neue Option -u oder
	26	# in der Konfigurationsdatei angegeben werden muss.
	27	# 08/02/01 - Siggi - Alle mrun-Meldungen ins englische uebersetzt,
	28	# Version 1.7
	29	# 05/05/02 - Siggi - Uebersetzungen mittels make-Mechanismus moeglich
	30	# 21/11/02 - Siggi - ENV-Variable XLFRTEOPTS wird vor Ausfuehrung auf
	31	# IBM gesetzt, um Record-Laenge der NAMELIST-Dateien,
	32	# zu bestimmen; Praeprozessoroptionen haben nun alle
	33	# die Form -Dtext=text, weil auf IBM sonst text durch
	34	# die leere Zeichenkette ersetzt wird
	35	# 01/08/03 - Siggi - Test of implementing dvrp+1PE on ibm
	36	# 07/01/04 - Siggi - additional preprocessor directive for ibm included
	37	# (-D$OMP=OMP) in order to avoid problems with
	38	# OMP_NUM_THREADS
	39	# 04/01/05 - Siggi - archiving on hanni and berni is done in separate
	40	# tar files, one for each node used
	41	# 07/01/04 - Siggi - old code for t3e, vpp and hpcs eliminated
	42	# 09/01/05 - Siggi - workaround for getting the IP address on gfdl3
	43	# 17/01/05 - Siggi - job for data transfer to local machine now within
	44	# job class c1 (HLRN)
	45	# 27/01/05 - Siggi - IP address workaround for gfdl3 removed
	46	# 28/01/05 - Siggi - tar-filenames on $PERM (HLRN) now include the
	47	# base filename and cycle number
	48	# 16/02/05 - Gerald - hababai validated
	49	# 14/03/05 - Siggi - abort on NEC after first runtime error (F_ERRCNT)
	50	# 29/03/05 - Marcus - berni*-en0 validated
	51	# 21/04/05 - Siggi - transfer-job-protocol for avs-data is no more stored
	52	# 24/04/05 - Siggi - netcdf support on lcmuk
	53	# 25/04/05 - Siggi - netcdf support on gfld3 (decalpha)
	54	# 11/05/05 - Siggi - files with non-numeric extensions (.xxx after cycle
	55	# number) can now be used within interactive runs
	56	# 12/05/05 - Siggi - netcdf support on ibm
	57	# 13/05/05 - Siggi - error in tar-filenames on $PERM (HLRN) removed
	58	# 18/05/05 - Siggi - netcdf support on nec
	59	# 24/05/05 - Siggi - netcdf support on ibms
	60	# 14/06/05 - Siggi - sleep interval after submit of restart job increased
	61	# to 30 seconds
	62	# 28/06/05 - Siggi - bora adjusted to new intel compiler (LD_LIBRARY_
	63	# PATH is temporarily set)
	64	# 07/09/05 - Siggi - setting of MP_EAGER_LIMIT switched off because it
	65	# caused warnings in the job protocol
	66	# 20/10/05 - Siggi - update of netcdf-version on decalpha (gfdl3)
	67	# 25/10/05 - Siggi - error in listing files for getting the cycle number
	68	# removed
	69	# 26/10/05 - Siggi - new paths for dvrp-library, transfer of catalogs
	70	# realized by file attribute trpe
	71	# 28/10/05 - Siggi - if existing, the file extension is output in case
	72	# of missing input file - no job abort in case of
	73	# missing input files with file extensions
	74	# (preliminary solution)
	75	# 31/10/05 - Siggi - data transfer from nech now within seperate job
	76	# 04/11/05 - Siggi - netcdf 3.6.0-p1 on ibmh/ibmb
	77	# 07/12/05 - Siggi - gallego and elephanta admitted
	78	# 30/12/05 - Siggi - gfdl5 (ibmy) admitted
	79	# 10/01/06 - Siggi - cpp directive for NetCDF 64bit support
	80	# 20/01/06 - Siggi - cpp directive for ibmy
	81	# 09/02/06 - Siggi - ibmy admitted for batch mode
	82	# 13/04/06 - Siggi - ostria admitted
	83	# 18/04/06 - Siggi - usage of OpenMP implemented (additional option -O)
	84	# 10/05/06 - Siggi - environment variable XLSMPOPTS set on IBM for
	85	# optimization of OpenMP runs (option stacksize is
	86	# necessary for large gridpoint numbers because
	87	# otherwise a segmentation fault occurs)
	88	# 23/05/05 - Siggi - lctit (SUN Fire X4600) admitted
	89	# 21/08/06 - Siggi - standard mrun path on scirocco is as on bora
	90	# 23/08/06 - Siggi - netcdf support for scirocco (notebook)
	91	# 20/09/06 - Marcus - add DVRP library on nech
	92	# 24/10/06 - Siggi - Environment variables for PALM steering are written
	93	# on local NAMELIST file ENVPAR
	94	# 24/11/06 - Siggi - levanto admitted, maestro switched to ifc 9.1
	95	# 28/11/06 - Siggi - call to interpret_config.x_levanto because of SuSe 10
	96	# 06/02/07 - Siggi - add DVRP library on lcmuk
	97	# 07/02/07 - Siggi - revision of source code handling, source code is
	98	# now expected in directory given by source_path and
	99	# not in the current working directory, source code
	100	# files provided in add_source_path are allways
	101	# compiled, source code to be compiled is allways
	102	# sampled in SOURCES_FOR_RUN_$fname which is a
	103	# subdirectory of the current working directory,
[5]	104	# -s TEST changed to -s WRITE_PERMIT or -s WP,
[1]	105	# adapted for RIAM (neck)
[5]	106	# bugfix for cycle number of output file (extout)
	107	# all hpmuk-related code removed
[14]	108	# 14/02/07 - Siggi - dvrp support for neck added
[21]	109	# 28/02/07 - Siggi - empty lines in configuration file are accepted
[26]	110	# mrun_path replaced by PALM_BIN,
	111	# all machines are calling interpret_config.x
[66]	112	# 14/03/07 - Siggi - fimm admitted, revision number added to terminal
	113	# output
[69]	114	# 16/03/07 - Siggi - adjustments for lctit
[78]	115	# 29/03/07 - Siggi - global revision transfered to batch job by new
	116	# option -G
[82]	117	# 30/03/07 - Siggi - compilation "by hand" removed, cpp-directives/options
	118	# + netcdf/dvrp-options are read from configuration
	119	# file, host identifier (local_host) is read from
	120	# config file, code related to ftp filetransfer
	121	# removed (incl. option -f)
[97]	122	# 19/06/07 - Siggi - time limit for cdata jobs increased
[102]	123	# 25/07/07 - Siggi - two sets of executables can be started on lcmuk,
	124	# if new option -Y (coupling) is given,
	125	# output of executables to aout_output removed,
	126	# messages are immediately written to stdout instead
[108]	127	# 03/08/07 - Marcus - add XOPT="-X $numprocs" for lcfimm
	128	# 09/08/07 - Marcus - workaround on lcfimm to propagate environment
	129	# variables out to the nodes in coupled mode -disabled-
	130	# 13/08/07 - Marcus - start local restart jobs per ssh on lcfimm
[109]	131	# 28/08/07 - Marcus - completely remove workaround on lcfimm to propagate
	132	# environment variables out to the nodes in coupled mode
[118]	133	# 15/10/07 - Siggi - Preliminary adjustments for lctit, based on Jin's
	134	# suggestions
[122]	135	# 19/10/07 - Marcus - further adjustments for lctit: add new optional
	136	# argument -g group_number, admit all sla* node_usage
	137	# queues
[129]	138	# 30/10/07 - Marcus - further adjustments for queues on lctit
[149]	139	# 14/02/08 - Bjoern - link of restart files with long file name possible
	140	# 28/02/08 - Marcus - If environment variable link_local_output is set to
	141	# true, mrun tries "ln -f" on local output and resorts
	142	# to "cp" or "cp -r" on error
[164]	143	# 15/04/08 - Siggi - argument -c introduced to most of the subjob calls,
	144	# which allows the user to choose his own job catalog
	145	# by setting job_catalog in the configuration file
	146	# (default is ~/job_queue),
[172]	147	# workaround for mpiexec with -env option,
[164]	148	# adjustments for lcxt4 (Bergen Center for Computational
	149	# Science)
[183]	150	# 22/05/08 - Marcus - If environment variable link_local_input is set to
	151	# true, mrun tries "ln -f" on local input and resorts
	152	# to "cp" or "cp -r" on error
[180]	153	# 27/05/08 - Siggi - PATH is set to PALM_BIN everywhere (missing so far)
[179]	154	# 14/07/08 - Siggi - adjustments for lcsgih
[188]	155	# 08/08/08 - Marcus - typo removed in lcxt4 branch
[204]	156	# 17/09/08 - Siggi - restart mechanism adjusted for lcsgi
[210]	157	# 02/10/08 - BjornM - argument "-Y" modified, adjustments for coupled runs
	158	# 21/10/08 - Siggi - bugfix for the case that -K has more than one
	159	# argument
[211]	160	# 08/11/08 - Siggi - bugfix for depository path settings in configuration
	161	# file
	162	# 11/11/08 - Siggi - rules for using user code files changed: user
	163	# code splitted into one file per subroutine,
	164	# user can provide his/her own makefile, only
	165	# default source-code filenames are allowed
[215]	166	# 14/11/08 - Siggi - cond1 and cond2 are part of the depository name
[221]	167	# 08/01/09 - Siggi - mpi module on sgi machines to be used can be set
	168	# with environment variable mpilib
[223]	169	# 13/01/09 - Siggi - totalview on hice/bice can be used in debug mode
	170	# using environment variable totalview
[246]	171	# 25/02/09 - Siggi - cputime-option (-t) is allowed to be omitted for
	172	# interactive runs
[253]	173	# 05/03/09 - Siggi - adjustments for new NEC-SX9 at RIAM (necriam)
[260]	174	# 16/03/09 - Siggi - dvrp_lib, dvrp_inc replaced by dvr_lib, dvr_inc,
	175	# dvr streaming server is automatically started using
	176	# new variable dvr_server and a configuration file
	177	# .dvrserver.config,
[261]	178	# processing of dvr output files using new script
	179	# process_dvr_output implemented, in order to make
	180	# dvr application more user friendly
[265]	181	# 20/03/09 - Marcus - update of n1ge command for lctit
[291]	182	# 16/04/09 - Siggi - new option -y for precursor (uncoupled) ocean runs
	183	# to be followed by a coupled atmosphere-ocean run
[343]	184	# 21/04/09 - Siggi - adjustments for new IBM at DKRZ, which is now ibmh
	185	# 24/06/09 - BjornM - adjustments for coupled/precursor runs on ibmy
[352]	186	# 08/07/09 - Siggi - default value for email_notification is none,
	187	# option -e added to subjob
[359]	188	# 19/08/09 - Marcus - MPI debug option
[362]	189	# 21/08/09 - Marcus - totalview on hice/bice also available for mvapich2
[367]	190	# 25/08/09 - BjornM - adpated for lck
[369]	191	# 26/08/09 - Siggi - additional environment variables for mpt on ice2
[372]	192	# 26/08/09 - Marcus - adjustment for ice2 on lcsgib
[397]	193	# 29/09/09 - Siggi - default value for archiving is false, option "-A"
	194	# switches on archiving, archiving on SGI-ICE enabled
[416]	195	# 16/10/09 - Carolin- adjustments for archiving on SGI-ICE of binary files;
	196	# adjustment for special1q
	197	# 18/12/09 - Carolin- modification of archiving (permq)
[440]	198	# 01/02/10 - Siggi - adapted for lcxt5m and lckyoto (Fujitsu HX600)
[475]	199	# 03/02/10 - Siggi - make options (mopts) to be set by configuration file
	200	# implemented
[493]	201	# 08/02/10 - Siggi - loading of modules now controlled via configuration
	202	# file (modules)
[503]	203	# 02/03/10 - Siggi - for each block in the configuration file, separate
	204	# utility programs are used, combine_plot_fields is
	205	# directly called and must not be given by an output
	206	# command in the configuration file any more
[540]	207	# 15/06/10 - Rieke - ssh from compute nodes to login nodes on hosts
	208	# lcsgih/lcsgib is done using the names
	209	# hicegate0/bicegate0 instead of the IP addresses
[544]	210	# 17/08/10 - BjornM - adjustments for interactive runs on lcxt4
[555]	211	# 07/09/10 - Siggi - bugfix for wrong netcdf/3.6.3 module on lcsgi
[622]	212	# 08/12/10 - Siggi - new handling of openmp/hybrid runs, option -O
	213	# has now argument threads_per_task
	214	# adjustments for Kyushu Univ. (lcrte, ibmku)
[635]	215	# 14/12/10 - Siggi - adjustments for new Tsubame system at Tokyo
	216	# institute of technology (lctit)
[759]	217	# 23/12/10 - Micha - different number of processors in ocean and
	218	# atmosphere is now allowed
[678]	219	# 02/02/10 - Siggi - further adjustments on Tsubame and concerning openMP
	220	# usage
[696]	221	# 09/03/10 - Siggi - adjustments for ibmkisti, mpt bugfix for netCDF4
[693]	222	# usage, totalview usage extended
[721]	223	# 17/03/11 - Siggi - adjustments for openmp usage on ibmkisti
	224	# 03/04/11 - Micha - added lckordi
	225	# 06/04/11 - BjornM - bugfix for runs with mpt on lcsgi
[738]	226	# 17/08/11 - Siggi - extensions for impi library
[759]	227	# 18/08/11 - Siggi - bugfix for local append of output files with suffix
	228	# (.nc)
[746]	229	# 18/08/11 - Marcus - support for Linux OS with German locale
[759]	230	# - properly report hosts on general Linux clusters
[755]	231	# 29/08/11 - BjornW - adapted for lcflow (ForWind cluster in Oldenburg)
[759]	232	# 29/08/11 - Carolin- initiating restart-run: adjustment of the path at
	233	# IMUK
	234	# 15/09/11 - Siggi - new option -w tp set the maximum number of parallel
	235	# io streams, option -T is obligatory from now on
[807]	236	# 25/01/12 - BjornM - a namelist file check prior to the model run has
	237	# been implemented (check_namelist_files).
	238	# New option -z can be used to skip the check
[815]	239	# 03/02/12 - BjornM - namelist file check temporary disabled
	240	# until severel bugs are fixed
[818]	241	# 08/02/12 - BjornM - bugfixes: skipping namelist file check in case of
	242	# restart runs. check is now possible with topography
	243	# and the -v (silent) option is considered
[821]	244	# 14/02/12 - BjornM/- bugfix: removed slash in \$queue, changed default
	245	# Farah queue for lcsgi to testq, since bigq is not
	246	# available on ice1
[1]	247
	248	# VARIABLENVEREINBARUNGEN + DEFAULTWERTE
	249
	250	set +o allexport # SICHERHEITSHALBER UNTERBINDEN, DA SONST EVTL. STAGEOUT
	251	# NICHT LAUEFT (TOO MANY ARGUMENTS - PROBLEM)
	252	set +o noclobber # EXISTIERENDE DATEIEN DUERFEN UEBERSCHRIEBEN WERDEN
	253
	254	AddFilenames=""
	255	additional_conditions=""
	256	add_source_path=""
	257	afname=""
	258	archive_save=true
	259	archive_system=none
[807]	260	check_namelist_files=true
[1]	261	compiler_name=""
	262	cond1=""
	263	cond2=""
	264	config_file=.mrun.config
[206]	265	coupled_dist=""
	266	coupled_mode="mpi1"
[1]	267	cpp_opts=""
[82]	268	cpp_options=""
[1]	269	cpumax=0
	270	cpurest=0
	271	delete_temporary_catalog=true
	272	do_batch=false
	273	do_compile=true
	274	do_remote=false
	275	do_stagein=true
	276	do_stageout=true
	277	do_trace=false
[352]	278	email_notification="none"
[1]	279	exclude=""
	280	executable=""
	281	execution_error=false
[67]	282	fimm=false
[1]	283	fname=test
	284	fromhost=""
[78]	285	global_revision=""
[122]	286	group_number=none
[1]	287	host=""
	288	host_file=""
	289	hp=""
	290	ignore_archive_error=false
	291	input_list=""
	292	interpreted_config_file=""
[164]	293	job_catalog="~/job_queue"
[1]	294	job_on_file=""
	295	keep_data_from_previous_run=false
[183]	296	link_local_input=false
[149]	297	link_local_output=false
[1]	298	localhost_realname=$(hostname)
	299	local_compile=false
[260]	300	local_dvrserver_running=.FALSE.
[1]	301	locat=normal
	302	mainprog=""
	303	makefile=""
[759]	304	max_par_io_str=""
[1]	305	mc=$0
	306	while [[ $(echo $mc \| grep -c "/") != 0 ]]
	307	do
	308	mc=`echo $mc \| cut -f2- -d"/"`
	309	done
[493]	310	module_calls=""
[1]	311	mrun_script_name=$mc
[82]	312	netcdf_inc=""
	313	netcdf_lib=""
[1]	314	netcdf_support=false
	315	node_usage=default
	316	numprocs=""
[206]	317	numprocs_atmos=0
	318	numprocs_ocean=0
[1]	319	OOPT=""
	320	openmp=false
	321	output_list=""
	322	package_list=""
	323	punkte="..........................................................."
[807]	324	p3d_suffix=""
[1]	325	queue=none
	326	read_from_config=""
[5]	327	restart_run=false
[622]	328	if [[ `hostname` = rte10 ]]
[1]	329	then
[622]	330	return_addres=133.5.185.60
[693]	331	echo "+++ WARNING: fixed return_addres = $return_addres is used !!!!!"
	332	elif [[ `hostname` = climate0 ]]
	333	then
	334	return_addres=165.132.26.68
	335	echo "+++ WARNING: fixed return_addres = $return_addres is used !!!!!"
[622]	336	else
	337	return_addres=$(nslookup `hostname` 2>&1 \| grep "Address:" \| tail -1 \| awk '{print $2}')
[1]	338	fi
	339	return_password=""
	340	return_username=$LOGNAME
	341	remotecall=false
	342	remote_username=""
[102]	343	run_coupled_model=false
[1]	344	run_mode=""
	345	scirocco=false
[397]	346	store_on_archive_system=false
[1]	347	striche=" ----------------------------------------------------------------------------"
	348	silent=false
	349	source_list=""
	350	source_path=SOURCE
	351	tasks_per_node=""
	352	threads_per_task=1
	353	tmpcreate=false
	354	tmp_data_catalog=""
	355	transfer_problems=false
	356	usern=$LOGNAME
[377]	357	use_openmp=false
[1]	358	working_directory=`pwd`
	359	TOPT=""
	360	XOPT=""
	361	zeit=$( date \| cut -c 12-19 )
	362
	363	typeset -i iec=0 iic=0 iin=0 ioc=0 iout=0 memory=0 stagein_anz=0 stageout_anz=0
[206]	364	typeset -i cputime i ii iia iii iio icycle inode ival jobges jobsek maxcycle minuten nodes pes sekunden tp1
[1]	365
[66]	366	typeset -R30 calltime
[1]	367	typeset -L20 spalte1
	368	typeset -L40 spalte2
	369	typeset -L60 spalte3
	370	typeset -L35 string1=`date`
	371	typeset -L12 string2=$usern
	372	typeset -L12 string3=$localhost_realname
	373	typeset -L12 string4
	374	typeset -L12 string5
[201]	375	typeset -L30 version="MRUN 2.0 Rev$Rev: 821 $"
[1]	376
	377
	378
	379	# EINZELNE VARIABLE FUER HAUPTPROGRAMM EXPORTIEREN
	380	export cpurest fname host localhost return_addres return_username remotecall tasks_per_node
	381
	382	# FOLGENDE VARIABLEN MUESSEN FUER DIE INTERPRETATION DER KONFIGURATIONSDATEI
	383	# EXPORTIERT WERDEN
	384	export afname config_file cpp_opts cpumax do_batch do_trace fname fromhost
[127]	385	export group_number input_list memory numprocs output_list queue run_mode
[1]	386
	387
	388
	389	# FEHLERBEHANDLUNG
	390	# BEI EXIT:
[83]	391	trap 'rm -rf $working_directory/tmp_mrun
[69]	392	if [[ $locat != localhost ]]
[1]	393	then
[22]	394	# if [[ ! -f ${mrun_path}/statistik/mrun_statistik ]]
	395	# then
	396	# cat > ${mrun_path}/statistik/mrun_statistik << %STATEND%
	397	#MRUN-calls on $localhost
	398	#
	399	#date and time user localhost remotehost termination mrun-command
	400	#--------------------------------------------------------------------------------------------------------------------
	401	#%STATEND%
	402	# chmod 666 ${mrun_path}/statistik/mrun_statistik
	403	# fi
	404	#
	405	# # EINTRAG IN DIE STATISTIK-DATEI
	406	# string4=$host
	407	# string5=$locat
	408	# if [[ "$job_on_file" = "" && $locat != control_c && $locat != user_abort ]]
	409	# then
	410	# if [[ $do_batch = true ]]
	411	# then
	412	# printf "$string1$string2$string3$string4$string5$mrun_com \n" >> ${mrun_path}/statistik/mrun_statistik
	413	# else
	414	# printf "$string1$string2$string3$string4$string5$mc \n" >> ${mrun_path}/statistik/mrun_statistik
	415	# fi
	416	# fi
[69]	417	echo " " > /dev/null
[1]	418	fi
	419
	420	if [[ $locat != normal && $locat != control_c && $locat != local_compile ]]
	421	then
	422
	423	# EVENTUELLE ERROR-KOMMANDOS ABARBEITEN
	424	(( i = 0 ))
	425	while (( i < iec ))
	426	do
	427	(( i = i + 1 ))
	428	printf "\n *** Execution of ERROR-command:\n"
	429	printf " >>> ${err_command[$i]}\n"
	430	eval ${err_command[$i]}
	431	done
	432	if [[ -n $interpreted_config_file ]] then
	433	rm -rf $interpreted_config_file
	434	fi
[71]	435	if [[ -n .mrun_environment ]] then
	436	rm -rf .mrun_environment
	437	fi
[1]	438	if [[ $tmpcreate = true ]]
	439	then
	440	printf "\n *** Contents of \"$TEMPDIR\":\n"
	441	ls -al; cd
	442	[[ $delete_temporary_catalog = true ]] && rm -rf $TEMPDIR
	443	fi
[260]	444	if [[ "$dvrserver_id" != "" ]]
	445	then
	446	echo "+++ killing dvrserver_id=$dvrserver_id"
	447	kill $dvrserver_id
	448	fi
[1]	449	if [[ -f ~/job_queue/JOBINFO.$QSUB_REQID ]]
	450	then
	451	rm -rf ~/job_queue/JOBINFO.$QSUB_REQID
	452	fi
	453	printf "\n\n+++ MRUN killed \n\n"
	454	elif [[ $locat != control_c ]]
	455	then
	456	printf "\n\n --> all actions finished\n\n"
	457	printf " Bye, bye $usern !!\n\n"
	458	fi' exit
	459
	460
	461	# BEI TERMINAL-BREAK:
[83]	462	trap 'rm -rf $working_directory/tmp_mrun
[807]	463	rm -rf $working_directory/tmp_check_namelist_files
[69]	464	[[ $tmpcreate = true ]] && (cd; rm -rf $TEMPDIR)
[1]	465	if [[ -f ~/job_queue/JOBINFO.$QSUB_REQID ]]
	466	then
	467	rm -rf ~/job_queue/JOBINFO.$QSUB_REQID
	468	fi
[260]	469	if [[ "$dvrserver_id" != "" ]]
	470	then
	471	echo "+++ killing dvrserver_id=$dvrserver_id"
	472	kill $dvrserver_id
	473	fi
[1]	474	printf "\n+++ MRUN killed by \"^C\" \n\n"
	475	locat=control_c
	476	exit
	477	' 2
	478
	479
[22]	480	# CHECK IF THE PATH FOR THE PALM BINARIES (SCRIPTS+UTILITY-PROGRAMS) HAS
	481	# BEEN SET
	482	if [[ "$PALM_BIN" = "" ]]
	483	then
	484	printf "\n +++ environment variable PALM_BIN has not been set"
	485	printf "\n please set it to the directory where the PALM scripts are located"
	486	locat=palm_bin; exit
	487	fi
[180]	488	export PATH=$PALM_BIN:$PATH
[22]	489
	490
[1]	491
	492	# SHELLSCRIPT-OPTIONEN EINLESEN UND KOMMANDO NEU ZUSAMMENSETZEN, FALLS ES
	493	# FUER FOLGEJOBS BENOETIGT WIRD
[807]	494	while getopts :a:AbBc:Cd:D:Fg:G:h:H:i:IkK:m:M:n:o:O:p:P:q:r:R:s:St:T:u:U:vw:xX:yY:z option
[1]	495	do
	496	case $option in
	497	(a) afname=$OPTARG;;
[397]	498	(A) store_on_archive_system=true; mc="$mc -A";;
[1]	499	(b) do_batch=true; mc="$mc -b";;
	500	(B) delete_temporary_catalog=false; mc="$mc -B";;
	501	(c) config_file=$OPTARG; mc="$mc -c$OPTARG";;
[5]	502	(C) restart_run=true; mc="$mc -C";;
[1]	503	(d) fname=$OPTARG; mc="$mc -d$OPTARG";;
[210]	504	(D) cpp_opts="$cpp_opts $OPTARG"; mc="$mc -D'$OPTARG'";;
[79]	505	(F) job_on_file="-D"; mc="$mc -F";;
[122]	506	(g) group_number=$OPTARG; mc="$mc -g$OPTARG";;
[78]	507	(G) global_revision=$OPTARG; mc="$mc -G'$OPTARG'";;
[1]	508	(h) host=$OPTARG; mc="$mc -h$OPTARG";;
	509	(H) fromhost=$OPTARG; mc="$mc -H$OPTARG";;
	510	(i) input_list=$OPTARG; mc="$mc -i'$OPTARG'";;
	511	(I) ignore_archive_error=true; mc="$mc -I";;
	512	(k) keep_data_from_previous_run=true; mc="$mc -k";;
[210]	513	(K) additional_conditions="$OPTARG"; mc="$mc -K'$OPTARG'";;
[1]	514	(m) memory=$OPTARG; mc="$mc -m$OPTARG";;
	515	(M) makefile=$OPTARG; mc="$mc -M$OPTARG";;
	516	(n) node_usage=$OPTARG; mc="$mc -n$OPTARG";;
	517	(o) output_list=$OPTARG; mc="$mc -o'$OPTARG'";;
[622]	518	(O) use_openmp=true; threads_per_task=$OPTARG; mc="$mc -O$OPTARG";;
[1]	519	(p) package_list=$OPTARG; mc="$mc -p'$OPTARG'";;
	520	(P) return_password=$OPTARG; mc="$mc -P$OPTARG";;
	521	(q) queue=$OPTARG; mc="$mc -q$OPTARG";;
	522	(r) run_mode=$OPTARG; mc="$mc -r'$OPTARG'";;
	523	(R) remotecall=true;return_addres=$OPTARG; mc="$mc -R$OPTARG";;
	524	(s) source_list=$OPTARG; mc="$mc -s'$OPTARG'";;
	525	(S) read_from_config=false; mc="$mc -S";;
	526	(t) cpumax=$OPTARG; mc="$mc -t$OPTARG";;
[759]	527	(T) mrun_tasks_per_node=$OPTARG; mc="$mc -T$OPTARG";;
[1]	528	(u) remote_username=$OPTARG; mc="$mc -u$OPTARG";;
	529	(U) return_username=$OPTARG; mc="$mc -U$OPTARG";;
	530	(v) silent=true; mc="$mc -v";;
[759]	531	(w) max_par_io_str=$OPTARG; mc="$mc -w$OPTARG";;
[1]	532	(x) do_trace=true;set -x; mc="$mc -x";;
	533	(X) numprocs=$OPTARG; mc="$mc -X$OPTARG";;
[291]	534	(y) ocean_file_appendix=true; mc="$mc -y";;
[206]	535	(Y) run_coupled_model=true; coupled_dist=$OPTARG; mc="$mc -Y'$OPTARG'";;
[807]	536	(z) check_namelist_files=false; mc="$mc -z";;
[1]	537	(\?) printf "\n +++ unknown option $OPTARG \n"
	538	printf "\n --> type \"$0 ?\" for available options \n"
	539	locat=parameter;exit;;
	540	esac
	541	done
	542
	543
	544	# EVTL. POSITIONSPARAMETER EINLESEN
	545	# ZUR ZEIT GIBT ES NUR DEN PARAMETER ? (=KURZINFO)
	546	shift OPTIND-1
	547
	548
	549	# KURZE AUFRUFBESCHREIBUNG WIRD HIER AUSGEGEBEN
	550	if [[ "$1" = "?" ]]
	551	then
	552	(printf "\n *** mrun can be called as follows:\n"
[291]	553	printf "\n $mrun_script_name -b -c.. -d.. -D.. -f.. -F -h.. -i.. -I -K.. -m.. -o.. -p.. -r.. -R -s.. -t.. -T.. -v -x -X.. -y -Y.. <modus>\n"
[1]	554	printf "\n Description of available options:\n"
	555	printf "\n Option Description Default-Value"
	556	printf "\n -a base name of input files equiv. -d"
[400]	557	printf "\n -A archiving when using file-attribute fl"
[1]	558	printf "\n -b batch-job on local machine ---"
	559	printf "\n -B do not delete temporary directory at end ---"
	560	printf "\n -c configuration file .mrun.config"
	561	printf "\n -d base name of files attached to program test"
	562	printf "\n -D preprocessor(cpp)-directives \"\" "
	563	printf "\n -F create remote job file only ---"
	564	printf "\n -h execution host $localhost_realname"
	565	printf "\n -i INPUT control list \"\" "
	566	printf "\n -I archiving errors of previous batch-jobs"
	567	printf "\n will be ignored"
	568	printf "\n -k keep data from previous run"
	569	printf "\n -K additional conditions for controling"
	570	printf "\n usage of conditional code and"
	571	printf "\n env-variables in configuration file \"\" "
	572	printf "\n -m memory demand in MB (batch-jobs) 0 MB"
	573	printf "\n -M Makefile name Makefile"
	574	printf "\n -n node usage (shared/not_shared) depending on -h"
	575	printf "\n -o OUTPUT control list \"\" "
	576	printf "\n -O use OpenMP ---"
	577	printf "\n -p software package list \"\" "
	578	printf "\n -q queue \"$queue\" "
	579	printf "\n -r run control list (combines -i -o) \"\" "
	580	printf "\n -s filenames of routines to be compiled \"\" "
	581	printf "\n must end with .f, .f90, .F, or .c !"
	582	printf "\n use \"..\" for more than one file and wildcards"
	583	printf "\n -s TEST compiles all files with w-permit"
	584	printf "\n -S config file interpreted by shellscript ---"
	585	printf "\n -t allowed cpu-time in seconds (batch) 0"
	586	printf "\n -T tasks per node depending on -h"
	587	printf "\n -u username on remote machine \"\" "
	588	printf "\n -v no prompt for confirmation ---"
[759]	589	printf "\n -w maximum parallel io streams as given by -X"
[1]	590	printf "\n -x tracing of mrun for debug purposes ---"
	591	printf "\n -X # of processors (on parallel machines) 1"
[291]	592	printf "\n -y add appendix \"_O\" to all local output"
	593	printf "\n files (ocean precursor runs followed by"
	594	printf "\n coupled atmosphere-ocean runs) ---"
[206]	595	printf "\n -Y run coupled model, \"#1 #2\" with"
	596	printf "\n #1 atmosphere and #2 ocean processors \"#/2 #/2\" depending on -X"
[1]	597	printf "\n "
	598	printf "\n Possible values of positional parameter <modus>:"
	599	printf "\n \"?\" - this outline \n\n") \| more
	600	exit
	601	elif [[ "$1" != "" ]]
	602	then
	603	printf "\n +++ positional parameter $1 unknown \n"
	604	locat=parameter; exit
	605	fi
	606
	607
	608
	609	# KURZE STARTMELDUNG
	610	printf "\n*** $version "
	611	printf "\n will be executed. Please wait ..."
	612
	613
[82]	614
	615	# PRUEFEN, OB KONFIGURATIONS-DATEI VORHANDEN
	616	if [[ ! -f $config_file ]]
	617	then
	618	printf "\n\n +++ configuration file: "
	619	printf "\n $config_file"
	620	printf "\n does not exist"
	621	locat=connect; exit
	622	fi
	623
	624
	625	# HOST-IDENTIFIER (local_host) AUS KONFIGURATIONSDATEI BESTIMMEN
	626	line=""
	627	grep "%host_identifier" $config_file > tmp_mrun
	628	while read line
	629	do
[206]	630	if [[ "$line" != "" && $(echo $line \| cut -c1) != "#" ]]
[82]	631	then
	632	HOSTNAME=`echo $line \| cut -d" " -s -f2`
	633	host_identifier=`echo $line \| cut -d" " -s -f3`
[83]	634	if [[ $localhost_realname = $HOSTNAME ]]
[82]	635	then
	636	localhost=$host_identifier
	637	break
	638	fi
	639	fi
	640	done < tmp_mrun
	641
	642	if [[ "$localhost" = "" ]]
	643	then
	644	printf "\n\n +++ no host identifier found in configuration file \"$config_file\""
	645	printf "\n for local host \"$localhost_realname\"."
	646	printf "\n Please add line"
	647	printf "\n \"\%host_identifier $localhost_realname <identifier>\""
	648	printf "\n to the configuration file."
	649	locat=localhost; exit
	650	fi
	651
	652
	653	# HOSTSPEZIFISCHE VARIABLEN SETZEN
	654	case $localhost_realname in
[397]	655	(r1\|r2\|h01\|b01) archive_system=tivoli;;
[82]	656	(cs*) archive_system=ut;;
	657	(fimm.bccs.uib.no) fimm=true;;
	658	(gate\|n-sx) PATH=$PALM_BIN:$PATH:/usr/bin/nqsII;;
	659	(scirocco) scirocco=true;;
	660	esac
	661
	662
	663
[1]	664	# BASISNAME DER INPUT-DATEIEN GLEICH ALLGEMEINEM BASISNAMEN SETZEN,
	665	# WENN NICHT VOM BENUTZER ANDERS BESTIMMT
	666	[[ "$afname" = "" ]] && afname=$fname
	667
	668
	669	# EVTL. RUN-MODUS DEN I/O-LISTEN HINZUFUEGEN
	670	if [[ "$run_mode" != "" ]]
	671	then
	672	input_list="$input_list $run_mode"
	673	output_list="$output_list $run_mode"
	674	fi
	675
	676
	677	# RECHNERNAMEN ABSPEICHERN, VON DEM AUS JOB GESTARTET WIRD,
	678	# ALLERDINGS NUR DANN, WENN NICHT PER OPTION -H BEREITS EIN WERT
	679	# ZUGEWIESEN WURDE (MRUN MACHT DIES IMMER, WENN ES SELBST BATCH-JOBS
	680	# STARTET)
	681	if [[ "$fromhost" = "" ]]
	682	then
	683	fromhost=$localhost
	684	fi
	685
	686
	687	# PRUEFEN, OB MRUN NUR TESTWEISE AUF DER LOKALEN MASCHINE KOMPILIEREN SOLL
	688	if [[ "$source_list" = LOCAL_COMPILE_TEST ]]
	689	then
	690	source_list=TEST
	691	local_compile=true
	692	host=$localhost
	693	fi
	694
	695
	696	# PRUEFEN, OB AUF REMOTE-MASCHINE GERECHNET WERDEN SOLL
	697	# WERT VON do_remote WIRD FUER DATEIVERBINDUNGEN BENOETIGT.
	698	# WENN AUF REMOTE-MASCHINE GERECHNET WIRD, IST GLEICHZEITIG KLAR,
	699	# DASS EIN BATCH-JOB GESTARTET WERDEN MUSS
	700	if [[ -n $host && "$host" != $localhost ]]
	701	then
	702	do_batch=true
	703	do_remote=true
	704	case $host in
[717]	705	(ibm\|ibmh\|ibmkisti\|ibmku\|ibms\|ibmy\|nech\|necriam\|lckyoto\|lcsgib\|lcsgih\|unics\|lcxt4\|lcxt5m\|lck\|lckordi) true;;
[1]	706	(*) printf "\n"
[83]	707	printf "\n +++ sorry: execution of batch jobs on remote host \"$host\""
	708	printf "\n is not available"
[1]	709	locat=nqs; (( iec = 0 )); exit;;
	710	esac
	711	else
	712	host=$localhost
	713	fi
	714
	715
	716	# ZUSATZBEDINGUNGEN (OPTION -K) AUSWERTEN
	717	if [[ -n $additional_conditions ]]
	718	then
[69]	719	# echo $additional_conditions \| cut -d" " -f1-3 \| read cond1 cond2 dummy
	720	cond1=`echo $additional_conditions \| cut -d" " -f1`
	721	cond2=`echo $additional_conditions \| cut -d" " -s -f2`
	722	dummy=`echo $additional_conditions \| cut -d" " -s -f3`
[1]	723	if [[ -n $dummy ]]
	724	then
	725	printf "\n +++ more than 2 additional conditions given for Option \"-K\""
	726	locat=options; exit
	727	fi
[503]	728	block=_$cond1
	729	[[ -n $cond2 ]] && block=${block}_$cond2
[1]	730	fi
	731
	732
[206]	733	# KOPPLUNGSEIGENSCHAFTEN (-Y) AUSWERTEN UND coupled_mode BESTIMMEN
	734	if [[ $run_coupled_model = true ]]
	735	then
	736
	737	if [[ -n $coupled_dist ]]
	738	then
	739
	740	numprocs_atmos=`echo $coupled_dist \| cut -d" " -s -f1`
	741	numprocs_ocean=`echo $coupled_dist \| cut -d" " -s -f2`
	742
	743	if (( $numprocs_ocean + $numprocs_atmos != $numprocs ))
	744	then
	745
	746	printf "\n +++ number of processors does not fit to specification by \"-Y\"."
	747	printf "\n PEs (total) : $numprocs"
	748	printf "\n PEs (atmosphere): $numprocs_atmos"
	749	printf "\n PEs (ocean) : $numprocs_ocean"
	750	locat=coupling; exit
	751
	752	fi
	753
	754	else
	755
	756	(( numprocs_ocean = $numprocs / 2 ))
	757	(( numprocs_atmos = $numprocs / 2 ))
	758
	759	fi
	760	coupled_dist=`echo "$numprocs_atmos $numprocs_ocean"`
	761
	762	# GET coupled_mode FROM THE CONFIG FILE
	763	line=""
	764	grep "%cpp_options.-D__mpi2.$host" $config_file > tmp_mrun
	765	while read line
	766	do
	767	if [[ "$line" != "" && $(echo $line \| cut -c1) != "#" && ( $(echo $line \| cut -d" " -s -f4) = $cond1 \|\| $(echo $line \| cut -d" " -s -f4) = $cond2 ) ]]
	768	then
	769	coupled_mode="mpi2"
	770	fi
	771	done < tmp_mrun
	772
	773	fi
	774
	775
[1]	776	# PRUEFEN, OB EVTL. BEI VORHERGEHENDEM LAUF (KETTENJOB) EINE
	777	# ARCHIVIERUNG FEHLGESCHLAGEN IST
	778	if [[ -f ~/job_queue/ARCHIVE_ERROR_$fname ]]
	779	then
	780	if [[ $ignore_archive_error = false ]]
	781	then
	782	printf "\n +++ data archiving of previous run failed"
	783	printf "\n see directory \~/job_queue on remote machine"
	784	locat=archive; exit
	785	else
	786	printf "\n +++ warning: data archiving in a previous run failed"
	787	printf "\n MRUN continues, trying to get backup copy"
	788	fi
	789	fi
	790
	791
	792
	793	# LESEN UND INTERPRETIEREN DER KONFIGURATIONS-DATEI VOM SHELLSCRIPT AUS
	794	# VORUEBERGEHEND ZWINGEND AUF LINUX-RECHNERN
	795	if [[ "$read_from_config" = false ]]
	796	then
	797
	798	[[ $silent = false ]] && printf "\n Reading the configuration file... "
	799	while read zeile
	800	do
	801	[[ $silent = false ]] && printf "."
	802
	803
	804	# ZUERST EVENTUELL VORKOMMENDE ENVIRONMENT-VARIABLEN DURCH IHRE WERTE
	805	# ERSETZEN
	806	eval zeile=\"$zeile\"
	807
	808
	809	# INTERPRETATION DER ZEILE
[21]	810	if [[ "$(echo $zeile)" = "" ]]
[1]	811	then
[21]	812	# LEERZEILE, KEINE AKTION
	813	continue
[1]	814
[21]	815	elif [[ "$(echo $zeile \| cut -c1)" = "#" ]]
	816	then
	817
[1]	818	# ZEILE IST KOMMENTARZEILE
	819	true
	820
	821	elif [[ "$(echo $zeile \| cut -c1)" = "%" ]]
	822	then
	823
	824	# ZEILE DEFINIERT ENVIRONMENT-VARIABLE
	825	zeile=$(echo $zeile \| cut -c2-)
[69]	826	# echo $zeile \| cut -d" " -f1-5 \| read var value for_host for_cond1 for_cond2
	827	var=`echo $zeile \| cut -d" " -f1`
	828	value=`echo $zeile \| cut -d" " -s -f2`
	829	for_host=`echo $zeile \| cut -d" " -s -f3`
	830	for_cond1=`echo $zeile \| cut -d" " -s -f4`
	831	for_cond2=`echo $zeile \| cut -d" " -s -f5`
[1]	832
	833	if [[ "$for_host" = "" \|\| ( "$for_host" = $host && "$for_cond1" = "$cond1" && "$for_cond2" = "$cond2" ) \|\| $(echo "$input_list$output_list"\|grep -c "$for_host") != 0 ]]
	834	then
	835
	836	# BEI COMPILER- CPP- ODER LINKEROPTIONEN EVTL ":" DURCH " "
	837	# ERSETZEN. "::" WIRD DURCH ":" ERSETZT.
	838	value=`echo $value \| sed 's/::/%DUM%/g' \| sed 's/:/ /g' \| sed 's/%DUM%/:/g'`
	839
	840
	841	# ENVIRONMENT-VARIABLE WIRD WERT AUS KONFIGURATIONSDATEI
	842	# ZUGEWIESEN, WENN SIE SELBST NOCH KEINEN WERT UEBER DIE
	843	# ENTSPRECHENDE SCRIPT-OPTION ERHALTEN HAT. SOLCHE
	844	# VARIABLEN HAETTEN DANN DEN WERT "" ODER IM INTEGER-FALL DEN
	845	# WERT 0. ALLGEMEINE REGEL ALSO: SCRIPT-OPTION GEHT UEBER
	846	# KONFIGURATIONSDATEI
	847	if [[ "$(eval echo \$$var)" = "" \|\| "$(eval echo \$$var)" = "0" ]]
	848	then
	849	eval $var=\$value
	850
	851	# EVTL. BILDSCHIRMAUSGABEN DER ENVIRONMENT-VARIABLEN
	852	if [[ $do_trace = true ]]
	853	then
	854	printf "\n*** ENVIRONMENT-VARIABLE $var = $value"
	855	fi
	856	fi
	857
	858	# WENN ENVIRONMENT-VARIABLE HOST VEREINBART, DANN SOFORT AUSWERTEN
	859	# WERT VON do-remote WIRD BEI DATEIVERBINDUNGEN BENOETIGT
	860	# WENN AUF REMOTE-MASCHINE GERECHNET WIRD, IST GLEICHZEITIG KLAR,
	861	# DASS EIN BATCH-JOB GESTARTET WERDEN MUSS
	862	if [[ $var = host ]]
	863	then
	864	if [[ -n $host && "$host" != $localhost ]]
	865	then
	866	do_batch=true
	867	do_remote=true
	868	case $host in
[717]	869	(ibm\|ibmh\|ibmkisti\|ibmku\|ibms\|ibmy\|lckyoto\|lcsgib\|lcsgih\|nech\|necriam\|unics\|lcxt4\|lcxt5m\|lck\|lckordi) true;;
[83]	870	(*) printf "\n +++ sorry: execution of batch jobs on remote host \"$host\""
	871	printf "\n is not available"
[1]	872	locat=nqs; exit;;
	873	esac
	874	else
	875	host=$localhost
	876	fi
	877	fi
	878
	879	# VOM BENUTZER DEFINIERTE ENVIRONMENT VARIABLEN MUESSEN PRINZIPIELL
	880	# EXPORTIERT WERDEN, DA SIE VIELLEICHT IN WEITER UNTEN AUFZURUFEN-
	881	# DEN PROGRAMMEN BENOETIGT WERDEN
	882	export $var
	883	fi
	884
	885
	886
	887
	888	elif [[ "$(echo $zeile \| cut -c1-3)" = "EC:" ]]
	889	then
	890
	891	# ZEILE DEFINIERT ERROR-KOMMANDO
	892	(( iec = iec + 1 ))
	893	zeile=$(echo $zeile \| cut -c4-)
	894	err_command[$iec]="$zeile"
	895
	896	elif [[ "$(echo $zeile \| cut -c1-3)" = "IC:" ]]
	897	then
	898
	899	# ZEILE DEFINIERT INPUT-KOMMANDO
	900	(( iic = iic + 1 ))
	901	zeile=$(echo $zeile \| cut -c4-)
	902	in_command[$iic]="$zeile"
	903
	904	elif [[ "$(echo $zeile \| cut -c1-3)" = "OC:" ]]
	905	then
	906
	907	# ZEILE DEFINIERT OUTPUT-KOMMANDO
	908	(( ioc = ioc + 1 ))
	909	zeile=$(echo $zeile \| cut -c4-)
	910	out_command[$ioc]="$zeile"
	911
	912	else
	913
	914	# ZEILE DEFINIERT DATEIVERBINDUNG. EINLESEN DER DATEIEIGENSCHAFTEN
	915	# s2a: in/out - Feld
	916	# s2b: loc - Feld (optional)
	917	# s2c: tr/ar - Feld (optional)
[69]	918	# echo $zeile \| cut -d" " -f1-2 \| read s1 s2
	919	s1=`echo $zeile \| cut -d" " -f1`
	920	s2=`echo $zeile \| cut -d" " -s -f2`
[1]	921	s2a=$(echo $s2 \| cut -d":" -f1)
	922	if [[ $(echo $s2 \| grep -c ":") = 0 ]]
	923	then
	924	s2b=""
	925	s2c=""
	926	else
[69]	927	# echo $s2 \| cut -d":" -f2-3 \| sed 's/:/ /g' \| read s2b s2c
	928	s2b=`echo $s2 \| cut -d":" -f2 \| sed 's/:/ /g'`
	929	s2c=`echo $s2 \| cut -d":" -s -f3 \| sed 's/:/ /g'`
[1]	930	fi
[69]	931	# echo $zeile \| cut -d" " -f3-6 \| read s3 s4 s5 s6
	932	s3=`echo $zeile \| cut -d" " -f3`
	933	s4=`echo $zeile \| cut -d" " -s -f4`
	934	s5=`echo $zeile \| cut -d" " -s -f5`
	935	s6=`echo $zeile \| cut -d" " -s -f6`
[1]	936
	937
	938	# ABSPEICHERN DER DATEIVERBINDUNG, FALLS IN INPUT- ODER OUTPUT-LIST
	939	# VERMERKT. VARIABLE S3 KANN AUCH LISTE ENTHALTEN (FELDTRENNER ":")
	940	# DATEIVERBINDUNG WIRD DANN NICHT ABGESPEICHERT UND GEPRUEFT, WENN
	941	# PROGRAMMLAUF AUF REMOTE-MASCHINE ERFOLGT UND DATEI NUR LOKAL VOR-
	942	# HANDEN SEIN MUSS (D.H. s2b = loc)
	943	IFSALT="$IFS"; IFS="$IFS:"
	944	if [[ "$s2a" = in && ! ( $do_remote = true && ( "$s2b" = loc \|\| "$s2b" = locopt ) ) ]]
	945	then
	946	found=false
	947	for actual in $input_list
	948	do
	949	for formal in $s3
	950	do
	951	[[ $actual = $formal \|\| "$formal" = "-" ]] && found=true
	952	done
	953	done
	954	if [[ $found = true ]]
	955	then
	956	(( iin = iin + 1 ))
	957	localin[$iin]=$s1; transin[$iin]=$s2b; actionin[$iin]=$s2c;
	958	typein[$iin]=$s3; pathin[$iin]=$s4; endin[$iin]=$s5;
	959	extin[$iin]=$s6
	960	fi
	961	elif [[ "$s2a" = out && ! ( $do_remote = true && "$s2b" = loc ) ]]
	962	then
	963	found=false
	964	for actual in $output_list
	965	do
	966	for formal in $s3
	967	do
	968	[[ $actual = $formal \|\| "$formal" = "-" ]] && found=true
	969	done
	970	done
	971	if [[ $found = true ]]
	972	then
	973	(( iout = iout + 1 ))
	974	localout[$iout]=$s1; actionout[$iout]=$s2c; typeout[$iout]=$s3;
	975	pathout[$iout]=$s4; endout[$iout]=$s5; extout[$iout]=$s6
	976	fi
	977	elif [[ "$s2a" != in && "$s2a" != out ]]
	978	then
	979	printf "\n +++ I/O-attribute in configuration file $config_file has the invalid"
	980	printf "\n value \"$s2\". Only \"in\" and \"out\" are allowed!"
	981	locat=connect; exit
	982	fi
	983	IFS="$IFSALT"
	984	fi
	985	done < $config_file
	986
	987	else
	988
	989
	990	# INTERPRETATION DER KONFIGURATIONSDATEI MITTELS FORTRAN 90 - PROGRAMM
	991	[[ $silent = false ]] && printf "..."
	992	export cond1 cond2 config_file do_remote do_trace input_list localhost output_list
	993	export interpreted_config_file=.icf.$RANDOM
	994
	995
[69]	996
	997	# ENVIRONMENT-VARIABLEN FUER INTERPRET_CONFIG UEBER NAMELIST_DATEI ZUR
	998	# VERFUEGUNG STELLEN
	999	cat > .mrun_environment << %%END%%
	1000	&mrun_environment cond1 = '$cond1', cond2 = '$cond2',
	1001	config_file = '$config_file', do_remote = '$do_remote',
	1002	do_trace = '$do_trace', host = '$host',
	1003	input_list = '$input_list', icf = '$interpreted_config_file',
	1004	localhost = '$localhost', output_list = '$output_list' /
	1005
	1006	%%END%%
	1007
	1008	# WERTE VON MRUN-OPTIONEN SICHERN UND DAMIT GEGEBENENFALLS SPAETER DIE
	1009	# IN DER KONFIGURAIONSDATEI ANGEGEBENEN WERTE UEBERSTEUERN
	1010	mrun_memory=$memory
[127]	1011	mrun_group_number=$group_number
[69]	1012	mrun_cpumax=$cpumax
	1013	mrun_numprocs=$numprocs
	1014
[251]	1015	if [[ $localhost_realname = "sx-fep" ]]
[66]	1016	then
[251]	1017	/home/COAR/NC/raasch/pub/interpret_config_necriam.x
[66]	1018	else
[503]	1019
	1020	if [[ "$host" != $localhost ]]
	1021	then
	1022
	1023	# REMOTE JOB FROM LOCAL HOST: JUST TAKE THE FIRST EXECUTABLE FOUND
	1024	interpret_config_executable=`ls -1 ${PALM_BIN}/interpret_config*.x 2>/dev/null`
	1025	if [[ $? != 0 ]]
	1026	then
	1027	printf "\n\n +++ no interpret_config found"
	1028	printf "\n run \"mbuild -u -h ...\" to generate utilities for this host"
	1029	locat=interpret_config; exit
	1030	fi
	1031	interpret_config_executable=`echo $interpret_config_executable \| cut -d" " -f1`
	1032	$interpret_config_executable
	1033
	1034	else
	1035
	1036	# CHECK, IF THERE IS AN EXECUTABLE FOR THE BLOCK
	1037	if [[ ! -f ${PALM_BIN}/interpret_config${block}.x ]]
	1038	then
	1039	printf "\n\n +++ no interpret_config found for given block \"$cond1 $cond2\""
	1040	printf "\n run \"mbuild -u -h ...\" to generate utilities for this block"
	1041	locat=interpret_config; exit
	1042	else
	1043	interpret_config${block}.x
	1044	fi
	1045
	1046	fi
[66]	1047	fi
[71]	1048	rm .mrun_environment
[1]	1049
[71]	1050
[1]	1051	# AUSFUEHRUNG DER GENERIERTEN SHELL-KOMMANDOS IN DIESER SHELL
	1052	chmod u+x $interpreted_config_file
	1053	export PATH=$PATH:.
	1054	. $interpreted_config_file
[71]	1055	rm $interpreted_config_file
[1]	1056
[71]	1057
[69]	1058	# OPTIONSWERTE UEBERSTEUERN KONFIGURATIONSDATEI
[759]	1059	[[ $mrun_memory != 0 ]] && memory=$mrun_memory
[127]	1060	[[ "$mrun_group_number" != "none" ]] && group_number=$mrun_group_number
[759]	1061	[[ $mrun_cpumax != 0 ]] && cpumax=$mrun_cpumax
	1062	[[ "$mrun_numprocs" != "" ]] && numprocs=$mrun_numprocs
	1063	[[ "$max_par_io_str" != "" ]] && maximum_parallel_io_streams=$max_par_io_str
	1064	[[ "$mrun_tasks_per_node" != "" ]] && tasks_per_node=$mrun_tasks_per_node
[69]	1065
[1]	1066	fi
	1067
	1068
	1069	# QUELLTEXTVERZEICHNIS AUF LOKALER MASCHINE AUS KONFIGURATIONSDATEI
	1070	# BESTIMMEN (WUERDE SONST EVTL. DAS VERZEICHNIS DES JEWEILS UNTER -h
	1071	# ANGEGEBENEN REMOTE-RECHNERS SEIN)
	1072	# BEI BATCH-JOBS SIND DIE ZU UEBERSETZENDEN PROGRAMMTEILE SCHON KOMPLETT
	1073	if [[ "$SOURCES_COMPLETE" = "" ]]
	1074	then
	1075
	1076	# ZUERST PRUEFEN, OB EIN GLOBALER QUELLTEXTPFAD FUER ALLE RECHNER
	1077	# VEREINBART WURDE
	1078	source_path=""
	1079	line=""
	1080	grep "%source_path" $config_file > tmp_mrun
	1081	while read line
	1082	do
	1083	if [[ "$line" != "" && $(echo $line \| cut -c1) != "#" ]]
	1084	then
	1085	if [[ "$(echo $line \| cut -d" " -f3)" = "" ]]
	1086	then
	1087	global_source_path=`echo $line \| cut -d" " -f2`
	1088	fi
	1089	fi
	1090	done < tmp_mrun
	1091
	1092	line=""
	1093	found=false
	1094	grep " $localhost" $config_file \| grep "%source_path" > tmp_mrun
	1095	while read line
	1096	do
	1097	if [[ "$line" != "" && $(echo $line \| cut -c1) != "#" ]]
	1098	then
	1099	if [[ $found = true ]]
	1100	then
	1101	printf "\n\n +++ more than one source path found in configuration file"
	1102	printf "\n for local host \"$localhost\" "
	1103	locat=source_path; exit
	1104	fi
	1105	source_path=`echo $line \| cut -d" " -f2`
	1106	found=true
	1107	fi
	1108	done < tmp_mrun
	1109	rm tmp_mrun
	1110
	1111	if [[ "$source_path" = "" ]]
	1112	then
	1113	if [[ "$global_source_path" != "" ]]
	1114	then
	1115	source_path=$global_source_path
	1116	else
	1117	printf "\n\n +++ no source path found in configuration file"
	1118	printf "\n for local host \"$localhost\" "
	1119	locat=source_path; exit
	1120	fi
	1121	fi
	1122	eval source_path=$source_path
	1123
	1124	if [[ ! -d $source_path ]]
	1125	then
	1126	printf "\n\n +++ source path \"$source_path\" on local host"
	1127	printf "\n \"$localhost\" does not exist"
	1128	locat=source_path; exit
	1129	fi
	1130
	1131	fi
	1132
	1133
[78]	1134	# GLOBALE REVISIONSNUMMER ERMITTELN (FORTSETZUNGSLAEUFEN WIRD DIESE
	1135	# DURCH OPTION -G MITGETEILT)
[693]	1136	if [[ "$global_revision" = "" && $host != "ibmkisti" ]]
[78]	1137	then
	1138	global_revision=`svnversion $source_path 2>/dev/null`
	1139	global_revision="Rev: $global_revision"
	1140	fi
[1]	1141
[78]	1142
[1]	1143	# NOCHMAL PRUEFEN, OB AUF REMOTE-MASCHINE GERECHNET WERDEN SOLL
	1144	# (HOST KANN IN KONFIGURATIONSDATEI ANDERS FESTGELEGT WORDEN SEIN)
	1145	# WERT VON do_remote WIRD FUER DATEIVERBINDUNGEN BENOETIGT.
	1146	# WENN AUF REMOTE-MASCHINE GERECHNET WIRD, IST GLEICHZEITIG KLAR,
	1147	# DASS EIN BATCH-JOB GESTARTET WERDEN MUSS
	1148	if [[ -n $host && "$host" != $localhost ]]
	1149	then
	1150	do_batch=true
	1151	do_remote=true
	1152	case $host in
[717]	1153	(ibm\|ibmh\|ibmkisti\|ibmku\|ibms\|ibmy\|lckyoto\|lcsgib\|lcsgih\|nech\|necriam\|unics\|lcxt4\|lcxt5m\|lck\|lckordi) true;;
[1]	1154	(*) printf "\n"
[83]	1155	printf "\n +++ sorry: execution of batch jobs on remote host \"$host\""
	1156	printf "\n is not available"
[1]	1157	locat=nqs; (( iec = 0 )); exit;;
	1158	esac
	1159	else
	1160	host=$localhost
	1161	fi
	1162
	1163
	1164	# PRUEFUNG EINIGER PROZESSORZAHLANGABEN BEI RECHNUNGEN AUF PARALLELRECHNERN
	1165	if [[ "$cond1" = parallel \|\| "$cond2" = parallel ]]
	1166	then
	1167
	1168	# PRUEFEN, OB DIE ANZAHL DER ZU VERWENDENDEN PES ANGEGEBEN WURDE
	1169	if [[ ! -n $numprocs ]]
	1170	then
	1171	printf "\n"
	1172	printf "\n +++ option \"-K parallel\" requires additional specification"
	1173	printf "\n of the number of processors to be used by"
	1174	printf "\n mrun-option \"-X\" or by environment-variable"
	1175	printf "\n \"numprocs\" in the configuration file"
	1176	locat=numprocs; (( iec = 0 )); exit
	1177	fi
	1178
	1179	# PRUEFEN, OB DIE PROZESSORANZAHL PRO KNOTEN ANGEGEBEN WURDE (GGF.
	1180	# DEFAULT-WERT SETZEN) UND OB SIE EIN GANZZAHLIGER TEILER DER
	1181	# GESAMTPROZESSORANZAHL IST
[759]	1182	if [[ "$tasks_per_node" = "" ]]
[1]	1183	then
[759]	1184	printf "\n"
	1185	printf "\n +++ option \"-T\" (tasks per node) is missing"
	1186	printf "\n set -T option or define tasks_per_node in the config file"
	1187	locat=tasks_per_node; (( iec = 0 )); exit
	1188	fi
	1189	(( ival = $tasks_per_node ))
	1190	(( pes = numprocs ))
[1]	1191	# if [[ $(echo $package_list \| grep -c dvrp_graphics+1PE) = 1 ]]
	1192	# then
	1193	# (( pes = pes - 1 ))
	1194	# fi
[759]	1195	(( ii = pes / ival ))
	1196	if (( pes - ii * ival > 0 ))
	1197	then
	1198	printf "\n"
	1199	printf "\n +++ tasks per node (option \"-T\") must be an integral"
	1200	printf "\n divisor of the total number of processors (option \"-X\")"
	1201	printf "\n values of this mrun-call: \"-T $tasks_per_node\" \"-X $numprocs\""
	1202	locat=tasks_per_node; (( iec = 0 )); exit
[1]	1203	fi
	1204
[759]	1205
[1]	1206	# IBMY HAT NUR EINEN KNOTEN
	1207	if [[ $host = ibmy ]]
	1208	then
	1209	if [[ "$tasks_per_node" != "" && "$tasks_per_node" != "$numprocs" ]]
	1210	then
	1211	printf "\n"
	1212	printf "\n +++ on ibmy, tasks per node (option \"-T\") must be equal to the"
	1213	printf "\n total number of processors (option \"-X\")"
	1214	printf "\n values of this mrun-call: \"-T $tasks_per_node\" \"-X $numprocs\""
	1215	locat=tasks_per_node; (( iec = 0 )); exit
	1216	fi
	1217	fi
	1218
[377]	1219
	1220	# SETTINGS FOR SUBJOB-COMMAND
[622]	1221	TOPT="-T $tasks_per_node"
[1]	1222	OOPT="-O $threads_per_task"
	1223
	1224	# GESAMTZAHL DER KNOTEN BESTIMMEN
	1225	if [[ "$tasks_per_node" != "" ]]
	1226	then
	1227	(( nodes = numprocs / ( tasks_per_node * threads_per_task ) ))
	1228	fi
	1229
	1230	# PRUEFEN, OB NODE USAGE EINEN ERLAUBTEN WERT HAT BZW. DEN DEFAULT
	1231	# SETZEN
	1232	if [[ $node_usage = default ]]
	1233	then
	1234	if [[ $host = ibms ]]
	1235	then
	1236	node_usage=shared
[181]	1237	elif [[ $(echo $host \| cut -c1-5) = lcsgi ]]
	1238	then
	1239	node_usage=singlejob
[1]	1240	else
	1241	node_usage=not_shared
	1242	fi
	1243	fi
[181]	1244
[635]	1245	if [[ $node_usage != shared && $node_usage != not_shared && $node_usage != singlejob && "$(echo $node_usage \| cut -c1-3)" != "sla" ]]
[1]	1246	then
	1247	printf "\n"
	1248	printf "\n +++ node usage (option \"-n\") is only allowed to be set"
	1249	printf "\n \"shared\" or \"not_shared\""
	1250	locat=tasks_per_node; (( iec = 0 )); exit
	1251	fi
	1252
	1253	fi
	1254
	1255	# PRUEFEN, OB HOSTFILE EXISTIERT
	1256	if [[ -n $hostfile ]]
	1257	then
	1258	if [[ ! -f $hostfile ]]
	1259	then
	1260	printf "\n"
	1261	printf "\n +++ hostfile \"$hostfile\" does not exist"
	1262	locat=hostfile; exit
	1263	fi
	1264	fi
	1265
	1266	# PRUEFEN, OB RHOSTS DATEI EXISTIERT. GEGEBENENFALLS ANLEGEN BZW. ERWEITERN
	1267	if [[ $host = ibmy && $do_remote = false ]]
	1268	then
	1269	if [[ ! -f $HOME/.rhosts ]]
	1270	then
	1271	echo "gfdl5.yonsei.ac.kr" > $HOME/.rhosts
	1272	printf "\n\n *** file:"
	1273	printf "\n $HOME/.rhosts"
	1274	printf "\n was created\n"
	1275	fi
	1276	if [[ $(grep -c gfdl5.yonsei.ac.kr $HOME/.rhosts) = 0 ]]
	1277	then
	1278	echo "gfdl5.yonsei.ac.kr" >> $HOME/.rhosts
	1279	printf "\n\n *** file:"
	1280	printf "\n $HOME/.rhosts"
	1281	printf "\n was extended by the name of the current host\n"
	1282	fi
	1283	fi
	1284
	1285
[759]	1286	# Set default value for the maximum number of parallel io streams
	1287	if [[ "$maximum_parallel_io_streams" = "" ]]
	1288	then
	1289	maximum_parallel_io_streams=$numprocs
	1290	fi
[1]	1291
[759]	1292
[1]	1293	# FALLS NICHT VORGEGEBEN, DEFAULT-QUEUE AUF DER ZIELMASCHINE FESTLEGEN
	1294	if [[ $queue = none ]]
	1295	then
	1296	case $host in
[312]	1297	(ibmh) queue=no_class;;
[693]	1298	(ibmkisti) queue=class.32plus;;
[1]	1299	(ibmy) queue=parallel;;
[440]	1300	(lckyoto) queue=ph;;
[821]	1301	(lcsgib) queue=testq;;
	1302	(lcsgih) queue=testq;;
[635]	1303	(lctit) queue=S;;
[1]	1304	(nech) queue=none;;
[253]	1305	(necriam) queue=SP;;
[1]	1306	(unics) queue=unics;;
	1307	esac
	1308	fi
	1309
	1310
	1311	# VOLLSTAENDIGE DATEINAMEN DER INPUT-FILES BILDEN,
	1312	# INPUT-DATEIEN AUF VORHANDENSEIN PRUEFEN UND EVTL. HOECHSTE ZYKLUSNUMMER
	1313	# ERMITTELN
	1314	(( i = 0 ))
	1315	while (( i < iin ))
	1316	do
	1317	(( i = i + 1 ))
	1318	(( maxcycle = 0 ))
	1319
	1320	# NAMENSBILDUNG (EVTL. IST FESTER DATEINAME VORGEGEBEN)
	1321	if [[ "${actionin[$i]}" = di ]]
	1322	then
	1323	remotepathin[$i]=${pathin[$i]}/${endin[$i]} # REMOTE-PFAD ERST AUF
	1324	# REM-MASCHINE AUSWERTEN
	1325	eval filename=${pathin[$i]}/${endin[$i]}
	1326	else
	1327	remotepathin[$i]=${pathin[$i]}/${afname}${endin[$i]} # REMOTE-PFAD ERST AUF
	1328	# REM-MASCHINE AUSWERTEN
	1329	eval filename=${pathin[$i]}/${afname}${endin[$i]}
	1330	fi
	1331
	1332	# PRUEFEN AUF VORHANDENSEIN
	1333	if [[ $(ls $filename* 2>&1 \| grep -c "not found") = 1 \|\| \
[746]	1334	$(ls $filename* 2>&1 \| grep -c "nicht gefunden") = 1 \|\| \
[1]	1335	$(ls $filename* 2>&1 \| grep -c "No such file") = 1 \|\| \
	1336	$(ls $filename* 2>&1 \| grep -c "does not exist") = 1 ]]
	1337	then
	1338	# DATEIEN MIT EXTENSION (Z.B. NC) MUESSEN NICHT ZWINGEND VORHANDEN
	1339	# SEIN, DESHALB IN SOLCHEN FAELLEN KEIN ABBRUCH. DIES IST NUR EINE
	1340	# VORUEBERGEHENDE LOESUNG (OKT 05). NICHT ZWINGEND BENOETIGTE
	1341	# EINGABEDATEIEN SOLLTEN EINE SPEZIELLE OPTION IN DER DATEI-
	1342	# VERBINDUNGSANWEISUNG BEKOMMEN (Z.B. inopt?)
	1343	if [[ "${transin[$i]}" != "locopt" ]]
	1344	then
	1345	printf "\n\n +++ INPUT-file: "
	1346	if [[ "${extin[$i]}" = "" ]]
	1347	then
	1348	printf "\n $filename"
	1349	else
	1350	printf "\n $filename.${extin[$i]}"
	1351	fi
	1352	printf "\n does not exist\n"
	1353	locat=input; exit
	1354	else
	1355	transin[$i]="unavailable"
	1356	fi
	1357	else
	1358
	1359	# ZYKLUSNUMMER FESTSTELLEN
	1360	ls -1 -d $filename > filelist 2>/dev/null
	1361	ls -1 -d $filename.* >> filelist 2>/dev/null
	1362	while read zeile
	1363	do
	1364	cycle=$(print $zeile \| cut -f2 -d".")
	1365	if [[ "$cycle" = "$zeile" ]]
	1366	then
	1367	(( icycle = 0 ))
	1368	elif [[ "$cycle" = "${extin[$i]}" ]]
	1369	then
	1370	(( icycle = 0 ))
	1371	else
	1372	(( icycle = $cycle ))
	1373	fi
	1374	if (( icycle > maxcycle ))
	1375	then
	1376	(( maxcycle = icycle ))
	1377	file_to_be_used=$zeile
	1378	fi
	1379	done <filelist
	1380	rm filelist
	1381
	1382	# EVTL. ZYKLUSNUMMER AN DATEINAMEN ANFUEGEN
	1383	if (( maxcycle > 0 ))
	1384	then
	1385	if [[ "${extin[$i]}" != " " && "${extin[$i]}" != "" ]]
	1386	then
	1387	filename=${filename}.$maxcycle.${extin[$i]}
	1388	else
	1389	filename=${filename}.$maxcycle
	1390	fi
	1391	else
	1392	if [[ "${extin[$i]}" != " " && "${extin[$i]}" != "" ]]
	1393	then
	1394	filename=${filename}.${extin[$i]}
	1395	fi
	1396	fi
	1397
	1398	# DATEINAMEN OHNE PFAD ABER MIT ZYKLUSNUMMER ABSPEICHERN,
	1399	# DA SPAETER BEI RUECKSPEICHERN VOM ARCHIVSYSTEM EVTL. BENOETIGT
	1400	absnamein[$i]=$filename
	1401	if (( maxcycle > 0 ))
	1402	then
	1403	if [[ "${actionin[$i]}" = di ]]
	1404	then
	1405	frelin[$i]=${endin[$i]}.$maxcycle
	1406	else
	1407	frelin[$i]=${afname}${endin[$i]}.$maxcycle
	1408	fi
	1409	else
	1410	if [[ "${actionin[$i]}" = di ]]
	1411	then
	1412	frelin[$i]=${endin[$i]}
	1413	else
	1414	frelin[$i]=${afname}${endin[$i]}
	1415	fi
	1416	fi
	1417
	1418	fi
	1419	done
	1420
	1421
	1422	# VOLLSTAENDIGE DATEINAMEN (OHNE $ ODER ~) DER OUTPUT-FILES BILDEN,
	1423	# OUTPUT-DATEIEN AUF VORHANDENSEIN PRUEFEN UND EVTL. HOECHSTE ZYKLUSNUMMER
	1424	# ERMITTELN ODER, FALLS NICHT VORHANDEN, PRUEFEN, OB SIE SICH ANLEGEN LASSEN
	1425	# DIESE AKTIONEN FINDEN NICHT STATT, WENN PROGRAMM AUF REMOTE-MASCHINE
	1426	# RECHNET UND DATEI ANSCHLIESSEND TRANSFERIERT WERDEN SOLL!
	1427	(( i = 0 ))
	1428	while (( i < iout ))
	1429	do
	1430	(( i = i + 1 ))
	1431	if [[ ! ( $fromhost != $localhost && ( "${actionout[$i]}" = tr \|\| "${actionout[$i]}" = tra \|\| "${actionout[$i]}" = trpe ) ) ]]
	1432	then
[260]	1433	if [[ "${actionout[$i]}" = tr ]]
[1]	1434	then
	1435	actionout[$i]=""
[260]	1436	elif [[ "${actionout[$i]}" = trpe ]]
	1437	then
	1438	actionout[$i]=pe
[1]	1439	elif [[ "${actionout[$i]}" = tra ]]
	1440	then
	1441	actionout[$i]=a
	1442	fi
	1443	(( maxcycle = 0 ))
	1444	eval filename=${pathout[$i]}/${fname}${endout[$i]}
	1445	eval catalogname=${pathout[$i]}
	1446	if [[ $(ls $filename* 2>&1 \| grep -c "not found") = 1 \|\| \
[746]	1447	$(ls $filename* 2>&1 \| grep -c "nicht gefunden") = 1 \|\| \
[1]	1448	$(ls $filename* 2>&1 \| grep -c "No such file") = 1 \|\| \
	1449	$(ls $filename* 2>&1 \| grep -c "does not exist") = 1 ]]
	1450	then
	1451
	1452	# OUTPUT-DATEI NICHT VORHANDEN. PRUEFEN, OB ANLEGEN MOEGLICH.
	1453	if cat /dev/null > $filename
	1454	then
	1455	rm $filename
	1456	else
	1457
	1458	# PRUEFEN, OB KATALOG VORHANDEN UND EVTL. DIESEN ANLEGEN
	1459	if [[ ! -d $catalogname ]]
	1460	then
	1461	if mkdir -p $catalogname
	1462	then
	1463	printf "\n\n *** directory:"
	1464	printf "\n $catalogname"
	1465	printf "\n was created\n"
	1466	else
	1467	printf "\n\n +++ OUTPUT-file:"
	1468	printf "\n $filename"
	1469	printf "\n cannot be created, because directory does not exist"
	1470	printf "\n and cannot be created either"
	1471	printf "\n"
	1472	locat=output ; exit
	1473	fi 2>/dev/null
	1474	else
	1475	printf "\n\n +++ OUTPUT-file:"
	1476	printf "\n $filename"
	1477	printf "\n cannot be created, although directory exists"
	1478	printf "\n"
	1479	locat=output ; exit
	1480	fi
	1481	fi 2>/dev/null
	1482	else
	1483
	1484	# ZYKLUSNUMMER FESTSTELLEN
	1485	ls -1 -d $filename > filelist 2>/dev/null
	1486	ls -1 -d $filename.* >> filelist 2>/dev/null
	1487	while read zeile
	1488	do
	1489	cycle=$(print $zeile \| cut -f2 -d".")
	1490	if [[ "$cycle" = "$zeile" \|\| "$cycle" = ${extout[$i]} ]]
	1491	then
	1492	(( icycle = 1 ))
	1493	else
	1494	(( icycle = $cycle + 1 ))
	1495	fi
	1496	if (( icycle > maxcycle ))
	1497	then
	1498	(( maxcycle = icycle ))
	1499	fi
	1500	done <filelist
	1501	rm filelist
	1502	fi
	1503
	1504	# EVTL. ZYKLUSNUMMER AN DATEINAMEN ANFUEGEN UND PRUEFEN, OB SICH
	1505	# DATEI AUCH WIRKLICH ANLEGEN LAESST. IM APPEND-FALL MUSS AN DEN
	1506	# HOECHSTEN VORHANDENEN ZYKLUS-ANGEHAENGT WERDEN. ANLAGEPRUEFUNG
	1507	# IST DANN NATUERLICH NICHT ZULAESSIG BZW. MOEGLICH
	1508	if [[ "${actionout[$i]}" != a ]]
	1509	then
	1510	if (( maxcycle > 0 ))
	1511	then
	1512	filename=${filename}.$maxcycle
	1513	if cat /dev/null > $filename
	1514	then
	1515	rm $filename
	1516	else
	1517	printf "\n +++ OUTPUT-file:"
	1518	printf "\n $filename"
	1519	printf "\n cannot be created"
	1520	locat=output ; exit
	1521	fi
	1522	fi
	1523	else
	1524	(( maxcycle = maxcycle - 1 ))
	1525	if (( maxcycle > 0 ))
	1526	then
	1527	filename=${filename}.$maxcycle
	1528	fi
	1529	fi
	1530
	1531	# DATEINAMEN OHNE PFAD ABER MIT ZYKLUSNUMMER ABSPEICHERN,
	1532	# DA SPAETER BEI ABLAGE AUF ARCHIVSYSTEM BZW. FUER
	1533	# DATEI OUTPUT_FILE_CONNECTIONS EVTL. BENOETIGT
	1534	pathout[$i]=$filename
	1535	if (( maxcycle > 0 ))
	1536	then
	1537	frelout[$i]=${fname}${endout[$i]}.$maxcycle
	1538	else
	1539	frelout[$i]=${fname}${endout[$i]}
	1540	fi
	1541
	1542	fi
	1543	done
	1544
	1545
[260]	1546	# DAS DVR-PAKET ERFORDERT EINE ENTSPRECHENDE BIBLIOTHEK
[83]	1547	if [[ $(echo $package_list \| grep -c dvrp_graphics) != 0 ]]
	1548	then
[260]	1549	if [[ "$dvr_inc" = "" ]]
[83]	1550	then
[260]	1551	printf "\n\n +++ no value for \"dvr_inc\" given in configuration file"
[83]	1552	printf "\n This is required for the dvrp_graphics package.\n"
[260]	1553	locat=dvr; exit
[83]	1554	fi
[260]	1555	if [[ "$dvr_lib" = "" ]]
[83]	1556	then
[260]	1557	printf "\n\n +++ no value for \"dvr_lib\" given in configuration file"
[83]	1558	printf "\n This is required for the dvrp_graphics package.\n"
[260]	1559	locat=dvr; exit
[83]	1560	fi
	1561	fi
	1562
	1563
[1]	1564	# PRUEFEN, OB ENTWEDER HAUPTPROGRAMM ODER NUR EIN AUSFUEHRBARES
	1565	# PROGRAMM VEREINBART WURDE (IN DIESEM FALL BRAUCHT IM WEITEREN NICHT
	1566	# UEBERSETZT ZU WERDEN)
	1567	if [[ "$mainprog" = "" && "$executable" = "" ]]
	1568	then
	1569	printf "\n +++ neither main program nor executable defined"
	1570	locat=source; exit
	1571	elif [[ "$mainprog" != "" && "$executable" != "" ]]
	1572	then
	1573	printf "\n +++ main program as well as executable defined"
	1574	locat=source; exit
	1575	elif [[ "$mainprog" = "" && "$executable" != "" ]]
	1576	then
	1577	do_compile=false
	1578	fi
	1579
	1580
[191]	1581	# SOURCE-VERZEICHNIS ZUM AUFSAMMELN DER ZU UEBERSETZENDEN PROGRAMMTEILE
	1582	# ERZEUGEN. HIERHIN WERDEN SPAETER IM FALL VON BATCH-JOBS AUCH DAS MRUN-
	1583	# SCRIPTS SOWIE DIE KONFIGURATIONSDATEI KOPIERT
[193]	1584	if [[ $restart_run != true && "$SOURCES_COMPLETE" = "" ]]
[191]	1585	then
	1586	rm -rf SOURCES_FOR_RUN_$fname
	1587	mkdir SOURCES_FOR_RUN_$fname
	1588	fi
	1589
	1590
[1]	1591	# ALLE ZU UEBERSETZENDEN PROGRAMMTEILE ZUSAMMENSAMMELN
	1592	# BEI BATCH-JOBS IST DIES NICHT NOETIG, WEIL DIE PROGRAMMTEILE BEREITS DURCH
	1593	# DEN MRUN-AUFRUF ERMITTELT SIND, DER DEN BATCH-JOB GENERIERT HAT, UND
	1594	# IM VERZEICHNIS SOURCES_FOR_RUN_... ABGELEGT SIND
	1595	if [[ $do_compile = true && "$SOURCES_COMPLETE" = "" ]]
	1596	then
	1597
[5]	1598	[[ "$source_list" = LM ]] && source_list=LOCALLY_MODIFIED
	1599	[[ "$source_list" = WP ]] && source_list=WRITE_PERMIT
[1]	1600
	1601
	1602	# EVTL. ZU UEBERSETZENDE PROGRAMMTEILE DADURCH BESTIMMEN, DASS GEPRUEFT
	1603	# WIRD, WELCHE DATEIEN WRITE-PERMIT BESITZEN (DATEIEN SOLLTEN DESHALB
	1604	# UNTER RCS-KONTROLLE STEHEN). MIT DER VARIABLEN EXCLUDE KOENNEN
	1605	# BESTIMMTE DATEIEN GRUNDSAETZLICH VON DER UEBERSETZUNG AUSGESCHLOSSEN
	1606	# WERDEN. DIES IST NUR BEI MANUELLEM START VON MRUN DURCH DEN BENUTZER
	1607	# MOEGLICH, DA BEI AUTOMATISCHEN RESTARTS TEST DURCH DIE TATSAECHLICH
	1608	# ZU UBERSETZENDEN DATEIEN ERSETZT IST.
	1609	# ALLE ERMITTELTEN DATEIEN WERDEN IM VERZEICHNIS SOURCES_FOR_RUN_...
	1610	# GESAMMELT
[5]	1611	if [[ "$source_list" = WRITE_PERMIT ]]
[1]	1612	then
	1613
	1614	source_list=""
	1615	cd $source_path
	1616
	1617
	1618	# ALLE MOEGLICHEN QUELLCODEDATEIEN AUFLISTEN
	1619	Names=$(ls -1 *.f90 2>&1)
	1620	[[ $(echo $Names \| grep -c '*.f90') = 0 ]] && Filenames="$Names"
	1621	Names=$(ls -1 *.F90 2>&1)
	1622	[[ $(echo $Names \| grep -c '*.F90') = 0 ]] && Filenames="$Filenames $Names"
	1623	Names=$(ls -1 *.F 2>&1)
	1624	[[ $(echo $Names \| grep -c '*.F') = 0 ]] && Filenames="$Filenames $Names"
	1625	Names=$(ls -1 *.f 2>&1)
	1626	[[ $(echo $Names \| grep -c '*.f') = 0 ]] && Filenames="$Filenames $Names"
	1627	Names=$(ls -1 *.c 2>&1)
	1628	[[ $(echo $Names \| grep -c '*.c') = 0 ]] && Filenames="$Filenames $Names"
	1629
	1630
	1631	# DATEIEN MIT WRITE-PERMIT NACH SOURCES_FOR_RUN_... KOPIEREN
	1632	for dateiname in $Filenames
	1633	do
	1634	if [[ -w $dateiname ]]
	1635	then
	1636	if [[ "$exclude" = "" \|\| $(echo $exclude \| grep -c $dateiname) = 0 ]]
	1637	then
	1638	cp $dateiname $working_directory/SOURCES_FOR_RUN_$fname
	1639	source_list=$source_list"$dateiname "
	1640	fi
	1641	fi
	1642	done
	1643
	1644	cd - > /dev/null
	1645
	1646
[5]	1647	elif [[ "$source_list" = LOCALLY_MODIFIED ]]
	1648	then
	1649
	1650	# MODIFIZIERTE DATEIEN DER SVN-ARBEITSKOPIE BESTIMMEN
	1651	source_list=""
	1652	cd $source_path
	1653
	1654
	1655	# PRUEFEN, OB VERZEICHNIS UEBERHAUPT UNTER SVN-KONTROLLE STEHT
	1656	if [[ ! -d .svn ]]
	1657	then
	1658	printf "\n\n +++ source directory"
	1659	printf "\n \"$source_path\" "
	1660	printf "\n is not under control of \"subversion\"."
[306]	1661	printf "\n Please do not use mrun-option \"-s LOCALLY_MODIFIED\"\n"
[5]	1662	fi
	1663
	1664
	1665	# ALLE MODIFIZIERTEN QUELLCODEDATEIEN AUFLISTEN
	1666	Filenames=""
[69]	1667	svn status > tmp_mrun
	1668	while read line
[5]	1669	do
	1670	firstc=`echo $line \| cut -c1`
	1671	if [[ $firstc = M \|\| $firstc = "?" ]]
	1672	then
	1673	Name=`echo "$line" \| cut -c8-`
	1674	extension=`echo $Name \| cut -d. -f2`
	1675	if [[ "$extension" = f90 \|\| "$extension" = F90 \|\| "$extension" = f \|\| "$extension" = F \|\| "$extension" = c ]]
	1676	then
	1677	Filenames="$Filenames "$Name
	1678	fi
	1679	fi
[69]	1680	done < tmp_mrun
[5]	1681
	1682
	1683	# DATEIEN NACH SOURCES_FOR_RUN_... KOPIEREN
	1684	for dateiname in $Filenames
	1685	do
	1686	cp $dateiname $working_directory/SOURCES_FOR_RUN_$fname
	1687	source_list=$source_list"$dateiname "
	1688	done
	1689
	1690	cd - > /dev/null
	1691
	1692
[1]	1693	# MITTELS OPTION -s ANGEGEBENE DATEIEN NACH SOURCES_FOR_RUN_... KOPIEREN
	1694	# BEI AUTOMATISCHEN FORTSETZUNGSLAEUFEN SIND DORT SCHON ALLE DATEIEN
	1695	# VORHANDEN
[5]	1696	elif [[ "$source_list" != "" && $restart_run != true ]]
[1]	1697	then
	1698
	1699	cd $source_path
	1700
	1701	for filename in $source_list
	1702	do
	1703
	1704	# QUELLTEXT-DATEI DARF KEINE PFADE BEINHALTEN
	1705	if [[ $(print $filename \| grep -c "/") != 0 ]]
	1706	then
	1707	printf "\n +++ source code file: $filename"
	1708	printf "\n must not contain (\"/\") "
	1709	locat=source; exit
	1710	fi
	1711
	1712	if [[ ! -f $filename ]]
	1713	then
	1714	printf "\n +++ source code file: $filename"
	1715	printf "\n does not exist"
	1716	locat=source; exit
	1717	else
	1718	cp $filename $working_directory/SOURCES_FOR_RUN_$fname
	1719	fi
	1720
	1721	done
	1722
	1723	cd - > /dev/null
	1724
	1725	fi
	1726
	1727
	1728	# PRUEFEN, OB ENTWEDER HAUPTPROGRAMM VORHANDEN UND ES EVTL. IN DER
	1729	# LISTE DER ZU UEBERSETZENDEN PROGRAMMTEILE MIT ENTHALTEN IST (WENN
	1730	# NICHT, WIRD ES DIESER LISTE HINZUGEFUEGT)
[5]	1731	if [[ $restart_run != true ]]
[1]	1732	then
	1733
	1734	if [[ ! -f "$source_path/$mainprog" ]]
	1735	then
	1736	printf "\n\n +++ main program: $mainprog"
	1737	printf "\n does not exist in source directory"
	1738	printf "\n \"$source_path\"\n"
	1739	locat=source; exit
	1740	else
	1741	if [[ $(echo $source_list \| grep -c $mainprog) = 0 ]]
	1742	then
	1743	cp $source_path/$mainprog SOURCES_FOR_RUN_$fname
	1744	source_list=${mainprog}" $source_list"
	1745	fi
	1746	fi
	1747	fi
	1748
	1749
[211]	1750	# MAKEFILE AUF VORHANDENSEIN PRUEFEN UND KOPIEREN
	1751	# BEI RESTART-LAEUFEN LIEGT ES SCHON IM VERZEICHNIS SOURCES_FOR_RUN...
	1752	if [[ "$restart_run" != true ]]
	1753	then
	1754	[[ "$makefile" = "" ]] && makefile=$source_path/Makefile
	1755	if [[ ! -f $makefile ]]
	1756	then
	1757	printf "\n +++ file \"$makefile\" does not exist"
	1758	locat=make; exit
	1759	else
	1760	cp $makefile SOURCES_FOR_RUN_$fname/Makefile
	1761	fi
	1762	fi
	1763
	1764
[1]	1765	# DATEIEN AUS ZUSAETZLICHEM QUELLVERZEICHNIS HINZUFUEGEN
[5]	1766	if [[ $restart_run != true && "$add_source_path" != "" ]]
[1]	1767	then
	1768
	1769	# GIBT ES DAS VERZEICHNIS UEBERHAUPT?
	1770	if [[ ! -d $add_source_path ]]
	1771	then
	1772	printf "\n\n +++ WARNING: additional source code directory"
	1773	printf "\n \"$add_source_path\" "
	1774	printf "\n does not exist or is not a directory."
	1775	printf "\n No source code will be used from this directory!\n"
	1776	add_source_path=""
	1777	sleep 3
	1778	else
	1779
	1780	cd $add_source_path
[211]	1781	found=false
[1]	1782
	1783	Names=$(ls -1 *.f90 2>&1)
	1784	[[ $(echo $Names \| grep -c '*.f90') = 0 ]] && AddFilenames="$Names"
	1785	Names=$(ls -1 *.F90 2>&1)
	1786	[[ $(echo $Names \| grep -c '*.F90') = 0 ]] && AddFilenames="$AddFilenames $Names"
	1787	Names=$(ls -1 *.F 2>&1)
	1788	[[ $(echo $Names \| grep -c '*.F') = 0 ]] && AddFilenames="$AddFilenames $Names"
	1789	Names=$(ls -1 *.f 2>&1)
	1790	[[ $(echo $Names \| grep -c '*.f') = 0 ]] && AddFilenames="$AddFilenames $Names"
	1791	Names=$(ls -1 *.c 2>&1)
	1792	[[ $(echo $Names \| grep -c '*.c') = 0 ]] && AddFilenames="$AddFilenames $Names"
	1793
	1794	cd - > /dev/null
	1795	cd SOURCES_FOR_RUN_$fname
	1796
[211]	1797	# COPY MAKEFILE IF EXISTING
	1798	if [[ -f $add_source_path/Makefile ]]
	1799	then
	1800	printf "\n\n *** user Makefile from directory"
	1801	printf "\n \"$add_source_path\" is used \n"
	1802	sleep 1
	1803	cp $add_source_path/Makefile .
	1804	fi
	1805
[1]	1806	for dateiname in $AddFilenames
	1807	do
	1808	if [[ -f $dateiname ]]
	1809	then
	1810	printf "\n +++ source code file \"$dateiname\" found in additional"
	1811	printf "\n source code directory \"$add_source_path\" "
[211]	1812	printf "\n but was also given with option \"-s\" which means that it should be taken"
	1813	printf "\n from directory \"$source_path\"."
[1]	1814	locat=source; exit
	1815	fi
	1816
	1817	cp $add_source_path/$dateiname .
	1818	source_list="$source_list $dateiname"
	1819
[211]	1820	# CHECK IF FILE IS CONTAINED IN MAKEFILE
	1821	if [[ $(grep -c $dateiname Makefile) = 0 ]]
	1822	then
	1823	printf "\n\n +++ user file \"$dateiname\" "
	1824	printf "\n is not listed in Makefile \n"
	1825	locat=source; exit
	1826	else
	1827
[1]	1828	# Default User-Interface von der Liste entfernen, falls Datei
	1829	# ein User-Interface enthaelt
[211]	1830	# if [[ $( cat $dateiname \| grep -c "END SUBROUTINE user_parin" ) != 0 ]]
	1831	# then
	1832	# if [[ $dateiname != user_interface.f90 && -f user_interface.f90 ]]
	1833	# then
	1834	# rm -rf user_interface.f90
	1835	# source_list=`echo $source_list \| sed -e 's/user_interface.f90//'`
	1836	# printf "\n\n *** default \"user_interface.f90\" removed from the files to be translated"
	1837	# printf "\n since a user-interface is found in file"
	1838	# printf "\n \"$add_source_path/$dateiname\" \n"
	1839	# sleep 3
	1840	# else
	1841	if [[ $found = false ]]
[1]	1842	then
[211]	1843	found=true
	1844	printf "\n\n *** following user file(s) added to the"
	1845	printf " files to be translated:\n "
[1]	1846	fi
[211]	1847	printf "$dateiname "
	1848	sleep 0.5
[1]	1849	fi
	1850	done
[211]	1851	[[ $found = true ]] && printf "\n"
[1]	1852	cd - > /dev/null
	1853	fi
	1854	fi
	1855
	1856
	1857	# ALLE UNTERPROGRAMME, DIE ZU VEREINBARTEN SOFTWAREPAKETEN GEHOEREN,
	1858	# DER LISTE DER ZU UEBERSETZENDEN DATEIEN HINZUFUEGEN
[5]	1859	if [[ $restart_run != true && -n $package_list ]]
[1]	1860	then
	1861
	1862	cd $source_path
	1863
	1864	for package in $package_list
	1865	do
	1866
	1867	[[ $package = "dvrp_graphics+1PE" ]] && package=dvrp_graphics
	1868
	1869	# ERMITTELE ALLE DATEIEN, DIE ZUM PAKET GEHOEREN
	1870	# FEHLERMELDUNGEN WERDEN ABGEFANGEN, DA * AUCH VERZEICHNISSNAMEN
	1871	# LIEFERT
	1872	package_source_list=`grep "defined( __$package " * 2>/dev/null \| cut -f1 -d:`
	1873
	1874
	1875	# FUEGE DIESE DATEIEN DER LISTE DER ZU UEBERSETZENDEN DATEIEN
	1876	# HINZU, FALLS SIE NOCH NICHT DAZUGEHOEREN
	1877	for source_list_name in $package_source_list
	1878	do
	1879	if [[ $(echo $source_list \| grep -c $source_list_name) = 0 ]]
	1880	then
	1881
	1882	# NUR DATEIEN MIT GUELTIGEN ENDUNGEN VERWENDEN
	1883	ending=`echo $source_list_name \| cut -f2 -d.`
	1884	if [[ "$ending" = f90 \|\| "$ending" = F90 \|\| "$ending" = f \|\| "$ending" = F \|\| "$ending" = c ]]
	1885	then
	1886	cp $source_list_name $working_directory/SOURCES_FOR_RUN_$fname
	1887	source_list="$source_list $source_list_name"
	1888	fi
	1889	fi
	1890	done
	1891	done
	1892
	1893	cd - > /dev/null
	1894	fi
	1895
	1896
	1897	# MAKEFILE AUF VORHANDENSEIN PRUEFEN UND KOPIEREN
	1898	# BEI RESTART-LAEUFEN LIEGT ES SCHON IM VERZEICHNIS SOURCES_FOR_RUN...
[211]	1899	# if [[ "$restart_run" != true ]]
	1900	# then
	1901	# [[ "$makefile" = "" ]] && makefile=$source_path/Makefile
	1902	# if [[ ! -f $makefile ]]
	1903	# then
	1904	# printf "\n +++ file \"$makefile\" does not exist"
	1905	# locat=make; exit
	1906	# else
	1907	# cp $makefile SOURCES_FOR_RUN_$fname/Makefile
	1908	# fi
	1909	# fi
[1]	1910
	1911	fi # do_compile=true
	1912
	1913
	1914	# FALLS PROGRAMMTEILE UEBERSETZT WERDEN SOLLEN, FOLGEN JETZT EINIGE
	1915	# UEBERPRUEFUNGEN UND DAS SETZEN DER PRAEPROZESSOR-DIREKTIVEN
	1916	if [[ $do_compile = true ]]
	1917	then
	1918
	1919	# PRAEPROZESSOR-DIREKTIVEN ZUM SELEKTIVEN AUSWAEHLEN VON CODETEILEN
	1920	# ZUSAMMENSETZEN
[83]	1921	# DIREKTIVEN ZUM AKTIVIEREN VON RECHNERSPEZIFISCHEM CODE
[1]	1922	if [[ $(echo $localhost \| cut -c1-3) = ibm ]]
	1923	then
[82]	1924	cpp_options="${cpp_options},-D__ibm=__ibm"
[1]	1925	elif [[ $(echo $localhost \| cut -c1-3) = nec ]]
	1926	then
[82]	1927	cpp_options="$cpp_options -D__nec"
	1928	elif [[ $(echo $localhost \| cut -c1-2) = lc ]]
	1929	then
	1930	cpp_options="$cpp_options -D__lc"
[1]	1931	else
[82]	1932	cpp_options="$cpp_options -D__$localhost"
[1]	1933	fi
[83]	1934
	1935	# DIREKTIVEN DIE DURCH OPTION -K BESTIMMT WERDEN (Z.B. PARALLEL)
[1]	1936	if [[ $(echo $localhost \| cut -c1-3) = ibm ]]
	1937	then
[82]	1938	[[ -n $cond1 ]] && cpp_options="${cpp_options},-D__$cond1=__$cond1"
	1939	[[ -n $cond2 ]] && cpp_options="${cpp_options},-D__$cond2=__$cond2"
[1]	1940	else
[82]	1941	[[ -n $cond1 ]] && cpp_options="$cpp_options -D__$cond1"
	1942	[[ -n $cond2 ]] && cpp_options="$cpp_options -D__$cond2"
[1]	1943	fi
[83]	1944
	1945	# DIREKTIVEN DIE SOFTWAREPAKETE AKTIVIEREN (OPTION -p)
[1]	1946	if [[ -n $package_list ]]
	1947	then
	1948	for package in $package_list
	1949	do
	1950	if [[ $(echo $localhost \| cut -c1-3) = ibm ]]
	1951	then
	1952	if [[ $package != "dvrp_graphics+1PE" ]]
	1953	then
[82]	1954	cpp_options="${cpp_options},-D__$package=__$package"
[1]	1955	else
[82]	1956	cpp_options="${cpp_options},-D__dvrp_graphics=__dvrp_graphics"
[1]	1957	export use_seperate_pe_for_dvrp_output=true
	1958	fi
	1959	else
	1960	if [[ $package != "dvrp_graphics+1PE" ]]
	1961	then
[82]	1962	cpp_options="$cpp_options -D__$package"
[1]	1963	else
[82]	1964	cpp_options="$cpp_options -D__dvrp_graphics"
[1]	1965	export use_seperate_pe_for_dvrp_output=true
	1966	fi
	1967	fi
	1968	done
	1969	fi
[83]	1970
	1971	# DIREKTIVEN DIE DURCH OPTION -D FESTGELEGT SIND
[1]	1972	if [[ -n $cpp_opts ]]
	1973	then
	1974	for popts in $cpp_opts
	1975	do
[82]	1976	if [[ $(echo $localhost \| cut -c1-3) = ibm ]]
	1977	then
	1978	cpp_options="${cpp_options},-D__$popts=__$popts"
	1979	else
	1980	cpp_options="$cpp_options -D__$popts"
	1981	fi
[1]	1982	done
	1983	fi
	1984
	1985	else
	1986
	1987
	1988	# BEI LOKALEN RECHNUNGEN PRUEFEN, OB EXECUTABLE VORHANDEN
	1989	if [[ $do_remote = false ]]
	1990	then
	1991	if [[ ! -f $executable ]]
	1992	then
	1993	printf "\n +++ executable file: $executable"
	1994	printf "\n does not exist"
	1995	locat=executable; exit
	1996	fi
	1997	fi
	1998	fi
	1999
	2000
	2001	# JOBMODUS FESTSTELLEN
[635]	2002	if [[ "$ENVIRONMENT" = BATCH ]]
[1]	2003	then
	2004	jobmo=BATCH
	2005	else
	2006	jobmo=INTERACTIVE
	2007	fi
	2008
	2009
[635]	2010	# no interactive runs on lctit
	2011	if [[ $host = lctit && $jobmo = INTERACTIVE && $do_batch = false ]]
	2012	then
	2013	printf "\n +++ no interactive runs allowed on host \"$host\" "
	2014	printf "\n please submit batch job using mrun option \"-b\" \n"
	2015	locat=normal; exit
	2016	fi
	2017
	2018
[1]	2019	# HOSTSPEZIFISCHE DEFAULT-COMPILER SETZEN, FALLS NICHT BEREITS
	2020	# DURCH BENUTZER ANDERWEITIG VEREINBART
	2021	if [[ "$compiler_name" = "" ]]
	2022	then
[82]	2023
[220]	2024	printf "\n +++ no compiler specified for \"$host $cond1 $cond2\""
[82]	2025	locat=compiler_name; exit
	2026
[1]	2027	fi
	2028
	2029
[251]	2030	# COMPILER AUF RIAMS NEC UEBERSCHREIBEN
	2031	[[ $localhost = necriam ]] && compiler_name=mpif90
[1]	2032
	2033
	2034
	2035	# TEMPORAEREN KATALOGNAMEN BESTIMMEN
	2036	kennung=$RANDOM
	2037	if [[ "$tmp_user_catalog" = "" ]]
	2038	then
[312]	2039	if [[ $localhost = ibmh ]]
[1]	2040	then
[312]	2041	tmp_user_catalog=$SCRATCH
[1]	2042	elif [[ $localhost = nech ]]
	2043	then
	2044	tmp_user_catalog=$WRKSHR
	2045	else
	2046	tmp_user_catalog=/tmp
	2047	fi
	2048	fi
	2049	TEMPDIR=$tmp_user_catalog/${usern}.$kennung
	2050
	2051
	2052	# KATALOGNAMEN FUER ZWISCHENSPEICHERUNG VON FORTSETZUNGSLAUFDATEIEN
	2053	# BESTIMMEN
	2054	if [[ "$tmp_data_catalog" = "" ]]
	2055	then
[622]	2056	if [[ $localhost = nech ]]
[1]	2057	then
	2058	tmp_data_catalog=$WRKSHR/mrun_restart_data
	2059	else
	2060	tmp_data_catalog=/tmp/mrun_restart_data
	2061	fi
	2062	fi
	2063
	2064
	2065	# EVENTUELL BEI LOKALEN RECHNUNGEN $-ZEICHEN IN ENVIRONMENT-VARIABLEN
	2066	# ERSETZEN
	2067	if [[ $do_remote = false && $do_compile = true ]]
	2068	then
	2069	eval fopts=\"$fopts\"
	2070	eval lopts=\"$lopts\"
	2071	fi
	2072
	2073
	2074
	2075	# COMPILE- UND LINK-OPTIONEN BESTIMMEN
[260]	2076	fopts="$fopts $netcdf_inc $dvr_inc"
	2077	lopts="$lopts $netcdf_lib $dvr_lib"
[1]	2078	ROPTS="$ropts"
[717]	2079	if [[ ( $(echo $host \| cut -c1-3) = nec \|\| $(echo $host \| cut -c1-3) = ibm \|\| $host = lckyoto \|\| $host = lcsgih \|\| $host = lcsgib \|\| $host = lctit \|\| $host = lcfimm \|\| $host = lcxt4 \|\| $host = lcxt5m \|\| $host = lck \|\| $host = lckordi) && -n $numprocs ]]
[1]	2080	then
	2081	XOPT="-X $numprocs"
	2082	fi
	2083
	2084
	2085
	2086	# PRUEFEN DER CPU-ZEIT. (CPUMAX WIRD ALS ENV-VARIABLE VOM HAUTPRO-
	2087	# GRAMM BENOETIGT
	2088	done=false
	2089	while [[ $done = false ]]
	2090	do
	2091	cputime=$cpumax
	2092	if (( $cputime == 0 ))
	2093	then
[246]	2094	if [[ $do_batch = true ]]
	2095	then
	2096	printf "\n +++ cpu-time is undefined"
	2097	printf "\n >>> Please type CPU-time in seconds as INTEGER:"
	2098	printf "\n >>> "
	2099	read cputime 1>/dev/null 2>&1
	2100	else
	2101	cputime=10000000 # NO CPU LIMIT FOR INTERACTIVE RUNS
	2102	fi
[1]	2103	else
	2104	done=true
	2105	fi
	2106	cpumax=$cputime
	2107	done
	2108
	2109	(( minuten = cputime / 60 ))
	2110	(( sekunden = cputime - minuten * 60 ))
	2111
	2112
	2113	# PRUEFEN DER KERNSPEICHERANFORDERUNG
	2114	if [[ $do_batch = true ]]
	2115	then
	2116	done=false
	2117	while [[ $done = false ]]
	2118	do
	2119	if (( memory == 0 ))
	2120	then
	2121	printf "\n +++ memory demand is undefined"
	2122	printf "\n >>> Please type memory in MByte per process as INTEGER:"
	2123	printf "\n >>> "
	2124	read memory 1>/dev/null 2>&1
	2125	else
	2126	done=true
	2127	fi
	2128	done
	2129	fi
	2130
	2131
	2132	# PRUEFEN, OB FUER REMOTE-RECHNUNGEN EIN BENUTZERNAME ANGEGEBEN WURDE
	2133	if [[ $do_remote = true && -z $remote_username ]]
	2134	then
	2135	while [[ -z $remote_username ]]
	2136	do
	2137	printf "\n +++ username on remote host \"$host\" is undefined"
	2138	printf "\n >>> Please type username:"
	2139	printf "\n >>> "
	2140	read remote_username
	2141	done
	2142	mc="$mc -u$remote_username"
	2143	fi
	2144
[221]	2145
[493]	2146	# set module load command and export for subjob
	2147	if [[ "$modules" != "" ]]
	2148	then
[678]	2149	if [[ $host = lctit ]]
	2150	then
	2151	export module_calls=". $modules"
	2152	else
	2153	export module_calls="module load ${modules};"
	2154	fi
[493]	2155	fi
	2156
[683]	2157	# bugfix for wrong netcdf module and for netCDF4 usage in case of mpt
[562]	2158	if [[ $host = lcsgib \|\| $host = lcsgih ]]
[555]	2159	then
	2160	if [[ $(echo $module_calls \| grep -c netcdf/3.6.3-intel) != 0 ]]
	2161	then
[562]	2162	export module_calls="$module_calls export LD_LIBRARY_PATH=/sw/dataformats/netcdf/3.6.3-intel/lib:\$LD_LIBRARY_PATH;"
[555]	2163	fi
[721]	2164	# if [[ $(echo $module_calls \| grep -c mpt) != 0 ]]
	2165	# then
	2166	# export module_calls="$module_calls export LD_LIBRARY_PATH=/sw/sgi/mpt/2011-02-07/lib:\$LD_LIBRARY_PATH;"
	2167	# echo "*** module_calls = $module_calls"
	2168	# fi
[555]	2169	fi
[493]	2170
[555]	2171
[221]	2172	# SET DEFAULT VALUE FOR MPI MODULE TO BE USED ON SGI-ALTIX
	2173	if [[ $host = lcsgib \|\| $host = lcsgih ]]
	2174	then
[493]	2175	if [[ $(echo $modules \| grep -c mpt ) != 0 ]]
	2176	then
	2177	mpilib=mpt
	2178	elif [[ $(echo $modules \| grep -c mvapich ) != 0 ]]
	2179	then
	2180	mpilib=mvapich
[738]	2181	elif [[ $(echo $modules \| grep -c impi ) != 0 ]]
	2182	then
	2183	mpilib=impi
[493]	2184	fi
[221]	2185	fi
	2186
	2187
	2188
[1]	2189	###########################################################################
	2190	# HEADER-AUSGABE
	2191	###########################################################################
	2192
	2193
	2194	calltime=$(date)
	2195	printf "\n"
	2196	# [[ $silent = false ]] && clear
	2197	printf "#--------------------------------------------------------------# \n"
	2198	printf "\| $version$calltime \| \n"
	2199	printf "\| \| \n"
	2200	spalte1="called on:"; spalte2=$localhost_realname
	2201	printf "\| $spalte1$spalte2 \| \n"
	2202	if [[ $local_compile = false ]]
	2203	then
	2204	if [[ $do_remote = true ]]
	2205	then
	2206	spalte1="execution on:"; spalte2="$host (username: $remote_username)"
	2207	else
	2208	spalte1="execution on:"; spalte2="$host ($localhost_realname)"
	2209	fi
	2210	else
	2211	spalte1="compiling test only!"; spalte2=""
	2212	fi
	2213	printf "\| $spalte1$spalte2 \| \n"
	2214	if [[ -n $numprocs ]]
	2215	then
	2216	spalte1="number of PEs:"; spalte2=$numprocs
	2217	printf "\| $spalte1$spalte2 \| \n"
	2218	fi
	2219	if [[ -n $tasks_per_node ]]
	2220	then
	2221	spalte1="tasks per node:"; spalte2="$tasks_per_node (number of nodes: $nodes)"
	2222	printf "\| $spalte1$spalte2 \| \n"
	2223	fi
[759]	2224	if [[ $maximum_parallel_io_streams != $numprocs ]]
	2225	then
	2226	spalte1="max par io streams:"; spalte2="$maximum_parallel_io_streams"
	2227	printf "\| $spalte1$spalte2 \| \n"
	2228	fi
[622]	2229	if [[ $use_openmp = true ]]
[1]	2230	then
	2231	spalte1="threads per task:"; spalte2="$threads_per_task"
	2232	printf "\| $spalte1$spalte2 \| \n"
	2233	fi
	2234	printf "\| \| \n"
	2235	if [[ $do_compile = true ]]
	2236	then
[475]	2237	if [[ "$mopts" != "" ]]
	2238	then
[493]	2239	spalte1="make options:"; spalte2=$mopts
[475]	2240	printf "\| $spalte1$spalte2 \| \n"
	2241	zeile=$(echo "$mopts" \| cut -c41-)
	2242	while [[ "$zeile" != "" ]]
	2243	do
	2244	spalte1=""
	2245	spalte2=$zeile
	2246	printf "\| $spalte1$spalte2 \| \n"
	2247	zeile=$(echo "$zeile" \| cut -c41-)
	2248	done
	2249	fi
	2250
[493]	2251	spalte1="cpp directives:"; spalte2=$cpp_options
[1]	2252	printf "\| $spalte1$spalte2 \| \n"
[82]	2253	zeile=$(echo "$cpp_options" \| cut -c41-)
[1]	2254	while [[ "$zeile" != "" ]]
	2255	do
	2256	spalte1=""
	2257	spalte2=$zeile
	2258	printf "\| $spalte1$spalte2 \| \n"
	2259	zeile=$(echo "$zeile" \| cut -c41-)
	2260	done
	2261
[493]	2262	spalte1="compiler options:"; spalte2="$fopts"
[1]	2263	printf "\| $spalte1$spalte2 \| \n"
[82]	2264	zeile=$(echo "$fopts" \| cut -c41-)
[1]	2265	while [[ "$zeile" != "" ]]
	2266	do
	2267	spalte1=""
	2268	spalte2=$zeile
	2269	printf "\| $spalte1$spalte2 \| \n"
	2270	zeile=$(echo "$zeile" \| cut -c41-)
	2271	done
	2272
[493]	2273	spalte1="linker options:"; spalte2=$lopts
[1]	2274	printf "\| $spalte1$spalte2 \| \n"
[82]	2275	zeile=$(echo "$lopts" \| cut -c41-)
[1]	2276	while [[ "$zeile" != "" ]]
	2277	do
	2278	spalte1=""
	2279	spalte2=$zeile
	2280	printf "\| $spalte1$spalte2 \| \n"
	2281	zeile=$(echo "$zeile" \| cut -c41-)
	2282	done
	2283
[493]	2284	spalte1="modules to be load:"; spalte2=$modules
	2285	printf "\| $spalte1$spalte2 \| \n"
	2286	zeile=$(echo "$modules" \| cut -c41-)
	2287	while [[ "$zeile" != "" ]]
	2288	do
	2289	spalte1=""
	2290	spalte2=$zeile
	2291	printf "\| $spalte1$spalte2 \| \n"
	2292	zeile=$(echo "$zeile" \| cut -c41-)
	2293	done
	2294
[1]	2295	spalte1="main program":; spalte2=$mainprog
	2296	printf "\| $spalte1$spalte2 \| \n"
	2297	else
	2298	spalte1=executable:; spalte2=$executable
	2299	printf "\| $spalte1$spalte2 \| \n"
	2300	fi
	2301	printf "\| \| \n"
	2302	spalte1="base name of files":; spalte2=$fname
	2303	printf "\| $spalte1$spalte2 \| \n"
	2304	if [[ $fname != $afname ]]
	2305	then
	2306	spalte1="base name of input files":; spalte2=$afname
	2307	printf "\| $spalte1$spalte2 \| \n"
	2308	fi
	2309	spalte1="INPUT control list":; spalte2=$input_list
	2310	printf "\| $spalte1$spalte2 \| \n"
	2311	spalte1="OUTPUT control list":; spalte2=$output_list
	2312	printf "\| $spalte1$spalte2 \| \n"
	2313
	2314	if [[ $do_batch = true \|\| "$LOADLBATCH" = yes ]]
	2315	then
	2316	spalte1="memory demand / PE":; spalte2="$memory MB"
	2317	printf "\| $spalte1$spalte2 \| \n"
	2318	spalte1=CPU-time:; spalte2="$minuten:$sekunden"
	2319	printf "\| $spalte1$spalte2 \| \n"
	2320	fi
	2321
	2322	if [[ $do_compile = true ]]
	2323	then
	2324	printf "\| \| \n"
	2325	printf "\| Files to be compiled: \| \n"
	2326	zeile=$source_list
	2327	while [[ "$zeile" != "" ]]
	2328	do
	2329	spalte3=$zeile
	2330	printf "\| $spalte3 \| \n"
	2331	zeile=$(echo "$zeile" \| cut -c61-)
	2332	done
	2333	fi
	2334	printf "#--------------------------------------------------------------#"
	2335
	2336
	2337
	2338	# BEDINGTE AUSGABE DER DATEIVERBINDUNGEN
	2339	if [[ $do_trace = true ]]
	2340	then
	2341	(( i = 0 ))
	2342	while (( i < iin ))
	2343	do
	2344	(( i = i + 1 ))
	2345	if (( i == 1 ))
	2346	then
	2347	printf "\n\n >>> INPUT-file assignments:\n"
	2348	fi
	2349	printf "\n ${localin[$i]} : ${absnamein[$i]}"
	2350	done
	2351	(( i = 0 ))
	2352	while (( i < iout ))
	2353	do
	2354	(( i = i + 1 ))
	2355	if (( i == 1 ))
	2356	then
	2357	printf "\n\n >>> OUTPUT-file assignments:\n"
	2358	fi
	2359	printf "\n ${localout[$i]} : ${pathout[$i]}"
	2360	done
	2361	(( i = 0 ))
	2362	while (( i < iic ))
	2363	do
	2364	(( i = i + 1 ))
	2365	if (( i == 1 ))
	2366	then
	2367	printf "\n\n >>> INPUT-commands:\n"
	2368	fi
	2369	printf "\n ${in_command[$i]}"
	2370	done
	2371	(( i = 0 ))
	2372	while (( i < ioc ))
	2373	do
	2374	(( i = i + 1 ))
	2375	if (( i == 1 ))
	2376	then
	2377	printf "\n\n >>> OUTPUT-commands:\n"
	2378	fi
	2379	printf "\n ${out_command[$i]}"
	2380	done
	2381	fi
	2382
	2383
	2384	# ABFRAGEN BEI AUFRUF AUF LOKALER MASCHINE
	2385	if [[ $remotecall = false && $silent = false && $jobmo != BATCH ]]
	2386	then
	2387	antwort=dummy
	2388	printf "\n\n"
	2389	while [[ "$antwort" != y && "$antwort" != Y && "$antwort" != n && "$antwort" != N ]]
	2390	do
	2391	printf " >>> everything o.k. (y/n) ? "
	2392	read antwort
	2393	done
	2394	if [[ $antwort = n \|\| $antwort = N ]]
	2395	then
	2396	locat=user_abort; (( iec = 0 )); exit
	2397	fi
	2398	if [[ $do_batch = true ]]
	2399	then
	2400	printf " >>> batch-job will be created and submitted"
	2401	else
	2402	if [[ $local_compile = false ]]
	2403	then
	2404	printf " >>> MRUN will now continue to execute on this machine"
	2405	else
	2406	printf " >>> a test compilation will now be carried out on this machine"
	2407	fi
	2408	fi
	2409	fi
	2410
[807]	2411	#### PERFORM PARAMETER FILE CHECK (COUPLED RUNS ARE NOT SUPPORTED YET)
[818]	2412	skip_check=false
	2413	if [[ $check_namelist_files == false \|\| $fromhost != $localhost \|\| $run_coupled_model == true \|\| $restart_run == true ]]
[807]	2414	then
[815]	2415	skip_check=true
	2416	fi
	2417	if [[ $skip_check == false ]]
	2418	then
[807]	2419	if [[ $ocean_file_appendix == true ]]
	2420	then
	2421	d3par_string="d3o#"
	2422	d3fpar_string="d3of"
	2423	else
	2424	d3par_string="d3#"
	2425	d3fpar_string="d3f"
	2426	fi
[1]	2427
[807]	2428	tmp_check=${working_directory}/tmp_check
[1]	2429
[807]	2430	# GET PARAMETER FILE SUFFIX (USUALLY "_P3D")
	2431	line=""
	2432	found=false
	2433	grep "$d3par_string" $config_file \| grep "PARIN" > $tmp_check
	2434	while read line1
	2435	do
	2436	line="$line1"
	2437	if [[ "$line" != "" && $(echo $line \| cut -c1) != "#" ]]
	2438	then
	2439	# EVENTUELLE DOPPELPUNKTE AUS OPTIONSSTRING ENTFERNEN
	2440	p3d_suffix=`echo $line \| tr -s " " \| cut -d" " -s -f5`
	2441	found=true
	2442	fi
	2443	done < $tmp_check
	2444
	2445	if [[ $found = false ]]
	2446	then
	2447	printf "\n +++ no PARIN list entry found for $d3par_string in the configuration file."
	2448
	2449	locat=check_namelist; exit
	2450	fi
	2451
	2452	# GET RESTART PARAMETER FILE SUFFIX (USUALLY "_P3DF"). THIS FILE MIGHT IS
	2453	# NOT NECESSARILY REQUIRED
	2454	line=""
	2455	found=false
	2456	grep "d3f" $config_file \| grep "PARIN" > $tmp_check
	2457	while read line1
	2458	do
	2459	line="$line1"
	2460	if [[ "$line" != "" && $(echo $line \| cut -c1) != "#" ]]
	2461	then
	2462	p3df_suffix=`echo $line \| tr -s " " \| cut -d" " -s -f5`
	2463	found=true
	2464	fi
	2465	done < $tmp_check
	2466
	2467	if [[ $found = false ]]
	2468	then
	2469	printf "\n +++ no PARIN list entry found for d3f in the configuration file."
	2470	fi
[818]	2471	# GET TOPOGRAPHY PARAMETER FILE SUFFIX (USUALLY "_P3DF"). THIS FILE MIGHT IS
	2472	# NOT NECESSARILY REQUIRED
	2473	line=""
	2474	found=false
	2475	grep "d3f" $config_file \| grep "TOPOGRAPHY_DATA" > $tmp_check
	2476	while read line1
	2477	do
	2478	line="$line1"
	2479	if [[ "$line" != "" && $(echo $line \| cut -c1) != "#" ]]
	2480	then
	2481	topo_suffix=`echo $line \| tr -s " " \| cut -d" " -s -f5`
	2482	found=true
	2483	fi
	2484	done < $tmp_check
[807]	2485
[818]	2486	if [[ $found = false ]]
	2487	then
	2488	printf "\n +++ no TOPOGRAPHY list entry found in the configuration file."
	2489	fi
	2490
[807]	2491	rm -rf ${working_directory}/tmp_check
	2492
	2493	# DEFINE VARIABLES FOR FREQUENTLY USED DIRECTORIES
	2494	check_depository="${working_directory}/trunk/UTIL"
	2495	check_sources="${working_directory}/tmp_check_namelist_files"
	2496
	2497	# CHECK IF THE A P3DF FILE MUST BE CHECKED
	2498	for item in $input_list
	2499	do
	2500	if [[ "$item" == "restart" \|\| "$item" == d3f \|\| "$item" == d3of ]]
	2501	then
	2502	check_restart=1
	2503	else
	2504	check_restart=0
	2505	fi
	2506	done
	2507
	2508	# ERROR IF NO PARAMETER FILE WAS FOUND, OTHERWISE PROCEED
	2509	if [[ ! -f $working_directory/JOBS/$fname/INPUT/${fname}${p3d_suffix} ]]
	2510	then
	2511	printf "\n\n +++ ERROR: parameter file (${fname}${p3d_suffix}) not found."
	2512	locat=check_namelist; exit
	2513	fi
	2514
	2515	# CHECK IF THE RESTART PARAMETER FILE EXISTS (IF NECESSARY)
	2516	if [[ $check_restart == 1 ]]
	2517	then
	2518	if [[ ! -f $working_directory/JOBS/$fname/INPUT/${fname}${p3df_suffix} ]]
	2519	then
	2520	printf "\n\n +++ WARNING: restart parameter file (${fname}${p3df_suffix}) is missing."
	2521	check_restart=0
	2522	answer=dummy
	2523	printf "\n\n"
	2524
[818]	2525	if [[ $silent == false ]]
[807]	2526	then
[818]	2527	while [[ "$answer" != c && "$answer" != C && "$answer" != a && "$answer" != A ]]
	2528	do
	2529	printf " >>> continue anyway (c(ontinue)/a(bort)) ? "
	2530	read answer
	2531	done
	2532	if [[ $answer = a \|\| $answer = A ]]
	2533	then
	2534	printf "\n +++ Aborting...."
	2535	locat=normal; exit
	2536	fi
[807]	2537	fi
	2538	else
	2539	check_restart=1
	2540	fi
	2541	fi
	2542
	2543	# CREATE TEMPORARY SOURCES_FOR_CHECK PATH
	2544	mkdir $check_sources
	2545	cd $check_sources
	2546
	2547	# CHECK FOR USER CODE, OTHERWISE USE THE PRECOMPILED CHECK_NAMELIST_FILES.X
	2548	if [[ -d $working_directory/USER_CODE/$fname ]]
	2549	then
	2550	printf "\n\n *** copying files from $check_depository"
	2551	cp $check_depository/check_namelist_files.tar ./
	2552
	2553	printf "\n\n *** untar of makefile and source files in $check_sources"
	2554	tar -xf check_namelist_files.tar > /dev/null 2>&1
	2555
	2556	printf "\n\n *** adding user code."
	2557	cp $working_directory/USER_CODE/$fname/* ./
	2558	touch check_namelist_files.f90
	2559
	2560	# GET COMPILER OPTIONS AND PERFORM MAKE
	2561	printf "\n\n *** compiling code if necessary...\n"
	2562	copts_check="-cpp -D_parallel -D__check"
	2563	make -f Makefile_check F90=$compiler_name_ser COPT="$copts_check"
	2564
	2565	# GET MAKE OUTPUT
	2566	if [[ $? != 0 ]]
	2567	then
[815]	2568	printf "\n +++ error during make."
	2569	answer=dummy
	2570	printf "\n\n"
[818]	2571	if [[ $silent == false ]]
[815]	2572	then
[818]	2573	while [[ "$answer" != c && "$answer" != C && "$answer" != a && "$answer" != A ]]
	2574	do
	2575	printf " >>> continue anyway (c(ontinue)/a(bort)) ? "
	2576	read answer
	2577	done
	2578	if [[ $answer = a \|\| $answer = A ]]
	2579	then
	2580	printf "\n +++ Aborting..."
	2581	rm -rf $check_sources
	2582	locat=normal; exit
	2583	else
	2584	skip_check=true
	2585	fi
[815]	2586	else
	2587	skip_check=true
	2588	fi
[807]	2589	fi
	2590	else
	2591	cp $PALM_BIN/check_namelist_files.x ./
	2592	fi
	2593
	2594	cp $working_directory/JOBS/$fname/INPUT/${fname}${p3d_suffix} ./PARIN
	2595	if [[ $check_restart == 1 ]]
	2596	then
	2597	cp $working_directory/JOBS/$fname/INPUT/${fname}${p3df_suffix} ./PARINF
	2598	fi
	2599
[818]	2600
	2601	if [[ -f $working_directory/JOBS/$fname/INPUT/${fname}${topo_suffix} && $skip_check == false ]]
	2602	then
	2603	printf "\n *** adding topography data"
	2604	cp $working_directory/JOBS/$fname/INPUT/${fname}${topo_suffix} ./TOPOGRAPHY_DATA
	2605
	2606	# IN CASE OF TOPOGRAPHY AND HIGH GRID POINT NUMBERS, THE STACK SIZE
	2607	# MUST BE INCREASED. THIS IS DUE TO THE ARRAY nzb_local AND topo_height,
	2608	# WHICH REQUIRE SUFFICIENT MEMORY
	2609	ulimit -s unlimited
	2610	fi
	2611
[807]	2612	# CREATE ENVPAR FILE, WHICH IS NEEDED BY CHECK_NAMELIST_FILES.X
	2613	cat > ENVPAR << %%END%%
	2614	&envpar run_identifier = '$fname', host = '$localhost',
	2615	write_binary = '$write_binary', tasks_per_node = $tasks_per_node,
	2616	maximum_parallel_io_streams = $maximum_parallel_io_streams,
	2617	maximum_cpu_time_allowed = ${cpumax}.,
	2618	revision = '$global_revision',
	2619	local_dvrserver_running = $local_dvrserver_running /
	2620
	2621	%%END%%
	2622
	2623	# SAFETY CHECK: ONLY PROCEED IF THE PARAMETER CHECK PROGRAM WAS PROPERLY COMPILED
[818]	2624	if [[ ! -f check_namelist_files.x && $skip_check == false ]] then
[807]	2625	printf "\n +++ WARNING: check_namelist_files.x not found."
	2626	answer=dummy
	2627	printf "\n\n"
[818]	2628
	2629	if [[ $silent == false ]]
[807]	2630	then
[818]	2631	while [[ "$answer" != c && "$answer" != C && "$answer" != a && "$answer" != A ]]
	2632	do
	2633	printf " >>> continue anyway (c(ontinue)/a(bort)) ? "
	2634	read answer
	2635	done
	2636	if [[ $answer = a \|\| $answer = A ]]
	2637	then
	2638	printf "\n +++ Aborting..."
	2639	rm -rf $check_sources
	2640	locat=normal; exit
	2641	else
	2642	printf "\n *** Skipping parameter file check."
	2643
	2644	fi
	2645	fi
	2646
[815]	2647	elif [[ $skip_check == false ]]
	2648	then
[807]	2649	# STARTING THE PARAMETER FILE CHECK
	2650	printf "\n\n *** starting parameter file check..."
	2651
	2652	# CHECKING THE P3D FILE
	2653	printf "\n\n (1) checking $fname$p3d_suffix"
	2654	echo "$numprocs 0" > VARIN
	2655	errors=`./check_namelist_files.x < VARIN 2>&1`
	2656
	2657	check_error=false
	2658	if [[ "$errors" == "" ]]
	2659	then
	2660	printf " --> o.k."
	2661	else
	2662	printf " --> failed."
	2663	check_error=true
	2664	printf "\n\n $errors"
	2665	fi
	2666
	2667	# CHECKING THE PD3F FILE IF NECESSARY
	2668	if [[ $check_restart == 1 && $check_error == false ]]
	2669	then
	2670	printf "\n\n (2) checking $fname$p3df_suffix"
	2671
	2672	# FIRST CHECK IF INITIALIZING_ACTIONS="READ_RESTART_DATA" IS SET IN &INIPAR LIST
	2673	found=false
	2674	cat PARINF\|while read line
	2675	do
	2676	line=$(echo $line\|sed 's/ //g')
	2677	if [[ $line == "&inipar" ]]
	2678	then
	2679	start_search=true
	2680	fi
	2681
	2682	if [[ $start_search == true ]]
	2683	then
	2684	if [[ $line == "initializing_actions='read_restart_data'" ]]
	2685	then
	2686	found=true
	2687	break
	2688	fi
	2689	fi
	2690
	2691	if [[ $line == "/" ]]
	2692	then
	2693	start_search=false
	2694	fi
	2695
	2696	done
	2697
	2698	if [[ $found = false ]]
	2699	then
	2700	printf "\n\n +++ ERROR: initializing_actions = 'read_restart_data' not found"
	2701	printf "\n in &inipar list in $fname$p3df_suffix."
	2702	rm -rf $check_sources
	2703	locat=check_namelist; exit
	2704	fi
	2705
	2706	echo "$numprocs 1" > VARIN
	2707	errors=`./check_namelist_files.x < VARIN 2>&1`
	2708
	2709	if [[ "$errors" == "" ]]
	2710	then
	2711	printf " --> o.k."
	2712	else
	2713	printf " --> failed."
	2714	check_error=true
	2715	printf "\n\n $errors"
	2716	fi
	2717	fi
	2718
	2719	# REPORT ERRORS AND CONTINUE/EXIT
	2720	if [[ $check_error == true ]]
	2721	then
	2722	printf "\n +++ errors found in the parameter file!\n"
	2723	answer=dummy
	2724	printf "\n\n"
	2725
	2726	while [[ "$answer" != c && "$answer" != C && "$answer" != a && "$answer" != A ]]
	2727	do
	2728	printf " >>> continue anyway (c(ontinue)/a(bort)) ? "
	2729	read answer
	2730	done
	2731
	2732	if [[ $answer = a \|\| $answer = A ]]
	2733	then
	2734	printf "\n +++ Aborting..."
	2735	rm -rf $check_sources
	2736	locat=normal; exit
	2737	fi
	2738	else
	2739	printf "\n\n *** parameter file(s) seem(s) to be o.k.\n"
	2740	fi
	2741	rm -rf $check_sources
	2742	fi
	2743	else
	2744	printf "\n\n +++ skipping parameter file check\n"
	2745	fi
	2746
[818]	2747	# DELETE TEMPORARY DIRECTORY AND FINISH NAMELIST FILE CHECK
	2748	rm -rf $check_sources
[807]	2749	cd $working_directory
	2750
	2751	# FALLS AUF DIESER MASCHINE GERECHNET WERDEN SOLL, WERDEN JETZT ENTSPRE-
	2752	# CHENDE AKTIONEN DURCHGEFUEHRT
[1]	2753	if [[ $do_batch = false ]]
	2754	then
	2755
	2756
	2757	# TEMPORAEREN KATALOG ERZEUGEN
	2758	mkdir -p $TEMPDIR
	2759	chmod go+rx $TEMPDIR
	2760	tmpcreate=true
	2761
[493]	2762	# set striping on lustre file system
	2763	# if [[ $localhost = lcsgih ]]
	2764	# then
	2765	# lfs setstripe -s 8192k -c 16 $TEMPDIR
	2766	# lfs getstripe $TEMPDIR
	2767	# fi
[1]	2768
[493]	2769
[1]	2770	# SAEMTLICHE QUELLTEXT-DATEIEN BZW. AUSFUEHRBARES PROGRAMM IN
	2771	# TEMPORAERES VERZEICHNIS KOPIEREN
	2772	if [[ $do_compile = true ]]
	2773	then
	2774
	2775	# ON NEC, COMPILATION IS DONE ON HOST CROSS VIA CROSS COMPILING
	2776	# CREATE A TEMPORARY DIRECTORY ON THAT MACHINE (HOME MOUNTED VIA NFS)
	2777	if [[ $localhost = nech ]]
	2778	then
	2779	TEMPDIR_COMPILE=$HOME/work/${usern}.$kennung
	2780	if mkdir -p $TEMPDIR_COMPILE
	2781	then
	2782	printf "\n *** \"$TEMPDIR_COMPILE\" "
	2783	printf "\n is generated as temporary directory for cross compiling\n"
	2784	else
	2785	printf "\n +++ creating directory \"$TEMPDIR_COMPILE\" "
	2786	printf "\n needed for cross compilation failed"
	2787	locat=compile
	2788	exit
	2789	fi
	2790	else
	2791	TEMPDIR_COMPILE=$TEMPDIR
	2792	fi
	2793
	2794
[82]	2795	# PFADNAMEN FUER DAS MAKE-DEPOSITORY ERMITTELN
	2796	line=""
	2797	grep "%depository_path" $config_file > tmp_mrun
	2798	while read line
	2799	do
	2800	if [[ "$line" != "" && $(echo $line \| cut -c1) != "#" ]]
	2801	then
	2802	if [[ "$(echo $line \| cut -d" " -s -f3)" = "" ]]
[1]	2803	then
[82]	2804	global_depository_path=`echo $line \| cut -d" " -s -f2`
[1]	2805	fi
[82]	2806	fi
	2807	done < tmp_mrun
[1]	2808
[82]	2809	line=""
	2810	grep " $localhost" $config_file \| grep "%depository_path" > tmp_mrun
	2811	while read line
	2812	do
	2813	if [[ "$line" != "" && $(echo $line \| cut -c1) != "#" ]]
[1]	2814	then
[211]	2815	if [[ "$(echo $line \| cut -d" " -s -f4)" = "$cond1" && "$(echo $line \| cut -d" " -s -f5)" = "$cond2" ]]
	2816	then
	2817	local_depository_path=`echo $line \| cut -d" " -s -f2`
	2818	fi
[1]	2819	fi
[82]	2820	done < tmp_mrun
[1]	2821
[82]	2822	if [[ "$local_depository_path" = "" ]]
	2823	then
	2824	if [[ "$global_depository_path" != "" ]]
[1]	2825	then
[82]	2826	local_depository_path=$global_depository_path
[1]	2827	else
[82]	2828	printf "\n\n +++ no depository path found in configuration file"
	2829	printf "\n for local host \"$localhost\" "
	2830	printf "\n please set \"\%depository_path\" in configuration file\n"
	2831	locat=config_file; exit
[1]	2832	fi
[82]	2833	fi
	2834	eval local_depository_path=$local_depository_path
[215]	2835	[[ "$cond1" != "" ]] && local_depository_path=${local_depository_path}_$cond1
	2836	[[ "$cond2" != "" ]] && local_depository_path=${local_depository_path}_$cond2
[1]	2837
[215]	2838
[82]	2839	basename=`print $mainprog \| cut -f1 -d"."`
	2840	eval make_depository=${local_depository_path}/${basename}_current_version.tar
	2841	if [[ ! -f $make_depository ]]
	2842	then
	2843	printf "\n"
	2844	printf "\n *** WARNING: make depository \"$make_depository\" not found"
	2845	printf "\n \"make\" will fail, if the Makefile or other source files are missing\n"
	2846	else
	2847	cp $make_depository $TEMPDIR_COMPILE
[1]	2848	cd $TEMPDIR_COMPILE
[82]	2849	tar -xf $make_depository > /dev/null 2>&1
[1]	2850	cd - > /dev/null
	2851	fi
[82]	2852
	2853	cp SOURCES_FOR_RUN_$fname/* $TEMPDIR_COMPILE
	2854
[1]	2855	else
[475]	2856
[1]	2857	cp $executable ${TEMPDIR}/a.out
[475]	2858
[1]	2859	fi
	2860
	2861
	2862	# WECHSEL IN TEMPORAEREN KATALOG
[475]	2863	cd $TEMPDIR
[1]	2864	printf "\n *** changed to temporary directory: $TEMPDIR"
	2865
	2866
	2867	# OUTPUT-DATEI-VERBINDUNGEN AUF TEMPORAERER DATEI ABLEGEN
	2868	# DIESE DATEI KANN VON SPAETER AUFZURUFENDEN BENUTZERPROZEDUREN GELESEN
	2869	# WERDEN, UM ZU LOKALEN DATEINAMEN GEHOERENDE PERMANENTE NAMEN ZU
	2870	# ERMITTELN
	2871	(( i = 0 ))
	2872	while (( i < iout ))
	2873	do
	2874	(( i = i + 1 ))
	2875	if [[ "${actionout[$i]}" = tr \|\| "${actionout[$i]}" = tra \|\| "${actionout[$i]}" = trpe ]]
	2876	then
	2877	printf "${localout[$i]} ${actionout[$i]}\n${pathout[$i]}\n${localhost}_${fname}${endout[$i]}\n" >> OUTPUT_FILE_CONNECTIONS
	2878	else
	2879	printf "${localout[$i]} ${actionout[$i]}\n${pathout[$i]}\n${frelout[$i]}\n" >> OUTPUT_FILE_CONNECTIONS
	2880	fi
	2881	done
	2882
	2883
	2884	# EVTL. UEBERSETZUNGSAKTIONEN STARTEN
	2885	if [[ $do_compile = true ]]
	2886	then
	2887
	2888
	2889	# COMPILING WITH MAKE (ON NEC COMPILER IS CALLED ON HOST CROSS)
[82]	2890	printf "\n\n\n *** compilation starts \n$striche\n"
	2891	printf " *** compilation with make using following options:\n"
[211]	2892	printf " make depository: $make_depository"
[475]	2893	if [[ "$mopts" != "" ]]
	2894	then
	2895	printf " make options: $mopts\n"
	2896	fi
[82]	2897	printf " compilername: $compiler_name\n"
[475]	2898	printf " compiler options: $fopts\n"
[493]	2899	printf " preprocessor directives: $cpp_options \n"
	2900	printf " linker options: $lopts \n"
	2901	if [[ "$modules" != "" ]]
	2902	then
	2903	printf " modules to be load: $modules \n"
	2904	fi
[82]	2905	printf " source code files: $source_list \n"
	2906
	2907	if [[ $localhost = nech ]]
[1]	2908	then
[493]	2909	ssh 136.172.44.192 -l $usern ". /SX/opt/etc/initsx.sh; $module_calls cd \$HOME/work/${usern}.$kennung; sxmake $mopts -f Makefile PROG=a.out F90=$compiler_name COPT=\"$cpp_options\" F90FLAGS=\"$fopts\" LDFLAGS=\"$lopts\" "
[82]	2910	cp $TEMPDIR_COMPILE/a.out .
	2911	[[ $? != 0 ]] && compile_error=true
	2912	rm -rf $TEMPDIR_COMPILE
[312]	2913	elif [[ $localhost = ibmh ]]
	2914	then
[325]	2915	printf " compiler is called via ssh on \"plogin1\" \n"
[493]	2916	ssh plogin1 -l $usern "export PATH=/sw/ibm/xlf/12.1.0.3/usr/bin:$PATH; $module_calls cd $TEMPDIR; make $mopts -f Makefile PROG=a.out F90=$compiler_name COPT=\"$cpp_options\" F90FLAGS=\"$fopts\" LDFLAGS=\"$lopts\" "
[312]	2917	[[ ! -f a.out ]] && compile_error=true
	2918	continue # ANDERENFALLS IST STATUS=1, FALLS A.OUT VORHANDEN
[197]	2919	elif [[ $localhost = lcsgib ]]
	2920	then
[540]	2921	printf " compiler is called via ssh on \"bicegate0\" \n"
	2922	ssh bicegate0 -l $usern ". /usr/share/modules/init/bash; $module_calls cd $TEMPDIR; make $mopts -f Makefile PROG=a.out F90=$compiler_name COPT=\"$cpp_options\" F90FLAGS=\"$fopts\" LDFLAGS=\"$lopts\" "
[197]	2923	[[ ! -f a.out ]] && compile_error=true
	2924	continue # ANDERENFALLS IST STATUS=1, FALLS A.OUT VORHANDEN
	2925	elif [[ $localhost = lcsgih ]]
	2926	then
[738]	2927	printf " compiler is called via ssh on \"hicegate0\" \n"
	2928	ssh hicegate0 -l $usern ". /usr/share/modules/init/bash; $module_calls cd $TEMPDIR; make $mopts -f Makefile PROG=a.out F90=$compiler_name COPT=\"$cpp_options\" F90FLAGS=\"$fopts\" LDFLAGS=\"$lopts\" 2>&1 "
[197]	2929	[[ ! -f a.out ]] && compile_error=true
	2930	continue # ANDERENFALLS IST STATUS=1, FALLS A.OUT VORHANDEN
[755]	2931	elif [[ $localhost = lcflow ]]
	2932	then
	2933	printf " compiler is called via ssh on \"flow\" \n"
	2934	ssh flow02.hpc.uni-oldenburg.de -l $usern ". /cm/local/apps/environment-modules/3.2.6/Modules/3.2.6/init/bash; echo $MODULEPATH; echo $MODULESHOME; MODULEPATH=/cm/shared/modulefiles/; module load netcdf/intel/64/4.1.3 intel/ics/64/2011.0.013 intel/impi/64/4.0.1.007 hdf5/intel/64/1.8.7; cd $TEMPDIR; make $mopts -f Makefile PROG=a.out F90=$compiler_name COPT=\"$cpp_options\" F90FLAGS=\"$fopts\" LDFLAGS=\"$lopts\" "
	2935	[[ ! -f a.out ]] && compile_error=true
	2936	continue # ANDERENFALLS IST STATUS=1, FALLS A.OUT VORHANDEN
[1]	2937	else
[544]	2938	[[ $localhost = lcxt4 ]] && . /opt/modules/default/init/ksh
[493]	2939	[[ "$module_calls" != "" ]] && eval $module_calls
[475]	2940	make $mopts -f Makefile PROG=a.out F90=$compiler_name COPT="$cpp_options" F90FLAGS="$fopts" LDFLAGS="$lopts"
[807]	2941
[82]	2942	fi
[1]	2943
	2944	if [[ $? != 0 \|\| "$compile_error" = true \|\| "$module_compile_error" = true ]]
	2945	then
	2946	printf "\n +++ error occured while compiling or linking"
	2947	locat=compile
[181]	2948
[182]	2949	# WORKAROUND: REMOVE IF CONSTRUCT LATER, BUT KEEP THE EXIT!
[197]	2950	# if [[ $localhost != lcsgib && $localhost != lcsgih ]]
	2951	# then
[181]	2952	exit
[197]	2953	# else
	2954	# locat=normal
	2955	# fi
[1]	2956	else
	2957	printf "$striche\n *** compilation finished \n"
	2958	fi
	2959	fi
	2960
	2961
	2962	# FALLS NUR TESTWEISE KOMPILIERT WERDEN SOLLTE, IST MRUN JETZT FERTIG
	2963	if [[ $local_compile = true ]]
	2964	then
	2965	cd $HOME
	2966	rm -rf $TEMPDIR
	2967	locat=local_compile; exit
	2968	fi
	2969
	2970
	2971	# BEREITSTELLEN DER INPUT-DATEIEN
	2972	# SCHLEIFE UEBER ALLE VOM BENUTZER ANGEGEBENEN DATEIEN
	2973	(( i = 0 ))
	2974	while (( i < iin ))
	2975	do
	2976	(( i = i + 1 ))
	2977	if (( i == 1 ))
	2978	then
	2979	printf "\n\n *** providing INPUT-files:\n$striche"
	2980	fi
	2981
	2982
	2983	# OPTIONALE DATEIEN BEI NICHTVORHANDENSEIN UEBERGEHEN
	2984	if [[ "${transin[$i]}" = unavailable ]]
	2985	then
	2986	if [[ "${extin[$i]}" = "" \|\| "${extin[$i]}" = " " ]]
	2987	then
	2988	printf "\n +++ WARNING: input file \"${pathin[$i]}/${afname}${endin[$i]}\" "
	2989	printf "\n is not available!"
	2990	else
	2991	printf "\n +++ WARNING: input file \"${pathin[$i]}/${afname}${endin[$i]}.${extin[$i]}\" "
	2992	printf "\n is not available!"
	2993	fi
	2994	continue
	2995	fi
	2996
	2997	# PRUEFEN, OB EINZELDATEI ODER DATEI PRO PROZESSOR
	2998	files_for_pes=false; datentyp=file
	2999	if [[ "${actionin[$i]}" = pe && -n $numprocs ]]
	3000	then
	3001	files_for_pes=true; datentyp=directory
	3002	actionin[$i]=""
	3003	elif [[ "${actionin[$i]}" = pe && ! -n $numprocs ]]
	3004	then
	3005	actionin[$i]=""
	3006	elif [[ "${actionin[$i]}" = arpe && -n $numprocs ]]
	3007	then
	3008	files_for_pes=true; datentyp=directory
	3009	actionin[$i]="ar"
	3010	elif [[ "${actionin[$i]}" = arpe && ! -n $numprocs ]]
	3011	then
	3012	actionin[$i]="ar"
	3013	elif [[ "${actionin[$i]}" = flpe && -n $numprocs ]]
	3014	then
	3015	files_for_pes=true; datentyp=directory
	3016	actionin[$i]="fl"
	3017	elif [[ "${actionin[$i]}" = flpe && ! -n $numprocs ]]
	3018	then
	3019	actionin[$i]="fl"
	3020	fi
	3021
	3022	if [[ $files_for_pes = true ]]
	3023	then
	3024	printf "\n >>> INPUT: ${absnamein[$i]}/.... to ${localin[$i]}"
	3025	else
	3026	printf "\n >>> INPUT: ${absnamein[$i]} to ${localin[$i]}"
	3027	fi
	3028
	3029	# INPUT-DATEI FUER EINEN FORTSETZUNGSLAUF. ES WIRD GEPRUEFT,
	3030	# OB DIESE DATEI NOCH AUF DEM TEMPORAEREN DATENKATALOG VORHANDEN
	3031	# IST. FALLS NICHT, WIRD VERSUCHT, SIE ANSCHLIESSEND VOM ARCHIV-
	3032	# SERVER ZU HOLEN
	3033	if [[ "${actionin[$i]}" = fl ]]
	3034	then
	3035	printf "\n $datentyp will be fetched from temporary directory \"${tmp_data_catalog}\" !"
	3036	if [[ $files_for_pes = false ]]
	3037	then
	3038	if [[ -f "$tmp_data_catalog/${frelin[$i]}" ]]
	3039	then
	3040	ln $tmp_data_catalog/${frelin[$i]} ${localin[$i]}
	3041	got_tmp[$i]=true
	3042	elif [[ -f "$WORK/${frelin[$i]}" && $ignore_archive_error = true ]]
	3043	then
	3044	printf "\n +++ $datentyp not found in \"$tmp_data_catalog\" !"
	3045	printf "\n *** trying to use backup copy in \"$WORK\" "
	3046	cp $WORK/${frelin[$i]} ${localin[$i]}
	3047	else
	3048	printf "\n +++ $datentyp not found in \"$tmp_data_catalog\" "
	3049	printf "\n or \"$tmp_data_catalog\" does not exist!"
	3050	printf "\n *** trying to get copy from archive"
	3051	actionin[$i]=ar
	3052	fi
	3053	else
	3054	if [[ -d "$tmp_data_catalog/${frelin[$i]}" ]]
	3055	then
	3056	mkdir ${localin[$i]}
[149]	3057	cd $tmp_data_catalog/${frelin[$i]}
	3058	for file in $(ls *)
	3059	do
	3060	ln $file $TEMPDIR/${localin[$i]}
	3061	done
	3062	cd $TEMPDIR
[1]	3063	got_tmp[$i]=true
	3064	elif [[ -d "$WORK/${frelin[$i]}" && $ignore_archive_error = true ]]
	3065	then
	3066	printf "\n +++ $datentyp not found in \"$tmp_data_catalog\" !"
	3067	printf "\n *** trying to use backup copy in \"$WORK\" "
	3068	cp -r $WORK/${frelin[$i]} ${localin[$i]}
	3069	else
	3070	printf "\n +++ $datentyp not found in \"$tmp_data_catalog\" "
	3071	printf "\n or \"$tmp_data_catalog\" does not exist!"
	3072	printf "\n *** trying to get copy from archive"
	3073	actionin[$i]=ar
	3074	fi
	3075	fi
	3076	fi
	3077
	3078
	3079	# DATEI LIEGT AUF ARCHIV-SERVER
	3080	if [[ "${actionin[$i]}" = ar ]]
	3081	then
	3082
	3083	if [[ $files_for_pes = false ]]
	3084	then
	3085	printf "\n file will be restored from archive-system ($archive_system)!"
	3086	else
	3087	printf "\n directory will be restored from archive-system ($archive_system)!"
	3088	fi
	3089
	3090	file_restored=false
	3091
	3092	if [[ $archive_system = asterix ]]
	3093	then
	3094	do_stagein=true
	3095	(( stagein_anz = 0 ))
	3096	while [[ $do_stagein = true ]]
	3097	do
	3098	if [[ $files_for_pes = false ]]
	3099	then
	3100	stagein -O ${frelin[$i]} > STAGEIN_OUTPUT
	3101	else
	3102	stagein -t -O ${frelin[$i]} > STAGEIN_OUTPUT
	3103	fi
	3104	cat STAGEIN_OUTPUT
	3105	if [[ $(grep -c "st.msg:i24" STAGEIN_OUTPUT) != 0 ]]
	3106	then
	3107	file_restored=true
	3108	do_stagein=false
	3109	else
	3110	(( stagein_anz = stagein_anz + 1 ))
	3111	if (( stagein_anz == 10 ))
	3112	then
	3113	printf "\n +++ stagein stoped after 10 tries"
	3114	locat=stage
	3115	exit
	3116	fi
	3117	printf "\n +++ restoring from archive failed, trying again:"
	3118	sleep 900
	3119	fi
	3120	done
	3121	elif [[ $archive_system = DMF ]]
	3122	then
	3123	if [[ $files_for_pes = false ]]
	3124	then
	3125	printf "\n +++ restoring of single files impossible with $archive_system !\n"
	3126	locat=DMF
	3127	exit
	3128	else
	3129	find $ARCHIVE/${frelin[$i]} -type m -print \| dmget
	3130	cp -r $ARCHIVE/${frelin[$i]} $PWD
	3131	file_restored=true
	3132	fi
	3133	elif [[ $archive_system = tivoli ]]
	3134	then
	3135	if [[ $files_for_pes = false ]]
	3136	then
[399]	3137	if [[ $localhost = lcsgih ]]
[1]	3138	then
[399]	3139	ssh $usern@hicedata.hlrn.de "cp $PERM/${frelin[$i]} $PWD"
[1]	3140	else
[399]	3141	ssh $usern@bicedata.hlrn.de "cp $PERM/${frelin[$i]} $PWD"
[1]	3142	fi
	3143	else
	3144	(( inode = 0 ))
	3145	while (( inode < nodes ))
	3146	do
[399]	3147	if [[ $localhost = lcsgih ]]
[1]	3148	then
[399]	3149	ssh $usern@hicedata.hlrn.de "cd $PWD; tar xf $PERM/${frelin[$i]}/${frelin[$i]}.node_$inode.tar"
[1]	3150	else
[399]	3151	ssh $usern@bicedata.hlrn.de "cd $PWD; tar xf $PERM/${frelin[$i]}/${frelin[$i]}.node_$inode.tar"
[1]	3152	fi
	3153	(( inode = inode + 1 ))
	3154	done
	3155	fi
	3156	file_restored=true
	3157	elif [[ $archive_system = ut ]]
	3158	then
	3159	if [[ $files_for_pes = false ]]
	3160	then
	3161	cp $UT/${frelin[$i]} .
	3162	else
	3163	(( inode = 0 ))
	3164	while (( inode < nodes ))
	3165	do
	3166	tar xf $UT/${frelin[$i]}/${frelin[$i]}.node_$inode.tar
	3167	(( inode = inode + 1 ))
	3168	done
	3169	fi
	3170	file_restored=true
	3171	else
	3172	printf "\n +++ archive_system=\"$archive_system\" restore impossible!"
	3173	locat=rearchive
	3174	exit
	3175	fi
	3176
	3177	if [[ $file_restored = true ]]
	3178	then
	3179
	3180	# DATEI AUCH AUF TEMPORAERES DATENVERZEICHNIS LEGEN, DAMIT
	3181	# SIE BEI WEITEREN ZUGRIFFEN NOCH VORHANDEN IST
	3182	[[ ! -d $tmp_data_catalog ]] && mkdir -p $tmp_data_catalog; chmod g+rx $tmp_data_catalog
	3183	if [[ $files_for_pes = false ]]
	3184	then
	3185	ln -f ${frelin[$i]} $tmp_data_catalog/${frelin[$i]}
	3186	else
	3187	mkdir $tmp_data_catalog/${frelin[$i]}
	3188	ln -f ${frelin[$i]}/* $tmp_data_catalog/${frelin[$i]}
	3189	fi
	3190	got_tmp[$i]=true
	3191
	3192	# DATEI UNTER LOKALEM NAMEN ZUR VERFUEGUNG STELLEN
	3193	mv ${frelin[$i]} ${localin[$i]}
	3194
	3195	fi
	3196	fi
	3197
	3198
	3199	# DATEI LIEGT IM VOM BENUTZER ANGEGEBENEN VERZEICHNIS
	3200	if [[ "${actionin[$i]}" = "" \|\| "${actionin[$i]}" = "di" \|\| "${actionin[$i]}" = "npe" ]]
	3201	then
	3202
	3203	if [[ "${actionin[$i]}" = "npe" && -n $numprocs ]]
	3204	then
	3205
	3206	# DATEI WIRD FUER DIE PROZESSOREN EINES PARALLERECHNERS BEREITGESTELLT
	3207	printf "\n file will be provided for $numprocs processors"
	3208	mkdir ${localin[$i]}
	3209	ival=$numprocs
	3210	(( ii = 0 ))
	3211	while (( ii <= ival-1 ))
	3212	do
	3213	if (( ii < 10 ))
	3214	then
	3215	cp ${absnamein[$i]} ${localin[$i]}/_000$ii
	3216	elif (( ii < 100 ))
	3217	then
	3218	cp ${absnamein[$i]} ${localin[$i]}/_00$ii
	3219	elif (( ii < 1000 ))
	3220	then
	3221	cp ${absnamein[$i]} ${localin[$i]}/_0$ii
	3222	else
	3223	cp ${absnamein[$i]} ${localin[$i]}/_$ii
	3224	fi
	3225	(( ii = ii + 1 ))
	3226	done
	3227
	3228	else
	3229
	3230	if [[ $files_for_pes = true ]]
	3231	then
	3232
	3233	# DIE DEN PROZESSOREN EINES PARALLELRECHNERS ZUGEHOERIGEN
	3234	# DATEIEN WERDEN BEREITGESTELLT, INDEM ZUERST DER GESAMTE
	3235	# KATALOGINHALT KOPIERT UND DANN DIE EINZELNEN DATEIEN
	3236	# PER MOVE UMBENANNT WERDEN
	3237	printf "\n providing $numprocs files for the respective processors"
	3238	mkdir ${localin[$i]}
[183]	3239	if [[ $link_local_input = true ]]
	3240	then
	3241	printf " using ln -f\n"
	3242	cd ${absnamein[$i]}
	3243	for file in $(ls *)
	3244	do
	3245	ln -f $file ${localin[$i]}
	3246	done
	3247	cd $TEMPDIR
	3248	fi
	3249	# If "ln -f" fails of if "$link_local_input = false" do a normal "cp -r"
	3250	if [[ ! -f "${localin[$i]}/_0000" ]]
	3251	then
	3252	if [[ $link_local_input = true ]]
	3253	then
	3254	printf " ln failed for .../_0000, using cp...\n"
	3255	fi
	3256	cp -r ${absnamein[$i]}/* ${localin[$i]}
	3257	fi
[1]	3258
	3259	else
	3260	# BEREITSTELLUNG AUF EINPROZESSORRECHNERN
[183]	3261	if [[ $link_local_input = true ]]
	3262	then
	3263	printf " using ln -f\n"
	3264	ln -f ${absnamein[$i]} ${localin[$i]}
	3265	fi
	3266	# If "ln -f" fails of if "$link_local_input = false" do a normal "cp"
	3267	if [[ ! -f "${localin[$i]}" ]]
	3268	then
	3269	if [[ $link_local_input = true ]]
	3270	then
	3271	printf " ln failed, using cp...\n"
	3272	fi
	3273	cp ${absnamein[$i]} ${localin[$i]}
	3274	fi
[1]	3275	fi
	3276	fi
	3277	fi
	3278
	3279	done
	3280	if (( i != 0 ))
	3281	then
	3282	printf "\n$striche\n *** all INPUT-files provided \n"
	3283	fi
	3284
	3285
	3286	# EVENTUELLE INPUT-KOMMANDOS ABARBEITEN
	3287	(( i = 0 ))
	3288	while (( i < iic ))
	3289	do
	3290	(( i = i + 1 ))
	3291	if (( i == 1 ))
	3292	then
	3293	printf "\n\n *** execution of INPUT-commands:\n$striche"
	3294	fi
	3295	printf "\n >>> ${in_command[$i]}"
	3296	eval ${in_command[$i]}
	3297	if (( i == iic ))
	3298	then
	3299	printf "\n$striche\n"
	3300	fi
	3301	done
	3302
	3303
	3304	# VERBLEIBENDE CPU-ZEIT BERECHNEN
	3305	cpurest=${cpumax}.
	3306
	3307
[260]	3308	# START DVR STREAMING SERVER
	3309	if [[ $(echo $package_list \| grep -c dvrp_graphics) != 0 ]]
	3310	then
	3311	if [[ "$dvr_server" != "" ]]
	3312	then
	3313
	3314	printf "\n\n *** preparing the dvr streaming server configuration file"
	3315
	3316	# Check, if a dvr server is already running
	3317	running_dvrserver_id=`echo $(ps -edaf \| grep .dvrserver.config \| grep -v grep) \| cut -d" " -f2`
	3318	if [[ "$running_dvrserver_id" != "" ]]
	3319	then
	3320
[261]	3321	printf "\n\n +++ WARNING: A dvr server with id=$running_dvrserver_id is already running!"
[260]	3322	printf "\n This server is used instead starting a new one!"
[261]	3323	printf "\n If required, script \"process_dvr_output\" has to be run manually."
[260]	3324
	3325	else
	3326
	3327	# COPY CONFIGURATION FILE FOR STREAMING SERVER FROM REPOSITORY TO HERE
	3328	if [[ -f ${PALM_BIN}/.dvrserver.config ]]
	3329	then
	3330	cp ${PALM_BIN}/.dvrserver.config .
	3331
	3332	# Entering the BASEDIR, UID and GID into this file
	3333	user_id=`id -u`
	3334	group_id=`id -g`
	3335	# & is needed as seperator, because TEMPDIR contains /
	3336	sed "s&<replace by dvr data directory>&${TEMPDIR}&g" .dvrserver.config > .dvrserver.1
	3337	sed "s/<replace by user id>/$user_id/g" .dvrserver.1 > .dvrserver.2
	3338	sed "s/<replace by group id>/$group_id/g" .dvrserver.2 > .dvrserver.3
	3339	mv .dvrserver.3 .dvrserver.config
	3340	rm .dvrserver.1 .dvrserver.2
	3341
	3342	# Start dvr server in background, get his id and print on terminal
	3343	$dvr_server .dvrserver.config >> DVR_LOGFILE 2>&1 &
	3344	dvrserver_id=`echo $(ps -edaf \| grep .dvrserver.config) \| cut -d" " -f2`
	3345	printf "\n *** streaming server with id=$dvrserver_id is started in background"
	3346	local_dvrserver_running=.TRUE.
	3347	else
	3348	printf "\n +++ missing file \".dvrserver.config\" in directory:"
	3349	printf "\n \"$PALM_BIN\" "
	3350	locat=dvr
	3351	exit
	3352	fi
	3353
	3354	fi
	3355
	3356	else
	3357	printf "\n\n --- INFORMATIVE: no dvr streaming server will be started"
	3358	fi
	3359	fi
	3360
	3361
	3362	# NAMELIST-DATEI MIT WERTEN VON ENVIRONMENT-VARIABLEN ERZEUGEN (ZU
	3363	# LESEN VON PALM)
	3364	cat > ENVPAR << %%END%%
	3365	&envpar run_identifier = '$fname', host = '$localhost',
	3366	write_binary = '$write_binary', tasks_per_node = $tasks_per_node,
[759]	3367	maximum_parallel_io_streams = $maximum_parallel_io_streams,
[260]	3368	maximum_cpu_time_allowed = ${cpumax}.,
	3369	revision = '$global_revision',
	3370	local_dvrserver_running = $local_dvrserver_running /
	3371
	3372	%%END%%
	3373
	3374
[1]	3375	# PROGRAMMSTART
	3376	printf "\n\n *** execution starts in directory\n \"`pwd`\"\n$striche\n"
	3377	PATH=$PATH:$TEMPDIR
	3378
[359]	3379	# MPI debug option (argument checking, slows down execution due to increased latency)
	3380	if [[ "$mpi_debug" = true ]]
	3381	then
	3382	export MPI_CHECK_ARGS=1
	3383	printf "\n +++ MPI_CHECK_ARGS=$MPI_CHECK_ARGS"
	3384	fi
	3385
[693]	3386	if [[ "$totalview" = true ]]
	3387	then
	3388	printf "\n *** totalview debugger will be used"
	3389	tv_opt="-tv"
	3390	else
	3391	tv_opt=""
	3392	fi
	3393
[1]	3394	if [[ "$cond1" = debug \|\| "$cond2" = debug ]]
	3395	then
	3396	if [[ "$ENVIRONMENT" = BATCH ]]
	3397	then
[211]	3398	if [[ $(echo $localhost \| cut -c1-5) != lcsgi ]]
	3399	then
	3400	printf "\n +++ debug is allowed in interactive mode only"
	3401	locat=debug
	3402	exit
	3403	fi
[1]	3404	fi
[5]	3405	if [[ $localhost = decalpha ]]
[1]	3406	then
	3407	dxladebug a.out
[622]	3408	elif [[ $localhost = ibmh ]]
[1]	3409	then
	3410
	3411	# SETUP THE IBM MPI ENVIRONMENT
	3412	export MP_SHARED_MEMORY=yes
	3413	export AIXTHREADS_SCOPE=S
	3414	export OMP_NUM_THREADS=$threads_per_task
	3415	export AUTHSTATE=files
	3416	export XLFRTEOPTS="nlwidth=132:err_recovery=no" # RECORD-LENGTH OF NAMELIST-OUTPUT
	3417
	3418	# FOLLOWING OPTIONS ARE MANDATORY FOR TOTALVIEW
	3419	export MP_ADAPTER_USE=shared
	3420	export MP_CPU_USE=multiple
	3421	export MP_TIMEOUT=1200
	3422
	3423	unset MP_TASK_AFFINITY
	3424
	3425	# SO FAR, TOTALVIEW NEEDS HOSTFILE MECHANISM FOR EXECUTION
	3426	#(( ii = 1 ))
	3427	#while (( ii <= $numprocs ))
	3428	#do
	3429	# echo $localhost_realname >> hostfile
	3430	# (( ii = ii + 1 ))
	3431	#done
	3432	#export MP_HOSTFILE=hostfile
	3433
	3434	if [[ "$LOADLBATCH" = yes ]]
	3435	then
	3436	totalview poe a.out $ROPTS
	3437	else
	3438	echo totalview poe -a a.out -procs $numprocs -rmpool 0 -nodes 1 $ROPTS
	3439	export TVDSVRLAUNCHCMD=ssh
	3440	totalview poe -a a.out -procs $numprocs -rmpool 0 -nodes 1 $ROPTS
	3441	fi
[211]	3442	elif [[ $(echo $localhost \| cut -c1-5) = lcsgi ]]
	3443	then
	3444	# CURRENTLY NO DEBUGGER ON LCSGI
	3445	if [[ $run_coupled_model = true ]]
	3446	then
	3447	printf "\n +++ no debug in coupled mode available on \"$localhost\" "
	3448	locat=debug
	3449	exit
[223]	3450	else
	3451	echo "no_coupling" > runfile_atmos
[211]	3452	fi
	3453	(( ii = $numprocs / $threads_per_task ))
	3454	export OMP_NUM_THREADS=$threads_per_task
	3455	echo "OMP_NUM_THREADS=$OMP_NUM_THREADS"
	3456	if [[ $threads_per_task != 1 ]]
	3457	then
	3458	printf "\n threads per task: $threads_per_task"
	3459	fi
	3460	printf "\n\n"
[223]	3461	if [[ $( echo $mpilib \| cut -c1-3 ) = mpt ]]
	3462	then
	3463	# export MPI_LAUNCH_TIMEOUT=360
	3464	if [[ "$totalview" = true ]]
	3465	then
	3466	printf "\n running totalview debugger"
[693]	3467	mpiexec_mpt $tv_opt -n $ii ./a.out $ROPTS < runfile_atmos
[223]	3468	else
	3469	mpiexec_mpt -np $ii ./a.out $ROPTS < runfile_atmos
	3470	fi
	3471	elif [[ $( echo $mpilib \| cut -c1-3 ) = mva ]]
	3472	then
	3473	# ulimit -s 300000 # A too large stack size causes problems
	3474	# export MV2_NUM_PORTS=2
	3475	# export MV2_CPU_MAPPING=0:1:2:3
[362]	3476	if [[ "$totalview" = true ]]
	3477	then
	3478	printf "\n running totalview debugger"
[693]	3479	mpiexec $tv_opt ./a.out $ROPTS < runfile_atmos
[362]	3480	else
	3481	mpiexec ./a.out $ROPTS < runfile_atmos
	3482	fi
[223]	3483	fi
[1]	3484	else
	3485	printf "\n +++ no debug available on \"$localhost\" "
	3486	printf "\n or not implemented in mrun so far"
	3487	locat=debug
	3488	exit
	3489	fi
[622]	3490
	3491	# end debug mode
[1]	3492	else
[622]	3493
	3494	# normal execution
[1]	3495	if [[ -n $numprocs ]]
	3496	then
	3497	# RUNNING THE PROGRAM ON PARALLEL MACHINES
	3498	if [[ $(echo $host \| cut -c1-3) = ibm ]]
	3499	then
	3500	# SETUP THE IBM MPI ENVIRONMENT
[693]	3501	if [[ $host != ibmh && $host != ibmkisti ]]
[312]	3502	then
	3503	export MP_SHARED_MEMORY=yes
	3504	export AIXTHREAD_SCOPE=S
	3505	export OMP_NUM_THREADS=$threads_per_task
	3506	export XLSMPOPTS="spins=0:yields=0:stack=20000000"
	3507	export AUTHSTATE=files
	3508	export XLFRTEOPTS="nlwidth=132:err_recovery=no" # RECORD-LENGTH OF NAMELIST-OUTPUT
	3509	# export MP_PRINTENV=yes
[1]	3510
[312]	3511	# TUNING-VARIABLEN ZUR VERBESSERUNG DER KOMMUNIKATION
	3512	# ZEIGEN ABER DERZEIT (SEP 04, FEDERATION) KAUM WIRKUNG
	3513	export MP_WAIT_MODE=poll
	3514	[[ $node_usage = not_shared ]] && export MP_SINGLE_THREAD=yes
	3515	fi
[693]	3516
	3517	if [[ $host = ibmkisti ]]
	3518	then
	3519	export LANG=en_US
	3520	export MP_SHARED_MEMORY=yes
[696]	3521	if [[ $threads_per_task = 1 ]]
	3522	then
	3523	export MP_SINGLE_THREAD=yes
	3524	export MEMORY_AFFINITY=MCM
	3525	else
	3526	export OMP_NUM_THREADS=$threads_per_task
	3527	fi
[693]	3528	fi
	3529
[1]	3530	if [[ "$LOADLBATCH" = yes ]]
	3531	then
	3532	printf "\n--- Control: OMP_NUM_THREADS = \"$OMP_NUM_THREADS\" \n"
	3533	if [[ "$cond1" = hpmcount \|\| "$cond2" = hpmcount ]]
	3534	then
[102]	3535	/opt/optibm/HPM_2_4_1/bin/hpmcount a.out $ROPTS
[1]	3536	else
[343]	3537	if [[ $run_coupled_model = false ]]
	3538	then
	3539	if [[ "$ocean_file_appendix" = true ]]
	3540	then
	3541	echo "precursor_ocean" > runfile_atmos
	3542	else
	3543	echo "precursor_atmos" > runfile_atmos
	3544	fi
	3545	else
	3546	(( iia = $numprocs_atmos / $threads_per_task ))
	3547	(( iio = $numprocs_ocean / $threads_per_task ))
	3548	printf "\n coupled run ($iia atmosphere, $iio ocean)"
	3549	printf "\n using $coupled_mode coupling"
	3550	printf "\n\n"
	3551	echo "coupled_run $iia $iio" > runfile_atmos
	3552	fi
	3553	poe ./a.out $ROPTS < runfile_atmos
[1]	3554	fi
	3555	else
[622]	3556	if [[ $localhost = ibmh \|\| $localhost = ibms ]]
[1]	3557	then
[102]	3558	poe a.out -procs $numprocs -nodes 1 -rmpool 0 $ROPTS
[693]	3559	elif [[ $localhost = ibmkisti \|\| $localhost = ibmku \|\| $localhost = ibmy ]]
[1]	3560	then
	3561	if [[ -f $hostfile ]]
	3562	then
	3563	cp $hostfile hostfile
	3564	else
	3565	(( ii = 1 ))
	3566	while (( ii <= $numprocs ))
	3567	do
	3568	echo $localhost_realname >> hostfile
	3569	(( ii = ii + 1 ))
	3570	done
	3571	fi
	3572	export MP_HOSTFILE=hostfile
[331]	3573	if [[ $run_coupled_model = false ]]
	3574	then
	3575	if [[ "$ocean_file_appendix" = true ]]
	3576	then
	3577	echo "precursor_ocean" > runfile_atmos
	3578	else
	3579	echo "precursor_atmos" > runfile_atmos
	3580	fi
	3581	else
	3582	(( iia = $numprocs_atmos / $threads_per_task ))
	3583	(( iio = $numprocs_ocean / $threads_per_task ))
	3584	printf "\n coupled run ($iia atmosphere, $iio ocean)"
	3585	printf "\n using $coupled_mode coupling"
	3586	printf "\n\n"
	3587	echo "coupled_run $iia $iio" > runfile_atmos
	3588	fi
[622]	3589	if [[ $localhost = ibmy ]]
	3590	then
	3591	./a.out -procs $tasks_per_node $ROPTS < runfile_atmos
	3592	else
	3593	poe ./a.out -procs $numprocs $ROPTS < runfile_atmos
	3594	fi
[331]	3595
[1]	3596	else
	3597	if [[ "$host_file" = "" ]]
	3598	then
	3599	printf "\n +++ no hostfile given in configuration file"
	3600	locat=config_file
	3601	exit
	3602	else
	3603	eval host_file=$host_file
	3604	fi
	3605	export MP_HOSTFILE=$host_file
[102]	3606	poe a.out -procs $numprocs -tasks_per_node $numprocs $ROPTS
[1]	3607	fi
	3608	fi
[251]	3609	elif [[ $host = nech \|\| $host = necriam ]]
[1]	3610	then
	3611	(( ii = nodes ))
	3612	if [[ $ii = 1 ]]
	3613	then
	3614	export F_ERRCNT=0 # acceptable number of errors before program is stopped
	3615	export MPIPROGINF=YES
[61]	3616	# export F_TRACE=YES\|FMT1\|FMT2 # output of ftrace informations to job protocol
[1]	3617	echo "*** execution on single node with mpirun"
[102]	3618	mpirun -np $numprocs ./a.out $ROPTS
[1]	3619	else
	3620	(( i = 0 ))
	3621	while (( i < ii ))
	3622	do
	3623	echo "-h $i -p $tasks_per_node -e ./mpi_exec_shell" >> multinode_config
	3624	(( i = i + 1 ))
	3625	done
	3626
	3627	echo "#!/bin/sh" > mpi_exec_shell
	3628	echo " " >> mpi_exec_shell
	3629	echo "set -u" >> mpi_exec_shell
	3630	echo "F_ERRCNT=0" >> mpi_exec_shell
	3631	echo "MPIPROGINV=YES" >> mpi_exec_shell
	3632	echo "OMP_NUM_THREADS=$threads_per_task" >> mpi_exec_shell
	3633	echo "cpurest=$cpurest" >> mpi_exec_shell
	3634	echo "fname=$fname" >> mpi_exec_shell
	3635	echo "localhost=$localhost" >> mpi_exec_shell
	3636	echo "return_addres=$return_addres" >> mpi_exec_shell
	3637	echo "return_username=$return_username" >> mpi_exec_shell
	3638	echo "tasks_per_node=$tasks_per_node" >> mpi_exec_shell
	3639	echo "write_binary=$write_binary" >> mpi_exec_shell
	3640	echo "use_seperate_pe_for_dvrp_output=$use_seperate_pe_for_dvrp_output" >> mpi_exec_shell
	3641	echo " " >> mpi_exec_shell
	3642	echo "export F_ERRCNT" >> mpi_exec_shell
	3643	echo "export MPIPROGINV" >> mpi_exec_shell
	3644	echo "export OMP_NUM_THREADS" >> mpi_exec_shell
	3645	echo "export cpurest" >> mpi_exec_shell
	3646	echo "export fname" >> mpi_exec_shell
	3647	echo "export localhost" >> mpi_exec_shell
	3648	echo "export return_addres" >> mpi_exec_shell
	3649	echo "export return_username" >> mpi_exec_shell
	3650	echo "export tasks_per_node" >> mpi_exec_shell
	3651	echo "export write_binary" >> mpi_exec_shell
	3652	echo "export use_seperate_pe_for_dvrp_output" >> mpi_exec_shell
	3653	echo " " >> mpi_exec_shell
[102]	3654	echo "exec ./a.out" >> mpi_exec_shell
[1]	3655
	3656	chmod u+x mpi_exec_shell
	3657	export MPIPROGINF=YES
	3658	mpirun -f multinode_config &
	3659	wait
	3660
	3661	fi
[440]	3662	elif [[ $(echo $host \| cut -c1-2) = lc && $host != lckyoto && $host != lctit ]]
[1]	3663	then
	3664
	3665	# COPY HOSTFILE FROM SOURCE DIRECTORY OR CREATE IT, IF IT
	3666	# DOES NOT EXIST
[179]	3667	if [[ $host != lcsgih && $host != lcsgib ]]
[1]	3668	then
[179]	3669	if [[ -f $hostfile ]]
	3670	then
	3671	cp $hostfile hostfile
[746]	3672	(( ii = $numprocs / $threads_per_task ))
[179]	3673	else
	3674	(( ii = 1 ))
	3675	while (( ii <= $numprocs / $threads_per_task ))
	3676	do
	3677	echo $localhost_realname >> hostfile
	3678	(( ii = ii + 1 ))
	3679	done
	3680	fi
	3681	eval zeile=\"`head -n $ii hostfile`\"
	3682	printf "\n *** running on: $zeile"
[1]	3683	fi
[179]	3684
[1]	3685	(( ii = $numprocs / $threads_per_task ))
	3686	export OMP_NUM_THREADS=$threads_per_task
[696]	3687	# echo "*** OMP_NUM_THREADS=$OMP_NUM_THREADS"
[1]	3688	if [[ $threads_per_task != 1 ]]
	3689	then
[405]	3690	# increase stack size to unlimited, because large runs
	3691	# may abort otherwise
	3692	ulimit -Ss unlimited
	3693	printf "\n threads per task: $threads_per_task stacksize: unlimited"
[1]	3694	fi
[102]	3695	if [[ $run_coupled_model = false ]]
[1]	3696	then
[291]	3697	if [[ "$ocean_file_appendix" = true ]]
	3698	then
	3699	echo "precursor_ocean" > runfile_atmos
	3700	else
	3701	echo "precursor_atmos" > runfile_atmos
	3702	fi
[102]	3703	printf "\n\n"
[179]	3704	if [[ $host = lcsgih \|\| $host = lcsgib ]]
[164]	3705	then
[221]	3706	if [[ $( echo $mpilib \| cut -c1-3 ) = mpt ]]
	3707	then
[366]	3708	# MPI_DSM_DISTRIBUTE not necessary when MPI_DSM_CPULIST is set
	3709	# export MPI_DSM_DISTRIBUTE=1
	3710	# MPI_DSM_CPULIST: pin MPI processes to cores
[377]	3711	if [[ $use_openmp = false ]]
	3712	then
[412]	3713	if [[ "$sgi_feature" = ice2 ]]
	3714	then
	3715	export MPI_DSM_CPULIST="0,4,1,5,2,6,3,7:allhosts"
	3716	else
	3717	export MPI_DSM_CPULIST="0,1,4,5,2,3,6,7:allhosts"
	3718	fi
[622]	3719	else
	3720	unset MPI_DSM_CPULIST
[377]	3721	fi
[366]	3722	# MPI_IB_RAILS: use both IB rails on ICE2
	3723	export MPI_BUFS_PER_HOST=512
	3724	export MPI_IB_RAILS=2
[369]	3725	# NECESSARY, IF MORE THAN 4096 PEs ARE USED
	3726	export MPI_CONNECTIONS_THRESHOLD=8192
[738]	3727	# echo "*** MPI_DSM_CPULIST=$MPI_DSM_CPULIST"
[562]	3728	export MPI_TYPE_DEPTH=20
[738]	3729	# echo "*** MPI_TYPE_DEPTH=$MPI_TYPE_DEPTH"
[683]	3730	export MPI_GROUP_MAX=64
[738]	3731	# echo "*** MPI_GROUP_MAX=$MPI_GROUP_MAX"
[693]	3732	if [[ $use_openmp = true ]]
	3733	then
	3734	echo " mpiexec -npernode $tasks_per_node ./a.out $ROPTS < runfile_atmos"
	3735	mpiexec -npernode $tasks_per_node $tv_opt ./a.out $ROPTS < runfile_atmos
	3736	else
	3737	mpiexec_mpt -np $ii $tv_opt ./a.out $ROPTS < runfile_atmos
	3738	fi
[388]	3739
	3740	# next is test for openmp usage
[622]	3741	# echo "mpiexec -npernode $tasks_per_node ./a.out $ROPTS < runfile_atmos"
	3742	# mpiexec -npernode $tasks_per_node ./a.out $ROPTS < runfile_atmos
[221]	3743	elif [[ $( echo $mpilib \| cut -c1-3 ) = mva ]]
	3744	then
[374]	3745	export MV2_NUM_PORTS=2
[376]	3746	# The default setting of MV2_CPU_MAPPING gives best results
[405]	3747	# export MV2_ENABLE_AFFINITY=1
[376]	3748	# export MV2_CPU_MAPPING=0,1,4,5,2,3,6,7
[412]	3749	# if [[ "$sgi_feature" = ice2 ]]
	3750	# then
	3751	# export MV2_CPU_MAPPING=0,4,1,5,2,6,3,7
	3752	# else
	3753	# export MV2_CPU_MAPPING=0,1,4,5,2,3,6,7
	3754	# fi
[678]	3755	if [[ $use_openmp = true ]]
	3756	then
	3757	unset MV2_CPU_MAPPING
	3758	export MV2_ENABLE_AFFINITY=0
	3759	fi
[412]	3760	echo "*** MV2_CPU_MAPPING=$MV2_CPU_MAPPING"
[678]	3761	echo "*** MV2_ENABLE_AFFINITY=$MV2_ENABLE_AFFINITY"
[622]	3762	if [[ $use_openmp = true ]]
[405]	3763	then
[678]	3764	echo " mpiexec -npernode $tasks_per_node ./a.out $ROPTS < runfile_atmos"
[693]	3765	mpiexec -npernode $tasks_per_node $tv_opt ./a.out $ROPTS < runfile_atmos
[405]	3766	else
[693]	3767	mpiexec -np $ii $tv_opt ./a.out $ROPTS < runfile_atmos
[405]	3768	fi
[738]	3769	elif [[ "$mpilib" = impi ]]
	3770	then
	3771	echo "mpirun -np $tasks_per_node inspxe-cl -r result -collect mi3 -- ./a.out < runfile_atmos"
	3772	mpirun -np $tasks_per_node inspxe-cl -r result -collect mi3 -- ./a.out < runfile_atmos
[221]	3773	fi
[437]	3774	elif [[ $host = lcxt4 \|\| $host = lcxt5m ]]
[179]	3775	then
[544]	3776	aprun -n $ii -N $tasks_per_node a.out $ROPTS < runfile_atmos
[755]	3777	elif [[ $host = lcflow ]]
	3778	then
	3779	mpiexec -machinefile $TMPDIR/machines -n $ii -env I_MPI_FABRICS shm:ofa a.out < runfile_atmos $ROPTS
[164]	3780	else
[223]	3781	mpiexec -machinefile hostfile -n $ii a.out < runfile_atmos $ROPTS
[164]	3782	fi
[1]	3783	else
[164]	3784
[206]	3785	# currently there is no full MPI-2 support on ICE and XT4
	3786	(( iia = $numprocs_atmos / $threads_per_task ))
	3787	(( iio = $numprocs_ocean / $threads_per_task ))
	3788	printf "\n coupled run ($iia atmosphere, $iio ocean)"
	3789	printf "\n using $coupled_mode coupling"
[102]	3790	printf "\n\n"
[164]	3791
[206]	3792	if [[ $coupled_mode = "mpi2" ]]
[164]	3793	then
[206]	3794	echo "atmosphere_to_ocean $iia $iio" > runfile_atmos
	3795	echo "ocean_to_atmosphere $iia $iio" > runfile_ocean
	3796	if [[ $host = lcsgih \|\| $host = lcsgib ]]
	3797	then
[197]	3798
[221]	3799	if [[ $( echo $mpilib \| cut -c1-3 ) = mpt ]]
	3800	then
	3801	# export MPI_LAUNCH_TIMEOUT=360
	3802	mpiexec_mpt -np $iia ./a.out $ROPTS < runfile_atmos &
	3803	mpiexec_mpt -np $iio ./a.out $ROPTS < runfile_ocean &
	3804	elif [[ $( echo $mpilib \| cut -c1-3 ) = mva ]]
	3805	then
	3806	# ulimit -s 300000 # A too large stack size causes problems
	3807	# export MV2_NUM_PORTS=2
	3808	# export MV2_CPU_MAPPING=0:1:2:3
	3809	mpiexec -n $iia ./a.out $ROPTS < runfile_atmos &
	3810	mpiexec -n $iio ./a.out $ROPTS < runfile_ocean &
	3811	fi
[197]	3812
[437]	3813	elif [[ $host = lcxt4 \|\| $host = lcxt5m ]]
[206]	3814	then
[197]	3815
[206]	3816	aprun -n $iia -N $tasks_per_node a.out < runfile_atmos $ROPTS &
	3817	aprun -n $iio -N $tasks_per_node a.out < runfile_ocean $ROPTS &
	3818
	3819	else
	3820	# WORKAROUND BECAUSE mpiexec WITH -env option IS NOT AVAILABLE ON SOME SYSTEMS
	3821	mpiexec -machinefile hostfile -n $iia a.out $ROPTS < runfile_atmos &
	3822	mpiexec -machinefile hostfile -n $iio a.out $ROPTS < runfile_ocean &
	3823	# mpiexec -machinefile hostfile -n $iia -env coupling_mode atmosphere_to_ocean a.out $ROPTS &
	3824	# mpiexec -machinefile hostfile -n $iio -env coupling_mode ocean_to_atmosphere a.out $ROPTS &
	3825	fi
	3826	wait
	3827
[164]	3828	else
	3829
[206]	3830	echo "coupled_run $iia $iio" > runfile_atmos
	3831	if [[ $host = lcsgih \|\| $host = lcsgib ]]
	3832	then
	3833
[221]	3834	if [[ $( echo $mpilib \| cut -c1-3 ) = mpt ]]
	3835	then
	3836	# export MPI_LAUNCH_TIMEOUT=360
	3837	mpiexec_mpt -np $ii ./a.out $ROPTS < runfile_atmos
	3838	elif [[ $( echo $mpilib \| cut -c1-3 ) = mva ]]
	3839	then
	3840	# ulimit -s 300000 # A too large stack size causes problems
	3841	# export MV2_NUM_PORTS=2
	3842	# export MV2_CPU_MAPPING=0:1:2:3
	3843	mpiexec ./a.out $ROPTS < runfile_atmos
	3844	fi
[206]	3845
[437]	3846	elif [[ $host = lcxt4 \|\| $host = lcxt5m ]]
[206]	3847	then
	3848
	3849	aprun -n $ii -N $tasks_per_node a.out < runfile_atmos $ROPTS
	3850
[718]	3851	elif [[ $host = lck \|\| $host = lckordi ]]
[367]	3852	then
	3853	mpiexec -n $ii ./a.out $ROPTS < runfile_atmos &
[206]	3854	fi
	3855	wait
[164]	3856	fi
[102]	3857
[206]	3858	fi
[1]	3859	elif [[ $host = decalpha ]]
	3860	then
[102]	3861	dmpirun -np $numprocs a.out $ROPTS
[440]	3862	elif [[ $host = lckyoto ]]
	3863	then
	3864	set -xv
	3865	export P4_RSHCOMMAND=plesh
	3866	echo " P4_RSHCOMMAND = $P4_RSHCOMMAND"
	3867	if [[ "$ENVIRONMENT" = BATCH ]]
	3868	then
[505]	3869	if [[ "$cond2" = fujitsu ]]
	3870	then
	3871	mpiexec -n $numprocs ./a.out $ROPTS # for fujitsu-compiler
	3872	elif [[ "cond2" = pgi ]]
	3873	then
	3874	mpirun -np $numprocs -machinefile ${QSUB_NODEINF} ./a.out $ROPTS
	3875	else
	3876	mpirun_rsh -np $numprocs -hostfile ${QSUB_NODEINF} MV2_USE_SRQ=0 ./a.out ${ROPTS} \|\| /bin/true
	3877	fi
[440]	3878	else
[505]	3879	if [[ "$cond2" = "" ]]
	3880	then
	3881	mpiruni_rsh -np $numprocs ./a.out $ROPTS # for intel
	3882	else
	3883	mpirun -np $numprocs ./a.out $ROPTS
	3884	fi
[440]	3885	fi
	3886	set +xv
[1]	3887	elif [[ $host = lctit ]]
	3888	then
[678]	3889	export OMP_NUM_THREADS=$threads_per_task
	3890	echo "OMP_NUM_THREADS=$OMP_NUM_THREADS"
	3891	if [[ "$threads_per_task" != 1 ]]
	3892	then
	3893	export MV2_ENABLE_AFFINITY=0
	3894	fi
	3895	echo "----- PBS_NODEFILE content:"
	3896	cat $PBS_NODEFILE
	3897	echo "-----"
	3898	(( ii = $numprocs / $threads_per_task ))
	3899	echo "mpirun -np $ii -hostfile $PBS_NODEFILE ./a.out $ROPTS"
	3900	mpirun -np $ii -hostfile $PBS_NODEFILE ./a.out $ROPTS
[1]	3901	else
[102]	3902	mpprun -n $numprocs a.out $ROPTS
[1]	3903	fi
	3904	[[ $? != 0 ]] && execution_error=true
	3905
	3906
	3907	# PERFORMANCE-AUSWERTUNG MIT APPRENTICE
	3908	if [[ "$cond1" = apprentice \|\| "$cond2" = apprentice ]]
	3909	then
	3910	apprentice
	3911	fi
	3912	else
[102]	3913	a.out $ROPTS
[1]	3914	fi
	3915	fi
	3916	if [[ $? != 0 \|\| $execution_error = true ]]
	3917	then
	3918
	3919	# ABBRUCH BEI LAUFZEITFEHLER
[102]	3920	# [[ ! ( "$cond1" = debug \|\| "$cond2" = debug ) ]] && cat aout_output*
[1]	3921	printf "\n +++ runtime error occured"
	3922	locat=execution
	3923	exit
	3924	else
[102]	3925	# [[ ! ( "$cond1" = debug \|\| "$cond2" = debug ) ]] && cat aout_output*
[1]	3926	printf "\n$striche\n *** execution finished \n"
[260]	3927
[261]	3928	# Stop the dvr streaming server and process the dvr output in order
	3929	# to create dvrs- and html-files containing all streams
[260]	3930	if [[ "$dvrserver_id" != "" ]]
	3931	then
	3932	kill $dvrserver_id
	3933	printf "\n *** dvr server with id=$dvrserver_id has been stopped"
[261]	3934
[262]	3935	# If there is a directory, data have been output by the
	3936	# streaming server. Otherwise, user has chosen dvrp_output=local
	3937	if [[ -d DATA_DVR ]]
	3938	then
[261]	3939
[262]	3940	# Add the current dvr configuration file to the dvr output
	3941	# directory
	3942	cp .dvrserver.config DATA_DVR
	3943
	3944	# Process the dvr output (option -s for also generating
	3945	# sequence mode data)
	3946	process_dvr_output -d DATA_DVR -f $fname -s
	3947
	3948	else
	3949
	3950	# Process the local output
	3951	process_dvr_output -l -d DATA_DVR -f $fname
	3952
	3953	fi
	3954
	3955	elif [[ $(echo $package_list \| grep -c dvrp_graphics) != 0 ]]
	3956	then
	3957
	3958	# Process dvr output generated in local mode (dvrp_output=local)
	3959	process_dvr_output -l -d DATA_DVR -f $fname
	3960
[260]	3961	fi
[1]	3962	fi
	3963
	3964
	3965
[503]	3966	# Call of combine_plot_fields in order to merge single files written
	3967	# by each PE into one file.
	3968	if [[ ! -f ${PALM_BIN}/combine_plot_fields${block}.x ]]
	3969	then
	3970	printf "\n\n\n +++ WARNING: no combine_plot_fields found for given block \"$cond1 $cond2\""
	3971	printf "\n 2d- and/or 3d-data may be incomplete!"
	3972	printf "\n Run \"mbuild -u -h $localhost\" to generate utilities for this block.\n"
	3973	else
	3974	printf "\n\n\n *** post-processing: now executing \"combine_plot_fields${block}.x\" ..."
	3975	combine_plot_fields${block}.x
	3976	fi
	3977
	3978
	3979
[1]	3980	# EVENTUELLE OUTPUT-KOMMANDOS ABARBEITEN
	3981	(( i = 0 ))
	3982	while (( i < ioc ))
	3983	do
	3984	(( i = i + 1 ))
	3985	if (( i == 1 ))
	3986	then
	3987	printf "\n\n *** execution of OUTPUT-commands:\n$striche"
	3988	fi
	3989	printf "\n >>> ${out_command[$i]}"
	3990	eval ${out_command[$i]}
	3991	if (( i == ioc ))
	3992	then
	3993	printf "\n$striche\n"
	3994	fi
	3995	done
	3996
	3997
	3998	# EVTL. INHALT DES AKTUELLEN VERZEICHNISSES AUSGEBEN
	3999	if [[ $do_trace = true ]]
	4000	then
	4001	printf "\n\n"
	4002	ls -al
	4003	fi
	4004
	4005
	4006
	4007	# OUTPUT-DATEIEN AN IHRE ZIELORTE KOPIEREN
	4008	(( i = 0 ))
	4009	while (( i < iout ))
	4010	do
	4011	(( i = i + 1 ))
	4012	if (( i == 1 ))
	4013	then
	4014	printf "\n\n *** saving OUTPUT-files:\n$striche"
	4015	fi
	4016
	4017	# PRUEFEN, OB EINZELDATEI ODER DATEI PRO PROZESSOR
	4018	files_for_pes=false; filetyp=file
	4019	if [[ "${actionout[$i]}" = pe && -n $numprocs ]]
	4020	then
	4021	files_for_pes=true; filetyp=directory
	4022	actionout[$i]=""
	4023	elif [[ "${actionout[$i]}" = pe && ! -n $numprocs ]]
	4024	then
	4025	actionout[$i]=""
	4026	elif [[ "${actionout[$i]}" = arpe && -n $numprocs ]]
	4027	then
	4028	files_for_pes=true; filetyp=directory
	4029	actionout[$i]="ar"
	4030	elif [[ "${actionout[$i]}" = arpe && ! -n $numprocs ]]
	4031	then
	4032	actionout[$i]="ar"
	4033	elif [[ "${actionout[$i]}" = flpe && -n $numprocs ]]
	4034	then
	4035	files_for_pes=true; filetyp=directory
	4036	actionout[$i]="fl"
	4037	elif [[ "${actionout[$i]}" = flpe && ! -n $numprocs ]]
	4038	then
	4039	actionout[$i]="fl"
	4040	elif [[ "${actionout[$i]}" = trpe && -n $numprocs ]]
	4041	then
	4042	files_for_pes=true; filetyp=directory
	4043	actionout[$i]="tr"
	4044	elif [[ "${actionout[$i]}" = trpe && ! -n $numprocs ]]
	4045	then
	4046	actionout[$i]="tr"
	4047	fi
	4048
	4049	if [[ ! -f ${localout[$i]} && $files_for_pes = false ]]
	4050	then
	4051	printf "\n +++ temporary OUTPUT-file ${localout[$i]} does not exist\n"
[260]	4052	elif [[ ! -d ${localout[$i]} && $files_for_pes = true ]]
[1]	4053	then
	4054	printf "\n +++ temporary OUTPUT-file ${localout[$i]}/.... does not exist\n"
	4055	else
	4056
	4057
	4058	# KOPIEREN PER FTP/SCP (IMMER IM BINAERMODUS, -M: FALLS ZIELKATALOG
	4059	# NICHT VORHANDEN, WIRD VERSUCHT, IHN ANZULEGEN), ABER NUR BEI
	4060	# REMOTE-RECHNUNGEN
	4061	if [[ "${actionout[$i]}" = tr ]]
	4062	then
	4063	if [[ $localhost != $fromhost ]]
	4064	then
	4065	if [[ $files_for_pes = false ]]
	4066	then
	4067	cps=""
	4068	cst=""
	4069	else
	4070	cps=-c
	4071	cst="/"
	4072	fi
[622]	4073	if [[ $localhost = nech ]]
[1]	4074	then
	4075
	4076	# TRANSFER IN EIGENSTAENDIGEM JOB
	4077	# ZUERST KOPIE DER DATEI INS TEMPORAERE DATENVERZEICHNIS
	4078	[[ ! -d $tmp_data_catalog/TRANSFER ]] && mkdir -p $tmp_data_catalog/TRANSFER
	4079	file_to_transfer=${fname}_${localout[$i]}_to_transfer_$kennung
	4080	if [[ $files_for_pes = false ]]
	4081	then
	4082	ln -f ${localout[$i]} $tmp_data_catalog/TRANSFER/$file_to_transfer
	4083	else
	4084	mkdir $tmp_data_catalog/TRANSFER/$file_to_transfer
	4085	ln ${localout[$i]}/* $tmp_data_catalog/TRANSFER/$file_to_transfer
	4086	fi
	4087
	4088	echo "set -x" > transfer_${localout[$i]}
	4089	echo "cd $tmp_data_catalog/TRANSFER" >> transfer_${localout[$i]}
[82]	4090
	4091	printf "\n >>> OUTPUT: ${localout[$i]}$cst by SCP in seperate job to"
	4092	printf "\n ${pathout[$i]}/${localhost}_${fname}${endout[$i]}$cst"
	4093	printf "\n or higher cycle\n"
	4094	echo "batch_scp $cps -b -m -u $return_username $return_addres $file_to_transfer \"${pathout[$i]}\" ${localhost}_${fname}${endout[$i]} ${extout[$i]}" >> transfer_${localout[$i]}
	4095
[1]	4096	echo "[[ \$? = 0 ]] && rm $file_to_transfer" >> transfer_${localout[$i]}
	4097
	4098	if [[ $localhost = nech ]]
	4099	then
[82]	4100	subjob -d -c /pf/b/$usern/job_queue -v -q pp -X 0 -m 1000 -t 900 transfer_${localout[$i]}
[1]	4101	else
	4102	if [[ "$LOGNAME" = b323013 ]]
	4103	then
[164]	4104	subjob -v -q c1 -X 0 -m 1000 -t 900 -c $job_catalog transfer_${localout[$i]}
[1]	4105	else
[164]	4106	subjob -d -v -q c1 -X 0 -m 1000 -t 900 -c $job_catalog transfer_${localout[$i]}
[1]	4107	fi
	4108	fi
	4109
	4110	else
	4111
	4112	# TRANSFER INNERHALB DIESES JOBS
	4113	transfer_failed=false
[82]	4114	printf "\n >>> OUTPUT: ${localout[$i]}$cst by SCP to"
	4115	printf "\n ${pathout[$i]}/${localhost}_${fname}${endout[$i]}$cst\n"
	4116	batch_scp $cps -b -m -u $return_username $return_addres ${localout[$i]} "${pathout[$i]}" ${localhost}_${fname}${endout[$i]} ${extout[$i]}
	4117	[[ $? != 0 ]] && transfer_failed=true
[1]	4118
	4119	# BEI FEHLGESCHLAGENEM TRANSFER SICHERUNGSKOPIE AUF
	4120	# LOKALER MASCHINE ANLEGEN
	4121	if [[ $transfer_failed = true ]]
	4122	then
	4123	printf " +++ transfer failed. Trying to save a copy on the local host under:\n"
	4124	printf " ${pathout[$i]}/${localhost}_${fname}${endout[$i]}_$kennung\n"
	4125
	4126	# ERSTMAL PRUEFEN, OB VERZEICHNIS EXISTIERT. GEGEBENENFALLS
	4127	# ANLEGEN.
	4128	eval local_catalog=${pathout[$i]}
	4129	if [[ ! -d $local_catalog ]]
	4130	then
	4131	printf " *** local directory does not exist. Trying to create:\n"
	4132	printf " $local_catalog \n"
	4133	mkdir -p $local_catalog
	4134	fi
	4135	eval cp ${localout[$i]} ${pathout[$i]}/${localhost}_${fname}${endout[$i]}_$kennung
	4136	transfer_problems=true
	4137	fi
	4138	fi
	4139	else
	4140
	4141	# WERTZUWEISUNG, SO DASS WEITER UNTEN NUR KOPIERT WIRD
	4142	actionout[$i]=""
	4143	fi
	4144	fi
	4145
	4146
	4147	# APPEND PER FTP/SCP (IMMER IM BINAERMODUS, -M: FALLS ZIELKATALOG
	4148	# NICHT VORHANDEN, WIRD VERSUCHT, IHN ANZULEGEN), ABER NUR BEI
	4149	# REMOTE-RECHNUNGEN
	4150	if [[ "${actionout[$i]}" = tra ]]
	4151	then
	4152	if [[ $localhost != $fromhost ]]
	4153	then
[622]	4154	if [[ $localhost = ibmh \|\| $localhost = nech ]]
[1]	4155	then
	4156
	4157	# TRANSFER IN EIGENSTAENDIGEM JOB
	4158	# ZUERST KOPIE DER DATEI INS TEMPORAERE DATENVERZEICHNIS
	4159	[[ ! -d $tmp_data_catalog/TRANSFER ]] && mkdir -p $tmp_data_catalog/TRANSFER
	4160	file_to_transfer=${fname}_${localout[$i]}_to_transfer_$kennung
	4161	ln -f ${localout[$i]} $tmp_data_catalog/TRANSFER/$file_to_transfer
	4162
	4163	echo "set -x" > transfer_${localout[$i]}
	4164	echo "cd $tmp_data_catalog/TRANSFER" >> transfer_${localout[$i]}
[82]	4165
	4166	printf "\n >>> OUTPUT: ${localout[$i]} append by SCP in seperate job to"
	4167	printf "\n ${pathout[$i]}/${localhost}_${fname}${endout[$i]}"
	4168	printf "\n or higher cycle\n"
	4169	echo "batch_scp -A -b -m -u $return_username $return_addres $file_to_transfer \"${pathout[$i]}\" ${localhost}_${fname}${endout[$i]} ${extout[$i]}" >> transfer_${localout[$i]}
	4170
[1]	4171	echo "[[ \$? = 0 ]] && rm $file_to_transfer" >> transfer_${localout[$i]}
	4172
	4173	if [[ $localhost = nech ]]
	4174	then
[82]	4175	subjob -d -c /pf/b/$usern/job_queue -v -q pp -X 0 -m 1000 -t 900 transfer_${localout[$i]}
[1]	4176	else
	4177	if [[ $LOGNAME = b323013 ]]
	4178	then
[164]	4179	subjob -v -q c1 -X 0 -m 1000 -t 900 -c $job_catalog transfer_${localout[$i]}
[1]	4180	else
[164]	4181	subjob -d -v -q c1 -X 0 -m 1000 -t 900 -c $job_catalog transfer_${localout[$i]}
[1]	4182	fi
	4183	fi
	4184
	4185	else
	4186
	4187	# TRANSFER INNERHALB DIESES JOBS
	4188	transfer_failed=false
[82]	4189	printf "\n >>> OUTPUT: ${localout[$i]} append by SCP to"
	4190	printf "\n ${pathout[$i]}/${localhost}_${fname}${endout[$i]}\n"
	4191	batch_scp -A -b -m -u $return_username $return_addres ${localout[$i]} "${pathout[$i]}" ${localhost}_${fname}${endout[$i]} ${extout[$i]}
	4192	[[ $? != 0 ]] && transfer_failed=true
[1]	4193
	4194	# BEI FEHLGESCHLAGENEM TRANSFER SICHERUNGSKOPIE AUF
	4195	# LOKALER MASCHINE ANLEGEN
	4196	if [[ $transfer_failed = true ]]
	4197	then
	4198	printf " +++ transfer failed. Trying to save a copy on the local host under:\n"
	4199	printf " ${pathout[$i]}/${localhost}_${fname}${endout[$i]}_$kennung\n"
	4200
	4201	# ERSTMAL PRUEFEN, OB VERZEICHNIS EXISTIERT. GEGEBENENFALLS
	4202	# ANLEGEN
	4203	eval local_catalog=${pathout[$i]}
	4204	if [[ ! -d $local_catalog ]]
	4205	then
	4206	printf " *** local directory does not exist. Trying to create:\n"
	4207	printf " $local_catalog \n"
	4208	mkdir -p $local_catalog
	4209	fi
	4210
	4211	eval cp ${localout[$i]} ${pathout[$i]}/${localhost}_${fname}${endout[$i]}_$kennung
	4212	transfer_problems=true
	4213	fi
	4214	fi
	4215	else
	4216
	4217	# WERTZUWEISUNG, SO DASS WEITER UNTEN NUR APPEND AUF
	4218	# LOKALER MASCHINE DURCHGEFUEHRT WIRD
	4219	actionout[$i]=a
	4220	fi
	4221	fi
	4222
	4223
	4224	# OUTPUT-DATEI FUER EINEN FORTSETZUNGSLAUF. DATEI WIRD PER
	4225	# LINK AUF DEN TEMPORAEREN DATENKATALOG GELEGT. OPTION -f WIRD
	4226	# VERWENDET, FALLS DORT NOCH EINE DATEI GLEICHEN NAMENS VORHANDEN
	4227	# SEIN SOLLTE. ANSCHLIESSEND WIRD SEINE ARCHIVIERUNG ANGESTOSSEN
	4228	if [[ "${actionout[$i]}" = fl ]]
	4229	then
	4230	[[ ! -d $tmp_data_catalog ]] && mkdir -p $tmp_data_catalog
	4231	chmod g+rx $tmp_data_catalog
	4232	if [[ $files_for_pes = false ]]
	4233	then
	4234	printf "\n >>> OUTPUT: ${localout[$i]} to"
	4235	printf "\n $tmp_data_catalog/${frelout[$i]} (temporary data catalog)\n"
	4236	ln -f ${localout[$i]} $tmp_data_catalog/${frelout[$i]}
	4237	else
	4238	printf "\n >>> OUTPUT: ${localout[$i]}/.... to"
	4239	printf "\n $tmp_data_catalog/${frelout[$i]} (temporary data catalog)\n"
	4240	mkdir $tmp_data_catalog/${frelout[$i]}
[149]	4241	cd ${localout[$i]}
	4242	for file in $(ls *)
	4243	do
	4244	ln -f $file $tmp_data_catalog/${frelout[$i]}
	4245	done
	4246	cd $TEMPDIR
[1]	4247	fi
	4248
	4249
	4250	# ARCHIVIERUNGSJOB WIRD ERZEUGT UND ABGESCHICKT
	4251	if [[ $store_on_archive_system = true ]]
	4252	then
	4253
	4254	if [[ $archive_system = asterix ]]
	4255	then
	4256	echo "cd $tmp_data_catalog" >> archive_${frelout[$i]}
	4257	if [[ $files_for_pes = false ]]
	4258	then
	4259	echo "stageout ${frelout[$i]} > STAGE_OUTPUT${i}_$kennung" >> archive_${frelout[$i]}
	4260	else
	4261	echo "stageout -t ${frelout[$i]} > STAGE_OUTPUT${i}_$kennung" >> archive_${frelout[$i]}
	4262	fi
	4263	echo "cat STAGE_OUTPUT${i}_$kennung" >> archive_${frelout[$i]}
	4264	echo "if [[ \$(grep -c \"st.msg:150\" STAGE_OUTPUT${i}_$kennung) != 0 ]]" >> archive_${frelout[$i]}
	4265	echo "then" >> archive_${frelout[$i]}
	4266	echo " do_stageout=false" >> archive_${frelout[$i]}
	4267	echo "else" >> archive_${frelout[$i]}
	4268	echo " echo \" +++ $filetyp ${frelout[$i]} could not be stored on archive-system\" " >> archive_${frelout[$i]}
	4269	echo " cat /dev/null > ~/job_queue/ARCHIVE_ERROR_$fname" >> archive_${frelout[$i]}
	4270	echo " cat STAGE_OUTPUT${i}_$kennung > ~/job_queue/archive_${frelout[$i]}_error" >> archive_${frelout[$i]}
	4271	echo " echo \" *** $filetyp ${frelout[$i]} will be copied to \$WORK as backup\" " >> archive_${frelout[$i]}
	4272	if [[ $files_for_pes = false ]]
	4273	then
	4274	echo " cp ${frelout[$i]} \$WORK" >> archive_${frelout[$i]}
	4275	else
	4276	echo " cp -r ${frelout[$i]} \$WORK/${frelout[$i]}" >> archive_${frelout[$i]}
	4277	fi
	4278	echo " echo \" *** $filetyp ${frelout[$i]} saved\" " >> archive_${frelout[$i]}
	4279	echo "fi" >> archive_${frelout[$i]}
	4280	echo "rm STAGE_OUTPUT${i}_$kennung" >> archive_${frelout[$i]}
	4281	elif [[ $archive_system = DMF ]]
	4282	then
	4283	echo "cd $tmp_data_catalog" >> archive_${frelout[$i]}
	4284	if [[ $files_for_pes = false ]]
	4285	then
	4286	printf "\n +++ archiving of single files impossible with $archive_system !\n"
	4287	locat=DMF
	4288	exit
	4289	else
	4290	# FUER RECHNER IN JUELICH. DORT KOENNTEN AUCH
	4291	# EINZELNE DATEIEN GESICHERT WERDEN (SPAETER KORR.)
	4292	echo "rm -rf \$ARCHIVE/${frelout[$i]}" >> archive_${frelout[$i]}
	4293	echo "cp -r ${frelout[$i]} \$ARCHIVE" >> archive_${frelout[$i]}
	4294	fi
	4295	elif [[ $archive_system = tivoli ]]
	4296	then
	4297	echo "cd $tmp_data_catalog" >> archive_${frelout[$i]}
	4298	if [[ $files_for_pes = false ]]
	4299	then
	4300	# EVENTUELL NOCH VORHANDENE DATEI IM ARCHIV LOSCHEN
	4301	echo "set -x" >> archive_${frelout[$i]}
	4302	echo "rm -rf \$PERM/${frelout[$i]}" >> archive_${frelout[$i]}
	4303	echo "cp ${frelout[$i]} \$PERM/${frelout[$i]}" >> archive_${frelout[$i]}
	4304	else
	4305
	4306	echo "set -x" >> archive_${frelout[$i]}
	4307	echo "rm -rf \$PERM/${frelout[$i]}/*" >> archive_${frelout[$i]}
	4308	echo "[[ ! -d \$PERM/${frelout[$i]} ]] && mkdir $PERM/${frelout[$i]}" >> archive_${frelout[$i]}
	4309	cd $tmp_data_catalog
	4310	all_files=`ls -1 ${frelout[$i]}/*`
	4311	cd - > /dev/null
	4312	(( inode = 0 ))
	4313	(( tp1 = tasks_per_node + 1 ))
	4314	while (( inode < nodes ))
	4315	do
	4316	# echo "*** all_files = \"$all_files\" "
	4317	files=`echo $all_files \| cut -d" " -f1-$tasks_per_node`
	4318	all_files=`echo $all_files \| cut -d" " -f$tp1-`
	4319	# echo "*** tasks_per_node = \"$tasks_per_node\" "
	4320	# echo "*** files = \"$files\" "
	4321	# echo "*** all_files = \"$all_files\" "
[416]	4322	echo "tar cvf \$PERM/${frelout[$i]}/${frelout[$i]}.node_$inode.tar $files" >> archive_${frelout[$i]}
	4323	### echo "tar cvf ${frelout[$i]}.node_$inode.tar $files" >> archive_${frelout[$i]}
	4324	### echo "cp ${frelout[$i]}.node_$inode.tar \$PERM/${frelout[$i]}/" >> archive_${frelout[$i]}
	4325	###echo "rm ${frelout[$i]}.node_$inode.tar" >> archive_${frelout[$i]}
[397]	4326	# echo "dsmc incremental \$PERM/${frelout[$i]}/${frelout[$i]}.node_$inode.tar" >> archive_${frelout[$i]}
	4327	# echo "dsmmigrate \$PERM/${frelout[$i]}/${frelout[$i]}.node_$inode.tar" >> archive_${frelout[$i]}
[1]	4328	(( inode = inode + 1 ))
	4329	done
	4330	# echo "rm -rf \$PERM/${frelout[$i]}.tar" >> archive_${frelout[$i]}
	4331	# echo "tar cvf \$PERM/${frelout[$i]}.tar ${frelout[$i]}" >> archive_${frelout[$i]}
	4332	fi
	4333	elif [[ $archive_system = ut ]]
	4334	then
	4335	echo "cd $tmp_data_catalog" >> archive_${frelout[$i]}
	4336	if [[ $files_for_pes = false ]]
	4337	then
	4338	# EVENTUELL NOCH VORHANDENE DATEI IM ARCHIV LOSCHEN
	4339	echo "set -x" >> archive_${frelout[$i]}
	4340	echo "rm -rf \$UT/${frelout[$i]}" >> archive_${frelout[$i]}
	4341	echo "cp ${frelout[$i]} \$UT/${frelout[$i]}" >> archive_${frelout[$i]}
	4342	else
	4343
	4344	echo "set -x" >> archive_${frelout[$i]}
	4345	echo "rm -rf \$UT/${frelout[$i]}/*" >> archive_${frelout[$i]}
	4346	echo "[[ ! -d \$UT/${frelout[$i]} ]] && mkdir $UT/${frelout[$i]}" >> archive_${frelout[$i]}
	4347	cd $tmp_data_catalog
	4348	all_files=`ls -1 ${frelout[$i]}/*`
	4349	cd - > /dev/null
	4350	(( inode = 0 ))
	4351	(( tp1 = tasks_per_node + 1 ))
	4352	while (( inode < nodes ))
	4353	do
	4354	files=`echo $all_files \| cut -d" " -f1-$tasks_per_node`
	4355	all_files=`echo $all_files \| cut -d" " -f$tp1-`
	4356	echo "tar cvf \$UT/${frelout[$i]}/${frelout[$i]}.node_$inode.tar $files" >> archive_${frelout[$i]}
	4357	(( inode = inode + 1 ))
	4358	done
	4359	fi
	4360	elif [[ $archive_system = none ]]
	4361	then
	4362	printf " +++ archiving on $localhost not available!\n"
	4363	fi
	4364
	4365	if [[ $archive_system != none ]]
	4366	then
[399]	4367	if [[ $localhost = lcsgih \|\| $localhost = lcsgib ]]
[1]	4368	then
[164]	4369	# subjob -d -v -q cdata -X 0 -m 1000 -t 43200 -c $job_catalog archive_${frelout[$i]}
[416]	4370	subjob -v -q permq -X 1 -T 1 -m 1000 -t 172800 -c $job_catalog archive_${frelout[$i]}
[1]	4371	elif [[ $localhost = nech ]]
	4372	then
[82]	4373	subjob -d -c /pf/b/$usern/job_queue -v -q pp -X 0 -m 1000 -t 7200 archive_${frelout[$i]}
[1]	4374	fi
	4375	printf " Archiving of $tmp_data_catalog/${frelout[$i]} initiated (batch job submitted)\n"
	4376	fi
	4377	else
[399]	4378	printf " +++ caution: option -A is switched off. No archiving on $archive_system!\n"
[1]	4379	fi
	4380
	4381
	4382	# LEERDATEI IM BENUTZERVERZEICHNIS ANLEGEN, DAMIT BEKANNT IST,
	4383	# WIE DIE HOECHSTE ZYKLUSNUMMER AUF DEM ARCHIV-SYSTEM LAUTET
	4384	if [[ $files_for_pes = false ]]
	4385	then
	4386	cat /dev/null > ${pathout[$i]}
	4387	else
	4388	mkdir -p ${pathout[$i]}
	4389	fi
	4390
	4391	fi
	4392
	4393
	4394	# KOPIEREN AUF LOKALER MASCHINE ZUM ARCHIV-SYSTEM
	4395	# AUSSERDEM MUSS LEERDATEI ANGELEGT WERDEN, DAMIT BEKANNT IST,
	4396	# WIE DIE HOECHSTE ZYKLUSNUMMER AUF DEM ARCHIV-SYSTEM LAUTET
	4397	# FALLS IRGENDEINE (VON MEHREREN) ARCHIVIERUNGEN SCHIEF GEHT,
	4398	# WIRD FLAG GESETZT UND BLEIBT AUCH BEI WEITEREN ORDNUNGS-
	4399	# GEMAESSEN ARCHIVIERUNGEN GESETZT
	4400	if [[ "${actionout[$i]}" = ar ]]
	4401	then
	4402	if [[ $files_for_pes = false ]]
	4403	then
	4404	printf "\n >>> OUTPUT: ${localout[$i]} to"
	4405	printf "\n ${pathout[$i]}"
	4406	printf "\n File will be copied to archive-system ($archive_system) !\n"
	4407	else
	4408	printf "\n >>> OUTPUT: ${localout[$i]}/_.... to"
	4409	printf "\n ${pathout[$i]}"
	4410	printf "\n Directory will be copied to archive-system ($archive_system) !\n"
	4411	fi
	4412	mv ${localout[$i]} ${frelout[$i]}
	4413
	4414	file_saved=false
	4415
	4416	if [[ $archive_system = asterix ]]
	4417	then
	4418	do_stageout=true
	4419	(( stageout_anz = 0 ))
	4420	while [[ $do_stageout = true ]]
	4421	do
	4422	if [[ $files_for_pes = false ]]
	4423	then
	4424	stageout ${frelout[$i]} > STAGE_OUTPUT
	4425	else
	4426	stageout -t ${frelout[$i]} > STAGE_OUTPUT
	4427	fi
	4428	cat STAGE_OUTPUT
	4429	if [[ $(grep -c "st.msg:150" STAGE_OUTPUT) != 0 ]]
	4430	then
	4431	file_saved=true
	4432	do_stageout=false
	4433	else
	4434	if [[ $files_for_pes = false ]]
	4435	then
	4436	printf "\n +++ file ${frelout[$i]} could not be saved on archive-system"
	4437	else
	4438	printf "\n +++ directory ${frelout[$i]} could not be saved on archive-system"
	4439	fi
	4440	(( stageout_anz = stageout_anz + 1 ))
	4441	if (( stageout_anz == 10 ))
	4442	then
	4443	printf "\n +++ stoped after 10 unsuccessful tries!"
	4444	archive_save=false
	4445	do_stageout=false
	4446	else
	4447	printf "\n *** new try to store on archive after 15 min:"
	4448	sleep 900
	4449	fi
	4450	fi
	4451	done
	4452	elif [[ $archive_system = DMF ]]
	4453	then
	4454	if [[ $files_for_pes = false ]]
	4455	then
	4456	printf "\n +++ archiving of single files impossible on $archive_system!\n"
	4457	locat=DMF
	4458	exit
	4459	else
	4460	rm -rf $ARCHIVE/${frelout[$i]}
	4461	cp -r ${frelout[$i]} $ARCHIVE
	4462	fi
	4463	file_saved=true
	4464	elif [[ $archive_system = tivoli ]]
	4465	then
	4466	# ARCHIVIERUNG NUR PER BATCH-JOB MOEGLICH
	4467	# DATEI MUSS ZWISCHENZEITLICH INS TEMPORAERE DATENVERZEICHNIS
	4468	# GELEGT WERDEN
	4469	[[ ! -d $tmp_data_catalog ]] && mkdir -p $tmp_data_catalog
	4470	chmod g+rx $tmp_data_catalog
	4471	if [[ $files_for_pes = false ]]
	4472	then
	4473	ln -f ${frelout[$i]} $tmp_data_catalog/${frelout[$i]}
	4474	else
	4475	mkdir $tmp_data_catalog/${frelout[$i]}
	4476	ln -f ${frelout[$i]}/* $tmp_data_catalog/${frelout[$i]}
	4477	fi
	4478
	4479	# BATCH JOB GENERIEREN UND ABSCHICKEN; DATEI MUSS WIEDER
	4480	# AUS TEMPORAEREM DATENVERZEICHNIS ENTFERNT WERDEN
	4481	echo "cd $tmp_data_catalog" > archive_${frelout[$i]}
	4482	if [[ $files_for_pes = false ]]
	4483	then
	4484	# EVENTUELL NOCH VORHANDENE DATEI IM ARCHIV LOSCHEN
	4485	echo "rm -rf \$PERM/${frelout[$i]}" >> archive_${frelout[$i]}
	4486	echo "cp ${frelout[$i]} \$PERM/${frelout[$i]}" >> archive_${frelout[$i]}
	4487	echo "rm -rf ${frelout[$i]}" >> archive_${frelout[$i]}
	4488	else
	4489	echo "rm -rf \$PERM/${frelout[$i]}.tar" >> archive_${frelout[$i]}
	4490	echo "tar cvf \$PERM/${frelout[$i]}.tar ${frelout[$i]}" >> archive_${frelout[$i]}
	4491	echo "rm -rf ${frelout[$i]}" >> archive_${frelout[$i]}
	4492	fi
	4493
[164]	4494	subjob -v -d -q cdata -X 0 -m 1000 -t 43200 -c $job_catalog archive_${frelout[$i]}
[1]	4495	printf " Archiving of $tmp_data_catalog/${frelout[$i]} initiated (batch job submitted)\n"
	4496	file_saved=true
	4497
	4498	elif [[ $archive_system = ut ]]
	4499	then
	4500	# ARCHIVIERUNG NUR PER BATCH-JOB MOEGLICH
	4501	# DATEI MUSS ZWISCHENZEITLICH INS TEMPORAERE DATENVERZEICHNIS
	4502	# GELEGT WERDEN
	4503	[[ ! -d $tmp_data_catalog ]] && mkdir -p $tmp_data_catalog
	4504	chmod g+rx $tmp_data_catalog
	4505	if [[ $files_for_pes = false ]]
	4506	then
	4507	ln -f ${frelout[$i]} $tmp_data_catalog/${frelout[$i]}
	4508	else
	4509	mkdir $tmp_data_catalog/${frelout[$i]}
	4510	ln -f ${frelout[$i]}/* $tmp_data_catalog/${frelout[$i]}
	4511	fi
	4512
	4513	# BATCH JOB GENERIEREN UND ABSCHICKEN; DATEI MUSS WIEDER
	4514	# AUS TEMPORAEREM DATENVERZEICHNIS ENTFERNT WERDEN
	4515	echo "cd $tmp_data_catalog" > archive_${frelout[$i]}
	4516	if [[ $files_for_pes = false ]]
	4517	then
	4518	# EVENTUELL NOCH VORHANDENE DATEI IM ARCHIV LOSCHEN
	4519	echo "rm -rf \$UT/${frelout[$i]}" >> archive_${frelout[$i]}
	4520	echo "cp ${frelout[$i]} \$UT/${frelout[$i]}" >> archive_${frelout[$i]}
	4521	echo "rm -rf ${frelout[$i]}" >> archive_${frelout[$i]}
	4522	else
	4523	echo "rm -rf \$UT/${frelout[$i]}.tar" >> archive_${frelout[$i]}
	4524	echo "tar cvf \$UT/${frelout[$i]}.tar ${frelout[$i]}" >> archive_${frelout[$i]}
	4525	echo "rm -rf ${frelout[$i]}" >> archive_${frelout[$i]}
	4526	fi
	4527
[82]	4528	subjob -v -c /pf/b/$usern/job_queue -d -q pp -X 0 -m 1000 -t 7200 archive_${frelout[$i]}
[1]	4529	printf " Archiving of $tmp_data_catalog/${frelout[$i]} initiated (batch job submitted)\n"
	4530	file_saved=true
	4531
	4532	else
	4533	printf "\n +++ archive_system=\"$archive_system\" archiving impossible!"
	4534	archive_save=false
	4535	fi
	4536	if [[ $file_saved = true ]]
	4537	then
	4538	if [[ $files_for_pes = false ]]
	4539	then
	4540	cat /dev/null > ${pathout[$i]}
	4541	else
	4542	mkdir -p ${pathout[$i]}
	4543	fi
	4544	fi
	4545	fi
	4546
	4547	# APPEND AUF LOKALER MASCHINE
	4548	if [[ "${actionout[$i]}" = "a" ]]
	4549	then
[741]	4550	if [[ "${extout[$i]}" != " " && "${extout[$i]}" != "" ]]
	4551	then
	4552	printf "\n >>> OUTPUT: ${localout[$i]} append to ${pathout[$i]}.${extout[$i]}\n"
	4553	cat ${localout[$i]} >> ${pathout[$i]}.${extout[$i]}
	4554	else
	4555	printf "\n >>> OUTPUT: ${localout[$i]} append to ${pathout[$i]}\n"
	4556	cat ${localout[$i]} >> ${pathout[$i]}
	4557	fi
[1]	4558	fi
	4559
	4560	# KOPIEREN AUF LOKALER MASCHINE
	4561	# ES MUSS KOPIERT WERDEN, DA MOVE NICHT UEBER FILESYSTEM HINAUS MOEGLICH
	4562	if [[ "${actionout[$i]}" = "" && $files_for_pes = false ]]
	4563	then
	4564
	4565	# KOPIEREN AUF EINPROZESSORMASCHINE
[5]	4566	if [[ "${extout[$i]}" != " " && "${extout[$i]}" != "" ]]
[1]	4567	then
	4568	printf "\n >>> OUTPUT: ${localout[$i]} to ${pathout[$i]}.${extout[$i]}\n"
[149]	4569	if [[ $link_local_output = true ]]
	4570	then
	4571	printf " using ln -f\n"
	4572	ln -f ${localout[$i]} ${pathout[$i]}.${extout[$i]}
	4573	fi
	4574	# If "ln -f" fails of if "$link_local_output = false" do a normal "cp"
	4575	if [[ ! -f "${pathout[$i]}.${extout[$i]}" ]]
	4576	then
	4577	if [[ $link_local_output = true ]]
	4578	then
	4579	printf " ln failed, using cp...\n"
	4580	fi
	4581	cp ${localout[$i]} ${pathout[$i]}.${extout[$i]}
	4582	fi
[1]	4583	else
	4584	printf "\n >>> OUTPUT: ${localout[$i]} to ${pathout[$i]}\n"
[149]	4585	if [[ $link_local_output = true ]]
	4586	then
	4587	printf " using ln -f\n"
	4588	ln -f ${localout[$i]} ${pathout[$i]}
	4589	fi
	4590	# If "ln -f" fails of if "$link_local_output = false" do a normal "cp"
	4591	if [[ ! -f "${pathout[$i]}" ]]
	4592	then
	4593	if [[ $link_local_output = true ]]
	4594	then
	4595	printf " ln failed, using cp...\n"
	4596	fi
	4597	cp ${localout[$i]} ${pathout[$i]}
	4598	fi
[1]	4599	fi
	4600
	4601	elif [[ "${actionout[$i]}" = "" && $files_for_pes = true ]]
	4602	then
	4603
	4604	# DIE DEN PROZESSOREN EINES PARALLELRECHNERS ZUGEHOERIGEN
	4605	# DATEIEN WERDEN ERST IN EINEM KATALOG GESAMMELT UND DIESER
	4606	# WIRD DANN KOPIERT
	4607	# PER MOVE UMBENANNT WERDEN
	4608	printf "\n >>> OUTPUT: ${localout[$i]}/_.... to ${pathout[$i]}\n"
[149]	4609	if [[ $link_local_output = true ]]
	4610	then
	4611	printf " using ln -f\n"
	4612	mkdir ${pathout[$i]}
	4613	cd ${localout[$i]}
	4614	for file in $(ls *)
	4615	do
	4616	ln -f $file ${pathout[$i]}
	4617	done
	4618	cd $TEMPDIR
	4619	fi
	4620	# If "ln -f" fails of if "$link_local_output = false" do a normal "cp -r"
	4621	if [[ ! -f "${pathout[$i]}/_0000" ]]
	4622	then
	4623	if [[ $link_local_output = true ]]
	4624	then
	4625	printf " ln failed for .../_0000, using cp...\n"
	4626	fi
	4627	cp -r ${localout[$i]} ${pathout[$i]}
	4628	fi
[1]	4629
	4630	fi
	4631	fi
	4632	done
	4633	if (( i != 0 ))
	4634	then
	4635	if [[ $transfer_problems = true ]]
	4636	then
	4637	printf "\n$striche\n *** OUTPUT-files saved"
	4638	printf "\n +++ WARNING: some data transfers failed! \n"
	4639	else
	4640	printf "\n$striche\n *** all OUTPUT-files saved \n"
	4641	fi
	4642	fi
	4643
	4644
	4645	# EVENTUELL FOLGEJOB STARTEN
	4646	# DATEI CONTINUE_RUN MUSS VOM BENUTZERPROGRAMM AUS ERZEUGT WERDEN
	4647	if [[ -f CONTINUE_RUN ]]
	4648	then
	4649
	4650	if [[ $archive_save = true ]]
	4651	then
	4652
	4653	# ZUERST IN MRUN-AUFRUF OPTIONEN FUER FORTSETZUNGSLAUF, FUER
	4654	# STILLES ABARBEITEN (OHNE INTERAKTIVE RUECKFAGEN) UND FUER
	4655	# BATCH-BETRIEB (NUR WICHTIG, FALLS BATCH AUF LOKALER MASCHINE
	4656	# DURCHGEFUEHRT WERDEN SOLL) EINFUEGEN, FALLS NICHT BEREITS VOR-
	4657	# HANDEN
	4658	[[ $(echo $mc \| grep -c "\-C") = 0 ]] && mc="$mc -C"
	4659	[[ $(echo $mc \| grep -c "\-v") = 0 ]] && mc="$mc -v"
	4660	[[ $(echo $mc \| grep -c "\-b") = 0 ]] && mc="$mc -b"
	4661	if [[ $(echo $mc \| grep -c "#") != 0 ]]
	4662	then
	4663	mc=`echo $mc \| sed 's/#/f/g'`
	4664	fi
	4665
	4666
	4667	# JOB STARTEN
	4668	printf "\n\n *** initiating restart-run on \"$return_addres\" using command:\n"
	4669	echo " $mc"
	4670	printf "\n$striche\n"
	4671	if [[ $localhost != $fromhost ]]
	4672	then
	4673
[693]	4674	if [[ $localhost = lcsgih \|\| $localhost = lcsgib \|\| $localhost = nech \|\| $localhost = ibmh \|\| $localhost = ibmkisti \|\| $localhost = ibmku \|\| $localhost = ibms ]]
[1]	4675	then
	4676	echo "*** ssh will be used to initiate restart-runs!"
	4677	echo " return_addres=\"$return_addres\" "
	4678	echo " return_username=\"$return_username\" "
[635]	4679	if [[ $(echo $return_addres \| grep -c "130.75.105") = 1 ]]
[1]	4680	then
[757]	4681	ssh $return_addres -l $return_username "PATH=\$PATH:$LOCAL_MRUN_PATH;export LD_LIBRARY_PATH=\$LD_LIBRARY_PATH:/muksoft/packages/intel/Compiler/11.1/073/lib/intel64/;export PALM_BIN=$LOCAL_MRUN_PATH;cd $LOCAL_PWD; $mc "
[1]	4682	else
[693]	4683	if [[ $localhost = ibmkisti ]]
	4684	then
	4685	ssh $usern@gaiad "ssh $return_addres -l $return_username \"PATH=\\\$PATH:$LOCAL_MRUN_PATH;export PALM_BIN=$LOCAL_MRUN_PATH;cd $LOCAL_PWD; $mc\" "
	4686	else
	4687	ssh $return_addres -l $return_username "PATH=\$PATH:$LOCAL_MRUN_PATH;export PALM_BIN=$LOCAL_MRUN_PATH;cd $LOCAL_PWD; $mc "
	4688	fi
[1]	4689	fi
	4690	else
	4691	printf "\n +++ no restart mechanism available for host \"$localhost\" "
	4692	locat=restart; exit
	4693	fi
	4694
	4695	# WARTEN, DAMIT SICH RESTART JOB IN QUEUE EINREIHEN KANN, BEVOR
	4696	# DER AKTUELLE JOB ENDET
[399]	4697	if [[ $queue = special1q ]]
	4698	then
	4699	sleep 120
	4700	else
	4701	sleep 30
	4702	fi
[1]	4703
	4704	else
	4705
	4706	# BEI RECHNUNGEN AUF LOKALER MASCHINE KANN MRUN DIREKT AUFGE-
[108]	4707	# RUFEN WERDEN, AUSSER AUF lcfimm
[1]	4708	cd $LOCAL_PWD
[108]	4709	if [[ $localhost = lcfimm ]]
	4710	then
	4711	ssh $return_addres -l $return_username "PATH=\$PATH:$LOCAL_MRUN_PATH;export PALM_BIN=$LOCAL_MRUN_PATH;cd $LOCAL_PWD; $mc "
	4712	else
	4713	eval $mc # ' MUESSEN AUSGEWERTET WERDEN
	4714	fi
[1]	4715	cd - > /dev/null
	4716	fi
	4717	printf "\n$striche\n *** restart-run initiated \n"
	4718
	4719
	4720	# EVENTUELL INPUT-DATEIEN, DIE VON TEMPORAEREM DATENVERZEICHNIS
	4721	# GEHOLT WORDEN SIND, LOESCHEN
	4722	(( i = 0 ))
	4723	while (( i < iin ))
	4724	do
	4725	(( i = i + 1 ))
	4726	if [[ "${got_tmp[$i]}" = true && $keep_data_from_previous_run = false ]]
	4727	then
	4728	rm -r $tmp_data_catalog/${frelin[$i]}
	4729	fi
	4730	done
	4731
	4732	else
	4733
	4734	printf "\n +++ no restart-run possible, since errors occured"
	4735	printf "\n during the archive process"
	4736	fi
	4737
	4738	fi
	4739
	4740
	4741
	4742	# EVTL. EMAIL-BENACHRICHTIGUNG UEBER ABGESCHLOSSENEN LAUF
[352]	4743	if [[ "$email_notification" != "none" ]]
[1]	4744	then
	4745
	4746	if [[ $localhost != $fromhost ]]
	4747	then
	4748	if [[ -f CONTINUE_RUN ]]
	4749	then
	4750	echo "PALM restart run necessary" > email_text
	4751	echo "description header of actual run:" >> email_text
	4752	cat CONTINUE_RUN >> email_text
	4753	echo "mrun-command to restart:" >> email_text
	4754	echo "$mc" >> email_text
	4755	else
	4756	echo "PALM run with base filename \"$fname\" on host \"$localhost\" finished" > email_text
	4757	fi
	4758	mail $email_notification < email_text
	4759	printf "\n *** email notification sent to \"$email_notification\" "
	4760	fi
	4761	fi
	4762
	4763
	4764
	4765	# ALLE ARBEITEN BEENDET. TEMPORAERER KATALOG KANN GELOESCHT WERDEN
	4766	cd $HOME
	4767	[[ $delete_temporary_catalog = true ]] && rm -rf $TEMPDIR
	4768
	4769	else
	4770
	4771
	4772	# FALLS AUF REMOTE-MASCHINE GERECHNET WERDEN SOLL, WERDEN JETZT ENTSPRE-
	4773	# CHENDE AKTIONEN DURCHGEFUEHRT
	4774
	4775	# MRUN-BEFEHL FUER REMOTE-MASCHINE ZUSAMMENSTELLEN
[821]	4776	mrun_com="$mrun_script_name -a $afname -c $config_file -d $fname -h $host -H $fromhost -m $memory -t $cpumax -q $queue -R $return_addres -U $return_username -u $remote_username"
[1]	4777	[[ "$cpp_opts" != "" ]] && mrun_com=${mrun_com}" -D \"$cpp_opts\""
[79]	4778	[[ "$global_revision" != "" ]] && mrun_com=${mrun_com}" -G \"$global_revision\""
[123]	4779	[[ $group_number != none ]] && mrun_com=${mrun_com}" -g $group_number"
[1]	4780	[[ $do_compile = true ]] && mrun_com=${mrun_com}" -s \"$source_list\""
	4781	[[ "$input_list" != "" ]] && mrun_com=${mrun_com}" -i \"$input_list\""
	4782	[[ $ignore_archive_error = true ]] && mrun_com=${mrun_com}" -I"
	4783	[[ $keep_data_from_previous_run = true ]] && mrun_com=${mrun_com}" -k"
	4784	[[ "$additional_conditions" != "" ]] && mrun_com=${mrun_com}" -K \"$additional_conditions\""
	4785	# [[ "$makefile" != "$source_path/Makefile" ]] && mrun_com=${mrun_com}" -M \"$makefile\""
	4786	[[ "$output_list" != "" ]] && mrun_com=${mrun_com}" -o \"$output_list\""
	4787	[[ "$read_from_config" = false ]] && mrun_com=${mrun_com}" -S"
	4788	[[ $do_trace = true ]] && mrun_com=${mrun_com}" -x"
	4789	[[ "$numprocs" != "" ]] && mrun_com=${mrun_com}" -X $numprocs"
	4790	if [[ $use_openmp = true ]]
	4791	then
[622]	4792	mrun_com=${mrun_com}" -O $threads_per_task"
[1]	4793	fi
[759]	4794	[[ "$tasks_per_node" != "" ]] && mrun_com=${mrun_com}" -T $tasks_per_node"
[397]	4795	[[ $store_on_archive_system = true ]] && mrun_com=${mrun_com}" -A"
[1]	4796	[[ $package_list != "" ]] && mrun_com=${mrun_com}" -p \"$package_list\""
	4797	[[ $return_password != "" ]] && mrun_com=${mrun_com}" -P $return_password"
	4798	[[ $delete_temporary_catalog = false ]] && mrun_com=${mrun_com}" -B"
[127]	4799	[[ $node_usage != default && "$(echo $node_usage \| cut -c1-3)" != "sla" && $node_usage != novice ]] && mrun_com=${mrun_com}" -n $node_usage"
[291]	4800	[[ "$ocean_file_appendix" = true ]] && mrun_com=${mrun_com}" -y"
[206]	4801	[[ $run_coupled_model = true ]] && mrun_com=${mrun_com}" -Y \"$coupled_dist\""
[759]	4802	[[ "$max_par_io_str" != "" ]] && mrun_com=${mrun_com}" -w $max_par_io_str"
[1]	4803	if [[ $do_remote = true ]]
	4804	then
	4805	printf "\n>>>> MRUN-command on execution host:\n>>>> $mrun_com \n"
	4806	fi
	4807
	4808
	4809	# ZUSAMMENSTELLUNG DES JOBSCRIPTS AUF DATEI jobfile
	4810	jobfile=jobfile.$RANDOM
	4811
	4812
	4813	# TEMPORAERES VERZEICHNIS GENERIEREN UND NACH DORT WECHSELN
	4814	echo "mkdir $TEMPDIR" >> $jobfile
	4815	echo "cd $TEMPDIR" >> $jobfile
	4816
	4817
	4818	# EVENTUELL FEHLERVERFOLGUNG AKTIVIEREN
	4819	if [[ $do_trace = true ]]
	4820	then
	4821	echo "set -x" >> $jobfile
	4822	else
	4823	echo "set +vx" >> $jobfile
	4824	fi
	4825
	4826
[191]	4827	# BEREITSTELLUNG VON QUELLTEXTEN, MRUN-SCRIPTS UND KONFIGURATIONS-
	4828	# DATEI FUER DEN JOB
[693]	4829	if [[ $( echo $host \| cut -c1-5 ) = lcsgi \|\| $host = ibmkisti ]]
[1]	4830	then
	4831
[191]	4832	# KONFIGURATIONSDATEI UND MRUN_SCRIPT IN DAS SOURCE-VERZEICHNIS
	4833	# KOPIEREN
	4834	if [[ $restart_run != true ]]
	4835	then
	4836	cp $config_file $working_directory/SOURCES_FOR_RUN_$fname
	4837	cp ${PALM_BIN}/$mrun_script_name $working_directory/SOURCES_FOR_RUN_$fname
	4838	fi
[1]	4839
[191]	4840	# SOURCE-VERZEICHNIS VOM LOKALEN RECHNER PER SCP TRANSFERIEREN
[693]	4841	# (BEI JOBS AUF LOKALEM RECHNER REICHT CP)
[193]	4842	echo "set -x" >> $jobfile
[693]	4843	if [[ $host = $localhost ]]
	4844	then
	4845	echo "cp -r $working_directory/SOURCES_FOR_RUN_$fname ." >> $jobfile
	4846	else
	4847	if [[ $host != ibmkisti ]]
	4848	then
	4849	echo "scp -r $return_username@$return_addres:$working_directory/SOURCES_FOR_RUN_$fname ." >> $jobfile
	4850	else
	4851	# on KISTI's IBM firewall is only opened on interactive node
	4852	echo "localdir=\`pwd\`" >> $jobfile
	4853	echo "ssh $remote_username@gaiad \"cd \$localdir; scp -r $return_username@$return_addres:$working_directory/SOURCES_FOR_RUN_$fname .\" " >> $jobfile
	4854	fi
	4855	fi
[193]	4856	echo "export SOURCES_COMPLETE=true" >> $jobfile
[1]	4857
[193]	4858	# QUELLTEXTE, MRUN-SCRIPT UND KONFIGURATIONSDATEI IN DAS AKTUELLE
	4859	# ARBEITSVERZEICHNIS VERSCHIEBEN
	4860	echo "mv SOURCES_FOR_RUN_$fname/$config_file . " >> $jobfile
	4861	echo "mv SOURCES_FOR_RUN_$fname/$mrun_script_name . " >> $jobfile
	4862	echo "execute_mrun=true" >> $jobfile
	4863	echo " " >> $jobfile
	4864
[191]	4865	else
	4866
	4867	# ABSPEICHERN DER QUELLTEXTE (NUR FALLS UEBERSETZT WERDEN SOLL)
	4868	# SOWIE GEGEBENENFALLS DES MAKEFILES
	4869	if [[ $do_compile = true ]]
	4870	then
	4871
	4872	source_catalog=SOURCES_FOR_RUN_$fname
	4873
	4874	# UNTERVERZEICHNIS FUER QUELLTEXTE UND MAKEFILE ANLEGEN
	4875	# MRUN WIRD DIESES VRZEICHNIS UEBER ENVIRONMENT VARIABLE
	4876	# MITGETEILT (UEBERSTEUERT ANGABE IN KONFIGURATIONSDATEI)
	4877	echo "mkdir SOURCES_FOR_RUN_$fname" >> $jobfile
	4878	echo "export SOURCES_COMPLETE=true" >> $jobfile
	4879	echo "cd SOURCES_FOR_RUN_$fname" >> $jobfile
	4880
	4881	for filename in $source_list
	4882	do
	4883	# ABDECKZEICHEN VERHINDERN, DASS ERSETZUNGEN ERFOLGEN
	4884	echo "cat > $filename << \"%END%\"" >> $jobfile
	4885	cat $source_catalog/$filename >> $jobfile
	4886	echo " " >> $jobfile
	4887	echo "%END%" >> $jobfile
	4888	echo " " >> $jobfile
	4889	done
	4890
[1]	4891	# ABDECKZEICHEN VERHINDERN, DASS ERSETZUNGEN ERFOLGEN
[191]	4892	echo "cat > Makefile << \"%END%\"" >> $jobfile
	4893	cat $source_catalog/Makefile >> $jobfile
[1]	4894	echo " " >> $jobfile
	4895	echo "%END%" >> $jobfile
	4896	echo " " >> $jobfile
	4897
[191]	4898	echo "cd - > /dev/null" >> $jobfile
[1]	4899
[191]	4900	fi
[1]	4901
	4902
[191]	4903	# ABSPEICHERN DER KONFIGURATIONSDATEI
	4904	# ABDECKZEICHEN VERHINDERN, DASS ERSETZUNGEN ERFOLGEN
	4905	echo "cat > $config_file << \"%END%\"" >> $jobfile
	4906	cat $config_file >> $jobfile
	4907	echo "%END%" >> $jobfile
	4908	echo " " >> $jobfile
[1]	4909
	4910
[191]	4911	# ABSPEICHERN DER AKTUELLEN MRUN-VERSION
	4912	# ABDECKZEICHEN VERHINDERN, DASS ERSETZUNGEN ERFOLGEN
[693]	4913	echo "cat > $mrun_script_name <<\"%END%\"" >> $jobfile
	4914	cat ${PALM_BIN}/$mrun_script_name >> $jobfile
	4915	echo "%END%" >> $jobfile
[191]	4916	echo "chmod u+x $mrun_script_name" >> $jobfile
	4917	echo "execute_mrun=true" >> $jobfile
	4918	echo " " >> $jobfile
[1]	4919
	4920	fi
	4921
[191]	4922
[1]	4923	# EVTL. BENOETIGTE INPUT-DATEIEN PER FTP HOLEN ODER DEM JOB DIREKT
	4924	# MITGEBEN UND AUF DEM REMOTE-RECHNER IM BENUTZERVERZEICHNIS ABLEGEN
	4925	# FALLS DIESES NICHT EXISTIERT, WIRD VERSUCHT, DAS JEWEILS LETZTE
	4926	# UNTERVERZEICHNIS DES PFADNAMENS ANZULEGEN
	4927	if [[ $do_remote = true ]]
	4928	then
	4929	(( i = 0 ))
	4930	while (( i < iin ))
	4931	do
	4932	(( i = i + 1 ))
	4933	echo "[[ ! -d ${pathin[$i]} ]] && mkdir -p ${pathin[$i]}" >> $jobfile
	4934	if [[ "${transin[$i]}" = job ]]
	4935	then
	4936	echo "cat > ${remotepathin[$i]} <<\"%END%\"" >> $jobfile
	4937	eval cat ${pathin[$i]}/${frelin[$i]} >> $jobfile
	4938	echo " " >> $jobfile
	4939	echo "%END%" >> $jobfile
	4940	else
[82]	4941	echo "batch_scp -b -o -g -s -u $return_username $return_addres ${remotepathin[$i]} \"${pathin[$i]}\" ${frelin[$i]}" >> $jobfile
[1]	4942	fi
	4943
	4944	# UEBERPRUEFEN, OB DATEI ANGELEGT WERDEN KONNTE
	4945	echo "if [[ \$? = 1 ]]" >> $jobfile
	4946	echo "then" >> $jobfile
	4947	echo " echo \" \" " >> $jobfile
	4948	echo " echo \"+++ file ${remotepathin[$i]} could not be created\" " >> $jobfile
	4949	echo " echo \" please check, if directory exists on $host!\" " >> $jobfile
	4950	echo " echo \"+++ MRUN will not be continued\" " >> $jobfile
	4951	echo " execute_mrun=false" >> $jobfile
	4952	echo "fi" >> $jobfile
	4953	done
	4954	fi
	4955
	4956	# ARBEITSKATALOG AUF DER LOKALEN MASCHINE FUER EVENTUELLE
	4957	# FORTSETZUNGSLAUEFE PER ENV-VARIABLE UEBERGEBEN
	4958	echo "LOCAL_PWD=$working_directory" >> $jobfile
	4959	echo "export LOCAL_PWD" >> $jobfile
	4960
	4961	# EBENSO LOKALEN MRUN-KATALOG UEBERGEBEN
[22]	4962	echo "LOCAL_MRUN_PATH=$PALM_BIN" >> $jobfile
[1]	4963	echo "export LOCAL_MRUN_PATH" >> $jobfile
	4964
[66]	4965	# WORKAROUND FUER RIAM-NEC-JOBS WEGEN PROFILE-SCHWIERIGKEITEN
[69]	4966	if [[ $localhost_realname = "gate" \|\| $localhost = lctit ]]
[66]	4967	then
	4968	echo "export PALM_BIN=$PALM_BIN" >> $jobfile
	4969	fi
[1]	4970
[693]	4971	# MRUN AUF ZIELRECHNER AUFRUFEN (queue is workaround for ibmkisti)
[1]	4972	echo "set -x" >> $jobfile
[693]	4973	echo "queue=$queue" >> $jobfile
[1]	4974	echo "[[ \$execute_mrun = true ]] && ./$mrun_com" >> $jobfile
	4975
	4976	echo 'ls -al; echo `pwd`' >> $jobfile
	4977	echo "cd \$HOME" >> $jobfile
	4978	echo "rm -rf $TEMPDIR" >> $jobfile
	4979
	4980
	4981
	4982
	4983	# JOB PER SUBJOB STARTEN
	4984	if [[ $silent = false ]]
	4985	then
	4986	printf "\n "
	4987	else
	4988	printf "\n\n"
	4989	fi
	4990
[352]	4991	subjob $job_on_file -h $host -u $remote_username -g $group_number -q $queue -m $memory -N $node_usage -t $cpumax $XOPT $TOPT $OOPT -n $fname -v -c $job_catalog -e $email_notification $jobfile
[1]	4992	rm -rf $jobfile
	4993
	4994
	4995	fi # ENDE REMOTE-TEIL
	4996
	4997
	4998
	4999	# ENDE DER PROZEDUR

Note: See TracBrowser for help on using the repository browser.

Download in other formats:

| Impressum | ©Leibniz Universität Hannover |