Home

Context Navigation

source: palm/trunk/SCRIPTS/palmrun @ 4719

Last change on this file since 4719 was 4536, checked in by raasch, 4 years ago
messages and debug output converted to PALM routines (restart_data_mpi_io_mod), binary version number set to 5.0, heeader output for restart data format added, restart data filesize and I/O transfer speed added in cpu_measures, handling of single restart files (created with MPI-I/O) added to palmrun, bugfix: preprocessor directive adjusted (virtual_measurement_mod), location message format changed
Property svn:executable set to ``* Property svn:keywords set to `Id Rev`
File size: 105.7 KB

Rev	Line
[2380]	1	#!/bin/bash
	2
	3	# palmrun - script for running PALM jobs
	4
	5	#--------------------------------------------------------------------------------#
[2696]	6	# This file is part of the PALM model system.
[2380]	7	#
	8	# PALM is free software: you can redistribute it and/or modify it under the terms
	9	# of the GNU General Public License as published by the Free Software Foundation,
	10	# either version 3 of the License, or (at your option) any later version.
	11	#
	12	# PALM is distributed in the hope that it will be useful, but WITHOUT ANY
	13	# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
	14	# A PARTICULAR PURPOSE. See the GNU General Public License for more details.
	15	#
	16	# You should have received a copy of the GNU General Public License along with
	17	# PALM. If not, see <http://www.gnu.org/licenses/>.
	18	#
[4481]	19	# Copyright 2017-2020 Leibniz Universitaet Hannover
[2380]	20	#--------------------------------------------------------------------------------#
	21	#
	22	# Current revisions:
	23	# ------------------
	24	#
[2410]	25	#
[2380]	26	# Former revisions:
	27	# -----------------
[3032]	28	# $Id: palmrun 4536 2020-05-17 17:24:13Z pavelkrc $
[4536]	29	# handling of single restart files (created with MPI-I/O) added
	30	#
	31	# 4481 2020-03-31 18:55:54Z maronga
[4443]	32	# add cycle number to initial files too (000),
	33	# remove compatibility for cycle numbers which are less than three digits wide
	34	#
	35	# 4425 2020-02-26 15:35:00Z raasch
[4425]	36	# hard coded path to the fileconnection file replaced by a path that is based on
	37	# the source path which has been given in the configuration file
	38	#
	39	# 4303 2019-11-25 08:55:18Z raasch
[4303]	40	# Abort if temporary working catalog cannot be created
	41	#
	42	# 3982 2019-05-16 11:07:31Z raasch
[3982]	43	# bugfix: -b is not added in automatically generated batch jobs for remote hosts
	44	#
	45	# 3917 2019-04-18 11:46:37Z raasch
[3917]	46	# in case of batch jobs on remote machines, copying files from SOURCES_FOR_RUN folder has
	47	# been rearranged to avoid that input files with original names (e.g. ..._p3d) are copied
	48	# into the temporary working catalog
	49	#
	50	# 3860 2019-04-04 06:23:27Z raasch
[3860]	51	# check added, that it is not allowed to set option -b if the configuration file has been setup
	52	# for running batch jobs on remote hosts,
	53	# local time added to terminal output before and after saving output files
	54	#
	55	# 3725 2019-02-07 10:11:02Z raasch
[3725]	56	# error messages for failed restarts extended
	57	#
	58	# 3682 2019-01-18 17:01:54Z knoop
[3665]	59	# ssh-call for submitting batch jobs on remote systems modified again to avoid output
	60	# of login messages on specific systems
	61	#
	62	# 3574 2018-11-28 13:14:39Z raasch
[3574]	63	# syntax-error in last commit fixed
	64	#
	65	# 3573 2018-11-28 13:03:44Z raasch
[3573]	66	# bugfixes: don't check the .palm.iofiles in restart runs + wrong cd-command
	67	# removed,
	68	# r3570 reverted
	69	#
	70	# 3570 2018-11-27 17:44:21Z kanani
[3570]	71	# Rename job-id by run_id
	72	#
	73	# 3555 2018-11-22 13:50:37Z raasch
[3555]	74	# sed-command to remove repeated space-characters replaced by tr-command, to
	75	# guarantee mac compatible
	76	#
	77	# 3549 2018-11-21 15:44:44Z raasch
[3549]	78	# ssh-call for submitting batch jobs on remote systems modified to avoid output
	79	# of login messages on specific systems
	80	#
	81	# 3534 2018-11-19 15:35:16Z raasch
[3534]	82	# batch_scp for sending back the job protocol file is called via login-node if
	83	# a login-node has been set in the config-file
	84	# ssh-calls rearranged to avoid output of system/user-profile messages
	85	#
	86	# 3505 2018-11-08 09:14:30Z raasch
[3505]	87	# job folders are created in case that they do not exist
	88	#
	89	# 3455 2018-10-30 14:12:31Z raasch
[3455]	90	# options -h and -d renamed -c and -r, respectively
	91	#
	92	# 3402 2018-10-23 11:49:24Z knoop
[3346]	93	# job_id is set for compatibility reasons with older versions of config files
	94	#
	95	# 3313 2018-10-06 15:22:48Z knoop
[3294]	96	# more detailed error message for outdated fileconnection file
	97	#
	98	# 3236 2018-09-10 09:07:06Z raasch
[3236]	99	# warning issued in case of file connection file version mismatch,
	100	# host_configuration renamed configuration_identifier,
	101	# "host identifier" in header output renamed "config. identifier",
	102	# jobname renamed run_identifier,
	103	# run_id renamed run_id_number
	104	# job_id renamed run_id
	105	# option -r renamed -i
	106	#
	107	# 3151 2018-07-19 08:45:38Z raasch
[3151]	108	# removed additional listing of files to be compiled, print PALM code revision
	109	# in header
	110	#
	111	# 3132 2018-07-16 11:44:58Z raasch
[3132]	112	# create consistent error messages in case of failed restarts,
	113	# check existence of remote job catalog
	114	#
	115	# 3043 2018-05-25 10:47:54Z raasch
[3043]	116	# check for setting of fast_io_catalog
	117	#
	118	# 3033 2018-05-23 15:26:19Z raasch
[3025]	119	# -T option not required for interactive runs (tasks per node =
	120	# total number of cores is assumed as default, if option is not given),
	121	# reformatting of a few messages,
	122	# bugfix: destination output directory is created if attribute "pe" is set in
	123	# .palm.iofiles
	124	#
	125	# 2718 2018-01-02 08:49:38Z maronga
[2993]	126	# bugfix of r2990
	127	#
	128	# 2718 2018-01-02 08:49:38Z maronga
[2990]	129	# bugfix: temporary SOURCES_FOR_RUN_CATALOG now created in base_directory and
	130	# not in the working directory from where palmrun is called
	131	#
	132	# 2718 2018-01-02 08:49:38Z maronga
[2957]	133	# svfout in the activation string will be replaced now by svfin in case of
	134	# automatic restarts
	135	#
	136	# 2718 2018-01-02 08:49:38Z maronga
[2906]	137	# Introduction of new ENVIRONMENT variables read_svf and write_svf to consider
	138	# the possible output or input of sky view factors
	139	#
	140	# 2718 2018-01-02 08:49:38Z maronga
[2834]	141	# "fname" renamed to "jobname"
	142	#
	143	# 2718 2018-01-02 08:49:38Z maronga
[2799]	144	# parameters are allowed in output commands (OC:)
	145	#
	146	# 2718 2018-01-02 08:49:38Z maronga
[2789]	147	# bugfix: exit if no queue has been given neither with option -q nor in the
	148	# configuration file
	149	#
	150	# 2718 2018-01-02 08:49:38Z maronga
[2762]	151	# file connection file from trunk/SCRIPTS is taken by default
	152	#
	153	# 2718 2018-01-02 08:49:38Z maronga
[2716]	154	# Corrected "Former revisions" section
	155	#
	156	# 2696 2017-12-14 17:12:51Z kanani
	157	# Change in file header (GPL part)
	158	#
	159	# 2693 2017-12-13 12:33:49Z raasch
[2693]	160	# replacement of variables given by {{<variable>}} in configuration file is now
	161	# done in a more general way, variable names are written in small letters
	162	#
	163	# 2670 2017-12-06 16:32:19Z raasch
[2670]	164	# test printouts removed
	165	#
	166	# 2669 2017-12-06 16:03:27Z raasch
[2669]	167	# file attributes in .palm.iofiles restructured, "loc" attribute completely
	168	# removed,
	169	# wildcard (*) allowed in .palm.iofiles as file activation string for output
	170	# files,
	171	# informative messages in case of missing optional input files shortened
	172	# bugfix: variable cycle explicitly interpreted with 10 as the number base
	173	#
	174	# 2638 2017-11-23 12:44:23Z raasch
[2610]	175	# use of wildcards in file connection statements enabled
	176	#
	177	# 2605 2017-11-09 15:31:46Z raasch
[2605]	178	# in case of remote jobs, input files with "job" or "jobopt" (new) attribute
	179	# will not be sent with the job file any more, but copied into the
[2610]	180	# SOURCES_FOR_RUN... folder on the remote host, before the job is submitted
[2605]	181	#
	182	# 2600 2017-11-01 14:11:20Z raasch
[2600]	183	# cycle numbers are made three digits wide
	184	#
	185	# 2566 2017-10-20 08:50:47Z raasch
[2566]	186	# execute command for combine_plot_fields added
	187	# "TEMPDIR" renamed "tempdir"
	188	# temporary working directory for local batch jobs is created immediately within
	189	# the user's palmrun call, due to a requirement of the "grid engine" batch
	190	# system, where the working directory is given with batch directive -wd and must
	191	# already exist when the job is submitted,
	192	# informative messages in non-trace mode reduced and partly reformatted
	193	#
	194	# 2551 2017-10-18 07:25:11Z raasch
[2551]	195	# TEMPDIR added as replacement string to be used in batch directives
	196	#
	197	# 2512 2017-10-04 08:26:59Z raasch
[2507]	198	# bugfix for determining cycle numbers of NetCDF input files
	199	#
	200	# 2506 2017-09-29 08:30:37Z raasch
[2506]	201	# option -V added to check for an existing SOURCES_FOR_RUN_... folder
	202	# host configuration added to SOURCES_FOR_RUN_... folder name
	203	# host_identifier renamed host_configuration
	204	# option -W added to allow for job dependencies
	205	#
	206	# 2501 2017-09-26 11:41:55Z raasch
[2501]	207	# default value for number of cores (option -X) set to 1
	208	# bugfix for mechanism which overwrites configuration file settings with values
	209	# provided by palmrun options
	210	#
	211	# 2499 2017-09-22 16:47:58Z kanani
[2499]	212	# option -h named configuration identifier
	213	#
	214	# 2480 2017-09-19 06:24:14Z maronga
[2480]	215	# bugfix for last revision
	216	#
	217	# 2479 2017-09-19 06:12:16Z raasch
[2479]	218	# option -A (project account number) added
	219	#
	220	# 2422 2017-09-08 08:25:41Z raasch
[2410]	221	# initial revision
[2380]	222	#
	223	#--------------------------------------------------------------------------------#
	224	# palmrun - script for running PALM jobs on local and remote hosts
	225	#--------------------------------------------------------------------------------#
	226
	227
	228
	229	# DECLARATION OF VARIABLES AND THEIR DEFAULT VALUES
	230
	231	set +o allexport # SUPPRESS EXPORT OF ALL VARIABLES, SINCE IN THE PAST THIS
	232	# LEAD TO PROBLEMS IN ROUTINES CALLED BY PALMRUN
	233	# (TOO MANY ARGUMENTS - PROBLEM)
	234	set +o noclobber # EXISTING FILES ARE ALLOWED TO BE OVERWRITTEN
	235
[2610]	236	typeset -i ibd=0 ibdt=0 iec=0 iic=0 iin=0 ioc=0 iout=0 nr_of_input_files=0
	237	typeset -i nr_of_output_files=0
[2380]	238	typeset -x -i memory=0 # HAS TO BE EXPORTED HERE, OTHERWISE AN UNKNOWN SIDE
	239	# SIDE EFFECT MAY CAUSE DATA LOSS WHEN GETOPTS IS READING THE
	240	# SCRIPT-OPTION ARGUMENTS
	241	typeset -i cores cputime cpu_hours cpu_minutes cpu_seconds i ii iia iii iio
	242	typeset -i icycle inode ival maxcycle mpi_tasks
	243	typeset -i nodes remaining_cores run_number tasks_per_node threads_per_task
	244
	245	activation_string_list=""
	246	AddFilenames=""
	247	combine_plot_fields=true
	248	compiler_name=""
	249	compiler_name_ser=""
	250	compiler_options=""
[3236]	251	configuration_identifier="default"
[2501]	252	cores=1
[2380]	253	cores_atmos=0
	254	cores_ocean=0
	255	coupled_dist=""
	256	cpp_options=""
	257	cpumax=0
[2506]	258	create_batch_job=false
[2380]	259	create_jobfile_only=false
	260	create_remote_batch_job=false
	261	dashes=" ----------------------------------------------------------------------------"
	262	defaultqueue=""
[2506]	263	delete_temporary_catalog=true
[2380]	264	do_compile=true
	265	do_trace=false
	266	executable=""
	267	execute_command=""
	268	execution_error=false
[4425]	269	fileconnection_file=""
[2380]	270	global_revision=""
	271	hostfile=""
	272	hp=""
	273	keep_data_from_previous_run=false
	274	link_local_input=false
	275	link_local_output=false
[2388]	276	linker_options=""
[2385]	277	local_jobcatalog=""
[2380]	278	locat=normal
	279	makefile=""
	280	max_par_io_str=""
	281	prc=$0
	282	while [[ $(echo $prc \| grep -c "/") != 0 ]]
	283	do
	284	prc=`echo $prc \| cut -f2- -d"/"`
	285	done
	286	module_calls=""
[2506]	287	palmrun_memory=""
[2380]	288	palmrun_script_name=$prc
	289	openmp=false
[2506]	290	previous_job=""
[2479]	291	project_account=""
[2380]	292	queue=none
[2906]	293	read_svf=""
[2380]	294	restart_run=false
	295	return_address=""
[2385]	296	remote_jobcatalog=""
[2380]	297	remote_username=""
	298	running_in_batch_mode=false
[3313]	299	running_in_test_mode=false
[2380]	300	run_coupled_model=false
	301	run_id=""
[3236]	302	run_id_number=""
	303	run_identifier=test
[2380]	304	silent=false
	305	source_list=""
	306	source_path=""
[2501]	307	tasks_per_node=0
[2380]	308	threads_per_task=1
	309	transfer_problems=false
[2384]	310	user_source_path=""
[2506]	311	use_existing_sources_folder=""
[2380]	312	use_openmp=false
[3033]	313	version="palmrun 1.0 Rev$Rev: 4536 $"
[2380]	314	working_directory=`pwd`
	315	write_binary=""
[2906]	316	write_svf=""
[2380]	317
	318
	319	# ERROR HANDLING IN CASE OF EXIT
[2404]	320	trap 'if [[ $locat != normal && $locat != control_c ]]
[2380]	321	then
	322
	323	# CARRY OUT ERROR-COMMANDS GIVEN IN THE CONFIGURATION FILE (EC:)
	324	(( i = 0 ))
	325	while (( i < iec ))
	326	do
	327	(( i = i + 1 ))
	328	printf "\n *** Execution of ERROR-command:\n"
	329	printf " >>> ${err_command[$i]}\n"
	330	eval ${err_command[$i]}
	331	done
[2566]	332	[[ $delete_temporary_catalog = true ]] && (cd; rm -rf $tempdir)
[3312]	333	printf "\n\n+++ palmrun crashed \n\n"
	334	exit 1
	335	elif [[ $locat != normal ]]
[2380]	336	then
[3312]	337	[[ $delete_temporary_catalog = true ]] && (cd; rm -rf $tempdir)
	338	printf "\n+++ palmrun killed by \"^C\" \n\n"
	339	exit 2
	340	else
[2566]	341	printf "\n --> palmrun finished\n\n"
[3312]	342	exit 0
[2380]	343	fi' exit
	344
	345
	346	# ACTIONS IN CASE OF TERMINAL-BREAK (CONTROL-C):
[3312]	347	trap 'locat=control_c
	348	exit 1
[2380]	349	' 2
	350
	351
	352	# READ SHELLSCRIPT-OPTIONS AND REBUILD THE PALMRUN-COMMAND STRING (prc),
	353	# WHICH WILL BE USED TO START RESTART-JOBS
[3455]	354	while getopts :a:A:bBc:CFG:i:jkm:M:O:q:r:R:s:t:T:u:U:vVw:W:xX:yY:zZ option
[2380]	355	do
	356	case $option in
	357	(a) activation_string_list=$OPTARG; prc="$prc -a'$OPTARG'";;
[2479]	358	(A) project_account=$OPTARG; prc="$prc -A'$OPTARG'";;
[2380]	359	(b) create_batch_job=true; prc="$prc -b";;
	360	(B) delete_temporary_catalog=false; prc="$prc -B";;
[3455]	361	(c) configuration_identifier=$OPTARG; prc="$prc -c$OPTARG";;
[2380]	362	(C) restart_run=true; prc="$prc -C";;
	363	(F) create_jobfile_only=true;;
	364	(G) global_revision=$OPTARG; prc="$prc -G'$OPTARG'";;
[3236]	365	(i) run_id_number=$OPTARG;;
[2380]	366	(j) running_in_batch_mode=true;;
	367	(k) keep_data_from_previous_run=true; prc="$prc -k";;
[2501]	368	(m) palmrun_memory=$OPTARG; prc="$prc -m$OPTARG";;
[2380]	369	(M) makefile=$OPTARG; prc="$prc -M$OPTARG";;
	370	(O) use_openmp=true; threads_per_task=$OPTARG; prc="$prc -O$OPTARG";;
	371	(q) queue=$OPTARG; prc="$prc -q$OPTARG";;
[3455]	372	(r) run_identifier=$OPTARG; prc="$prc -r$OPTARG";;
[2380]	373	(R) return_address=$OPTARG;;
	374	(s) source_list=$OPTARG;;
[2501]	375	(t) palmrun_cpumax=$OPTARG; prc="$prc -t$OPTARG";;
[2380]	376	(T) palmrun_tasks_per_node=$OPTARG; prc="$prc -T$OPTARG";;
	377	(u) remote_username=$OPTARG; prc="$prc -u$OPTARG";;
	378	(U) return_username=$OPTARG; prc="$prc -U$OPTARG";;
	379	(v) silent=true; prc="$prc -v";;
[2506]	380	(V) use_existing_sources_folder="-V";;
[2380]	381	(w) max_par_io_str=$OPTARG; prc="$prc -w$OPTARG";;
[2506]	382	(W) previous_job=$OPTARG;;
[2380]	383	(x) do_trace=true;set -x; prc="$prc -x";;
[2501]	384	(X) palmrun_cores=$OPTARG; prc="$prc -X$OPTARG";;
[2380]	385	(y) ocean_file_appendix=true; prc="$prc -y";;
	386	(Y) run_coupled_model=true; coupled_dist=$OPTARG; prc="$prc -Y'$OPTARG'";;
[3313]	387	(z) running_in_test_mode=true;;
[2380]	388	(Z) combine_plot_fields=false; prc="$prc -Z";;
	389	(\?) printf "\n +++ unknown option $OPTARG \n"
	390	printf "\n --> type \"$0 ?\" for available options \n"
	391	locat=parameter;exit;;
	392	esac
	393	done
	394
	395
	396	# SKIP GIVEN OPTIONS TO READ POSITIONAL PARAMETER, IF GIVEN
	397	# CURRENTLY ONLY PARAMETER ? (TO OUTPUT A SHORT COMMAND INFO) IS ALLOWED
	398	(( to_shift = $OPTIND - 1 ))
	399	shift $to_shift
	400
	401	# PRINT SHORT DESCRIPTION OF PALMRUN OPTIONS
	402	if [[ "$1" = "?" ]]
	403	then
	404	(printf "\n *** Description of available palmrun options:\n"
	405	printf "\n Option Description Default-Value"
	406	printf "\n -a activation string list \"\" "
[2479]	407	printf "\n -A project account number ---"
[2380]	408	printf "\n -b batch-job on local machine ---"
	409	printf "\n -B do not delete temporary directory at end ---"
[3455]	410	printf "\n -c configuration identifier \"default\" "
[2380]	411	printf "\n -F create batch job file only ---"
	412	printf "\n -k keep data from previous run"
	413	printf "\n -m memory demand in MB (batch-jobs) 0 MB"
	414	printf "\n -M Makefile name Makefile"
	415	printf "\n -O threads per openMP task ---"
	416	printf "\n -q queue \"$queue\" "
[3455]	417	printf "\n -r run identifier test"
[2380]	418	printf "\n -s filenames of routines to be compiled \"\" "
	419	printf "\n must end with .f, .f90, .F, or .c !"
	420	printf "\n use \"..\" for more than one file and wildcards"
	421	printf "\n -s LM compiles all locally modified files"
	422	printf "\n -t allowed cpu-time in seconds (batch) 0"
[2384]	423	printf "\n -T tasks per node ---"
[2380]	424	printf "\n -u username on remote machine \"\" "
	425	printf "\n -v no prompt for confirmation ---"
[2506]	426	printf "\n -V check if SOURCES_FOR_RUN_... exists ---"
[2380]	427	printf "\n -w maximum parallel io streams as given by -X"
[2506]	428	printf "\n -W name of job to wait for ---"
[2380]	429	printf "\n -x tracing of palmrun for debug purposes ---"
	430	printf "\n -X # of processors (on parallel machines) 1"
	431	printf "\n -y add appendix \"_O\" to all local output"
	432	printf "\n files (ocean precursor runs followed by"
	433	printf "\n coupled atmosphere-ocean runs) ---"
	434	printf "\n -Y run coupled model, \"#1 #2\" with"
	435	printf "\n #1 atmosphere and #2 ocean processors \"#/2 #/2\" depending on -X"
	436	printf "\n -Z skip combine_plot_fields at the end of "
	437	printf "\n the simulation ---"
	438	printf "\n "
	439	printf "\n Possible values of positional parameter <modus>:"
	440	printf "\n \"?\" - this outline \n\n") \| more
	441	exit
	442	elif [[ "$1" != "" ]]
	443	then
	444	printf "\n +++ positional parameter $1 unknown \n"
	445	locat=parameter; exit
	446	fi
	447
	448
	449
	450	# SHORT STARTING MESSAGE
	451	printf "\n*** $version "
	452	printf "\n will be executed. Please wait ..."
	453
	454
[2506]	455	# BUILD THE CONFIGURATION-FILE NAME AND THE SOURCES_FOR_RUN-FOLDER NAME
[3236]	456	config_file=.palm.config.$configuration_identifier
	457	sources_for_run_catalog=SOURCES_FOR_RUN_${configuration_identifier}_$run_identifier
[2380]	458
	459
	460	# CHECK, IF CONFIGURATION FILE EXISTS
	461	if [[ ! -f $config_file ]]
	462	then
	463	printf "\n\n +++ configuration file: "
	464	printf "\n $config_file"
	465	printf "\n does not exist"
	466	locat=connect; exit
	467	fi
	468
	469
[2404]	470	# CHECK, IF THE ACTIVATION_STRING_LIST HAS BEEN GIVEN
	471	if [[ "$activation_string_list" = "" ]]
	472	then
	473	printf "\n\n +++ no activation string list given: "
	474	printf "\n please set palmrun option \"-a\" "
	475	locat=palmrun_option; exit
	476	fi
	477
	478
	479	# SET VARIABLE TO ACTIVATE PALM BINARY OUTPUT FOR RESTARTS
	480	if [[ $(echo $activation_string_list \| grep -c "restart") != 0 ]]
	481	then
	482	write_binary=true
	483	else
	484	write_binary=false
	485	fi
	486
[2906]	487	# SET VARIABLE TO ACTIVATE WRITING OF SVF DATA
	488	if [[ $(echo $activation_string_list \| grep -c "svfout") != 0 ]]
	489	then
	490	write_svf=true
	491	else
	492	write_svf=false
	493	fi
[2404]	494
[2906]	495	# SET VARIABLE TO ACTIVATE READING OF SVF DATA
	496	if [[ $(echo $activation_string_list \| grep -c "svfin") != 0 ]]
	497	then
	498	read_svf=true
	499	else
	500	read_svf=false
	501	fi
	502
	503
[2380]	504	# READ AND EVALUATE THE CONFIGURATION-FILE
	505	[[ $silent = false ]] && printf "\n\n Reading the configuration file... "
	506
	507	# READ VARIABLE SETTINGS FROM CONFIG FILE LINE BY LINE
	508	while read line
	509	do
	510
	511	# FIRST REPLACE ENVIRONMENT-VARIABLES BY THEIR RESPECTIVE VALUES
	512	eval line=\"$line\"
	513
	514
	515	# INTERPRET THE LINE
	516	if [[ "$(echo $line)" = "" ]]
	517	then
	518
	519	# EMPTY LINE, NO ACTION
	520	continue
	521
	522	elif [[ "$(echo $line \| cut -c1)" = "#" ]]
	523	then
	524
	525	# LINE IS A COMMENT LINE
	526	continue
	527
	528	elif [[ "$(echo $line \| cut -c1)" = "%" ]]
	529	then
	530
	531	# LINE DEFINES AN ENVIRONMENT-VARIABLE
	532	var=`echo $line \| cut -d" " -s -f1 \| cut -c2-`
	533	value=`echo $line \| cut -d" " -s -f2-`
	534
	535
	536	# VALUE FROM THE CONFIGURATION-FILE IS ASSIGNED TO THE
	537	# ENVIRONMENT-VARIABLE, BUT ONLY IF NO VALUE HAS BEEN ALREADY
	538	# ASSIGNED WITHIN THIS SCRIPT (E.G. BY SCRIPT-OPTIONS).
	539	# NON-ASSIGNED VARIABLES HAVE VALUE "" OR 0 (IN CASE OF INTEGER).
	540	# HENCE THE GENERAL RULE IS: SCRIPT-OPTION OVERWRITES THE
	541	# CONFIGURATION-FILE.
	542	if [[ "$(eval echo \$$var)" = "" \|\| "$(eval echo \$$var)" = "0" ]]
	543	then
	544	eval export $var="\$value"
	545
	546	# TERMINAL OUTPUT OF ENVIRONMENT-VARIABLES, IF TRACEBACK IS SWITCHED on
	547	if [[ $do_trace = true ]]
	548	then
	549	printf "\n*** ENVIRONMENT-VARIABLE $var = $value"
	550	fi
	551	fi
	552
	553	elif [[ "$(echo $line \| cut -c1-3)" = "BD:" ]]
	554	then
	555
	556	# LINE DEFINES BATCH-DIRECTIVE
	557	(( ibd = ibd + 1 ))
	558	line=$(echo $line \| cut -c4-)
	559	batch_directive[$ibd]="$line"
	560
	561	elif [[ "$(echo $line \| cut -c1-4)" = "BDT:" ]]
	562	then
	563
	564	# LINE DEFINES BATCH-DIRECTIVE FOR SENDING BACK THE JOBFILE FROM A
	565	# REMOTE TO A LOCAL HOST
	566	(( ibdt = ibdt + 1 ))
	567	line=$(echo $line \| cut -c5-)
	568	batch_directive_transfer[$ibdt]="$line"
	569
	570	elif [[ "$(echo $line \| cut -c1-3)" = "EC:" ]]
	571	then
	572
	573	# LINE DEFINES ERROR-COMMAND
	574	(( iec = iec + 1 ))
	575	line=$(echo $line \| cut -c4-)
	576	err_command[$iec]="$line"
	577
	578	elif [[ "$(echo $line \| cut -c1-3)" = "IC:" ]]
	579	then
	580
	581	# LINE DEFINES INPUT-COMMAND
	582	(( iic = iic + 1 ))
	583	line=$(echo $line \| cut -c4-)
	584	in_command[$iic]="$line"
	585
	586	elif [[ "$(echo $line \| cut -c1-3)" = "OC:" ]]
	587	then
	588
	589	# LINE DEFINES OUTPUT-COMMAND
	590	(( ioc = ioc + 1 ))
	591	line=$(echo $line \| cut -c4-)
	592	out_command[$ioc]="$line"
	593
	594	else
	595
	596	# SKIP ALL OTHER LINES
	597	continue
	598
	599	fi
	600
	601	done < $config_file
	602
	603
	604	# CHECK SETTING OF REQUIRED PARAMETERS
[3043]	605	if [[ "$fast_io_catalog" = "" ]]
	606	then
[3860]	607	printf "\n\n +++ no fast_io_catalog found in $config_file"
[3043]	608	printf "\n Please add line \"fast_io_catalog ...\" to that file."
	609	locat=config_file; exit
	610	fi
[2380]	611	if [[ "$compiler_name" = "" ]]
	612	then
[3860]	613	printf "\n\n +++ no compiler name found in $config_file"
[2380]	614	printf "\n Please add line \"compiler_name ...\" to that file."
	615	locat=config_file; exit
	616	fi
	617	if [[ "$compiler_name_ser" = "" ]]
	618	then
[3860]	619	printf "\n\n +++ no compiler name for non-paralle compilation found in $config_file"
[2380]	620	printf "\n Please add line \"compiler_name_ser ...\" to that file."
	621	locat=config_file; exit
	622	fi
	623	if [[ "$compiler_options" = "" ]]
	624	then
[3860]	625	printf "\n\n +++ no compiler options found in $config_file"
[2380]	626	printf "\n Please add line \"compiler_options ...\" to that file."
	627	locat=config_file; exit
	628	fi
	629	if [[ "$execute_command" = "" ]]
	630	then
[3860]	631	printf "\n\n +++ no execute command found in $config_file"
[2380]	632	printf "\n Please add line \"execute_command ...\" to that file."
	633	locat=config_file; exit
	634	fi
	635
	636	if [[ "$hostfile" != "" ]]
	637	then
	638	if [[ $hostfile != auto && ! -f $hostfile ]]
	639	then
[3860]	640	printf "\n\n +++ no hostfile \"$hostfile\" found"
[3151]	641	printf "\n Please check line \"hostfile ...\" in $config_file"
[2380]	642	locat=config_file; exit
	643	fi
	644	fi
	645
	646
	647	# DETERMINE THE CALL STATUS
	648	if [[ "$return_address" != "" ]]
	649	then
	650
	651	# I AM RUNNING ON A REMOTE HOST, WHICH ALSO MEANS THAT I AM RUNNING IN
	652	# BATCH MODE AND ...
	653	running_on_remote=true
	654
	655	else
	656
	657	# I HAVE BEEN CALLED INTERACTIVELY ON THIS HOST
	658	if [[ "$remote_ip" != "" ]]
	659	then
	660
	661	# I HAVE TO CREATE A BATCH JOB TO RUN PALM ON THE REMOTE HOST
	662	create_remote_batch_job=true
	663
[3132]	664	# CHECK THAT THE REMOTE JOBCATALOG HAS BEEN SET
	665	if [[ "$remote_jobcatalog" = "" ]]
	666	then
[3860]	667	printf "\n\n +++ no remote jobcatalog found in $config_file"
[3132]	668	printf "\n Please add line \"remote_jobcatalog ...\" to that file."
	669	locat=config_file; exit
	670	fi
	671
[3860]	672	# CHECK THAT OPTION -b (SUBMITTING BATCH JOB ON LOCAL HOST) HASN'T BEEN SET
	673	if [[ $create_batch_job = true ]]
	674	then
	675	printf "\n\n +++ option -b must not be set because configuration file has been setup"
	676	printf "\n for running jobs on a remote host"
	677	locat=config_file; exit
	678	fi
	679
[2380]	680	fi
	681	running_on_remote=false
	682	fi
	683
	684
[4425]	685	# SET THE DEFAULT FILE CONNECTION FILE (WHICH IS IN THE REPOSITORY)
	686	fileconnection_file=${source_path}/../SCRIPTS/.palm.iofiles
[2380]	687
[4425]	688	# CHECK, IF USER PROVIDES OWN FILE CONNECTION FILE
	689	if [[ -f .palm.iofiles ]]
	690	then
	691
	692	# CHECK VERSION MISMATCH
	693	# RESTART RUNS DON'T REQUIRE A CHECK, BECAUSE THEY USE A COPY OF THAT
	694	# FILE WHICH HAS ALREADY BEEN CHECKED WITHIN THE INITAL PALMRUN CALL
	695	if [[ $running_in_batch_mode != true && $restart_run != true ]]
	696	then
	697	if [[ $(head -n1 $fileconnection_file) != $(head -n1 .palm.iofiles) ]]
	698	then
	699	printf "\n\n +++ WARNING: A file connection file has been found in your"
	700	printf "\n working directory, but its revision does not match"
	701	printf "\n the revision of the default (trunk) version."
	702	printf "\n You may need to update your connection file"
	703	printf "\n \"${working_directory}/.palm.iofiles\" !"
	704	fi
	705	fi
	706
	707	fileconnection_file=.palm.iofiles
	708
	709	fi
	710
	711
	712	# CHECK, IF FILE CONNECTION FILE EXISTS
	713	if [[ ! -f $fileconnection_file ]]
	714	then
	715	printf "\n\n +++ file connection file: "
	716	printf "\n $fileconnection_file"
	717	printf "\n does not exist"
	718	locat=connect; exit
	719	fi
	720
	721
[2380]	722	# READ AND EVALUATE THE I/O-FILE LIST
	723	[[ $silent = false ]] && printf "\n Reading the I/O files... "
	724
	725	# READ THE FILE CONNECTION FILE LINE BY LINE
	726	while read line
	727	do
	728
[2669]	729	# REPLACE REPEATING SPACES BETWEEN THE COLUMNS BY A SINGLE SPACE
[3555]	730	# HERE, TR IS USED INSTEAD OF SED TO GUARANTEE MAC COMPATIBILITY
	731	line=`echo "$line" \| tr -s " "`
[2380]	732
	733	# INTERPRET THE LINE
	734	if [[ "$(echo $line)" = "" ]]
	735	then
	736	# EMPTY LINE, NO ACTION
	737	continue
	738
	739	elif [[ "$(echo $line \| cut -c1)" = "#" ]]
	740	then
	741
	742	# LINE IS A COMMENT LINE
	743	true
	744
	745	else
	746
	747	# LINE DEFINES FILE CONNECTION. READ THE FILE ATTRIBUTES.
	748	# s2a: in/out - field
[2669]	749	# s2b: action - field (optional)
	750	s1=`echo "$line" \| cut -d" " -f1`
	751	s2=`echo "$line" \| cut -d" " -s -f2`
[2380]	752	if [[ $(echo $s2 \| grep -c ":") = 0 ]]
	753	then
[2669]	754	s2a=$s2
[2380]	755	s2b=""
	756	else
[2669]	757	s2a=`echo $s2 \| cut -d":" -f1`
	758	s2b=`echo $s2 \| cut -d":" -f2`
[2380]	759	fi
[2669]	760	s3=`echo "$line" \| cut -d" " -f3 \| sed 's/*/wildcard /g'`
	761	s4=`echo "$line" \| cut -d" " -s -f4`
	762	eval s4=\"$s4\" # REPLACE ENVIRONMENT-VARIABLES IN PATH BY THEIR RESPECTIVE VALUES
	763	s5=`echo "$line" \| cut -d" " -s -f5`
	764	s6=`echo "$line" \| cut -d" " -s -f6`
[2380]	765
	766
	767	# STORE FILE CONNECTION, IF ACTIVATED BY ACTIVATION-STRING FROM
	768	# INPUT- OR OUTPUT-LIST.
	769	# VARIABLE S3 MAY CONTAIN A LIST OF ACTIVATION STRINGS (FIELD-SEPERATOR ":").
	770	# IF EXECUTION IS SCHEDULED FOR A REMOTE-MACHINE AND THE FILE IS ONLY
[2669]	771	# LOCALLY REQUIRED ON THAT MACHINE (I.E. s2b != tr), THE FILE CONNECTION
[2380]	772	# IS NOT CHECKED AND STORED.
[2610]	773	IFSALT="$IFS"; IFS="$IFS:" # ADD ":" AS FIELD SEPARATOR
[2669]	774	if [[ ( "$s2a" = in \|\| "$s2a" = inopt ) && ! ( $create_remote_batch_job = true && "$s2b" != tr ) ]]
[2380]	775	then
	776	found=false
	777	for actual in $activation_string_list
	778	do
	779	for formal in $s3
	780	do
	781	[[ $actual = $formal \|\| "$formal" = "-" ]] && found=true
	782	done
	783	done
	784	if [[ $found = true ]]
	785	then
	786	(( iin = iin + 1 ))
[2669]	787	localin_pre[$iin]=$s1; actionin_pre[$iin]=$s2b;
[2610]	788	pathin_pre[$iin]=$s4; endin_pre[$iin]=$s5; extin_pre[$iin]=$s6
[2669]	789	if [[ "$s2a" = inopt ]]
	790	then
	791	optin_pre[$iin]=yes
	792	else
	793	optin_pre[$iin]=no
	794	fi
[2605]	795
	796	# FILES WITH JOB-ATTRIBUTE ARE STORED IN THE SOURCES_FOR_RUN
	797	# FOLDER IF THE JOB IS RUNNING ON A REMOTE HOST
[2669]	798	if [[ $running_on_remote = true && "$s2b" = tr ]]
[2605]	799	then
[2610]	800	pathin_pre[$iin]=${fast_io_catalog}/${sources_for_run_catalog}
[2605]	801	fi
	802
[2610]	803	# CHECK FOR MULTIPLE FILES, SET A RESPECTIVE FLAG AND REMOVE
	804	# THE WILDCARD FROM THE ENDING
	805	if [[ "${s5: -1}" = "*" ]]
	806	then
[2669]	807	if [[ "$s2b" = "di" ]]
[2610]	808	then
	809	printf "\n +++ wildcards (*) not allowed with \"di\" file attribute."
	810	printf "\n see file \"$fileconnection_file\", line"
	811	printf "\n$line"
	812	locat=iofiles_file; exit
	813	fi
	814	multin[$iin]=true
	815	string=${endin_pre[$iin]}
	816	endin_pre[$iin]="${string%?}"
	817	else
	818	multin[$iin]=false
	819	fi
[2380]	820	fi
[2669]	821	elif [[ "$s2a" = out && ! ( $create_remote_batch_job = true ) ]]
[2380]	822	then
	823	found=false
	824	for actual in $activation_string_list
	825	do
	826	for formal in $s3
	827	do
[2670]	828	[[ $actual = $formal \|\| $formal = wildcard ]] && found=true
[2380]	829	done
	830	done
	831	if [[ $found = true ]]
	832	then
	833	(( iout = iout + 1 ))
[2669]	834	localout_pre[$iout]=$s1; actionout_pre[$iout]=$s2b
[2610]	835	pathout_pre[$iout]=$s4; endout_pre[$iout]=$s5; extout_pre[$iout]=$s6
	836
[2669]	837	# CHECK IF WILDCARD IS USED AS ACTIVATION STRING
	838	# IN SUCH CASES, NO WARNING WILL LATER BE OUTPUT IF LOCAL FILES DO NOT EXIST
	839	if [[ $formal = wildcard ]]
	840	then
	841	warnout_pre[$iout]=false
	842	else
	843	warnout_pre[$iout]=true
	844	fi
	845
[2610]	846	# CHECK FOR MULTIPLE FILES, SET A RESPECTIVE FLAG AND REMOVE
	847	# THE WILDCARD FROM THE LOCAL FILENAME
	848	if [[ "${s1: -1}" = "*" ]]
	849	then
[2669]	850	if [[ "$s2b" = "di" ]]
[2610]	851	then
	852	printf "\n +++ wildcards (*) not allowed with \"di\" file attribute."
	853	printf "\n see file \"$fileconnection_file\", line"
	854	printf "\n$line"
	855	locat=iofiles_file; exit
	856	fi
	857	multout[$iout]=true
	858	string=${localout_pre[$iout]}
	859	localout_pre[$iout]="${string%?}"
	860	else
	861	multout[$iout]=false
	862	fi
[2380]	863	fi
[2669]	864	elif [[ "$s2a" != in && "$s2a" != inopt && "$s2a" != out ]]
[2380]	865	then
[2669]	866	printf "\n +++ I/O-attribute in file $fileconnection_file has invalid"
	867	printf "\n value \"$s2\". Only \"in\", \"inopt\", and \"out\" are allowed!"
[2380]	868	locat=connect; exit
	869	fi
	870	IFS="$IFSALT"
	871	fi
	872
	873	done < $fileconnection_file
	874
	875
	876
[2501]	877	# VALUES OF PALMRUN-OPTIONS OVERWRITE THOSE FROM THE CONFIGURATION-FILE
[3025]	878	[[ "$palmrun_memory" != "" ]] && memory=$palmrun_memory
	879	[[ "$palmrun_cpumax" != "" ]] && cpumax=$palmrun_cpumax
	880	[[ "$palmrun_cores" != "" ]] && cores=$palmrun_cores
[2501]	881	[[ "$max_par_io_str" != "" ]] && maximum_parallel_io_streams=$max_par_io_str
	882	[[ "$palmrun_tasks_per_node" != "" ]] && tasks_per_node=$palmrun_tasks_per_node
	883
	884
	885
[2380]	886	# EVALUATE MODEL COUPLING FEATURES (OPTION -Y)
	887	if [[ $run_coupled_model = true ]]
	888	then
	889
	890	cores_atmos=`echo $coupled_dist \| cut -d" " -s -f1`
	891	cores_ocean=`echo $coupled_dist \| cut -d" " -s -f2`
	892
	893	if (( $cores_ocean + $cores_atmos != $cores ))
	894	then
	895
	896	printf "\n +++ number of processors does not fit to specification by \"-Y\"."
	897	printf "\n PEs (total) : $cores"
	898	printf "\n PEs (atmosphere): $cores_atmos"
	899	printf "\n PEs (ocean) : $cores_ocean"
	900	locat=coupling; exit
	901
	902	fi
	903
	904	fi
	905
	906
	907	# IF I AM IN BATCH MODE, CHECK IF EXECUTABLE AND OTHER REQUIRED FILES
	908	# HAVE BEEN GENERATED BY PALMBUILD AND STORED IN THE SOURCES_FOR_RUN_...
	909	# FOLDER
	910	if [[ $running_in_batch_mode = true ]]
	911	then
	912
[2506]	913	if [[ ! -d ${fast_io_catalog}/${sources_for_run_catalog} ]]
[2380]	914	then
[2506]	915	printf "\n +++ directory ${fast_io_catalog}/${sources_for_run_catalog} is missing"
[2380]	916	printf "\n Please check the output of the palmrun-call"
	917	printf "\n that you did on your local host."
	918	locat=SOURCES_FOR_RUN; exit
	919	fi
	920
	921	else
	922
	923	# CREATE THE SOURCES_FOR_RUN_... FOLDER, BUT NOT IF I AM PART OF AN
	924	# AUTOMATIC RESTART RUN
	925	# AUTOMATIC RESTART RUNS JUST ACCESS THE DIRECTORY CREATED BY THE INITIAL RUN
	926	if [[ $restart_run = false ]]
	927	then
	928
	929	# COLLECT FILES TO BE COMPILED IN THE SOURCES_FOR_RUN_... FOLDER ON
	930	# THE LOCAL HOST
	931	if [[ ! -d $source_path ]]
	932	then
	933	printf "\n\n +++ source path \"$source_path\" on local host"
	934	printf "\n \"$(hostname)\" does not exist"
	935	locat=source_path; exit
	936	fi
	937
[2990]	938	rm -rf ${base_directory}/${sources_for_run_catalog}
	939	mkdir -p ${base_directory}/${sources_for_run_catalog}
[2380]	940
	941	if [[ "$source_list" = LM ]]
	942	then
	943
	944	# DETERMINE MODIFIED FILES OF THE SVN WORKING COPY
	945	source_list=""
	946	cd $source_path
	947
	948
[2512]	949	# CHECK, IF TRUNK-DIRECTORY IS UNDER SVN CONTROL
	950	if [[ ! -d ../.svn ]]
[2380]	951	then
	952	printf "\n\n +++ source directory"
	953	printf "\n \"$source_path\" "
	954	printf "\n is not under control of \"subversion\"."
[2512]	955	printf "\n Please do not use palmrun-option \"-s LM\"\n"
[2380]	956	fi
	957
	958
	959	# LIST ALL MODIFIED SOURCE CODE FILES
	960	Filenames=""
[2404]	961	svn status > tmp_svnstatus
[2380]	962	while read line
	963	do
	964	firstc=`echo $line \| cut -c1`
	965	if [[ $firstc = M \|\| $firstc = "?" ]]
	966	then
	967	Name=`echo "$line" \| cut -c8-`
	968	extension=`echo $Name \| cut -d. -f2`
	969	if [[ "$extension" = f90 \|\| "$extension" = F90 \|\| "$extension" = f \|\| "$extension" = F \|\| "$extension" = c ]]
	970	then
	971	Filenames="$Filenames "$Name
	972	fi
	973	fi
[2404]	974	done < tmp_svnstatus
	975	rm -rf tmp_svnstatus
[2380]	976
	977
	978	# COPY FILES TO SOURCES_FOR_RUN_...
	979	for filename in $Filenames
	980	do
[2990]	981	cp $filename ${base_directory}/${sources_for_run_catalog}
[2380]	982	source_list=$source_list"$filename "
	983	done
	984
	985	cd - > /dev/null
	986
	987
	988	# COPY FILES GIVEN BY OPTION -s TO DIRECTORY SOURCES_FOR_RUN_...
	989	elif [[ "$source_list" != "" ]]
	990	then
	991
	992	cd $source_path
	993
	994	for filename in $source_list
	995	do
	996
	997	# SOURCE CODE FILE IS NOT ALLOWED TO INCLUDE PATH
	998	if [[ $(echo $filename \| grep -c "/") != 0 ]]
	999	then
	1000	printf "\n +++ source code file: $filename"
	1001	printf "\n must not contain (\"/\") "
	1002	locat=source; exit
	1003	fi
	1004
	1005	if [[ ! -f $filename ]]
	1006	then
	1007	printf "\n +++ source code file: $filename"
	1008	printf "\n does not exist"
	1009	locat=source; exit
	1010	else
[2990]	1011	cp $filename ${base_directory}/${sources_for_run_catalog}
[2380]	1012	fi
	1013
	1014	done
	1015
	1016	cd - > /dev/null
	1017
	1018	fi
	1019
	1020	# CHECK, IF MAKEFILE EXISTS AND COPY IT TO THE SOURCES_FOR_RUN... DIRECTORY
	1021	[[ "$makefile" = "" ]] && makefile=$source_path/Makefile
	1022	if [[ ! -f $makefile ]]
	1023	then
	1024	printf "\n +++ file \"$makefile\" does not exist"
	1025	locat=make; exit
	1026	else
[2990]	1027	cp $makefile ${base_directory}/${sources_for_run_catalog}/Makefile
[2380]	1028	fi
	1029
	1030
	1031	# COPY FILES FROM OPTIONAL SOURCE PATH GIVEN IN THE CONFIGURATION FILE
[2384]	1032	if [[ "$user_source_path" != "" ]]
[2380]	1033	then
	1034
	1035	# DOES THE DIRECTORY EXIST?
[2384]	1036	if [[ ! -d $user_source_path ]]
[2380]	1037	then
	1038
	1039	printf "\n\n *** INFORMATIVE: additional source code directory"
[2384]	1040	printf "\n \"$user_source_path\" "
[2380]	1041	printf "\n does not exist or is not a directory."
	1042	printf "\n No source code will be used from this directory!\n"
[2384]	1043	user_source_path=""
[2380]	1044	if [[ $silent == false ]]
	1045	then
	1046	sleep 2
	1047	fi
	1048
	1049	else
	1050
[2384]	1051	cd $user_source_path
[2380]	1052
	1053	Names=$(ls -1 *.f90 2>&1)
	1054	[[ $(echo $Names \| grep -c '*.f90') = 0 ]] && AddFilenames="$Names"
	1055	Names=$(ls -1 *.F90 2>&1)
	1056	[[ $(echo $Names \| grep -c '*.F90') = 0 ]] && AddFilenames="$AddFilenames $Names"
	1057	Names=$(ls -1 *.F 2>&1)
	1058	[[ $(echo $Names \| grep -c '*.F') = 0 ]] && AddFilenames="$AddFilenames $Names"
	1059	Names=$(ls -1 *.f 2>&1)
	1060	[[ $(echo $Names \| grep -c '*.f') = 0 ]] && AddFilenames="$AddFilenames $Names"
	1061	Names=$(ls -1 *.c 2>&1)
	1062	[[ $(echo $Names \| grep -c '*.c') = 0 ]] && AddFilenames="$AddFilenames $Names"
	1063
	1064	cd - > /dev/null
[2990]	1065	cd ${base_directory}/${sources_for_run_catalog}
[2380]	1066
	1067	# COPY MAKEFILE IF EXISTING
[2384]	1068	if [[ -f $user_source_path/Makefile ]]
[2380]	1069	then
	1070	printf "\n\n *** user Makefile from directory"
[2384]	1071	printf "\n \"$user_source_path\" is used \n"
[2380]	1072	if [[ $silent == false ]]
	1073	then
	1074	sleep 1
	1075	fi
[2384]	1076	cp $user_source_path/Makefile .
[2380]	1077	fi
	1078
	1079	for filename in $AddFilenames
	1080	do
	1081	if [[ -f $filename ]]
	1082	then
	1083	printf "\n +++ source code file \"$filename\" found in additional"
[2384]	1084	printf "\n source code directory \"$user_source_path\" "
[2380]	1085	printf "\n but was also given with option \"-s\" which means that it should be taken"
	1086	printf "\n from directory \"$source_path\"."
	1087	locat=source; exit
	1088	fi
	1089
[2384]	1090	cp $user_source_path/$filename .
[2380]	1091	source_list="$source_list $filename"
	1092
	1093	# CHECK IF FILE IS CONTAINED IN MAKEFILE
	1094	if [[ $(grep -c $filename Makefile) = 0 ]]
	1095	then
	1096	printf "\n\n +++ user file \"$filename\" "
	1097	printf "\n is not listed in Makefile \n"
	1098	locat=source; exit
	1099	fi
	1100	done
	1101	cd - > /dev/null
	1102	fi
	1103	fi
	1104
	1105	# COPY CONFIGURATION FILES
[2990]	1106	cp $config_file ${base_directory}/${sources_for_run_catalog}
	1107	cp $fileconnection_file ${base_directory}/${sources_for_run_catalog}
[2380]	1108
	1109	# COPY SHELLSCRIPTS
[2990]	1110	cp ${source_path}/../SCRIPTS/palmrun ${base_directory}/${sources_for_run_catalog}
	1111	cp ${source_path}/../SCRIPTS/batch_scp ${base_directory}/${sources_for_run_catalog}
[2380]	1112
	1113	fi
	1114
	1115	fi
	1116
[2605]	1117
[2380]	1118	# GET THE GLOBAL REVISION-NUMBER OF THE SVN-REPOSITORY
	1119	# (HANDED OVER TO RESTART-RUNS USING OPTION -G)
	1120	if [[ "$global_revision" = "" ]]
	1121	then
	1122	global_revision=`svnversion $source_path 2>/dev/null`
	1123	global_revision="Rev: $global_revision"
	1124	fi
	1125
	1126
	1127	# IN CASE OF PARALLEL EXECUTION, CHECK SOME SPECIFICATIONS CONCERNING PROCESSOR NUMBERS
	1128	if [[ -n $cores ]]
	1129	then
	1130
	1131	# CHECK, IF THE NUMBER OF CORES PER NODE HAS BEEN GIVEN UND IF IT IS AN
	1132	# INTEGRAL DIVISOR OF THE TOTAL NUMBER OF CORES GIVEN BY OPTION -X
[2501]	1133	if [[ $tasks_per_node = 0 ]]
[2380]	1134	then
[3025]	1135	if [[ $create_batch_job = true \|\| $create_remote_batch_job = true ]]
	1136	then
	1137	printf "\n"
	1138	printf "\n +++ option \"-T\" (tasks per node) is missing"
	1139	printf "\n set -T option or define tasks_per_node in the config file"
	1140	locat=tasks_per_node; (( iec = 0 )); exit
	1141	else
	1142	# DEFAULT FOR INTERACTIVE RUN
	1143	tasks_per_node=$cores
	1144	fi
[2380]	1145	fi
	1146
	1147	if (( cores < tasks_per_node ))
	1148	then
	1149	printf "\n"
	1150	printf "\n +++ tasks per node (-T) cannot exceed total number of cores (-X)"
	1151	printf "\n given values: -T $tasks_per_node -X $cores"
	1152	locat=tasks_per_node; (( iec = 0 )); exit
	1153	fi
	1154
	1155	(( nodes = cores / ( tasks_per_node * threads_per_task ) ))
	1156	(( mpi_tasks = cores / threads_per_task ))
	1157	[[ $mpi_tasks = 0 ]] && (( mpi_tasks = 1 ))
	1158	(( ii = cores / tasks_per_node ))
	1159	(( remaining_cores = cores - ii * tasks_per_node ))
	1160	if (( remaining_cores > 0 ))
	1161	then
	1162	printf "\n"
	1163	printf "\n +++ WARNING: tasks per node (option \"-T\") is not an integral"
	1164	printf "\n divisor of the total number of cores (option \"-X\")"
	1165	printf "\n values of this palmrun-call: \"-T $tasks_per_node\" \"-X $cores\""
	1166	printf "\n One of the nodes is filled with $remaining_cores instead of $tasks_per_node tasks"
	1167	(( nodes = nodes + 1 ))
	1168	fi
	1169
	1170	fi
	1171
	1172
	1173	# SET DEFAULT VALUE FOR THE MAXIMUM NUMBER OF PARALLEL IO STREAMS
	1174	if [[ "$maximum_parallel_io_streams" = "" ]]
	1175	then
	1176	maximum_parallel_io_streams=$cores
	1177	fi
	1178
	1179
	1180	# SET PORT NUMBER OPTION FOR CALLS OF SSH/SCP AND batch_scp SCRIPT
	1181	if [[ "$scp_port" != "" ]]
	1182	then
	1183	PORTOPT="-P $scp_port"
	1184	SSH_PORTOPT="-p $scp_port"
	1185	fi
	1186
	1187
	1188	# DETERMINE THE SSH-OPTION IN CASE THAT AN SSH-KEY IS EXPLICITLY GIVEN IN THE
	1189	# CONFIG-FILE
	1190	if [[ "$ssh_key" != "" ]]
	1191	then
	1192	ssh_key="-i $HOME/.ssh/$ssh_key"
	1193	fi
	1194
	1195
	1196	# SET QUEUE, IF NOT GIVEN
	1197	if [[ $create_batch_job = true \|\| $create_remote_batch_job = true ]]
	1198	then
	1199
	1200	if [[ $queue = none && "$defaultqueue" = "" ]]
	1201	then
	1202	printf "\n"
	1203	printf "\n +++ no default queue given in configuration file and no queue"
	1204	printf "\n given with option -q"
[2789]	1205	locat=queue; exit
[2380]	1206	fi
	1207	if [[ $queue = none ]]
	1208	then
	1209	queue=$defaultqueue
	1210	fi
	1211
	1212	fi
	1213
	1214
	1215	# GENERATE FULL FILENAMES OF INPUT-FILES, INCLUDING THEIR PATH
	1216	# CHECK, IF INPUT-FILES EXIST, AND DETERMINE HIGHEST CYCLE NUMBER (IF CYCLES EXIST)
	1217	(( i = 0 ))
[2610]	1218	(( nr_of_input_files = 0 ))
[2380]	1219	while (( i < iin ))
	1220	do
	1221	(( i = i + 1 ))
	1222
	1223	# GENERATE PATH AND FULL FILE NAME (then-BRANCH: FIXED FULL NAME IS GIVEN, I.E. THE
	1224	# FILE IDENTIFIER IS NOT PART OF THE FILENAME))
[2610]	1225	if [[ "${actionin_pre[$i]}" = di ]]
[2380]	1226	then
[2610]	1227	eval filename=${pathin_pre[$i]}/${endin_pre[$i]}
[2380]	1228	else
[3236]	1229	eval filename=${pathin_pre[$i]}/${run_identifier}${endin_pre[$i]}
[2380]	1230	fi
	1231
[2610]	1232
[2380]	1233	# CHECK IF FILE EXISTS
	1234	if ! ls $filename* 1>/dev/null 2>&1
	1235	then
	1236
[2669]	1237	# FILES WITH ATTRIBUTE opt ARE OPTIONAL. NO ABORT, IF THEY DO NOT EXIST.
	1238	if [[ "${optin_pre[$i]}" != "yes" ]]
[2380]	1239	then
	1240	printf "\n\n +++ INPUT-file: "
[2610]	1241	if [[ "${extin_pre[$i]}" = "" \|\| "${extin_pre[$i]}" = " " ]]
[2380]	1242	then
	1243	printf "\n $filename"
	1244	else
[2610]	1245	printf "\n $filename.${extin_pre[$i]}"
[2380]	1246	fi
	1247	printf "\n does not exist\n"
	1248	locat=input; exit
	1249	else
[2610]	1250	(( nr_of_input_files = nr_of_input_files + 1 ))
	1251	localin[$nr_of_input_files]="${localin_pre[$i]}"
[2669]	1252	optin[$nr_of_input_files]="${optin_pre[$i]}"
	1253	actionin[$nr_of_input_files]="unavailable"
[2610]	1254	pathin[$nr_of_input_files]="${pathin_pre[$i]}"
	1255	endin[$nr_of_input_files]="${endin_pre[$i]}"
	1256	extin[$nr_of_input_files]="${extin_pre[$i]}"
[2380]	1257	fi
[2605]	1258
[2380]	1259	else
	1260
[3236]	1261	# FIRST CHECK FOR MULTIPLE NAMES WITH THE SAME BASENAME
	1262	# ($run_identifier) AND CREATE A LIST FOR THE DETECTED BASENAME
	1263	# ENDINGS
[2610]	1264	if [[ "${multin[$i]}" = true ]]
	1265	then
	1266	# DETERMINE THE EXISTING EXTENSIONS FROM THE LIST OF FILES
	1267	ls -1 -d ${filename} > filelist 2>/dev/null
	1268	ls -1 -d ${filename}.* >> filelist 2>/dev/null
	1269	ls -1 -d ${filename}_* >> filelist 2>/dev/null
[2380]	1270
[2610]	1271	endings=""
	1272	while read line
	1273	do
	1274	# filename without path (i.e. after the last "/")
	1275	basefilename=$(basename ${line})
[2380]	1276
[2610]	1277	# check if there is an extension and remove it
	1278	ext=${basefilename##*.}
	1279	if [[ "$ext" = "${extin_pre[$i]}" ]]
	1280	then
	1281	basefilename=${basefilename%.*}
	1282	fi
[2380]	1283
[2610]	1284	# check for an existing cycle number and remove it
	1285	cycle=${basefilename##*.}
	1286	if [[ $cycle =~ ^-?[0-9]+$ ]]
	1287	then
	1288	basefilename=${basefilename%.*}
	1289	fi
[2600]	1290
[3236]	1291	# remove the run_identifier from the beginning
	1292	length_run_identifier=${#run_identifier}
	1293	ending=${basefilename:${length_run_identifier}}
[2610]	1294
	1295	# remove the ending given in the .iofiles from the beginning
	1296	endingstring="${endin_pre[$i]}"
	1297	length_ending=${#endingstring}
	1298	ending=${ending:${length_ending}}
	1299
	1300	if [[ "$ending" = "" ]]
[2600]	1301	then
[2610]	1302	# standard ending as given in the .iofiles
	1303	if [[ $(echo $endings \| grep -c DEFAULT) = 0 ]]
	1304	then
	1305	endings="$endings DEFAULT"
	1306	fi
[2600]	1307	else
[2610]	1308	# ending must start with "_", otherwise its a different file
	1309	if [[ "${ending:0:1}" = "_" ]]
	1310	then
	1311	if [[ $(echo $endings \| grep -c "$ending") = 0 ]]
	1312	then
	1313	endings="$endings $ending"
	1314	fi
	1315	fi
[2600]	1316	fi
[2610]	1317
	1318	done <filelist
[2380]	1319
[2610]	1320	rm filelist
[2380]	1321
[2600]	1322	else
[2610]	1323
	1324	# SINGLE NAME
	1325	endings=DEFAULT
	1326
[2600]	1327	fi
	1328
[2610]	1329	# FOR EACH BASENAME ENDING CREATE AN ENTRY IN THE FINAL INPUT FILE LIST
	1330	for ending in $endings
	1331	do
	1332
	1333	# DEFAULT MEANS THAT THE ENDING GIVEN IN .iofiles SHALL BE USED
	1334	if [[ $ending = DEFAULT ]]
[2380]	1335	then
[2610]	1336	ending=""
	1337	fi
	1338
	1339	# NEW ENTRY (ENDING IS ALSO ADDED TO LOCAL FILENAME READ BY PALM!)
	1340	(( nr_of_input_files = nr_of_input_files + 1 ))
	1341	localin[$nr_of_input_files]="${localin_pre[$i]}"$ending
[2669]	1342	optin[$nr_of_input_files]="${optin_pre[$i]}"
[2610]	1343	actionin[$nr_of_input_files]="${actionin_pre[$i]}"
	1344	pathin[$nr_of_input_files]="${pathin_pre[$i]}"
	1345	endin[$nr_of_input_files]="${endin_pre[$i]}"$ending
	1346	extin[$nr_of_input_files]="${extin_pre[$i]}"
	1347
	1348
	1349	# GENERATE PATH AND FULL FILE NAME (then-BRANCH: FIXED FULL NAME IS GIVEN, I.E. THE
	1350	# FILE IDENTIFIER IS NOT PART OF THE FILENAME))
	1351	if [[ "${actionin[$nr_of_input_files]}" = di ]]
	1352	then
	1353	eval filename=${pathin[$nr_of_input_files]}/${endin[$nr_of_input_files]}
[2380]	1354	else
[3236]	1355	eval filename=${pathin[$nr_of_input_files]}/${run_identifier}${endin[$nr_of_input_files]}
[2380]	1356	fi
[2610]	1357
	1358	# DETERMINE THE FILE'S CYCLE NUMBER
	1359	(( maxcycle = 0 ))
	1360	ls -1 -d $filename > filelist 2>/dev/null
	1361	ls -1 -d $filename.* >> filelist 2>/dev/null
	1362	while read line
	1363	do
[4443]	1364	# filename without path (i.e. after the last "/")
	1365	basefilename=$(basename ${line})
[2610]	1366
[4443]	1367	# check if there is an extension
	1368	extension=${basefilename##*.}
	1369	if [[ "$extension" = "${extin[$nr_of_input_files]}" ]]
	1370	then
	1371	basefilename=${basefilename%.*}
	1372	fi
[2610]	1373
[4443]	1374	# check for an existing cycle number
	1375	cycle=${basefilename##*.}
	1376	if [[ $cycle =~ ^-?[0-9]+$ ]]
	1377	then
	1378	# NUMBERS WITH LEADING ZEROS ARE INTERPRETED AS OCTAL NUMBERS
	1379	# 10# EXPLICITLY SPECIFIES THE NUMBER BASE AS 10
	1380	(( icycle = $((10#$cycle)) ))
	1381	explicit_cycle_number=true
	1382	else
	1383	(( icycle = 0 ))
	1384	explicit_cycle_number=false
	1385	fi
[2610]	1386
[4443]	1387	if (( icycle > maxcycle ))
	1388	then
	1389	(( maxcycle = icycle ))
	1390	fi
[2610]	1391
	1392	done <filelist
	1393	rm filelist
[4443]	1394
[2610]	1395
	1396	# MAKE CYCLE NUMBER THREE DIGITS WIDE
[4443]	1397	cyclestring=`printf "%03d" $maxcycle`
	1398
[2610]	1399
	1400	# APPEND CYCLE NUMBER TO FILENAME
[4443]	1401	if [[ $explicit_cycle_number = true ]]
[2380]	1402	then
[2610]	1403	if [[ "${extin[$nr_of_input_files]}" != " " && "${extin[$nr_of_input_files]}" != "" ]]
	1404	then
	1405	filename=${filename}.$cyclestring.${extin[$nr_of_input_files]}
	1406	else
	1407	filename=${filename}.$cyclestring
	1408	fi
[2380]	1409	else
[2610]	1410	if [[ "${extin[$nr_of_input_files]}" != " " && "${extin[$nr_of_input_files]}" != "" ]]
	1411	then
	1412	filename=${filename}.${extin[$nr_of_input_files]}
	1413	fi
[2380]	1414	fi
[2610]	1415
	1416	# STORE FILENAME WITHOUT PATH BUT WITH CYCLE NUMBER,
	1417	# IS LATER USED FOR TRANSFERRING FILES WIHIN THE JOB (SEE END OF FILE)
	1418	absnamein[$nr_of_input_files]=$filename
[4443]	1419	if [[ $explicit_cycle_number = true ]]
[2380]	1420	then
[2610]	1421	if [[ "${actionin[$nr_of_input_files]}" = di ]]
	1422	then
	1423	frelin[$nr_of_input_files]=${endin[$nr_of_input_files]}.$cyclestring
	1424	else
[3236]	1425	frelin[$nr_of_input_files]=${run_identifier}${endin[$nr_of_input_files]}.$cyclestring
[2610]	1426	fi
[2380]	1427	else
[2610]	1428	if [[ "${actionin[$nr_of_input_files]}" = di ]]
	1429	then
	1430	frelin[$nr_of_input_files]=${endin[$nr_of_input_files]}
	1431	else
[3236]	1432	frelin[$nr_of_input_files]=${run_identifier}${endin[$nr_of_input_files]}
[2610]	1433	fi
[2380]	1434	fi
	1435
[2610]	1436	done
	1437
[2380]	1438	fi
[2610]	1439
[2380]	1440	done
	1441
[2610]	1442
[2380]	1443	# GENERATE FULL FILENAMES OF OUTPUT-FILES (WITHOUT $ OR ~),
	1444	# CHECK, IF OUTPUT-FILES EXIST, AND DETERMINE HIGHEST CYCLE NUMBER (IF CYCLES EXIST),
	1445	# OR, IN CASE THAT FILE DOES NOT EXIST, CHECK, IF IT CAN BE CREATED
[2610]	1446	# THESE ACTIONS ARE NOT CARRIED OUT, IF FILES SHALL BE TRANSFERRED FROM THE REMOTE TO
[2380]	1447	# THE LOCAL HOST (BECAUSE THEIR IS NO DIRECT ACCESS TO THE LOCAL DIRECTORIES FROM THE
	1448	# REMOTE HOST)
	1449	(( i = 0 ))
	1450	while (( i < iout ))
	1451	do
	1452	(( i = i + 1 ))
[2610]	1453	if [[ ! ( $running_on_remote = true && ( "${actionout_pre[$i]}" = tr \|\| "${actionout_pre[$i]}" = tra \|\| "${actionout_pre[$i]}" = trpe ) ) ]]
[2380]	1454	then
[2610]	1455	if [[ "${actionout_pre[$i]}" = tr ]]
[2380]	1456	then
[2610]	1457	actionout_pre[$i]=""
	1458	elif [[ "${actionout_pre[$i]}" = trpe ]]
[2380]	1459	then
[2610]	1460	actionout_pre[$i]=pe
	1461	elif [[ "${actionout_pre[$i]}" = tra ]]
[2380]	1462	then
[2610]	1463	actionout_pre[$i]=a
[2380]	1464	fi
	1465	(( maxcycle = 0 ))
[3236]	1466	eval filename=${pathout_pre[$i]}/${run_identifier}${endout_pre[$i]}
[2610]	1467	eval catalogname=${pathout_pre[$i]}
[2380]	1468	if ! ls $filename* 1>/dev/null 2>&1
	1469	then
	1470
	1471	# IF OUTPUT-FILE DOES NOT EXIST CHECK, IF IT CAN BE CREATED
	1472	if cat /dev/null > $filename
	1473	then
	1474	rm $filename
	1475	else
	1476
	1477	# CHECK, IF THE DIRECTORY WHERE FILE SHALL BE COPIED TO EXISTS
	1478	# IF IT DOES NOT EXIST, TRY TO CREATE IT
	1479	if [[ ! -d $catalogname ]]
	1480	then
	1481	if mkdir -p $catalogname
	1482	then
	1483	printf "\n\n *** directory:"
	1484	printf "\n $catalogname"
	1485	printf "\n was created\n"
	1486	else
	1487	printf "\n\n +++ OUTPUT-file:"
	1488	printf "\n $filename"
	1489	printf "\n cannot be created, because directory does not exist"
	1490	printf "\n and cannot be created either"
	1491	printf "\n"
	1492	locat=output ; exit
	1493	fi 2>/dev/null
	1494	else
	1495	printf "\n\n +++ OUTPUT-file:"
	1496	printf "\n $filename"
	1497	printf "\n cannot be created, although directory exists"
	1498	printf "\n"
	1499	locat=output ; exit
	1500	fi
	1501	fi 2>/dev/null
	1502
	1503	fi
	1504
	1505	fi
	1506	done
	1507
	1508
	1509	# DETERMINE THE NAME OF PALMRUN'S TEMPORARY WORKING DIRECTORY
	1510	if [[ $running_in_batch_mode = false ]]
	1511	then
[3236]	1512	run_id_number=$RANDOM
	1513	run_id=${run_identifier}.$run_id_number
[2380]	1514
[3236]	1515	tempdir=$fast_io_catalog/$run_id
[3346]	1516
	1517	# FOR COMPATIBILITY REASONS WITH OLDER VERSIONS SET JOB_ID
	1518	export job_id=$run_id
	1519
[2380]	1520	fi
	1521
	1522
	1523	# CHECK SETTINGS REQUIRED FOR BATCH JOBS
	1524	if [[ $create_batch_job = true \|\| $create_remote_batch_job = true ]]
	1525	then
	1526
	1527	# CHECK, IF JOB DIRECTIVES HAVE BEEN GIVEN IN CONFIGURATION FILE
	1528	if [[ $ibd = 0 ]]
	1529	then
	1530	printf "\n"
	1531	printf "\n +++ no batch directives found in configuration file"
	1532	locat=config_file_batch_directives; (( iec = 0 )); exit
	1533	fi
	1534
	1535	# CHECK IF CPUTIME IS GIVEN FOR JOB
	1536	done=false
[2384]	1537	cputime=$cpumax
[2380]	1538	while [[ $done = false ]]
	1539	do
[2384]	1540	if (( cputime == 0 ))
[2380]	1541	then
	1542	printf "\n +++ cpu-time is undefined"
	1543	printf "\n >>> Please type CPU-time in seconds as INTEGER:"
	1544	printf "\n >>> "
[2384]	1545	read cputime 1>/dev/null 2>&1
[2380]	1546	else
	1547	done=true
	1548	fi
	1549	done
[2420]	1550	cpumax=$cputime
[2380]	1551
	1552	# CHECK THE MEMORY DEMAND
	1553	done=false
	1554	while [[ $done = false ]]
	1555	do
	1556	if (( memory == 0 ))
	1557	then
	1558	printf "\n +++ memory demand is undefined"
	1559	printf "\n >>> Please type memory in MByte per process as INTEGER:"
	1560	printf "\n >>> "
	1561	read memory 1>/dev/null 2>&1
	1562	else
	1563	done=true
	1564	fi
	1565	done
	1566
	1567	# IN CASE OF REMOTE-JOBS CHECK, IF A USERNAME FOR THE REMOTE HOST IS GIVEN
	1568	if [[ $create_remote_batch_job = true && -z $remote_username ]]
	1569	then
	1570	while [[ -z $remote_username ]]
	1571	do
[2420]	1572	printf "\n +++ username on remote host with IP \"$remote_ip\" is undefined"
[2380]	1573	printf "\n >>> Please type username:"
	1574	printf "\n >>> "
	1575	read remote_username
	1576	done
	1577	fi
	1578
	1579	else
	1580
[2420]	1581	if [[ $running_in_batch_mode = false ]]
	1582	then
	1583	cputime=10000000 # NO LIMT FOR INTERACTIVE RUNS
	1584	cpumax=$cputime
	1585	else
	1586	cputime=$cpumax
	1587	fi
[2380]	1588
	1589	fi
	1590
	1591
	1592	# CALCULATE HOURS/MINUTES/SECONDS, E.G. FOR BATCH-DIRECTIVES
	1593	(( cpu_hours = cputime / 3600 ))
	1594	(( resttime = cputime - cpu_hours * 3600 ))
	1595	(( cpu_minutes = resttime / 60 ))
	1596	(( cpu_seconds = resttime - cpu_minutes * 60 ))
	1597	timestring=${cpu_hours}:${cpu_minutes}:${cpu_seconds}
	1598
	1599
	1600	# OUTPUT OF THE PALMRUN-HEADER
	1601	calltime=$(date)
	1602	printf "\n"
	1603	printf "#------------------------------------------------------------------------# \n"
	1604	printf "\| %-35s%35s \| \n" "$version" "$calltime"
[3151]	1605	printf "\| %-35s%35s \| \n" "PALM code $global_revision" " "
[2380]	1606	printf "\| \| \n"
	1607	column1="called on:"; column2=$(hostname)
	1608	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
	1609	if [[ $create_remote_batch_job = true ]]
	1610	then
[3236]	1611	column1="execution on:"; column2="$configuration_identifier (username: $remote_username)"
[2380]	1612	else
	1613	if [[ $running_on_remote = true ]]
	1614	then
[3236]	1615	column1="config. identifier:"; column2="$configuration_identifier (execute on IP: $remote_ip)"
[2380]	1616	else
[3236]	1617	column1="config. identifier:"; column2="$configuration_identifier (execute on IP: $local_ip)"
[2380]	1618	fi
	1619	fi
	1620	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
	1621
	1622	column1="running in:"
	1623	if [[ $running_in_batch_mode = true ]]
	1624	then
	1625	column2="batch job mode"
	1626	else
	1627	if [[ $create_batch_job = true \|\| $create_remote_batch_job = true ]]
	1628	then
	1629	column2="job creation mode"
	1630	else
	1631	column2="interactive run mode"
	1632	fi
	1633	fi
	1634	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
	1635
[2479]	1636	if [[ $running_in_batch_mode = true \|\| $create_batch_job = true \|\| $create_remote_batch_job = true ]]
	1637	then
[2480]	1638	if [[ "$project_account" != "" ]]
[2479]	1639	then
	1640	column1="project account number:"
	1641	column2="$project_account"
	1642	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
	1643	fi
	1644	fi
	1645
[2380]	1646	if [[ -n $cores ]]
	1647	then
	1648	if [[ $run_coupled_model = false ]]
	1649	then
	1650	column1="number of cores:"; column2=$cores
	1651	else
	1652	column1="number of cores:"; column2="$cores (atmosphere: $cores_atmos, ocean: $cores_ocean)"
	1653	fi
	1654	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
	1655	fi
	1656	if [[ -n $tasks_per_node ]]
	1657	then
	1658	column1="tasks per node:"; column2="$tasks_per_node (number of nodes: $nodes)"
	1659	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
	1660	if (( remaining_cores > 0 ))
	1661	then
	1662	column1=" "; column2="one of the nodes only filled with $remaining_cores tasks"
	1663	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
	1664	fi
	1665	fi
	1666	if [[ $maximum_parallel_io_streams != $cores ]]
	1667	then
	1668	column1="max par io streams:"; column2="$maximum_parallel_io_streams"
	1669	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
	1670	fi
	1671	if [[ $use_openmp = true ]]
	1672	then
	1673	column1="threads per task:"; column2="$threads_per_task"
	1674	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
	1675	fi
[2384]	1676	if [[ $create_batch_job = true \|\| $create_remote_batch_job = true \|\| $running_in_batch_mode = true ]]
	1677	then
[2506]	1678	column1="memory demand / PE":; column2="$memory MB"
	1679	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
[2384]	1680	column1="job cpu time (h:m:s):"; column2="$timestring"
	1681	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
	1682	fi
[2380]	1683	printf "\| \| \n"
	1684	if [[ "$source_list" != "" ]]
	1685	then
	1686	if [[ "$make_options" != "" ]]
	1687	then
	1688	column1="make options:"; column2=$(echo "$make_options" \| cut -c-45)
	1689	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
	1690	line=$(echo "$make_options" \| cut -c46-)
	1691	while [[ "$line" != "" ]]
	1692	do
	1693	column1=""
	1694	column2=$(echo "$line" \| cut -c-45)
	1695	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
	1696	line=$(echo "$line" \| cut -c46-)
	1697	done
	1698	fi
	1699	fi
	1700
	1701	column1="cpp directives:"; column2=$(echo "$cpp_options" \| cut -c-45)
	1702	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
	1703	line=$(echo "$cpp_options" \| cut -c46-)
	1704	while [[ "$line" != "" ]]
	1705	do
	1706	column1=""
	1707	column2=$(echo "$line" \| cut -c-45)
	1708	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
	1709	line=$(echo "$line" \| cut -c46-)
	1710	done
	1711
	1712	column1="compiler options:"; column2=$(echo "$compiler_options" \| cut -c-45)
	1713	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
	1714	line=$(echo "$compiler_options" \| cut -c46-)
	1715	while [[ "$line" != "" ]]
	1716	do
	1717	column1=""
	1718	column2=$(echo "$line" \| cut -c-45)
	1719	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
	1720	line=$(echo "$line" \| cut -c46-)
	1721	done
	1722
[2388]	1723	column1="linker options:"; column2=$(echo "$linker_options" \| cut -c-45)
[2380]	1724	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
[2388]	1725	line=$(echo "$linker_options" \| cut -c46-)
[2380]	1726	while [[ "$line" != "" ]]
	1727	do
	1728	column1=""
	1729	column2=$(echo "$line" \| cut -c-45)
	1730	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
	1731	line=$(echo "$line" \| cut -c46-)
	1732	done
	1733
	1734	if [[ "$login_init_cmd" != "" ]]
	1735	then
	1736	column1="login init commands:"; column2=$(echo "$login_init_cmd" \| cut -c-45)
	1737	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
	1738	line=$(echo "$login_init_cmd" \| cut -c46-)
	1739	while [[ "$line" != "" ]]
	1740	do
	1741	column1=""
	1742	column2=$(echo "$line" \| cut -c-45)
	1743	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
	1744	line=$(echo "$line" \| cut -c46-)
	1745	done
	1746	fi
	1747
	1748	if [[ "$module_commands" != "" ]]
	1749	then
	1750	column1="module commands:"; column2=$(echo "$module_commands" \| cut -c-45)
	1751	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
	1752	line=$(echo "$module_commands" \| cut -c46-)
	1753	while [[ "$line" != "" ]]
	1754	do
	1755	column1=""
	1756	column2=$(echo "$line" \| cut -c-45)
	1757	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
	1758	line=$(echo "$line" \| cut -c46-)
	1759	done
	1760	fi
	1761	printf "\| \| \n"
[3236]	1762	column1="run identifier:"; column2=$run_identifier
[2380]	1763	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
	1764	column1="activation string list:"; column2=$(echo $activation_string_list)
	1765	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
	1766
	1767	if [[ "$ocean_file_appendix" = true ]]
	1768	then
	1769	printf "\| %-35s%-35s \| \n" "suffix \"_O\" is added to local files" " "
	1770	fi
	1771
	1772	if [[ "$source_list" != "" ]]
	1773	then
	1774	printf "\| \| \n"
	1775	printf "\| Files to be compiled: \| \n"
	1776	line=$source_list
	1777	while [[ "$line" != "" ]]
	1778	do
	1779	linestart=$(echo $line \| cut -c-70)
	1780	printf "\| %-70s \| \n" "$linestart"
	1781	line=$(echo "$line" \| cut -c71-)
	1782	done
	1783	fi
	1784	printf "#------------------------------------------------------------------------#"
	1785
	1786
	1787
	1788	# OUTPUT OF FILE CONNECTIONS IN CASE OF TRACEBACK
	1789	if [[ $do_trace = true ]]
	1790	then
	1791	(( i = 0 ))
[2610]	1792	while (( i < nr_of_input_files ))
[2380]	1793	do
	1794	(( i = i + 1 ))
	1795	if (( i == 1 ))
	1796	then
	1797	printf "\n\n >>> INPUT-file assignments:\n"
	1798	fi
	1799	printf "\n ${localin[$i]} : ${absnamein[$i]}"
	1800	done
	1801	(( i = 0 ))
	1802	while (( i < iout ))
	1803	do
	1804	(( i = i + 1 ))
	1805	if (( i == 1 ))
	1806	then
	1807	printf "\n\n >>> OUTPUT-file assignments:\n"
	1808	fi
	1809	printf "\n ${localout[$i]} : ${pathout[$i]}"
	1810	done
	1811	(( i = 0 ))
	1812	while (( i < iic ))
	1813	do
	1814	(( i = i + 1 ))
	1815	if (( i == 1 ))
	1816	then
	1817	printf "\n\n >>> INPUT-commands:\n"
	1818	fi
	1819	printf "\n ${in_command[$i]}"
	1820	done
	1821	(( i = 0 ))
	1822	while (( i < ioc ))
	1823	do
	1824	(( i = i + 1 ))
	1825	if (( i == 1 ))
	1826	then
	1827	printf "\n\n >>> OUTPUT-commands:\n"
	1828	fi
	1829	printf "\n ${out_command[$i]}"
	1830	done
	1831	fi
	1832
	1833	# QUERY FOR CONTINUE
	1834	if [[ $silent = false && $running_in_batch_mode = false ]]
	1835	then
	1836	antwort=dummy
	1837	printf "\n\n"
	1838	printf " >>> everything o.k. (y/n) ? "
	1839	while read antwort
	1840	do
	1841	if [[ "$antwort" != y && "$antwort" != Y && "$antwort" != n && "$antwort" != N ]]
	1842	then
	1843	printf " >>> everything o.k. (y/n) ? "
	1844	else
	1845	break
	1846	fi
	1847	done
	1848	if [[ $antwort = n \|\| $antwort = N ]]
	1849	then
	1850	locat=user_abort; (( iec = 0 )); exit
	1851	fi
	1852	if [[ $create_batch_job = true \|\| $create_remote_batch_job = true ]]
	1853	then
[2566]	1854	printf "\n *** batch-job will be created and submitted"
[2380]	1855	else
[2566]	1856	printf "\n *** PALMRUN will now continue to execute on this machine"
[2380]	1857	fi
	1858	fi
	1859
	1860
	1861
	1862	# PROVIDE FILES TO EXECUTE PALM AND CREATE THE EXECUTABLE
	1863	if [[ $restart_run = false && $running_in_batch_mode = false ]]
	1864	then
	1865
	1866	if [[ $create_batch_job = true \|\| $create_remote_batch_job = true ]]
	1867	then
[2566]	1868	printf "\n\n *** creating executable and other sources for the remote host\n"
[2380]	1869	else
[2566]	1870	printf "\n\n *** creating executable and other sources for the local host\n"
[2380]	1871	fi
	1872
[2422]	1873	# FIRST CHECK, IF A MAKE DEPOSITORY EXISTS, AND IF NOT, ASK THE USER IF
	1874	# IT SHALL BE CREATED
	1875	ask_for_make_depository=false
	1876	if [[ $create_remote_batch_job = true ]]
	1877	then
	1878
	1879	line=`grep %base_directory $config_file`
[3236]	1880	make_depository=`echo $line \| cut -d" " -s -f2`/MAKE_DEPOSITORY_${configuration_identifier}
[3534]	1881	ssh -q $ssh_key ${remote_username}@${remote_ip} "[[ ! -d ${make_depository} ]] && echo depository not found" 2>&1 \| tee ${configuration_identifier}_last_make_protokoll
[2422]	1882
[3236]	1883	if [[ $(grep -c "depository not found" ${configuration_identifier}_last_make_protokoll) != 0 ]]
[2422]	1884	then
	1885	printf "\n\n +++ make depository \"${make_depository}\""
	1886	printf "\n on remote host not found!"
	1887	ask_for_make_depository=true
	1888	fi
[3236]	1889	rm ${configuration_identifier}_last_make_protokoll
[2422]	1890
	1891	else
	1892
	1893	# CHECK FOR MAKE_DEPOSITORY ON THE LOCAL HOST
[3236]	1894	make_depository=${base_directory}/MAKE_DEPOSITORY_${configuration_identifier}
[2422]	1895	if [[ ! -d ${make_depository} ]]
	1896	then
	1897	printf "\n\n +++ make depository \"${make_depository}\""
	1898	printf "\n on local host not found!"
	1899	ask_for_make_depository=true
	1900	fi
	1901
	1902	fi
	1903
	1904	if [[ $ask_for_make_depository = true ]]
	1905	then
	1906
	1907	antwort=dummy
	1908	printf "\n\n"
	1909	printf " >>> Create a new one (y/n) ? "
	1910	while read antwort
	1911	do
	1912	if [[ "$antwort" != y && "$antwort" != Y && "$antwort" != n && "$antwort" != N ]]
	1913	then
	1914	printf " >>> Create a new one (y/n) ? "
	1915	else
	1916	break
	1917	fi
	1918	done
	1919	if [[ $antwort = n \|\| $antwort = N ]]
	1920	then
	1921	locat=user_abort; (( iec = 0 )); exit
	1922	fi
	1923
[2566]	1924	if [[ $do_trace = true ]]
	1925	then
[3455]	1926	palmbuild -c $configuration_identifier
[2566]	1927	else
[3455]	1928	palmbuild -v -c $configuration_identifier
[2566]	1929	fi
[2422]	1930
[3312]	1931	if [[ ${PIPESTATUS[0]} != 0 ]]
[2422]	1932	then
	1933
	1934	# ABORT IN CASE OF COMPILATION PROBLEMS
	1935	printf "\n +++ error while compiling for the MAKE_DEPOSITORY"
	1936	locat=make_depository
	1937	exit
	1938	else
	1939	echo " *** now continue with creating executable and other sources"
	1940	fi
	1941
	1942	fi
	1943
[2566]	1944	# NOW CREATE THE SOURCES_FOR_RUN FOLDER
[3455]	1945	palmbuild -v $use_existing_sources_folder -c $configuration_identifier -r $run_identifier
[2380]	1946
[3312]	1947	if [[ ${PIPESTATUS[0]} != 0 ]]
[2380]	1948	then
	1949
	1950	# ABORT IN CASE OF COMPILATION PROBLEMS
	1951	printf "\n +++ error while creating executable and/or other sources"
	1952	locat=execution
[2990]	1953	rm -rf ${base_directory}/${sources_for_run_catalog}
[2380]	1954	exit
	1955
	1956	else
	1957
[2566]	1958	printf " *** executable and other sources created\n"
[2990]	1959	rm -rf ${base_directory}/${sources_for_run_catalog}
[2380]	1960
	1961	fi
	1962
	1963	fi
	1964
	1965
[2669]	1966	# WHEN CREATING A REMOTE BATCH JOB, THOSE INPUT FILES WITH TRANSFER-ATTRIBUT
	1967	# WILL BE COPIED TO THE REMOTE HOST
[2605]	1968	if [[ $create_remote_batch_job = true ]]
	1969	then
	1970	(( i = 0 ))
[2610]	1971	while (( i < nr_of_input_files ))
[2605]	1972	do
	1973	(( i = i + 1 ))
[2669]	1974	if [[ "${actionin[$i]}" = tr ]]
[2605]	1975	then
	1976	eval inputfile=${pathin[$i]}/${frelin[$i]}
	1977	scp -q $ssh_key $PORTOPT $inputfile ${remote_username}@${remote_ip}:${fast_io_catalog}/${sources_for_run_catalog}/${frelin[$i]}
	1978	fi
	1979	done
	1980	if (( i > 0 ))
	1981	then
[3025]	1982	printf "\n\n *** input files have been copied to the remote host\n"
[2605]	1983	fi
	1984	fi
	1985
	1986
[2380]	1987	# NOW PERFORM THOSE ACTIONS REQUIRED TO EXECUTE THE PROGRAM (PALM) ON THIS MACHINE
	1988	# (COMPILING/LINKING, EXECUTING, COPYING I/O FILES)
	1989	if [[ $create_batch_job = false && $create_remote_batch_job = false ]]
	1990	then
	1991
	1992	# CHANGE TO THE TEMPORARY WORKING DIRECTORY
	1993	if [[ $running_in_batch_mode = false ]]
	1994	then
[4443]	1995
[3917]	1996	# CREATE THE DIRECTORY
[2566]	1997	mkdir -p $tempdir
[4303]	1998	if [[ $? != 0 ]]
	1999	then
	2000	printf "\n +++ temporary working directory cannot be created."
	2001	printf "\n Check setting of variable fast_io_catalog in your config file."
	2002	locat=execution
	2003	exit
	2004	fi
[2566]	2005	chmod go+rx $tempdir
	2006	cd $tempdir
	2007	printf "\n *** changed to temporary directory: $tempdir"
[2380]	2008
	2009	else
	2010
	2011	# IN BATCH MODE PALMRUN IS CALLED FROM TEMPDIR
[2566]	2012	printf "\n *** running in temporary directory: $tempdir"
[2380]	2013
	2014	fi
	2015
	2016
	2017	# PROVIDE THE INPUT FILES
	2018	# LOOP OVER ALL ACTIVATED FILES (LISTED IN THE CONFIGURATION FILE)
[2669]	2019	optional_files_missing=false
[2380]	2020	(( i = 0 ))
[2610]	2021	while (( i < nr_of_input_files ))
[2380]	2022	do
	2023	(( i = i + 1 ))
	2024	if (( i == 1 ))
	2025	then
	2026	printf "\n\n *** providing INPUT-files:\n$dashes"
	2027	fi
	2028
	2029
	2030	# SKIP OPTIONAL FILES, IF THEY DO NOT EXIST
[2669]	2031	if [[ "${actionin[$i]}" = unavailable ]]
[2380]	2032	then
[2669]	2033	optional_files_missing=true
[2380]	2034	continue
	2035	fi
	2036
	2037	# CHECK FOR SINGLE FILE (SERIAL RUN) OR DIRECTORY (ONE FILE PER CORE FOR PARELLEL EXECUTION)
	2038	files_for_cores=false; filetype=file
	2039	if [[ "${actionin[$i]}" = pe && -n $cores ]]
	2040	then
	2041	files_for_cores=true; filetype=files
	2042	actionin[$i]=""
	2043	elif [[ "${actionin[$i]}" = pe && ! -n $cores ]]
	2044	then
	2045	actionin[$i]=""
	2046	elif [[ "${actionin[$i]}" = lnpe && -n $cores ]]
	2047	then
	2048	files_for_cores=true; filetype=files
	2049	actionin[$i]="ln"
[4536]	2050	# FILE HAS BEEN FOUND INSTEAD OF DIRECTORY
	2051	if [[ -f "${absnamein[$i]}" ]]
	2052	then
	2053	files_for_cores=false; filetype=file
	2054	fi
[2380]	2055	elif [[ "${actionin[$i]}" = lnpe && ! -n $cores ]]
	2056	then
	2057	actionin[$i]="ln"
	2058	fi
	2059
	2060	if [[ $files_for_cores = true ]]
	2061	then
	2062	printf "\n >>> INPUT: ${absnamein[$i]}/.... to ${localin[$i]}"
	2063	else
	2064	printf "\n >>> INPUT: ${absnamein[$i]} to ${localin[$i]}"
	2065	fi
	2066
	2067	# INPUT-FILES TO BE LINKED
	2068	if [[ "${actionin[$i]}" = ln ]]
	2069	then
	2070
	2071	printf "\n $filetype will be linked"
	2072	if [[ $files_for_cores = false ]]
	2073	then
	2074	if [[ -f "${absnamein[$i]}" ]]
	2075	then
	2076	ln ${absnamein[$i]} ${localin[$i]}
	2077	got_tmp[$i]=true
	2078	fi
	2079	else
	2080	if [[ -d "${absnamein[$i]}" ]]
	2081	then
[2396]	2082	mkdir -p ${localin[$i]}
[2380]	2083	cd ${absnamein[$i]}
	2084	for file in $(ls *)
	2085	do
[2566]	2086	ln $file $tempdir/${localin[$i]}
[2380]	2087	done >\|/dev/null 2>&1
[2566]	2088	cd $tempdir
[2380]	2089	fi
	2090
	2091	# IF "ln -f" HAS FAILED DO A NORMAL COPY "cp -r"
	2092	if [[ ! -f "${localin[$i]}/_000000" ]]
	2093	then
	2094	printf "\n --- WARNING: ln failed, using cp instead (might be time consuming...)"
	2095	cp -r ${absnamein[$i]}/* ${localin[$i]}
	2096	fi
	2097
	2098	got_tmp[$i]=true
	2099	fi
	2100	fi
	2101
	2102	# FILE IS STORED IN THE RESPECTIVE DIRECTORY GIVEN IN THE CONFIGURATION FILE
[2669]	2103	if [[ "${actionin[$i]}" = "" \|\| "${actionin[$i]}" = "di" \|\| "${actionin[$i]}" = "tr" \|\| "${actionin[$i]}" = "npe" ]]
[2380]	2104	then
	2105
	2106	if [[ "${actionin[$i]}" = "npe" && -n $cores ]]
	2107	then
	2108
	2109	# FILE COPIES ARE PROVIDED FOR ALL CORES
	2110	# EACH FILE GETS A UNIQUE FILENAME WITH A FOUR DIGIT NUMBER
	2111	printf "\n file will be provided for $cores processors"
[2396]	2112	mkdir -p ${localin[$i]}
[2380]	2113	ival=$cores
	2114	(( ii = 0 ))
	2115	while (( ii <= ival-1 ))
	2116	do
	2117	if (( ii < 10 ))
	2118	then
	2119	cp ${absnamein[$i]} ${localin[$i]}/_000$ii
	2120	elif (( ii < 100 ))
	2121	then
	2122	cp ${absnamein[$i]} ${localin[$i]}/_00$ii
	2123	elif (( ii < 1000 ))
	2124	then
	2125	cp ${absnamein[$i]} ${localin[$i]}/_0$ii
	2126	else
	2127	cp ${absnamein[$i]} ${localin[$i]}/_$ii
	2128	fi
	2129	(( ii = ii + 1 ))
	2130	done
	2131
	2132	else
	2133
	2134	if [[ $files_for_cores = true ]]
	2135	then
	2136
	2137	# PROVIDE FILES FOR EACH CORE
	2138	# FIRST CREATE THE LOCAL DIRECTORY, THEN COPY FILES
	2139	# FROM THE PERMANENT DIRECTORY BY LINKING THEM TO THE LOCAL ONE
	2140	printf "\n providing $cores files for the respective cores"
[2396]	2141	mkdir -p ${localin[$i]}
[2380]	2142	if [[ $link_local_input = true ]]
	2143	then
	2144	printf " files will be linked\n"
	2145	cd ${absnamein[$i]}
	2146	for file in $(ls *)
	2147	do
	2148	ln -f $file ${localin[$i]}
	2149	done
[2566]	2150	cd $tempdir
[2380]	2151	fi
	2152
	2153	# IF "ln -f" FAILED OR IF "$link_local_input = false" DO A NORMAL "cp -r"
	2154	if [[ ! -f "${localin[$i]}/_000000" ]]
	2155	then
	2156	if [[ $link_local_input = true ]]
	2157	then
	2158	printf "\n --- WARNING: ln failed, using cp instead (might be time consuming...)"
	2159	fi
	2160	cp -r ${absnamein[$i]}/* ${localin[$i]}
	2161	fi
	2162
	2163	else
	2164
	2165	# PROVIDE FILE FOR RUNS ON A SINGLE CORE
	2166	if [[ $link_local_input = true ]]
	2167	then
	2168	printf " file will be linked\n"
	2169	ln -f ${absnamein[$i]} ${localin[$i]}
	2170	fi
[4443]	2171	# IF "ln -f" FAILS OR IF "$link_local_input = false" DO A NORMAL "cp"
[2380]	2172	if [[ ! -f "${localin[$i]}" ]]
	2173	then
	2174	if [[ $link_local_input = true ]]
	2175	then
	2176	printf "\n --- WARNING: ln failed, using cp instead (might be time consuming...)"
	2177	fi
[2669]	2178	if [[ $running_on_remote = true && "${actionin[$i]}" = tr ]]
[2605]	2179	then
	2180	mv ${absnamein[$i]} ${localin[$i]}
	2181	else
	2182	cp ${absnamein[$i]} ${localin[$i]}
	2183	fi
[2380]	2184	fi
	2185	fi
	2186	fi
	2187	fi
	2188
	2189	done
	2190	if (( i != 0 ))
	2191	then
[2669]	2192	if [[ $optional_files_missing = true ]]
	2193	then
	2194	printf "\n *** INFORMATIVE: some optional INPUT-files are not present"
	2195	fi
[2380]	2196	printf "\n$dashes\n *** all INPUT-files provided \n"
	2197	fi
	2198
[3917]	2199	# NOW COPY FILES (.f90, .o, config files, etc.) FROM SOURCES_FOR_RUN_... TO THE TEMPORARY
	2200	# WORKING DIRECTORY
	2201	cp ${fast_io_catalog}/${sources_for_run_catalog}/{,.[!.]} $tempdir
[2380]	2202
[3917]	2203
[2380]	2204	# EXECUTE INPUT-COMMANDS GIVEN IN THE CONFIGURATION FILE
	2205	(( i = 0 ))
	2206	while (( i < iic ))
	2207	do
	2208	(( i = i + 1 ))
	2209	if (( i == 1 ))
	2210	then
	2211	printf "\n\n *** execution of INPUT-commands:\n$dashes"
	2212	fi
	2213	printf "\n >>> ${in_command[$i]}"
	2214	eval ${in_command[$i]}
	2215	if (( i == iic ))
	2216	then
	2217	printf "\n$dashes\n"
	2218	fi
	2219	done
	2220
	2221
[3313]	2222	# CHECK IF THE PROGRESS BAR NEEDS TO BE DISABLED
	2223	if [[ $running_in_batch_mode = true \|\| $running_in_test_mode = true ]]
	2224	then
	2225	progress_bar_disabled=true
	2226	else
	2227	progress_bar_disabled=false
	2228	fi
	2229
	2230
[2380]	2231	# CREATE THE NAMELIST-FILE WITH VALUES OF ENVIRONMENT-VARIABLES REQUIRED BY PALM
	2232	# (FILE ENVPAR WILL BE READ BY PALM)
	2233	cat > ENVPAR << EOF
[3236]	2234	&envpar run_identifier = '$run_identifier', host = '$configuration_identifier',
[2906]	2235	write_svf = .${write_svf}., write_binary = .${write_binary}.,
	2236	read_svf = .${read_svf}., tasks_per_node = $tasks_per_node,
[2380]	2237	maximum_parallel_io_streams = $maximum_parallel_io_streams,
	2238	maximum_cpu_time_allowed = ${cpumax}.,
	2239	revision = '$global_revision',
[3313]	2240	progress_bar_disabled = .${progress_bar_disabled}. /
[2380]	2241
	2242	EOF
	2243
	2244
	2245	# STARTING THE EXECUTABLE
	2246	printf "\n\n *** execution starts in directory\n \"`pwd`\"\n$dashes\n"
[2566]	2247	PATH=$PATH:$tempdir
[2380]	2248
	2249
	2250	# REPLACE PARAMETERS IN THE EXECUTION COMMAND WITH REAL VALUES
[2693]	2251	line=`echo "${execute_command}" \| sed 's/{{/$/g' \| sed 's/}}//g'`
	2252	line2=`echo "${execute_command}" \| sed 's/{{mpi_tasks}}/1/g' \| sed 's/{{tasks_per_node}}/1/g' \| sed 's/palm/combine_plot_fields.x/g'`
[2380]	2253	eval line=\"$line\"
	2254	execute_command="$line"
	2255
	2256
	2257	# EXECUTION COMMAND FOR COMBINE_PLOT_FIELDS
[2566]	2258	if [[ "$execute_command_for_combine" = "" ]]
	2259	then
	2260	eval line2=\"$line2\"
	2261	execute_command_for_combine="$line2"
	2262	fi
	2263
[2380]	2264
	2265
	2266	# PROVIDE A HOSTFILE, IF REQUIRED
	2267	if [[ "$hostfile" != "" ]]
	2268	then
	2269
	2270	if [[ $hostfile = auto ]]
	2271	then
	2272	# CREATE A NEW HOSTFILE
	2273	(( ii = 1 ))
	2274	while (( ii <= cores / threads_per_task ))
	2275	do
	2276	echo $(hostname) >> hostfile
	2277	(( ii = ii + 1 ))
	2278	done
	2279	if (( cores / threads_per_task == 0 ))
	2280	then
	2281	echo $(hostname) >> hostfile
	2282	fi
	2283
	2284	else
	2285	cp $hostfile hostfile
	2286	fi
	2287	eval line=\"`head -n $ii hostfile`\"
	2288	printf "\n *** running on: $line"
	2289	fi
	2290
	2291
	2292
	2293	# SET THE NUMBER OF OPENMP-THREADS
	2294	if [[ $use_openmp = true ]]
	2295	then
	2296	export OMP_NUM_THREADS=$threads_per_task
	2297	printf "\n *** number of OpenMP threads per MPI-task: $OMP_NUM_THREADS"
	2298	else
	2299	export OMP_NUM_THREADS=1
	2300	fi
	2301
	2302
[2610]	2303	# PROVIDE DATA FOR ATMOSPHERE OCEAN COUPLING
[2380]	2304	if [[ $run_coupled_model = false ]]
	2305	then
	2306	if [[ "$ocean_file_appendix" = true ]]
	2307	then
	2308	echo "precursor_ocean" > coupling_steering
	2309	else
	2310	echo "precursor_atmos" > coupling_steering
	2311	fi
	2312	else
	2313	(( iia = $cores_atmos / $threads_per_task ))
	2314	(( iio = $cores_ocean / $threads_per_task ))
	2315	printf "\n coupled run ($iia atmosphere, $iio ocean)"
	2316	printf "\n\n"
	2317	echo "coupled_run $iia $iio" > coupling_steering
	2318	fi
	2319
	2320	printf "\n *** execute command:"
	2321	printf "\n \"$execute_command\" \n\n"
	2322
	2323
[3393]	2324	if [[ $progress_bar_disabled = true ]]
[2380]	2325	then
[3402]	2326	$execute_command < coupling_steering &> >(grep -v --line-buffered -e '^STOP 1$' -e '^1$' &> >(tee STDOUT) )
[3393]	2327	exit_code=${PIPESTATUS[0]}
	2328	else
	2329	$execute_command < coupling_steering &> >(tee STDOUT)
	2330	exit_code=${PIPESTATUS[0]}
	2331	fi
[2380]	2332
[3393]	2333	if [[ ${exit_code} != 0 ]]
	2334	then
	2335
[2380]	2336	# ABORT IN CASE OF RUNTIME ERRORS
	2337	printf "\n +++ runtime error occured"
	2338	locat=execution
	2339	exit
	2340
	2341	else
	2342
	2343	printf "\n$dashes\n *** execution finished \n"
	2344
	2345	fi
	2346
	2347
	2348	# CALL OF combine_plot_fields IN ORDER TO MERGE SINGLE FILES WRITTEN
	2349	# BY EACH CORE INTO ONE FILE
	2350	if [[ ! -f combine_plot_fields.x ]]
	2351	then
	2352
	2353	printf "\n\n\n +++ WARNING: no combine_plot_fields found"
	2354	printf "\n 2d- and/or 3d-data may be incomplete!"
	2355	printf "\n Your previous palmbuild may have failed. Please check.\n"
	2356
	2357	elif [[ "$combine_plot_fields" == true ]]
	2358	then
	2359
	2360	printf "\n\n\n *** post-processing: now executing \"$execute_command_for_combine\" ..."
	2361	$execute_command_for_combine
	2362
	2363	else
	2364
	2365	# TEMPORARY SOLUTION TO SKIP combine_plot_fields. THIS IS REQUIRED IN CASE OF HUGE AMOUNT OF
[2605]	2366	# DATA OUTPUT
[2380]	2367	printf "\n\n\n *** post-processing: skipping combine_plot_fields (-Z option set) ..."
	2368	fi
	2369
	2370
	2371
	2372	# EXECUTE OUTPUT-COMMANDS GIVEN IN THE CONFIGURATION FILE
	2373	(( i = 0 ))
	2374	while (( i < ioc ))
	2375	do
	2376	(( i = i + 1 ))
	2377	if (( i == 1 ))
	2378	then
	2379	printf "\n\n *** execution of OUTPUT-commands:\n$dashes"
	2380	fi
[2799]	2381
	2382	# REPLACE PARAMETERS IN THE OUTPUT COMMAND WITH REAL VALUES
	2383	out_command[$i]=`echo "${out_command[$i]}" \| sed 's/{{/$/g' \| sed 's/}}//g'`
	2384
[2380]	2385	printf "\n >>> ${out_command[$i]}"
	2386	eval ${out_command[$i]}
	2387	if (( i == ioc ))
	2388	then
	2389	printf "\n$dashes\n"
	2390	fi
	2391	done
	2392
	2393
[2610]	2394	# IN A FIRST PASS, ADD ADDITIONAL OUTPUT FILE CONNECTIONS IN CASE OF
	2395	# WILDCARDS
[2380]	2396	(( i = 0 ))
[2610]	2397	(( nr_of_output_files = 0 ))
	2398
[2380]	2399	while (( i < iout ))
	2400	do
[2610]	2401
[2380]	2402	(( i = i + 1 ))
[2610]	2403
	2404	# FIRST CHECK FOR MULTIPLE NAMES WITH THE SAME LOCAL NAME AND
	2405	# CREATE A LIST FOR THE DETECTED ENDINGS
	2406	if [[ "${multout[$i]}" = true ]]
	2407	then
	2408	# DETERMINE THE EXISTING EXTENSIONS FROM THE LIST OF FILES
	2409	ls -1 -d ${localout_pre[$i]} > filelist 2>/dev/null
	2410	ls -1 -d ${localout_pre[$i]}_* >> filelist 2>/dev/null
	2411
	2412	endings="DEFAULT"
	2413	while read line
	2414	do
	2415	# remove the local name from the beginning
	2416	localnamestring="${localout_pre[$i]}"
	2417	length_localname=${#localnamestring}
	2418	ending=${line:${length_localname}}
	2419
	2420	if [[ "$ending" != "" ]]
	2421	then
	2422	endings="$endings $ending"
	2423	fi
	2424
	2425	done <filelist
	2426
	2427	rm filelist
	2428
	2429	else
	2430
	2431	# SINGLE NAME
	2432	endings=DEFAULT
	2433
	2434	fi
	2435
	2436	# FOR EACH BASENAME ENDING CREATE AN ENTRY IN THE FINAL OUTPUT FILE LIST
	2437	for ending in $endings
	2438	do
	2439
	2440	# DEFAULT MEANS THAT THE ENDING GIVEN IN .iofiles SHALL BE USED
	2441	if [[ $ending = DEFAULT ]]
	2442	then
	2443	ending=""
	2444	fi
	2445
	2446	# NEW ENTRY (ENDING IS ALSO ADDED TO LOCAL FILENAME READ BY PALM!)
	2447	(( nr_of_output_files = nr_of_output_files + 1 ))
	2448	localout[$nr_of_output_files]="${localout_pre[$i]}"$ending
	2449	transout[$nr_of_output_files]="${transout_pre[$i]}"
	2450	actionout[$nr_of_output_files]="${actionout_pre[$i]}"
	2451	pathout[$nr_of_output_files]="${pathout_pre[$i]}"
	2452	endout[$nr_of_output_files]="${endout_pre[$i]}"$ending
	2453	extout[$nr_of_output_files]="${extout_pre[$i]}"
[2669]	2454	warnout[$nr_of_output_files]="${warnout_pre[$i]}"
[2610]	2455
	2456	done
	2457
	2458	done
	2459
	2460
	2461
	2462
	2463	# COPY LOCAL OUTPUT-FILES TO THEIR PERMANENT DESTINATIONS
	2464	(( i = 0 ))
	2465	while (( i < nr_of_output_files ))
	2466	do
	2467	(( i = i + 1 ))
[2380]	2468	if (( i == 1 ))
	2469	then
[3860]	2470	timestring=`date +%H:%M:%S`
	2471	printf "\n\n *** saving OUTPUT-files: local time: $timestring"
[2380]	2472
	2473	# GET RUN NUMBER ASSIGNED BY PALM
	2474	if [[ -f RUN_NUMBER ]]
	2475	then
	2476	read run_number < RUN_NUMBER
	2477	printf "\n *** PALM generated run_number = "$run_number" will be used as unified cycle number for all output files"
	2478	usecycle_option="-U $run_number"
	2479	else
	2480	run_number=0
	2481	usecycle_option=""
	2482	fi
	2483	if [[ $running_on_remote = true && "$remote_loginnode" != "" ]]
	2484	then
[2404]	2485	printf "\n *** in case of SCP transfers to local host"
	2486	printf "\n they will be done via remote login-node \"$remote_loginnode\" "
[2380]	2487	fi
	2488	printf "\n$dashes"
	2489	fi
	2490
	2491	if [[ ! ( $running_on_remote = true && ( "${actionout[$i]}" = tr \|\| "${actionout[$i]}" = tra \|\| "${actionout[$i]}" = trpe ) ) ]]
	2492	then
	2493
[3236]	2494	eval filename=${pathout[$i]}/${run_identifier}${endout[$i]}
[2610]	2495
	2496	# DETERMINE THE CYCLE NUMBER
	2497	ls -1 -d $filename > filelist 2>/dev/null
	2498	ls -1 -d $filename.* >> filelist 2>/dev/null
[4443]	2499	(( maxcycle = 0 ))
[2610]	2500	while read line
	2501	do
	2502
	2503	# filename without path (i.e. after the last "/")
	2504	basefilename=$(basename ${line})
	2505
	2506	# check if there is an extension
	2507	extension=${basefilename##*.}
	2508	if [[ "$extension" = "${extout[$i]}" ]]
	2509	then
	2510	basefilename=${basefilename%.*}
	2511	fi
	2512
	2513	# check for an existing cycle number
	2514	cycle=${basefilename##*.}
	2515	if [[ $cycle =~ ^-?[0-9]+$ ]]
	2516	then
[2669]	2517	# NUMBERS WITH LEADING ZEROS ARE INTERPRETED AS OCTAL NUMBERS
	2518	# 10# EXPLICITLY SPECIFIES THE NUMBER BASE AS 10
	2519	(( icycle = $((10#$cycle)) + 1 ))
[2610]	2520	else
	2521	(( icycle = 1 ))
	2522	fi
	2523
	2524	if (( icycle > maxcycle ))
	2525	then
	2526	(( maxcycle = icycle ))
	2527	fi
	2528
	2529	done <filelist
	2530	rm filelist
	2531
	2532
	2533	# SET THE CYCLE NUMBER
	2534	# IN CASE OF FILE-APPEND, IT MUST BE THE HIGHEST EXISTING CYCLE NUMBER
	2535	if [[ "${actionout[$i]}" = a ]]
	2536	then
[4443]	2537	if (( maxcycle > 0 ))
	2538	then
	2539	(( maxcycle = maxcycle - 1 ))
	2540	fi
[2610]	2541	fi
	2542
	2543	(( cycnum[$i] = maxcycle ))
	2544	pathout[$i]=$filename
	2545
	2546
	2547	# ADD CYCLE NUMBER TO FILENAME
	2548	# IN APPEND MODE, FILES KEEP THEIR CURRENT CYCLE NUMBER
[2380]	2549	if [[ "${actionout[$i]}" != "a" ]]
	2550	then
	2551	# SET RUN NUMBER AS CYCLE NUMBER, IF THERE IS NOT A CONFLICT
	2552	# WITH AN EXISTING CYCLE NUMBER
	2553	if (( run_number >= cycnum[$i] ))
	2554	then
	2555	(( cycnum[$i] = run_number ))
	2556	else
	2557	if (( run_number > 0 ))
	2558	then
	2559	printf "\n --- INFORMATIVE: The following file cannot get a unified cycle number"
	2560	fi
	2561	fi
	2562	fi
[4443]	2563	cyclestring=`printf "%03d" ${cycnum[$i]}`
	2564	pathout[$i]=${pathout[$i]}.$cyclestring
	2565	fi
[2380]	2566
	2567	# CHECK FOR SINGLE FILE (SERIAL RUN) OR DIRECTORY (ONE FILE PER CORE FOR PARELLEL EXECUTION)
	2568	files_for_cores=false; filetype=file
	2569	link_local_output=false
	2570	if [[ "${actionout[$i]}" = pe && -n $cores ]]
	2571	then
	2572	files_for_cores=true; filetype=directory
	2573	actionout[$i]=""
	2574	elif [[ "${actionout[$i]}" = pe && ! -n $cores ]]
	2575	then
	2576	actionout[$i]=""
	2577	elif [[ "${actionout[$i]}" = lnpe && -n $cores ]]
	2578	then
	2579	files_for_cores=true; filetype=directory
	2580	link_local_output=true
	2581	actionout[$i]=""
[4536]	2582	# FILE HAS BEEN FOUND INSTEAD OF DIRECTORY
	2583	if [[ -f ${localout[$i]} ]]
	2584	then
	2585	files_for_cores=false; filetype=file
	2586	fi
[2380]	2587	elif [[ "${actionout[$i]}" = lnpe && ! -n $cores ]]
	2588	then
[4536]	2589	link_local_output=true
[2380]	2590	actionout[$i]=""
	2591	elif [[ "${actionout[$i]}" = trpe && -n $cores ]]
	2592	then
	2593	files_for_cores=true; filetype=directory
	2594	actionout[$i]="tr"
	2595	elif [[ "${actionout[$i]}" = trpe && ! -n $cores ]]
	2596	then
	2597	actionout[$i]="tr"
	2598	fi
	2599
	2600	if [[ ! -f ${localout[$i]} && $files_for_cores = false ]]
	2601	then
[2669]	2602	if [[ ${warnout[$i]} = true ]]
	2603	then
	2604	printf "\n +++ temporary OUTPUT-file ${localout[$i]} does not exist\n"
	2605	fi
[2380]	2606	elif [[ ! -d ${localout[$i]} && $files_for_cores = true ]]
	2607	then
[2669]	2608	if [[ ${warnout[$i]} = true ]]
	2609	then
	2610	printf "\n +++ temporary OUTPUT-file ${localout[$i]}/.... does not exist\n"
	2611	fi
[2380]	2612	else
	2613
	2614
	2615	# COPY VIA SCP TO LOCAL HOST (ALWAYS IN BINARY MODE USING batch_scp option -m)
	2616	# IF TARGET DIRECTORY DOES NOT EXISTS, TRY TO CREATE IT
	2617	if [[ "${actionout[$i]}" = tr \|\| "${actionout[$i]}" = tra ]]
	2618	then
	2619	if [[ $running_on_remote = true ]]
	2620	then
	2621
	2622	# SET OPTIONS FOR TRANSFER
	2623	if [[ "${actionout[$i]}" = tr ]]
	2624	then
	2625	if [[ $files_for_cores = false ]]
	2626	then
	2627	catalog_option=""
	2628	catalog_string=""
	2629	else
	2630	catalog_option="-c"
	2631	catalog_string="/"
	2632	fi
	2633	append_option=""
	2634	append_string=""
	2635	else
	2636	append_option="-A"
	2637	append_string="append"
	2638	fi
	2639
	2640	transfer_failed=false
	2641	printf "\n >>> OUTPUT: ${localout[$i]}$catalog_string $append_string by SCP to"
[3236]	2642	printf "\n ${pathout[$i]}/${configuration_identifier}_${run_identifier}${endout[$i]}$catalog_string\n"
[2380]	2643
	2644	# TRANSFER VIA SCP
	2645	if [[ "$remote_loginnode" != "" ]]
	2646	then
[3534]	2647	ssh -q $remote_username@$remote_loginnode "cd $tempdir; ${fast_io_catalog}/${sources_for_run_catalog}/batch_scp $PORTOPT $catalog_option $append_option -b -m $usecycle_option -u $local_username $return_address ${localout[$i]} \"${pathout[$i]}\" ${configuration_identifier}_${run_identifier}${endout[$i]} ${extout[$i]}"
[2380]	2648	else
[3236]	2649	batch_scp $PORTOPT $catalog_option $append_option -b -m $usecycle_option -u $local_username $return_address ${localout[$i]} "${pathout[$i]}" ${configuration_identifier}_${run_identifier}${endout[$i]} ${extout[$i]}
[2380]	2650	fi
[3312]	2651	[[ ${PIPESTATUS[0]} != 0 ]] && transfer_failed=true
[2380]	2652
	2653
	2654	# IF TRANSFER FAILED, CREATE BACKUP COPY ON THIS MACHINE
	2655	if [[ $transfer_failed = true ]]
	2656	then
	2657	printf " +++ transfer failed. Trying to save a copy on this host under:\n"
[3236]	2658	printf " ${pathout[$i]}/${configuration_identifier}_${run_identifier}${endout[$i]}_$run_id_number\n"
[2380]	2659
	2660	# FIRST CHECK, IF DIRECTORY EXISTS, AND CREATE IT, IF NECESSARY
	2661	eval local_catalog=${pathout[$i]}
	2662	if [[ ! -d $local_catalog ]]
	2663	then
	2664	printf " *** local directory does not exist. Trying to create:\n"
	2665	printf " $local_catalog \n"
	2666	mkdir -p $local_catalog
	2667	fi
[3236]	2668	eval cp ${localout[$i]} ${pathout[$i]}/${configuration_identifier}_${run_identifier}${endout[$i]}_$run_id_number
[2380]	2669	transfer_problems=true
	2670	fi
	2671
	2672	else
	2673
	2674	# UNSET actionout. DUE TO THIS SETTING, FILE WILL LATER JUST
	2675	# BE COPIED OR APPENDED ON THIS MACHINE
	2676	if [[ "${actionout[$i]}" = tr ]]
	2677	then
	2678	actionout[$i]=""
	2679	else
	2680	actionout[$i]="a"
	2681	fi
	2682	fi
	2683	fi
	2684
	2685
	2686	# APPEND ON THIS MACHINE
	2687	if [[ "${actionout[$i]}" = "a" ]]
	2688	then
	2689	if [[ "${extout[$i]}" != " " && "${extout[$i]}" != "" ]]
	2690	then
	2691	printf "\n >>> OUTPUT: ${localout[$i]} append to"
	2692	printf "\n ${pathout[$i]}.${extout[$i]}\n"
	2693	cat ${localout[$i]} >> ${pathout[$i]}.${extout[$i]}
	2694	else
	2695	printf "\n >>> OUTPUT: ${localout[$i]} append to"
	2696	printf "\n ${pathout[$i]}\n"
	2697	cat ${localout[$i]} >> ${pathout[$i]}
	2698	fi
	2699	fi
	2700
	2701	# COPY ON THIS MACHINE
	2702	# COPY HAS TO BE USED, BECAUSE MOVE DOES NOT WORK IF FILE-ORIGIN AND TARGET ARE
	2703	# ON DIFFERENT FILE-SYSTEMS
	2704	if [[ "${actionout[$i]}" = "" && $files_for_cores = false ]]
	2705	then
	2706
	2707	# COPY IN CASE OF RUNS ON SINGLE CORES
	2708	if [[ "${extout[$i]}" != " " && "${extout[$i]}" != "" ]]
	2709	then
	2710	printf "\n >>> OUTPUT: ${localout[$i]} to"
	2711	printf "\n ${pathout[$i]}.${extout[$i]}\n"
[4536]	2712	if [[ $link_local_output = true ]]
[2380]	2713	then
[4536]	2714	printf " file will be linked\n"
	2715	ln -f ${localout[$i]} ${pathout[$i]}.${extout[$i]}
	2716	fi
	2717	# If "ln -f" fails of if "$link_local_output = false" do a normal "cp"
	2718	if [[ ! -f "${pathout[$i]}.${extout[$i]}" ]]
	2719	then
	2720	if [[ $link_local_output = true ]]
	2721	then
[2380]	2722	printf " --- WARNING: ln failed, using cp instead (might be time consuming...)\n"
[4536]	2723	fi
	2724	cp ${localout[$i]} ${pathout[$i]}.${extout[$i]}
[2380]	2725	fi
	2726	else
	2727	printf "\n >>> OUTPUT: ${localout[$i]} to"
	2728	printf "\n ${pathout[$i]}\n"
[4536]	2729	if [[ $link_local_output = true ]]
[2380]	2730	then
[4536]	2731	printf " file will be linked\n"
	2732	ln -f ${localout[$i]} ${pathout[$i]}
	2733	fi
	2734	# If "ln -f" fails of if "$link_local_output = false" do a normal "cp"
	2735	if [[ ! -f "${pathout[$i]}" ]]
	2736	then
	2737	if [[ $link_local_output = true ]]
	2738	then
[2380]	2739	printf " --- WARNING: ln failed, using cp instead (might be time consuming...)\n"
[4536]	2740	fi
	2741	cp ${localout[$i]} ${pathout[$i]}
[2380]	2742	fi
	2743	fi
	2744
	2745	elif [[ "${actionout[$i]}" = "" && $files_for_cores = true ]]
	2746	then
	2747
	2748	# FILES FROM THE DIFFERENT CORES ARE MOVED WITH ln-COMMAND TO THE PERMANENT DIRECTORY
	2749	# AS A FIRST STEP, THE PERMANENT DIRECTORY IS CREATED
	2750	printf "\n >>> OUTPUT: ${localout[$i]}/_.... to"
	2751	printf "\n ${pathout[$i]}\n"
[4536]	2752	if [[ $link_local_output = true ]]
	2753	then
	2754	printf " files will be linked\n"
	2755	mkdir -p ${pathout[$i]}
	2756	cd ${localout[$i]}
	2757	for file in $(ls *)
	2758	do
	2759	ln -f $file ${pathout[$i]}
	2760	done >\|/dev/null 2>&1
	2761	cd $tempdir
	2762	fi
[2380]	2763
	2764	# IF "ln -f" HAS FAILED OR IF "$link_local_output = false" DO A NORMAL COPY "cp -r"
[4536]	2765	if [[ ! -f "${pathout[$i]}/_000000" ]]
[2380]	2766	then
[4536]	2767	if [[ $link_local_output = true ]]
	2768	then
[2380]	2769	printf " --- WARNING: ln failed, using cp instead (might be time consuming...)\n"
[4536]	2770	fi
	2771	[[ ! -d "${pathout[$i]}" ]] && mkdir ${pathout[$i]}
	2772	cp -r ${localout[$i]}/* ${pathout[$i]}
[2380]	2773	fi
	2774
	2775	fi
	2776	fi
	2777	done
	2778
	2779	if (( i != 0 ))
	2780	then
[3860]	2781	timestring=`date +%H:%M:%S`
[2380]	2782	if [[ $transfer_problems = true ]]
	2783	then
[3860]	2784	printf "\n$dashes\n *** OUTPUT-files saved local time: $timestring"
[2380]	2785	printf "\n +++ WARNING: some data transfers failed! \n"
	2786	else
[3860]	2787	printf "\n$dashes\n *** all OUTPUT-files saved local time: $timestring \n"
[2380]	2788	fi
	2789	fi
	2790
	2791
	2792	# IF REQUIRED, START A RESTART-JOB
	2793	# FILE CONTINUE_RUN MUST HAVE BEEN CREATED BY THE EXECUTABLE (PALM)
	2794	if [[ -f CONTINUE_RUN ]]
	2795	then
	2796
	2797	# ADD RESTART-OPTIONS TO THE PALMRUN-CALL (IF THEY ARE NOT USED ALREADY):
	2798	# -C TELLS PALMRUN THAT IT IS A RESTART-RUN
	2799	# -v SILENT MODE WITHOUT INTERACTIVE QUERIES
	2800	# -b START A BATCH JOB
	2801	[[ $(echo $prc \| grep -c "\-C") = 0 ]] && prc="$prc -C"
	2802	[[ $(echo $prc \| grep -c "\-v") = 0 ]] && prc="$prc -v"
[3982]	2803	[[ $(echo $prc \| grep -c "\-b") = 0 && $running_on_remote = false ]] && prc="$prc -b"
[2380]	2804
[2957]	2805	# REPLACE SVFOUT IN THE ACTIVATION STRINGS (GIVEN WITH OPTION -a)
	2806	# SO THAT RESTARTS USE SVF DATA CREATED BY THE INITIAL RUN
	2807	if [[ $(echo $prc \| grep -c "svfout") != 0 ]]
	2808	then
	2809	prc=`echo $prc \| sed 's/svfout/svfin/g'`
	2810	fi
[2380]	2811
	2812	# REPLACE THE HASH IN THE ACTIVATION STRINGS (GIVEN WITH OPTION -a)
	2813	# SO THAT RESTARTS ACCESS DIFFERENT FILES THAN THE INITIAL RUN
	2814	if [[ $(echo $prc \| grep -c "#") != 0 ]]
	2815	then
[2381]	2816	prc=`echo $prc \| sed 's/#/r/g'`
[2380]	2817	fi
	2818
	2819
	2820	# START THE RESTART-JOB
	2821	printf "\n\n *** initiating restart-run on \"$local_ip\" using command:\n"
	2822	echo " $prc"
	2823	printf "\n$dashes\n"
	2824	if [[ $running_on_remote = true ]]
	2825	then
	2826
	2827	echo "*** ssh will be used to initiate restart-runs!"
	2828	echo " return_address=\"$return_address\" "
	2829	echo " return_username=\"$local_username\" "
	2830
	2831	if [[ "$remote_loginnode" != "" ]]
	2832	then
[3534]	2833	ssh -q $remote_username@$remote_loginnode "ssh -q $SSH_PORTOPT $local_username@$return_address \" PATH=\\\$PATH:$LOCAL_PALMRUN_PATH; cd $LOCAL_PWD; $prc\" " \| tee palmrun_restart.log
[2380]	2834	else
[3534]	2835	ssh -q $SSH_PORTOPT $local_username@$return_address " PATH=\\\$PATH:$LOCAL_PALMRUN_PATH; cd $LOCAL_PWD; $prc" \| tee palmrun_restart.log
[2380]	2836	fi
	2837
	2838	# WAIT TO ALLOW THE RESTART-JOB TO BE QUEUED, BEFORE THE CURRENT JOB IS FINISHED
	2839	sleep 30
	2840
	2841	else
	2842
	2843	# START THE RESTART JOB ON THE LOCAL HOST
[3132]	2844	eval $prc \| tee palmrun_restart.log # THE ' MUST BE EVALUATED
[2380]	2845
	2846	fi
	2847
[3132]	2848	# CHECK, IF RESTART JOB HAS BEEN STARTED
[3725]	2849	if [[ $( grep -c "+++ palmrun crashed" palmrun_restart.log ) != 0 ]]
[3132]	2850	then
	2851	printf "\n$dashes\n +++ creating restart run failed \n"
	2852	locat=create_restart
[3455]	2853	rm palmrun_restart.log
[3132]	2854	exit
[3725]	2855	elif [[ $( grep -c "*** palmrun finished" palmrun_restart.log ) != 1 ]]
	2856	then
	2857	printf "\n$dashes\n +++ creating restart run failed, probably due to network problems\n"
	2858	locat=create_restart
	2859	rm palmrun_restart.log
	2860	exit
[3132]	2861	else
	2862	printf "\n$dashes\n *** restart run initiated \n"
	2863	rm palmrun_restart.log
	2864	fi
[2380]	2865
[3132]	2866
[2380]	2867	# DELETE INPUT-(RESTART)FILES, WHICH HAVE BEEN FETCHED FROM THE TEMPORARY DATA
	2868	# DIRECTORY, BACAUSE THEY ARE NOT REQUIRED BY THE RESTART-JOB.
	2869	# THIS IS DONE IN ORDER TO AVOID EXCEEDING DISC QUOTAS OR DISC SPACE (RESTART-FILES
	2870	# MAY BE VERY HUGE)
	2871	(( i = 0 ))
[2610]	2872	while (( i < nr_of_input_files ))
[2380]	2873	do
	2874	(( i = i + 1 ))
	2875	if [[ "${got_tmp[$i]}" = true && $keep_data_from_previous_run = false ]]
	2876	then
	2877	rm -r ${absnamein[$i]}
	2878	fi
	2879	done
	2880
	2881	fi
	2882
	2883
	2884	# ALL ACTIONS FINISHED, TEMPORARY WORKING-DIRECTORY CAN BE DELETED
	2885	cd $HOME
[2566]	2886	[[ $delete_temporary_catalog = true ]] && rm -rf $tempdir
[2380]	2887
	2888
	2889	else
	2890
	2891
	2892	# PREPARING ACTIONS,
	2893	# IF A BATCH-JOB IS TO BE GENERATED AND TO BE STARTED ON A LOCAL OR REMOTE-MACHINE
	2894
	2895	# BUILD THE PALMRUN-COMMAND TO BE CALLED IN THE BATCH-JOB
[3455]	2896	palmrun_com="$palmrun_script_name -r $run_identifier -c $configuration_identifier -m $memory -t $cpumax -q $queue -i $run_id_number -U $local_username"
[2380]	2897	[[ "$activation_string_list" != "" ]] && palmrun_com=${palmrun_com}" -a \"$activation_string_list\""
	2898	[[ "$global_revision" != "" ]] && palmrun_com=${palmrun_com}" -G \"$global_revision\""
	2899	[[ $keep_data_from_previous_run = true ]] && palmrun_com=${palmrun_com}" -k"
	2900	[[ $do_trace = true ]] && palmrun_com=${palmrun_com}" -x"
	2901	[[ "$cores" != "" ]] && palmrun_com=${palmrun_com}" -X $cores"
	2902	[[ $use_openmp = true ]] && palmrun_com=${palmrun_com}" -O $threads_per_task"
[2501]	2903	[[ $tasks_per_node != 0 ]] && palmrun_com=${palmrun_com}" -T $tasks_per_node"
[2380]	2904	[[ $delete_temporary_catalog = false ]] && palmrun_com=${palmrun_com}" -B"
	2905	[[ "$ocean_file_appendix" = true ]] && palmrun_com=${palmrun_com}" -y"
	2906	[[ $run_coupled_model = true ]] && palmrun_com=${palmrun_com}" -Y \"$coupled_dist\""
	2907	[[ "$combine_plot_fields" = false ]] && palmrun_com=${palmrun_com}" -Z"
	2908	[[ "$max_par_io_str" != "" ]] && palmrun_com=${palmrun_com}" -w $max_par_io_str"
[2479]	2909	[[ "$project_account" != "" ]] && palmrun_com=${palmrun_com}" -A $project_account"
[2380]	2910	if [[ $create_remote_batch_job = true ]]
	2911	then
	2912	palmrun_com=${palmrun_com}" -j -u $remote_username -R $local_ip"
[2420]	2913	if [[ $do_trace = true ]]
	2914	then
	2915	printf "\n *** PALMRUN-command on remote host:\n $palmrun_com \n"
	2916	fi
[2380]	2917	elif [[ $create_batch_job = true ]]
	2918	then
	2919	palmrun_com=${palmrun_com}" -j"
[2420]	2920	if [[ $do_trace = true ]]
	2921	then
	2922	printf "\n *** PALMRUN-command on local host:\n $palmrun_com \n"
	2923	fi
[2380]	2924	fi
	2925
	2926
	2927	# DETERMINE THE FULL PATHS FOR THE JOB PROTOCOL FILES ON THE LOCAL AND
	2928	# REMOTE HOST
[3236]	2929	job_protocol_file_local=${local_jobcatalog}/${configuration_identifier}_${run_id}
[2380]	2930	job_protocol_file=$job_protocol_file_local
	2931	if [[ $create_remote_batch_job = true ]]
	2932	then
[3236]	2933	job_protocol_file_remote=${remote_jobcatalog}/${configuration_identifier}_${run_id}
[2380]	2934	job_protocol_file=$job_protocol_file_remote
	2935	job_transfer_protocol_file=${remote_jobcatalog}/last_job_transfer_protocol
[3236]	2936	scpjob_file=${remote_jobcatalog}/scpjob.$run_id_number
[2380]	2937	fi
	2938
	2939
	2940	# BUILD THE JOB-SCRIPTS ON FILE jobfile
[3236]	2941	jobfile=jobfile.$run_id_number
[2380]	2942
	2943
	2944	# FIRST CREATE THE BATCH DIRECTIVES
	2945	(( i = 0 ))
	2946	while (( i < ibd ))
	2947	do
	2948	(( i = i + 1 ))
[2693]	2949	line=`echo "${batch_directive[$i]}" \| sed 's/{{/$/g' \| sed 's/}}//g'`
[2380]	2950	eval line=\"$line\"
	2951	echo "$line" >> $jobfile
	2952	done
	2953	echo " " >> $jobfile
	2954
	2955
	2956	# FOR BATCH JOBS ON REMOTE HOSTS, ADD THE JOBFILE TO SEND BACK THE JOB
	2957	# PROTOCOL
	2958	if [[ $create_remote_batch_job = true ]]
	2959	then
[3236]	2960	echo "set +vx" >> $jobfile
	2961	echo "trap '" >> $jobfile
	2962	echo "set +vx" >> $jobfile
	2963	echo "cd ${remote_jobcatalog}" >> $jobfile
	2964	echo "cat > scpjob.$run_id_number << %%END%%" >> $jobfile
[2380]	2965
	2966	# ADD THE BATCH DIRECTIVES
	2967	(( i = 0 ))
	2968	while (( i < ibdt ))
	2969	do
	2970	(( i = i + 1 ))
[2693]	2971	line=`echo "${batch_directive_transfer[$i]}" \| sed 's/{{/$/g' \| sed 's/}}//g'`
[2380]	2972	eval line=\"$line\"
	2973	echo "$line" >> $jobfile
	2974	done
	2975	echo " " >> $jobfile
	2976
	2977	echo "set -x" >> $jobfile
[3534]	2978	if [[ "$remote_loginnode" != "" ]]
	2979	then
	2980	echo "ssh -q $remote_username@$remote_loginnode \"cd ${remote_jobcatalog}; ${fast_io_catalog}/${sources_for_run_catalog}/batch_scp $PORTOPT -d -w 10 -u $local_username $local_ip $job_protocol_file_remote \\\"$local_jobcatalog\\\" ${configuration_identifier}_${run_identifier}\" " >> $jobfile
	2981	else
	2982	echo "${fast_io_catalog}/${sources_for_run_catalog}/batch_scp $PORTOPT -d -w 10 -u $local_username $local_ip $job_protocol_file_remote \"$local_jobcatalog\" ${configuration_identifier}_${run_identifier}" >> $jobfile
	2983	fi
[2380]	2984	echo "%%END%%" >> $jobfile
	2985	echo "echo \" *** submitting job for transfering the job protocol file to $local_ip\" " >> $jobfile
[3534]	2986	echo "$submit_command $scpjob_file" >> $jobfile
[2380]	2987	echo "rm $scpjob_file" >> $jobfile
[2404]	2988	echo "rm -rf $job_transfer_protocol_file" >> $jobfile
[2380]	2989	echo "set -x" >> $jobfile
	2990	echo " ' exit" >> $jobfile
	2991	fi
	2992
	2993
	2994	# ACTIVATE ERROR-TRACEBACK
	2995	if [[ $do_trace = true ]]
	2996	then
	2997	echo "set -x" >> $jobfile
	2998	else
	2999	echo "set +vx" >> $jobfile
	3000	fi
	3001
	3002
	3003	# INITIALIZE THE ENVIRONMENT AND LOAD MODULES
	3004	if [[ "$login_init_cmd" != "" ]]
	3005	then
	3006	echo "$login_init_cmd" >> $jobfile
	3007	fi
	3008	if [[ "$module_commands" != "" ]]
	3009	then
	3010	echo "$module_commands" >> $jobfile
	3011	fi
	3012
	3013
	3014	# CREATE TEMPORARY DIRECTORY AND SWITCH TO IT
[2566]	3015	if [[ $create_remote_batch_job = true ]]
	3016	then
	3017	echo "mkdir $tempdir" >> $jobfile
[4303]	3018	echo "if [[ \$? != 0 ]]" >> $jobfile
	3019	echo "then" >> $jobfile
	3020	echo " echo \"+++ temporary working directory cannot be created.\" " >> $jobfile
	3021	echo " echo \" Check setting of variable fast_io_catalog in your config file.\" " >> $jobfile
	3022	echo " exit" >> $jobfile
	3023	echo "fi" >> $jobfile
[2566]	3024	echo "chmod go+rx $tempdir" >> $jobfile
	3025	else
	3026	# DIRECTORY FOR LOCAL BATCH JOBS IS CREATED NOW, DUE TO A
	3027	# REQUIREMENT OF THE GRID ENGINE BATCH SYSTEM (WORKING DIR IS GIVEN IN
	3028	# BATCH DIRECTIVE -wd AND MUST ALREADY EXIST WHEN THE JOB IS SUBMITTED)
	3029	mkdir $tempdir
[4303]	3030	if [[ $? != 0 ]]
	3031	then
	3032	printf "\n +++ temporary working directory cannot be created."
	3033	printf "\n Check setting of variable fast_io_catalog in your config file."
	3034	locat=execution
	3035	exit
	3036	fi
[2566]	3037	chmod go+rx $tempdir
	3038	fi
	3039	echo "cd $tempdir" >> $jobfile
	3040	echo "export tempdir=$tempdir" >> $jobfile
[3917]	3041	echo "cp ${fast_io_catalog}/${sources_for_run_catalog}/.[!.]* ." >> $jobfile
	3042	echo "export PATH=.:${fast_io_catalog}/${sources_for_run_catalog}:\$PATH" >> $jobfile
[2380]	3043	echo "export execute_palmrun=true" >> $jobfile
	3044
	3045
	3046	# PROVIDE NAME OF THE CURRENT WORKING-DIRECTORY ON THE LOCAL MACHINE (FROM WHERE THE JOB IS
	3047	# STARTED) BY SETTING AN ENVIRONMENT-VARIABLE. THIS INFORMATION IS USED IN THE JOB BY PALMRUN
	3048	# IN CASE THAT RESTART-RUNS HAVE TO BE GENERATED
	3049	echo "LOCAL_PWD=$working_directory" >> $jobfile
	3050	echo "export LOCAL_PWD" >> $jobfile
	3051
	3052
	3053	# PROVIDE THE PATH OF THE LOCAL PALMRUN-SCRIPT FOR THE SAME REASON
[2420]	3054	echo "LOCAL_PALMRUN_PATH=${source_path}/../SCRIPTS" >> $jobfile
[2380]	3055	echo "export LOCAL_PALMRUN_PATH" >> $jobfile
	3056
	3057
	3058	# CALL PALMRUN WITHIN THE JOB
	3059	# AS FINAL ACTION, REMOVE THE TEMPORARY DIRECTORY CREATED AT THE BEGINNING OF THE JOB
	3060	echo "set -x" >> $jobfile
	3061	echo "[[ \$execute_palmrun = true ]] && $palmrun_com" >> $jobfile
	3062
	3063
	3064	# TRANSFER JOBFILE TO THE TARGET HOST
	3065	if [[ $create_jobfile_only = false ]]
	3066	then
	3067
[3505]	3068	# CREATE LOCAL JOB FOLDER
	3069	eval local_jobcatalog=$local_jobcatalog
	3070	if [[ ! -d $local_jobcatalog ]]
	3071	then
	3072	echo " "
	3073	echo " *** local jobcatalog \"$local_jobcatalog\" does not exist and will be created now"
	3074	mkdir $local_jobcatalog
	3075	fi
	3076
[2380]	3077	if [[ $create_remote_batch_job = true ]]
	3078	then
	3079
[2420]	3080	echo " "
[2566]	3081	echo " *** transfer of job to remote host via scp"
[3505]	3082
	3083	# CREATE REMOTE JOB FOLDER, IF IT DOES NOT EXIST
[3534]	3084	ssh -q $ssh_key $SSH_PORTOPT ${remote_username}@${remote_ip} "[[ ! -d $remote_jobcatalog ]] && mkdir $remote_jobcatalog" 2>&1
[3505]	3085
	3086	# COPY THE JOB FILE
[2420]	3087	if [[ $do_trace = true ]]
	3088	then
[3236]	3089	echo " scp $ssh_key $PORTOPT $jobfile ${remote_username}@${remote_ip}:${remote_jobcatalog}/${configuration_identifier}_${run_id}"
[2420]	3090	fi
[3236]	3091	scp $ssh_key $PORTOPT $jobfile ${remote_username}@${remote_ip}:${remote_jobcatalog}/${configuration_identifier}_${run_id} > /dev/null
[2380]	3092
[3505]	3093	# SUBMIT THE JOB
[3574]	3094	printf " *** submit the job (output of submit command, \n"
[3573]	3095	printf " e.g. the job-id given by the batch-system, may follow) \n"
[2420]	3096	if [[ $do_trace = true ]]
	3097	then
[3665]	3098	echo " ssh -q $ssh_key $SSH_PORTOPT ${remote_username}@${remote_ip} \"cd $remote_jobcatalog; $submit_command ${configuration_identifier}_${run_id}; rm ${configuration_identifier}_${run_id}\" "
[2420]	3099	fi
[3665]	3100	ssh -q $ssh_key $SSH_PORTOPT ${remote_username}@${remote_ip} "cd $remote_jobcatalog; $submit_command ${configuration_identifier}_${run_id}; rm ${configuration_identifier}_${run_id}"
[2380]	3101
	3102	elif [[ $create_batch_job = true ]]
	3103	then
	3104
[2385]	3105	eval local_jobcatalog=$local_jobcatalog
[3236]	3106	cp $jobfile ${local_jobcatalog}/${configuration_identifier}_${run_id}
[2385]	3107	cd $local_jobcatalog
[2380]	3108	echo " "
[2420]	3109	echo " *** submit the job"
	3110	if [[ $do_trace = true ]]
	3111	then
[3236]	3112	echo "$submit_command ${configuration_identifier}_${run_id}"
[2420]	3113	fi
[3236]	3114	$submit_command ${configuration_identifier}_${run_id}
	3115	rm ${configuration_identifier}_${run_id}
[2380]	3116	cd - > /dev/null
	3117
	3118	fi
	3119
	3120	rm -rf $jobfile
	3121
	3122	else
	3123
	3124	printf "\n *** jobfile created under name \"$jobfile\" "
	3125	printf "\n no batch-job has been sent!"
	3126
	3127	fi
	3128
	3129	fi # END OF REMOTE-PART

Note: See TracBrowser for help on using the repository browser.

Download in other formats:

| Impressum | ©Leibniz Universität Hannover |