Home

Context Navigation

source: palm/trunk/SCRIPTS/palmrun @ 4796

Last change on this file since 4796 was 4536, checked in by raasch, 5 years ago
messages and debug output converted to PALM routines (restart_data_mpi_io_mod), binary version number set to 5.0, heeader output for restart data format added, restart data filesize and I/O transfer speed added in cpu_measures, handling of single restart files (created with MPI-I/O) added to palmrun, bugfix: preprocessor directive adjusted (virtual_measurement_mod), location message format changed
Property svn:executable set to ``* Property svn:keywords set to `Id Rev`
File size: 105.7 KB

Line
1	#!/bin/bash
2
3	# palmrun - script for running PALM jobs
4
5	#--------------------------------------------------------------------------------#
6	# This file is part of the PALM model system.
7	#
8	# PALM is free software: you can redistribute it and/or modify it under the terms
9	# of the GNU General Public License as published by the Free Software Foundation,
10	# either version 3 of the License, or (at your option) any later version.
11	#
12	# PALM is distributed in the hope that it will be useful, but WITHOUT ANY
13	# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
14	# A PARTICULAR PURPOSE. See the GNU General Public License for more details.
15	#
16	# You should have received a copy of the GNU General Public License along with
17	# PALM. If not, see <http://www.gnu.org/licenses/>.
18	#
19	# Copyright 2017-2020 Leibniz Universitaet Hannover
20	#--------------------------------------------------------------------------------#
21	#
22	# Current revisions:
23	# ------------------
24	#
25	#
26	# Former revisions:
27	# -----------------
28	# $Id: palmrun 4536 2020-05-17 17:24:13Z gronemeier $
29	# handling of single restart files (created with MPI-I/O) added
30	#
31	# 4481 2020-03-31 18:55:54Z maronga
32	# add cycle number to initial files too (000),
33	# remove compatibility for cycle numbers which are less than three digits wide
34	#
35	# 4425 2020-02-26 15:35:00Z raasch
36	# hard coded path to the fileconnection file replaced by a path that is based on
37	# the source path which has been given in the configuration file
38	#
39	# 4303 2019-11-25 08:55:18Z raasch
40	# Abort if temporary working catalog cannot be created
41	#
42	# 3982 2019-05-16 11:07:31Z raasch
43	# bugfix: -b is not added in automatically generated batch jobs for remote hosts
44	#
45	# 3917 2019-04-18 11:46:37Z raasch
46	# in case of batch jobs on remote machines, copying files from SOURCES_FOR_RUN folder has
47	# been rearranged to avoid that input files with original names (e.g. ..._p3d) are copied
48	# into the temporary working catalog
49	#
50	# 3860 2019-04-04 06:23:27Z raasch
51	# check added, that it is not allowed to set option -b if the configuration file has been setup
52	# for running batch jobs on remote hosts,
53	# local time added to terminal output before and after saving output files
54	#
55	# 3725 2019-02-07 10:11:02Z raasch
56	# error messages for failed restarts extended
57	#
58	# 3682 2019-01-18 17:01:54Z knoop
59	# ssh-call for submitting batch jobs on remote systems modified again to avoid output
60	# of login messages on specific systems
61	#
62	# 3574 2018-11-28 13:14:39Z raasch
63	# syntax-error in last commit fixed
64	#
65	# 3573 2018-11-28 13:03:44Z raasch
66	# bugfixes: don't check the .palm.iofiles in restart runs + wrong cd-command
67	# removed,
68	# r3570 reverted
69	#
70	# 3570 2018-11-27 17:44:21Z kanani
71	# Rename job-id by run_id
72	#
73	# 3555 2018-11-22 13:50:37Z raasch
74	# sed-command to remove repeated space-characters replaced by tr-command, to
75	# guarantee mac compatible
76	#
77	# 3549 2018-11-21 15:44:44Z raasch
78	# ssh-call for submitting batch jobs on remote systems modified to avoid output
79	# of login messages on specific systems
80	#
81	# 3534 2018-11-19 15:35:16Z raasch
82	# batch_scp for sending back the job protocol file is called via login-node if
83	# a login-node has been set in the config-file
84	# ssh-calls rearranged to avoid output of system/user-profile messages
85	#
86	# 3505 2018-11-08 09:14:30Z raasch
87	# job folders are created in case that they do not exist
88	#
89	# 3455 2018-10-30 14:12:31Z raasch
90	# options -h and -d renamed -c and -r, respectively
91	#
92	# 3402 2018-10-23 11:49:24Z knoop
93	# job_id is set for compatibility reasons with older versions of config files
94	#
95	# 3313 2018-10-06 15:22:48Z knoop
96	# more detailed error message for outdated fileconnection file
97	#
98	# 3236 2018-09-10 09:07:06Z raasch
99	# warning issued in case of file connection file version mismatch,
100	# host_configuration renamed configuration_identifier,
101	# "host identifier" in header output renamed "config. identifier",
102	# jobname renamed run_identifier,
103	# run_id renamed run_id_number
104	# job_id renamed run_id
105	# option -r renamed -i
106	#
107	# 3151 2018-07-19 08:45:38Z raasch
108	# removed additional listing of files to be compiled, print PALM code revision
109	# in header
110	#
111	# 3132 2018-07-16 11:44:58Z raasch
112	# create consistent error messages in case of failed restarts,
113	# check existence of remote job catalog
114	#
115	# 3043 2018-05-25 10:47:54Z raasch
116	# check for setting of fast_io_catalog
117	#
118	# 3033 2018-05-23 15:26:19Z raasch
119	# -T option not required for interactive runs (tasks per node =
120	# total number of cores is assumed as default, if option is not given),
121	# reformatting of a few messages,
122	# bugfix: destination output directory is created if attribute "pe" is set in
123	# .palm.iofiles
124	#
125	# 2718 2018-01-02 08:49:38Z maronga
126	# bugfix of r2990
127	#
128	# 2718 2018-01-02 08:49:38Z maronga
129	# bugfix: temporary SOURCES_FOR_RUN_CATALOG now created in base_directory and
130	# not in the working directory from where palmrun is called
131	#
132	# 2718 2018-01-02 08:49:38Z maronga
133	# svfout in the activation string will be replaced now by svfin in case of
134	# automatic restarts
135	#
136	# 2718 2018-01-02 08:49:38Z maronga
137	# Introduction of new ENVIRONMENT variables read_svf and write_svf to consider
138	# the possible output or input of sky view factors
139	#
140	# 2718 2018-01-02 08:49:38Z maronga
141	# "fname" renamed to "jobname"
142	#
143	# 2718 2018-01-02 08:49:38Z maronga
144	# parameters are allowed in output commands (OC:)
145	#
146	# 2718 2018-01-02 08:49:38Z maronga
147	# bugfix: exit if no queue has been given neither with option -q nor in the
148	# configuration file
149	#
150	# 2718 2018-01-02 08:49:38Z maronga
151	# file connection file from trunk/SCRIPTS is taken by default
152	#
153	# 2718 2018-01-02 08:49:38Z maronga
154	# Corrected "Former revisions" section
155	#
156	# 2696 2017-12-14 17:12:51Z kanani
157	# Change in file header (GPL part)
158	#
159	# 2693 2017-12-13 12:33:49Z raasch
160	# replacement of variables given by {{<variable>}} in configuration file is now
161	# done in a more general way, variable names are written in small letters
162	#
163	# 2670 2017-12-06 16:32:19Z raasch
164	# test printouts removed
165	#
166	# 2669 2017-12-06 16:03:27Z raasch
167	# file attributes in .palm.iofiles restructured, "loc" attribute completely
168	# removed,
169	# wildcard (*) allowed in .palm.iofiles as file activation string for output
170	# files,
171	# informative messages in case of missing optional input files shortened
172	# bugfix: variable cycle explicitly interpreted with 10 as the number base
173	#
174	# 2638 2017-11-23 12:44:23Z raasch
175	# use of wildcards in file connection statements enabled
176	#
177	# 2605 2017-11-09 15:31:46Z raasch
178	# in case of remote jobs, input files with "job" or "jobopt" (new) attribute
179	# will not be sent with the job file any more, but copied into the
180	# SOURCES_FOR_RUN... folder on the remote host, before the job is submitted
181	#
182	# 2600 2017-11-01 14:11:20Z raasch
183	# cycle numbers are made three digits wide
184	#
185	# 2566 2017-10-20 08:50:47Z raasch
186	# execute command for combine_plot_fields added
187	# "TEMPDIR" renamed "tempdir"
188	# temporary working directory for local batch jobs is created immediately within
189	# the user's palmrun call, due to a requirement of the "grid engine" batch
190	# system, where the working directory is given with batch directive -wd and must
191	# already exist when the job is submitted,
192	# informative messages in non-trace mode reduced and partly reformatted
193	#
194	# 2551 2017-10-18 07:25:11Z raasch
195	# TEMPDIR added as replacement string to be used in batch directives
196	#
197	# 2512 2017-10-04 08:26:59Z raasch
198	# bugfix for determining cycle numbers of NetCDF input files
199	#
200	# 2506 2017-09-29 08:30:37Z raasch
201	# option -V added to check for an existing SOURCES_FOR_RUN_... folder
202	# host configuration added to SOURCES_FOR_RUN_... folder name
203	# host_identifier renamed host_configuration
204	# option -W added to allow for job dependencies
205	#
206	# 2501 2017-09-26 11:41:55Z raasch
207	# default value for number of cores (option -X) set to 1
208	# bugfix for mechanism which overwrites configuration file settings with values
209	# provided by palmrun options
210	#
211	# 2499 2017-09-22 16:47:58Z kanani
212	# option -h named configuration identifier
213	#
214	# 2480 2017-09-19 06:24:14Z maronga
215	# bugfix for last revision
216	#
217	# 2479 2017-09-19 06:12:16Z raasch
218	# option -A (project account number) added
219	#
220	# 2422 2017-09-08 08:25:41Z raasch
221	# initial revision
222	#
223	#--------------------------------------------------------------------------------#
224	# palmrun - script for running PALM jobs on local and remote hosts
225	#--------------------------------------------------------------------------------#
226
227
228
229	# DECLARATION OF VARIABLES AND THEIR DEFAULT VALUES
230
231	set +o allexport # SUPPRESS EXPORT OF ALL VARIABLES, SINCE IN THE PAST THIS
232	# LEAD TO PROBLEMS IN ROUTINES CALLED BY PALMRUN
233	# (TOO MANY ARGUMENTS - PROBLEM)
234	set +o noclobber # EXISTING FILES ARE ALLOWED TO BE OVERWRITTEN
235
236	typeset -i ibd=0 ibdt=0 iec=0 iic=0 iin=0 ioc=0 iout=0 nr_of_input_files=0
237	typeset -i nr_of_output_files=0
238	typeset -x -i memory=0 # HAS TO BE EXPORTED HERE, OTHERWISE AN UNKNOWN SIDE
239	# SIDE EFFECT MAY CAUSE DATA LOSS WHEN GETOPTS IS READING THE
240	# SCRIPT-OPTION ARGUMENTS
241	typeset -i cores cputime cpu_hours cpu_minutes cpu_seconds i ii iia iii iio
242	typeset -i icycle inode ival maxcycle mpi_tasks
243	typeset -i nodes remaining_cores run_number tasks_per_node threads_per_task
244
245	activation_string_list=""
246	AddFilenames=""
247	combine_plot_fields=true
248	compiler_name=""
249	compiler_name_ser=""
250	compiler_options=""
251	configuration_identifier="default"
252	cores=1
253	cores_atmos=0
254	cores_ocean=0
255	coupled_dist=""
256	cpp_options=""
257	cpumax=0
258	create_batch_job=false
259	create_jobfile_only=false
260	create_remote_batch_job=false
261	dashes=" ----------------------------------------------------------------------------"
262	defaultqueue=""
263	delete_temporary_catalog=true
264	do_compile=true
265	do_trace=false
266	executable=""
267	execute_command=""
268	execution_error=false
269	fileconnection_file=""
270	global_revision=""
271	hostfile=""
272	hp=""
273	keep_data_from_previous_run=false
274	link_local_input=false
275	link_local_output=false
276	linker_options=""
277	local_jobcatalog=""
278	locat=normal
279	makefile=""
280	max_par_io_str=""
281	prc=$0
282	while [[ $(echo $prc \| grep -c "/") != 0 ]]
283	do
284	prc=`echo $prc \| cut -f2- -d"/"`
285	done
286	module_calls=""
287	palmrun_memory=""
288	palmrun_script_name=$prc
289	openmp=false
290	previous_job=""
291	project_account=""
292	queue=none
293	read_svf=""
294	restart_run=false
295	return_address=""
296	remote_jobcatalog=""
297	remote_username=""
298	running_in_batch_mode=false
299	running_in_test_mode=false
300	run_coupled_model=false
301	run_id=""
302	run_id_number=""
303	run_identifier=test
304	silent=false
305	source_list=""
306	source_path=""
307	tasks_per_node=0
308	threads_per_task=1
309	transfer_problems=false
310	user_source_path=""
311	use_existing_sources_folder=""
312	use_openmp=false
313	version="palmrun 1.0 Rev$Rev: 4536 $"
314	working_directory=`pwd`
315	write_binary=""
316	write_svf=""
317
318
319	# ERROR HANDLING IN CASE OF EXIT
320	trap 'if [[ $locat != normal && $locat != control_c ]]
321	then
322
323	# CARRY OUT ERROR-COMMANDS GIVEN IN THE CONFIGURATION FILE (EC:)
324	(( i = 0 ))
325	while (( i < iec ))
326	do
327	(( i = i + 1 ))
328	printf "\n *** Execution of ERROR-command:\n"
329	printf " >>> ${err_command[$i]}\n"
330	eval ${err_command[$i]}
331	done
332	[[ $delete_temporary_catalog = true ]] && (cd; rm -rf $tempdir)
333	printf "\n\n+++ palmrun crashed \n\n"
334	exit 1
335	elif [[ $locat != normal ]]
336	then
337	[[ $delete_temporary_catalog = true ]] && (cd; rm -rf $tempdir)
338	printf "\n+++ palmrun killed by \"^C\" \n\n"
339	exit 2
340	else
341	printf "\n --> palmrun finished\n\n"
342	exit 0
343	fi' exit
344
345
346	# ACTIONS IN CASE OF TERMINAL-BREAK (CONTROL-C):
347	trap 'locat=control_c
348	exit 1
349	' 2
350
351
352	# READ SHELLSCRIPT-OPTIONS AND REBUILD THE PALMRUN-COMMAND STRING (prc),
353	# WHICH WILL BE USED TO START RESTART-JOBS
354	while getopts :a:A:bBc:CFG:i:jkm:M:O:q:r:R:s:t:T:u:U:vVw:W:xX:yY:zZ option
355	do
356	case $option in
357	(a) activation_string_list=$OPTARG; prc="$prc -a'$OPTARG'";;
358	(A) project_account=$OPTARG; prc="$prc -A'$OPTARG'";;
359	(b) create_batch_job=true; prc="$prc -b";;
360	(B) delete_temporary_catalog=false; prc="$prc -B";;
361	(c) configuration_identifier=$OPTARG; prc="$prc -c$OPTARG";;
362	(C) restart_run=true; prc="$prc -C";;
363	(F) create_jobfile_only=true;;
364	(G) global_revision=$OPTARG; prc="$prc -G'$OPTARG'";;
365	(i) run_id_number=$OPTARG;;
366	(j) running_in_batch_mode=true;;
367	(k) keep_data_from_previous_run=true; prc="$prc -k";;
368	(m) palmrun_memory=$OPTARG; prc="$prc -m$OPTARG";;
369	(M) makefile=$OPTARG; prc="$prc -M$OPTARG";;
370	(O) use_openmp=true; threads_per_task=$OPTARG; prc="$prc -O$OPTARG";;
371	(q) queue=$OPTARG; prc="$prc -q$OPTARG";;
372	(r) run_identifier=$OPTARG; prc="$prc -r$OPTARG";;
373	(R) return_address=$OPTARG;;
374	(s) source_list=$OPTARG;;
375	(t) palmrun_cpumax=$OPTARG; prc="$prc -t$OPTARG";;
376	(T) palmrun_tasks_per_node=$OPTARG; prc="$prc -T$OPTARG";;
377	(u) remote_username=$OPTARG; prc="$prc -u$OPTARG";;
378	(U) return_username=$OPTARG; prc="$prc -U$OPTARG";;
379	(v) silent=true; prc="$prc -v";;
380	(V) use_existing_sources_folder="-V";;
381	(w) max_par_io_str=$OPTARG; prc="$prc -w$OPTARG";;
382	(W) previous_job=$OPTARG;;
383	(x) do_trace=true;set -x; prc="$prc -x";;
384	(X) palmrun_cores=$OPTARG; prc="$prc -X$OPTARG";;
385	(y) ocean_file_appendix=true; prc="$prc -y";;
386	(Y) run_coupled_model=true; coupled_dist=$OPTARG; prc="$prc -Y'$OPTARG'";;
387	(z) running_in_test_mode=true;;
388	(Z) combine_plot_fields=false; prc="$prc -Z";;
389	(\?) printf "\n +++ unknown option $OPTARG \n"
390	printf "\n --> type \"$0 ?\" for available options \n"
391	locat=parameter;exit;;
392	esac
393	done
394
395
396	# SKIP GIVEN OPTIONS TO READ POSITIONAL PARAMETER, IF GIVEN
397	# CURRENTLY ONLY PARAMETER ? (TO OUTPUT A SHORT COMMAND INFO) IS ALLOWED
398	(( to_shift = $OPTIND - 1 ))
399	shift $to_shift
400
401	# PRINT SHORT DESCRIPTION OF PALMRUN OPTIONS
402	if [[ "$1" = "?" ]]
403	then
404	(printf "\n *** Description of available palmrun options:\n"
405	printf "\n Option Description Default-Value"
406	printf "\n -a activation string list \"\" "
407	printf "\n -A project account number ---"
408	printf "\n -b batch-job on local machine ---"
409	printf "\n -B do not delete temporary directory at end ---"
410	printf "\n -c configuration identifier \"default\" "
411	printf "\n -F create batch job file only ---"
412	printf "\n -k keep data from previous run"
413	printf "\n -m memory demand in MB (batch-jobs) 0 MB"
414	printf "\n -M Makefile name Makefile"
415	printf "\n -O threads per openMP task ---"
416	printf "\n -q queue \"$queue\" "
417	printf "\n -r run identifier test"
418	printf "\n -s filenames of routines to be compiled \"\" "
419	printf "\n must end with .f, .f90, .F, or .c !"
420	printf "\n use \"..\" for more than one file and wildcards"
421	printf "\n -s LM compiles all locally modified files"
422	printf "\n -t allowed cpu-time in seconds (batch) 0"
423	printf "\n -T tasks per node ---"
424	printf "\n -u username on remote machine \"\" "
425	printf "\n -v no prompt for confirmation ---"
426	printf "\n -V check if SOURCES_FOR_RUN_... exists ---"
427	printf "\n -w maximum parallel io streams as given by -X"
428	printf "\n -W name of job to wait for ---"
429	printf "\n -x tracing of palmrun for debug purposes ---"
430	printf "\n -X # of processors (on parallel machines) 1"
431	printf "\n -y add appendix \"_O\" to all local output"
432	printf "\n files (ocean precursor runs followed by"
433	printf "\n coupled atmosphere-ocean runs) ---"
434	printf "\n -Y run coupled model, \"#1 #2\" with"
435	printf "\n #1 atmosphere and #2 ocean processors \"#/2 #/2\" depending on -X"
436	printf "\n -Z skip combine_plot_fields at the end of "
437	printf "\n the simulation ---"
438	printf "\n "
439	printf "\n Possible values of positional parameter <modus>:"
440	printf "\n \"?\" - this outline \n\n") \| more
441	exit
442	elif [[ "$1" != "" ]]
443	then
444	printf "\n +++ positional parameter $1 unknown \n"
445	locat=parameter; exit
446	fi
447
448
449
450	# SHORT STARTING MESSAGE
451	printf "\n*** $version "
452	printf "\n will be executed. Please wait ..."
453
454
455	# BUILD THE CONFIGURATION-FILE NAME AND THE SOURCES_FOR_RUN-FOLDER NAME
456	config_file=.palm.config.$configuration_identifier
457	sources_for_run_catalog=SOURCES_FOR_RUN_${configuration_identifier}_$run_identifier
458
459
460	# CHECK, IF CONFIGURATION FILE EXISTS
461	if [[ ! -f $config_file ]]
462	then
463	printf "\n\n +++ configuration file: "
464	printf "\n $config_file"
465	printf "\n does not exist"
466	locat=connect; exit
467	fi
468
469
470	# CHECK, IF THE ACTIVATION_STRING_LIST HAS BEEN GIVEN
471	if [[ "$activation_string_list" = "" ]]
472	then
473	printf "\n\n +++ no activation string list given: "
474	printf "\n please set palmrun option \"-a\" "
475	locat=palmrun_option; exit
476	fi
477
478
479	# SET VARIABLE TO ACTIVATE PALM BINARY OUTPUT FOR RESTARTS
480	if [[ $(echo $activation_string_list \| grep -c "restart") != 0 ]]
481	then
482	write_binary=true
483	else
484	write_binary=false
485	fi
486
487	# SET VARIABLE TO ACTIVATE WRITING OF SVF DATA
488	if [[ $(echo $activation_string_list \| grep -c "svfout") != 0 ]]
489	then
490	write_svf=true
491	else
492	write_svf=false
493	fi
494
495	# SET VARIABLE TO ACTIVATE READING OF SVF DATA
496	if [[ $(echo $activation_string_list \| grep -c "svfin") != 0 ]]
497	then
498	read_svf=true
499	else
500	read_svf=false
501	fi
502
503
504	# READ AND EVALUATE THE CONFIGURATION-FILE
505	[[ $silent = false ]] && printf "\n\n Reading the configuration file... "
506
507	# READ VARIABLE SETTINGS FROM CONFIG FILE LINE BY LINE
508	while read line
509	do
510
511	# FIRST REPLACE ENVIRONMENT-VARIABLES BY THEIR RESPECTIVE VALUES
512	eval line=\"$line\"
513
514
515	# INTERPRET THE LINE
516	if [[ "$(echo $line)" = "" ]]
517	then
518
519	# EMPTY LINE, NO ACTION
520	continue
521
522	elif [[ "$(echo $line \| cut -c1)" = "#" ]]
523	then
524
525	# LINE IS A COMMENT LINE
526	continue
527
528	elif [[ "$(echo $line \| cut -c1)" = "%" ]]
529	then
530
531	# LINE DEFINES AN ENVIRONMENT-VARIABLE
532	var=`echo $line \| cut -d" " -s -f1 \| cut -c2-`
533	value=`echo $line \| cut -d" " -s -f2-`
534
535
536	# VALUE FROM THE CONFIGURATION-FILE IS ASSIGNED TO THE
537	# ENVIRONMENT-VARIABLE, BUT ONLY IF NO VALUE HAS BEEN ALREADY
538	# ASSIGNED WITHIN THIS SCRIPT (E.G. BY SCRIPT-OPTIONS).
539	# NON-ASSIGNED VARIABLES HAVE VALUE "" OR 0 (IN CASE OF INTEGER).
540	# HENCE THE GENERAL RULE IS: SCRIPT-OPTION OVERWRITES THE
541	# CONFIGURATION-FILE.
542	if [[ "$(eval echo \$$var)" = "" \|\| "$(eval echo \$$var)" = "0" ]]
543	then
544	eval export $var="\$value"
545
546	# TERMINAL OUTPUT OF ENVIRONMENT-VARIABLES, IF TRACEBACK IS SWITCHED on
547	if [[ $do_trace = true ]]
548	then
549	printf "\n*** ENVIRONMENT-VARIABLE $var = $value"
550	fi
551	fi
552
553	elif [[ "$(echo $line \| cut -c1-3)" = "BD:" ]]
554	then
555
556	# LINE DEFINES BATCH-DIRECTIVE
557	(( ibd = ibd + 1 ))
558	line=$(echo $line \| cut -c4-)
559	batch_directive[$ibd]="$line"
560
561	elif [[ "$(echo $line \| cut -c1-4)" = "BDT:" ]]
562	then
563
564	# LINE DEFINES BATCH-DIRECTIVE FOR SENDING BACK THE JOBFILE FROM A
565	# REMOTE TO A LOCAL HOST
566	(( ibdt = ibdt + 1 ))
567	line=$(echo $line \| cut -c5-)
568	batch_directive_transfer[$ibdt]="$line"
569
570	elif [[ "$(echo $line \| cut -c1-3)" = "EC:" ]]
571	then
572
573	# LINE DEFINES ERROR-COMMAND
574	(( iec = iec + 1 ))
575	line=$(echo $line \| cut -c4-)
576	err_command[$iec]="$line"
577
578	elif [[ "$(echo $line \| cut -c1-3)" = "IC:" ]]
579	then
580
581	# LINE DEFINES INPUT-COMMAND
582	(( iic = iic + 1 ))
583	line=$(echo $line \| cut -c4-)
584	in_command[$iic]="$line"
585
586	elif [[ "$(echo $line \| cut -c1-3)" = "OC:" ]]
587	then
588
589	# LINE DEFINES OUTPUT-COMMAND
590	(( ioc = ioc + 1 ))
591	line=$(echo $line \| cut -c4-)
592	out_command[$ioc]="$line"
593
594	else
595
596	# SKIP ALL OTHER LINES
597	continue
598
599	fi
600
601	done < $config_file
602
603
604	# CHECK SETTING OF REQUIRED PARAMETERS
605	if [[ "$fast_io_catalog" = "" ]]
606	then
607	printf "\n\n +++ no fast_io_catalog found in $config_file"
608	printf "\n Please add line \"fast_io_catalog ...\" to that file."
609	locat=config_file; exit
610	fi
611	if [[ "$compiler_name" = "" ]]
612	then
613	printf "\n\n +++ no compiler name found in $config_file"
614	printf "\n Please add line \"compiler_name ...\" to that file."
615	locat=config_file; exit
616	fi
617	if [[ "$compiler_name_ser" = "" ]]
618	then
619	printf "\n\n +++ no compiler name for non-paralle compilation found in $config_file"
620	printf "\n Please add line \"compiler_name_ser ...\" to that file."
621	locat=config_file; exit
622	fi
623	if [[ "$compiler_options" = "" ]]
624	then
625	printf "\n\n +++ no compiler options found in $config_file"
626	printf "\n Please add line \"compiler_options ...\" to that file."
627	locat=config_file; exit
628	fi
629	if [[ "$execute_command" = "" ]]
630	then
631	printf "\n\n +++ no execute command found in $config_file"
632	printf "\n Please add line \"execute_command ...\" to that file."
633	locat=config_file; exit
634	fi
635
636	if [[ "$hostfile" != "" ]]
637	then
638	if [[ $hostfile != auto && ! -f $hostfile ]]
639	then
640	printf "\n\n +++ no hostfile \"$hostfile\" found"
641	printf "\n Please check line \"hostfile ...\" in $config_file"
642	locat=config_file; exit
643	fi
644	fi
645
646
647	# DETERMINE THE CALL STATUS
648	if [[ "$return_address" != "" ]]
649	then
650
651	# I AM RUNNING ON A REMOTE HOST, WHICH ALSO MEANS THAT I AM RUNNING IN
652	# BATCH MODE AND ...
653	running_on_remote=true
654
655	else
656
657	# I HAVE BEEN CALLED INTERACTIVELY ON THIS HOST
658	if [[ "$remote_ip" != "" ]]
659	then
660
661	# I HAVE TO CREATE A BATCH JOB TO RUN PALM ON THE REMOTE HOST
662	create_remote_batch_job=true
663
664	# CHECK THAT THE REMOTE JOBCATALOG HAS BEEN SET
665	if [[ "$remote_jobcatalog" = "" ]]
666	then
667	printf "\n\n +++ no remote jobcatalog found in $config_file"
668	printf "\n Please add line \"remote_jobcatalog ...\" to that file."
669	locat=config_file; exit
670	fi
671
672	# CHECK THAT OPTION -b (SUBMITTING BATCH JOB ON LOCAL HOST) HASN'T BEEN SET
673	if [[ $create_batch_job = true ]]
674	then
675	printf "\n\n +++ option -b must not be set because configuration file has been setup"
676	printf "\n for running jobs on a remote host"
677	locat=config_file; exit
678	fi
679
680	fi
681	running_on_remote=false
682	fi
683
684
685	# SET THE DEFAULT FILE CONNECTION FILE (WHICH IS IN THE REPOSITORY)
686	fileconnection_file=${source_path}/../SCRIPTS/.palm.iofiles
687
688	# CHECK, IF USER PROVIDES OWN FILE CONNECTION FILE
689	if [[ -f .palm.iofiles ]]
690	then
691
692	# CHECK VERSION MISMATCH
693	# RESTART RUNS DON'T REQUIRE A CHECK, BECAUSE THEY USE A COPY OF THAT
694	# FILE WHICH HAS ALREADY BEEN CHECKED WITHIN THE INITAL PALMRUN CALL
695	if [[ $running_in_batch_mode != true && $restart_run != true ]]
696	then
697	if [[ $(head -n1 $fileconnection_file) != $(head -n1 .palm.iofiles) ]]
698	then
699	printf "\n\n +++ WARNING: A file connection file has been found in your"
700	printf "\n working directory, but its revision does not match"
701	printf "\n the revision of the default (trunk) version."
702	printf "\n You may need to update your connection file"
703	printf "\n \"${working_directory}/.palm.iofiles\" !"
704	fi
705	fi
706
707	fileconnection_file=.palm.iofiles
708
709	fi
710
711
712	# CHECK, IF FILE CONNECTION FILE EXISTS
713	if [[ ! -f $fileconnection_file ]]
714	then
715	printf "\n\n +++ file connection file: "
716	printf "\n $fileconnection_file"
717	printf "\n does not exist"
718	locat=connect; exit
719	fi
720
721
722	# READ AND EVALUATE THE I/O-FILE LIST
723	[[ $silent = false ]] && printf "\n Reading the I/O files... "
724
725	# READ THE FILE CONNECTION FILE LINE BY LINE
726	while read line
727	do
728
729	# REPLACE REPEATING SPACES BETWEEN THE COLUMNS BY A SINGLE SPACE
730	# HERE, TR IS USED INSTEAD OF SED TO GUARANTEE MAC COMPATIBILITY
731	line=`echo "$line" \| tr -s " "`
732
733	# INTERPRET THE LINE
734	if [[ "$(echo $line)" = "" ]]
735	then
736	# EMPTY LINE, NO ACTION
737	continue
738
739	elif [[ "$(echo $line \| cut -c1)" = "#" ]]
740	then
741
742	# LINE IS A COMMENT LINE
743	true
744
745	else
746
747	# LINE DEFINES FILE CONNECTION. READ THE FILE ATTRIBUTES.
748	# s2a: in/out - field
749	# s2b: action - field (optional)
750	s1=`echo "$line" \| cut -d" " -f1`
751	s2=`echo "$line" \| cut -d" " -s -f2`
752	if [[ $(echo $s2 \| grep -c ":") = 0 ]]
753	then
754	s2a=$s2
755	s2b=""
756	else
757	s2a=`echo $s2 \| cut -d":" -f1`
758	s2b=`echo $s2 \| cut -d":" -f2`
759	fi
760	s3=`echo "$line" \| cut -d" " -f3 \| sed 's/*/wildcard /g'`
761	s4=`echo "$line" \| cut -d" " -s -f4`
762	eval s4=\"$s4\" # REPLACE ENVIRONMENT-VARIABLES IN PATH BY THEIR RESPECTIVE VALUES
763	s5=`echo "$line" \| cut -d" " -s -f5`
764	s6=`echo "$line" \| cut -d" " -s -f6`
765
766
767	# STORE FILE CONNECTION, IF ACTIVATED BY ACTIVATION-STRING FROM
768	# INPUT- OR OUTPUT-LIST.
769	# VARIABLE S3 MAY CONTAIN A LIST OF ACTIVATION STRINGS (FIELD-SEPERATOR ":").
770	# IF EXECUTION IS SCHEDULED FOR A REMOTE-MACHINE AND THE FILE IS ONLY
771	# LOCALLY REQUIRED ON THAT MACHINE (I.E. s2b != tr), THE FILE CONNECTION
772	# IS NOT CHECKED AND STORED.
773	IFSALT="$IFS"; IFS="$IFS:" # ADD ":" AS FIELD SEPARATOR
774	if [[ ( "$s2a" = in \|\| "$s2a" = inopt ) && ! ( $create_remote_batch_job = true && "$s2b" != tr ) ]]
775	then
776	found=false
777	for actual in $activation_string_list
778	do
779	for formal in $s3
780	do
781	[[ $actual = $formal \|\| "$formal" = "-" ]] && found=true
782	done
783	done
784	if [[ $found = true ]]
785	then
786	(( iin = iin + 1 ))
787	localin_pre[$iin]=$s1; actionin_pre[$iin]=$s2b;
788	pathin_pre[$iin]=$s4; endin_pre[$iin]=$s5; extin_pre[$iin]=$s6
789	if [[ "$s2a" = inopt ]]
790	then
791	optin_pre[$iin]=yes
792	else
793	optin_pre[$iin]=no
794	fi
795
796	# FILES WITH JOB-ATTRIBUTE ARE STORED IN THE SOURCES_FOR_RUN
797	# FOLDER IF THE JOB IS RUNNING ON A REMOTE HOST
798	if [[ $running_on_remote = true && "$s2b" = tr ]]
799	then
800	pathin_pre[$iin]=${fast_io_catalog}/${sources_for_run_catalog}
801	fi
802
803	# CHECK FOR MULTIPLE FILES, SET A RESPECTIVE FLAG AND REMOVE
804	# THE WILDCARD FROM THE ENDING
805	if [[ "${s5: -1}" = "*" ]]
806	then
807	if [[ "$s2b" = "di" ]]
808	then
809	printf "\n +++ wildcards (*) not allowed with \"di\" file attribute."
810	printf "\n see file \"$fileconnection_file\", line"
811	printf "\n$line"
812	locat=iofiles_file; exit
813	fi
814	multin[$iin]=true
815	string=${endin_pre[$iin]}
816	endin_pre[$iin]="${string%?}"
817	else
818	multin[$iin]=false
819	fi
820	fi
821	elif [[ "$s2a" = out && ! ( $create_remote_batch_job = true ) ]]
822	then
823	found=false
824	for actual in $activation_string_list
825	do
826	for formal in $s3
827	do
828	[[ $actual = $formal \|\| $formal = wildcard ]] && found=true
829	done
830	done
831	if [[ $found = true ]]
832	then
833	(( iout = iout + 1 ))
834	localout_pre[$iout]=$s1; actionout_pre[$iout]=$s2b
835	pathout_pre[$iout]=$s4; endout_pre[$iout]=$s5; extout_pre[$iout]=$s6
836
837	# CHECK IF WILDCARD IS USED AS ACTIVATION STRING
838	# IN SUCH CASES, NO WARNING WILL LATER BE OUTPUT IF LOCAL FILES DO NOT EXIST
839	if [[ $formal = wildcard ]]
840	then
841	warnout_pre[$iout]=false
842	else
843	warnout_pre[$iout]=true
844	fi
845
846	# CHECK FOR MULTIPLE FILES, SET A RESPECTIVE FLAG AND REMOVE
847	# THE WILDCARD FROM THE LOCAL FILENAME
848	if [[ "${s1: -1}" = "*" ]]
849	then
850	if [[ "$s2b" = "di" ]]
851	then
852	printf "\n +++ wildcards (*) not allowed with \"di\" file attribute."
853	printf "\n see file \"$fileconnection_file\", line"
854	printf "\n$line"
855	locat=iofiles_file; exit
856	fi
857	multout[$iout]=true
858	string=${localout_pre[$iout]}
859	localout_pre[$iout]="${string%?}"
860	else
861	multout[$iout]=false
862	fi
863	fi
864	elif [[ "$s2a" != in && "$s2a" != inopt && "$s2a" != out ]]
865	then
866	printf "\n +++ I/O-attribute in file $fileconnection_file has invalid"
867	printf "\n value \"$s2\". Only \"in\", \"inopt\", and \"out\" are allowed!"
868	locat=connect; exit
869	fi
870	IFS="$IFSALT"
871	fi
872
873	done < $fileconnection_file
874
875
876
877	# VALUES OF PALMRUN-OPTIONS OVERWRITE THOSE FROM THE CONFIGURATION-FILE
878	[[ "$palmrun_memory" != "" ]] && memory=$palmrun_memory
879	[[ "$palmrun_cpumax" != "" ]] && cpumax=$palmrun_cpumax
880	[[ "$palmrun_cores" != "" ]] && cores=$palmrun_cores
881	[[ "$max_par_io_str" != "" ]] && maximum_parallel_io_streams=$max_par_io_str
882	[[ "$palmrun_tasks_per_node" != "" ]] && tasks_per_node=$palmrun_tasks_per_node
883
884
885
886	# EVALUATE MODEL COUPLING FEATURES (OPTION -Y)
887	if [[ $run_coupled_model = true ]]
888	then
889
890	cores_atmos=`echo $coupled_dist \| cut -d" " -s -f1`
891	cores_ocean=`echo $coupled_dist \| cut -d" " -s -f2`
892
893	if (( $cores_ocean + $cores_atmos != $cores ))
894	then
895
896	printf "\n +++ number of processors does not fit to specification by \"-Y\"."
897	printf "\n PEs (total) : $cores"
898	printf "\n PEs (atmosphere): $cores_atmos"
899	printf "\n PEs (ocean) : $cores_ocean"
900	locat=coupling; exit
901
902	fi
903
904	fi
905
906
907	# IF I AM IN BATCH MODE, CHECK IF EXECUTABLE AND OTHER REQUIRED FILES
908	# HAVE BEEN GENERATED BY PALMBUILD AND STORED IN THE SOURCES_FOR_RUN_...
909	# FOLDER
910	if [[ $running_in_batch_mode = true ]]
911	then
912
913	if [[ ! -d ${fast_io_catalog}/${sources_for_run_catalog} ]]
914	then
915	printf "\n +++ directory ${fast_io_catalog}/${sources_for_run_catalog} is missing"
916	printf "\n Please check the output of the palmrun-call"
917	printf "\n that you did on your local host."
918	locat=SOURCES_FOR_RUN; exit
919	fi
920
921	else
922
923	# CREATE THE SOURCES_FOR_RUN_... FOLDER, BUT NOT IF I AM PART OF AN
924	# AUTOMATIC RESTART RUN
925	# AUTOMATIC RESTART RUNS JUST ACCESS THE DIRECTORY CREATED BY THE INITIAL RUN
926	if [[ $restart_run = false ]]
927	then
928
929	# COLLECT FILES TO BE COMPILED IN THE SOURCES_FOR_RUN_... FOLDER ON
930	# THE LOCAL HOST
931	if [[ ! -d $source_path ]]
932	then
933	printf "\n\n +++ source path \"$source_path\" on local host"
934	printf "\n \"$(hostname)\" does not exist"
935	locat=source_path; exit
936	fi
937
938	rm -rf ${base_directory}/${sources_for_run_catalog}
939	mkdir -p ${base_directory}/${sources_for_run_catalog}
940
941	if [[ "$source_list" = LM ]]
942	then
943
944	# DETERMINE MODIFIED FILES OF THE SVN WORKING COPY
945	source_list=""
946	cd $source_path
947
948
949	# CHECK, IF TRUNK-DIRECTORY IS UNDER SVN CONTROL
950	if [[ ! -d ../.svn ]]
951	then
952	printf "\n\n +++ source directory"
953	printf "\n \"$source_path\" "
954	printf "\n is not under control of \"subversion\"."
955	printf "\n Please do not use palmrun-option \"-s LM\"\n"
956	fi
957
958
959	# LIST ALL MODIFIED SOURCE CODE FILES
960	Filenames=""
961	svn status > tmp_svnstatus
962	while read line
963	do
964	firstc=`echo $line \| cut -c1`
965	if [[ $firstc = M \|\| $firstc = "?" ]]
966	then
967	Name=`echo "$line" \| cut -c8-`
968	extension=`echo $Name \| cut -d. -f2`
969	if [[ "$extension" = f90 \|\| "$extension" = F90 \|\| "$extension" = f \|\| "$extension" = F \|\| "$extension" = c ]]
970	then
971	Filenames="$Filenames "$Name
972	fi
973	fi
974	done < tmp_svnstatus
975	rm -rf tmp_svnstatus
976
977
978	# COPY FILES TO SOURCES_FOR_RUN_...
979	for filename in $Filenames
980	do
981	cp $filename ${base_directory}/${sources_for_run_catalog}
982	source_list=$source_list"$filename "
983	done
984
985	cd - > /dev/null
986
987
988	# COPY FILES GIVEN BY OPTION -s TO DIRECTORY SOURCES_FOR_RUN_...
989	elif [[ "$source_list" != "" ]]
990	then
991
992	cd $source_path
993
994	for filename in $source_list
995	do
996
997	# SOURCE CODE FILE IS NOT ALLOWED TO INCLUDE PATH
998	if [[ $(echo $filename \| grep -c "/") != 0 ]]
999	then
1000	printf "\n +++ source code file: $filename"
1001	printf "\n must not contain (\"/\") "
1002	locat=source; exit
1003	fi
1004
1005	if [[ ! -f $filename ]]
1006	then
1007	printf "\n +++ source code file: $filename"
1008	printf "\n does not exist"
1009	locat=source; exit
1010	else
1011	cp $filename ${base_directory}/${sources_for_run_catalog}
1012	fi
1013
1014	done
1015
1016	cd - > /dev/null
1017
1018	fi
1019
1020	# CHECK, IF MAKEFILE EXISTS AND COPY IT TO THE SOURCES_FOR_RUN... DIRECTORY
1021	[[ "$makefile" = "" ]] && makefile=$source_path/Makefile
1022	if [[ ! -f $makefile ]]
1023	then
1024	printf "\n +++ file \"$makefile\" does not exist"
1025	locat=make; exit
1026	else
1027	cp $makefile ${base_directory}/${sources_for_run_catalog}/Makefile
1028	fi
1029
1030
1031	# COPY FILES FROM OPTIONAL SOURCE PATH GIVEN IN THE CONFIGURATION FILE
1032	if [[ "$user_source_path" != "" ]]
1033	then
1034
1035	# DOES THE DIRECTORY EXIST?
1036	if [[ ! -d $user_source_path ]]
1037	then
1038
1039	printf "\n\n *** INFORMATIVE: additional source code directory"
1040	printf "\n \"$user_source_path\" "
1041	printf "\n does not exist or is not a directory."
1042	printf "\n No source code will be used from this directory!\n"
1043	user_source_path=""
1044	if [[ $silent == false ]]
1045	then
1046	sleep 2
1047	fi
1048
1049	else
1050
1051	cd $user_source_path
1052
1053	Names=$(ls -1 *.f90 2>&1)
1054	[[ $(echo $Names \| grep -c '*.f90') = 0 ]] && AddFilenames="$Names"
1055	Names=$(ls -1 *.F90 2>&1)
1056	[[ $(echo $Names \| grep -c '*.F90') = 0 ]] && AddFilenames="$AddFilenames $Names"
1057	Names=$(ls -1 *.F 2>&1)
1058	[[ $(echo $Names \| grep -c '*.F') = 0 ]] && AddFilenames="$AddFilenames $Names"
1059	Names=$(ls -1 *.f 2>&1)
1060	[[ $(echo $Names \| grep -c '*.f') = 0 ]] && AddFilenames="$AddFilenames $Names"
1061	Names=$(ls -1 *.c 2>&1)
1062	[[ $(echo $Names \| grep -c '*.c') = 0 ]] && AddFilenames="$AddFilenames $Names"
1063
1064	cd - > /dev/null
1065	cd ${base_directory}/${sources_for_run_catalog}
1066
1067	# COPY MAKEFILE IF EXISTING
1068	if [[ -f $user_source_path/Makefile ]]
1069	then
1070	printf "\n\n *** user Makefile from directory"
1071	printf "\n \"$user_source_path\" is used \n"
1072	if [[ $silent == false ]]
1073	then
1074	sleep 1
1075	fi
1076	cp $user_source_path/Makefile .
1077	fi
1078
1079	for filename in $AddFilenames
1080	do
1081	if [[ -f $filename ]]
1082	then
1083	printf "\n +++ source code file \"$filename\" found in additional"
1084	printf "\n source code directory \"$user_source_path\" "
1085	printf "\n but was also given with option \"-s\" which means that it should be taken"
1086	printf "\n from directory \"$source_path\"."
1087	locat=source; exit
1088	fi
1089
1090	cp $user_source_path/$filename .
1091	source_list="$source_list $filename"
1092
1093	# CHECK IF FILE IS CONTAINED IN MAKEFILE
1094	if [[ $(grep -c $filename Makefile) = 0 ]]
1095	then
1096	printf "\n\n +++ user file \"$filename\" "
1097	printf "\n is not listed in Makefile \n"
1098	locat=source; exit
1099	fi
1100	done
1101	cd - > /dev/null
1102	fi
1103	fi
1104
1105	# COPY CONFIGURATION FILES
1106	cp $config_file ${base_directory}/${sources_for_run_catalog}
1107	cp $fileconnection_file ${base_directory}/${sources_for_run_catalog}
1108
1109	# COPY SHELLSCRIPTS
1110	cp ${source_path}/../SCRIPTS/palmrun ${base_directory}/${sources_for_run_catalog}
1111	cp ${source_path}/../SCRIPTS/batch_scp ${base_directory}/${sources_for_run_catalog}
1112
1113	fi
1114
1115	fi
1116
1117
1118	# GET THE GLOBAL REVISION-NUMBER OF THE SVN-REPOSITORY
1119	# (HANDED OVER TO RESTART-RUNS USING OPTION -G)
1120	if [[ "$global_revision" = "" ]]
1121	then
1122	global_revision=`svnversion $source_path 2>/dev/null`
1123	global_revision="Rev: $global_revision"
1124	fi
1125
1126
1127	# IN CASE OF PARALLEL EXECUTION, CHECK SOME SPECIFICATIONS CONCERNING PROCESSOR NUMBERS
1128	if [[ -n $cores ]]
1129	then
1130
1131	# CHECK, IF THE NUMBER OF CORES PER NODE HAS BEEN GIVEN UND IF IT IS AN
1132	# INTEGRAL DIVISOR OF THE TOTAL NUMBER OF CORES GIVEN BY OPTION -X
1133	if [[ $tasks_per_node = 0 ]]
1134	then
1135	if [[ $create_batch_job = true \|\| $create_remote_batch_job = true ]]
1136	then
1137	printf "\n"
1138	printf "\n +++ option \"-T\" (tasks per node) is missing"
1139	printf "\n set -T option or define tasks_per_node in the config file"
1140	locat=tasks_per_node; (( iec = 0 )); exit
1141	else
1142	# DEFAULT FOR INTERACTIVE RUN
1143	tasks_per_node=$cores
1144	fi
1145	fi
1146
1147	if (( cores < tasks_per_node ))
1148	then
1149	printf "\n"
1150	printf "\n +++ tasks per node (-T) cannot exceed total number of cores (-X)"
1151	printf "\n given values: -T $tasks_per_node -X $cores"
1152	locat=tasks_per_node; (( iec = 0 )); exit
1153	fi
1154
1155	(( nodes = cores / ( tasks_per_node * threads_per_task ) ))
1156	(( mpi_tasks = cores / threads_per_task ))
1157	[[ $mpi_tasks = 0 ]] && (( mpi_tasks = 1 ))
1158	(( ii = cores / tasks_per_node ))
1159	(( remaining_cores = cores - ii * tasks_per_node ))
1160	if (( remaining_cores > 0 ))
1161	then
1162	printf "\n"
1163	printf "\n +++ WARNING: tasks per node (option \"-T\") is not an integral"
1164	printf "\n divisor of the total number of cores (option \"-X\")"
1165	printf "\n values of this palmrun-call: \"-T $tasks_per_node\" \"-X $cores\""
1166	printf "\n One of the nodes is filled with $remaining_cores instead of $tasks_per_node tasks"
1167	(( nodes = nodes + 1 ))
1168	fi
1169
1170	fi
1171
1172
1173	# SET DEFAULT VALUE FOR THE MAXIMUM NUMBER OF PARALLEL IO STREAMS
1174	if [[ "$maximum_parallel_io_streams" = "" ]]
1175	then
1176	maximum_parallel_io_streams=$cores
1177	fi
1178
1179
1180	# SET PORT NUMBER OPTION FOR CALLS OF SSH/SCP AND batch_scp SCRIPT
1181	if [[ "$scp_port" != "" ]]
1182	then
1183	PORTOPT="-P $scp_port"
1184	SSH_PORTOPT="-p $scp_port"
1185	fi
1186
1187
1188	# DETERMINE THE SSH-OPTION IN CASE THAT AN SSH-KEY IS EXPLICITLY GIVEN IN THE
1189	# CONFIG-FILE
1190	if [[ "$ssh_key" != "" ]]
1191	then
1192	ssh_key="-i $HOME/.ssh/$ssh_key"
1193	fi
1194
1195
1196	# SET QUEUE, IF NOT GIVEN
1197	if [[ $create_batch_job = true \|\| $create_remote_batch_job = true ]]
1198	then
1199
1200	if [[ $queue = none && "$defaultqueue" = "" ]]
1201	then
1202	printf "\n"
1203	printf "\n +++ no default queue given in configuration file and no queue"
1204	printf "\n given with option -q"
1205	locat=queue; exit
1206	fi
1207	if [[ $queue = none ]]
1208	then
1209	queue=$defaultqueue
1210	fi
1211
1212	fi
1213
1214
1215	# GENERATE FULL FILENAMES OF INPUT-FILES, INCLUDING THEIR PATH
1216	# CHECK, IF INPUT-FILES EXIST, AND DETERMINE HIGHEST CYCLE NUMBER (IF CYCLES EXIST)
1217	(( i = 0 ))
1218	(( nr_of_input_files = 0 ))
1219	while (( i < iin ))
1220	do
1221	(( i = i + 1 ))
1222
1223	# GENERATE PATH AND FULL FILE NAME (then-BRANCH: FIXED FULL NAME IS GIVEN, I.E. THE
1224	# FILE IDENTIFIER IS NOT PART OF THE FILENAME))
1225	if [[ "${actionin_pre[$i]}" = di ]]
1226	then
1227	eval filename=${pathin_pre[$i]}/${endin_pre[$i]}
1228	else
1229	eval filename=${pathin_pre[$i]}/${run_identifier}${endin_pre[$i]}
1230	fi
1231
1232
1233	# CHECK IF FILE EXISTS
1234	if ! ls $filename* 1>/dev/null 2>&1
1235	then
1236
1237	# FILES WITH ATTRIBUTE opt ARE OPTIONAL. NO ABORT, IF THEY DO NOT EXIST.
1238	if [[ "${optin_pre[$i]}" != "yes" ]]
1239	then
1240	printf "\n\n +++ INPUT-file: "
1241	if [[ "${extin_pre[$i]}" = "" \|\| "${extin_pre[$i]}" = " " ]]
1242	then
1243	printf "\n $filename"
1244	else
1245	printf "\n $filename.${extin_pre[$i]}"
1246	fi
1247	printf "\n does not exist\n"
1248	locat=input; exit
1249	else
1250	(( nr_of_input_files = nr_of_input_files + 1 ))
1251	localin[$nr_of_input_files]="${localin_pre[$i]}"
1252	optin[$nr_of_input_files]="${optin_pre[$i]}"
1253	actionin[$nr_of_input_files]="unavailable"
1254	pathin[$nr_of_input_files]="${pathin_pre[$i]}"
1255	endin[$nr_of_input_files]="${endin_pre[$i]}"
1256	extin[$nr_of_input_files]="${extin_pre[$i]}"
1257	fi
1258
1259	else
1260
1261	# FIRST CHECK FOR MULTIPLE NAMES WITH THE SAME BASENAME
1262	# ($run_identifier) AND CREATE A LIST FOR THE DETECTED BASENAME
1263	# ENDINGS
1264	if [[ "${multin[$i]}" = true ]]
1265	then
1266	# DETERMINE THE EXISTING EXTENSIONS FROM THE LIST OF FILES
1267	ls -1 -d ${filename} > filelist 2>/dev/null
1268	ls -1 -d ${filename}.* >> filelist 2>/dev/null
1269	ls -1 -d ${filename}_* >> filelist 2>/dev/null
1270
1271	endings=""
1272	while read line
1273	do
1274	# filename without path (i.e. after the last "/")
1275	basefilename=$(basename ${line})
1276
1277	# check if there is an extension and remove it
1278	ext=${basefilename##*.}
1279	if [[ "$ext" = "${extin_pre[$i]}" ]]
1280	then
1281	basefilename=${basefilename%.*}
1282	fi
1283
1284	# check for an existing cycle number and remove it
1285	cycle=${basefilename##*.}
1286	if [[ $cycle =~ ^-?[0-9]+$ ]]
1287	then
1288	basefilename=${basefilename%.*}
1289	fi
1290
1291	# remove the run_identifier from the beginning
1292	length_run_identifier=${#run_identifier}
1293	ending=${basefilename:${length_run_identifier}}
1294
1295	# remove the ending given in the .iofiles from the beginning
1296	endingstring="${endin_pre[$i]}"
1297	length_ending=${#endingstring}
1298	ending=${ending:${length_ending}}
1299
1300	if [[ "$ending" = "" ]]
1301	then
1302	# standard ending as given in the .iofiles
1303	if [[ $(echo $endings \| grep -c DEFAULT) = 0 ]]
1304	then
1305	endings="$endings DEFAULT"
1306	fi
1307	else
1308	# ending must start with "_", otherwise its a different file
1309	if [[ "${ending:0:1}" = "_" ]]
1310	then
1311	if [[ $(echo $endings \| grep -c "$ending") = 0 ]]
1312	then
1313	endings="$endings $ending"
1314	fi
1315	fi
1316	fi
1317
1318	done <filelist
1319
1320	rm filelist
1321
1322	else
1323
1324	# SINGLE NAME
1325	endings=DEFAULT
1326
1327	fi
1328
1329	# FOR EACH BASENAME ENDING CREATE AN ENTRY IN THE FINAL INPUT FILE LIST
1330	for ending in $endings
1331	do
1332
1333	# DEFAULT MEANS THAT THE ENDING GIVEN IN .iofiles SHALL BE USED
1334	if [[ $ending = DEFAULT ]]
1335	then
1336	ending=""
1337	fi
1338
1339	# NEW ENTRY (ENDING IS ALSO ADDED TO LOCAL FILENAME READ BY PALM!)
1340	(( nr_of_input_files = nr_of_input_files + 1 ))
1341	localin[$nr_of_input_files]="${localin_pre[$i]}"$ending
1342	optin[$nr_of_input_files]="${optin_pre[$i]}"
1343	actionin[$nr_of_input_files]="${actionin_pre[$i]}"
1344	pathin[$nr_of_input_files]="${pathin_pre[$i]}"
1345	endin[$nr_of_input_files]="${endin_pre[$i]}"$ending
1346	extin[$nr_of_input_files]="${extin_pre[$i]}"
1347
1348
1349	# GENERATE PATH AND FULL FILE NAME (then-BRANCH: FIXED FULL NAME IS GIVEN, I.E. THE
1350	# FILE IDENTIFIER IS NOT PART OF THE FILENAME))
1351	if [[ "${actionin[$nr_of_input_files]}" = di ]]
1352	then
1353	eval filename=${pathin[$nr_of_input_files]}/${endin[$nr_of_input_files]}
1354	else
1355	eval filename=${pathin[$nr_of_input_files]}/${run_identifier}${endin[$nr_of_input_files]}
1356	fi
1357
1358	# DETERMINE THE FILE'S CYCLE NUMBER
1359	(( maxcycle = 0 ))
1360	ls -1 -d $filename > filelist 2>/dev/null
1361	ls -1 -d $filename.* >> filelist 2>/dev/null
1362	while read line
1363	do
1364	# filename without path (i.e. after the last "/")
1365	basefilename=$(basename ${line})
1366
1367	# check if there is an extension
1368	extension=${basefilename##*.}
1369	if [[ "$extension" = "${extin[$nr_of_input_files]}" ]]
1370	then
1371	basefilename=${basefilename%.*}
1372	fi
1373
1374	# check for an existing cycle number
1375	cycle=${basefilename##*.}
1376	if [[ $cycle =~ ^-?[0-9]+$ ]]
1377	then
1378	# NUMBERS WITH LEADING ZEROS ARE INTERPRETED AS OCTAL NUMBERS
1379	# 10# EXPLICITLY SPECIFIES THE NUMBER BASE AS 10
1380	(( icycle = $((10#$cycle)) ))
1381	explicit_cycle_number=true
1382	else
1383	(( icycle = 0 ))
1384	explicit_cycle_number=false
1385	fi
1386
1387	if (( icycle > maxcycle ))
1388	then
1389	(( maxcycle = icycle ))
1390	fi
1391
1392	done <filelist
1393	rm filelist
1394
1395
1396	# MAKE CYCLE NUMBER THREE DIGITS WIDE
1397	cyclestring=`printf "%03d" $maxcycle`
1398
1399
1400	# APPEND CYCLE NUMBER TO FILENAME
1401	if [[ $explicit_cycle_number = true ]]
1402	then
1403	if [[ "${extin[$nr_of_input_files]}" != " " && "${extin[$nr_of_input_files]}" != "" ]]
1404	then
1405	filename=${filename}.$cyclestring.${extin[$nr_of_input_files]}
1406	else
1407	filename=${filename}.$cyclestring
1408	fi
1409	else
1410	if [[ "${extin[$nr_of_input_files]}" != " " && "${extin[$nr_of_input_files]}" != "" ]]
1411	then
1412	filename=${filename}.${extin[$nr_of_input_files]}
1413	fi
1414	fi
1415
1416	# STORE FILENAME WITHOUT PATH BUT WITH CYCLE NUMBER,
1417	# IS LATER USED FOR TRANSFERRING FILES WIHIN THE JOB (SEE END OF FILE)
1418	absnamein[$nr_of_input_files]=$filename
1419	if [[ $explicit_cycle_number = true ]]
1420	then
1421	if [[ "${actionin[$nr_of_input_files]}" = di ]]
1422	then
1423	frelin[$nr_of_input_files]=${endin[$nr_of_input_files]}.$cyclestring
1424	else
1425	frelin[$nr_of_input_files]=${run_identifier}${endin[$nr_of_input_files]}.$cyclestring
1426	fi
1427	else
1428	if [[ "${actionin[$nr_of_input_files]}" = di ]]
1429	then
1430	frelin[$nr_of_input_files]=${endin[$nr_of_input_files]}
1431	else
1432	frelin[$nr_of_input_files]=${run_identifier}${endin[$nr_of_input_files]}
1433	fi
1434	fi
1435
1436	done
1437
1438	fi
1439
1440	done
1441
1442
1443	# GENERATE FULL FILENAMES OF OUTPUT-FILES (WITHOUT $ OR ~),
1444	# CHECK, IF OUTPUT-FILES EXIST, AND DETERMINE HIGHEST CYCLE NUMBER (IF CYCLES EXIST),
1445	# OR, IN CASE THAT FILE DOES NOT EXIST, CHECK, IF IT CAN BE CREATED
1446	# THESE ACTIONS ARE NOT CARRIED OUT, IF FILES SHALL BE TRANSFERRED FROM THE REMOTE TO
1447	# THE LOCAL HOST (BECAUSE THEIR IS NO DIRECT ACCESS TO THE LOCAL DIRECTORIES FROM THE
1448	# REMOTE HOST)
1449	(( i = 0 ))
1450	while (( i < iout ))
1451	do
1452	(( i = i + 1 ))
1453	if [[ ! ( $running_on_remote = true && ( "${actionout_pre[$i]}" = tr \|\| "${actionout_pre[$i]}" = tra \|\| "${actionout_pre[$i]}" = trpe ) ) ]]
1454	then
1455	if [[ "${actionout_pre[$i]}" = tr ]]
1456	then
1457	actionout_pre[$i]=""
1458	elif [[ "${actionout_pre[$i]}" = trpe ]]
1459	then
1460	actionout_pre[$i]=pe
1461	elif [[ "${actionout_pre[$i]}" = tra ]]
1462	then
1463	actionout_pre[$i]=a
1464	fi
1465	(( maxcycle = 0 ))
1466	eval filename=${pathout_pre[$i]}/${run_identifier}${endout_pre[$i]}
1467	eval catalogname=${pathout_pre[$i]}
1468	if ! ls $filename* 1>/dev/null 2>&1
1469	then
1470
1471	# IF OUTPUT-FILE DOES NOT EXIST CHECK, IF IT CAN BE CREATED
1472	if cat /dev/null > $filename
1473	then
1474	rm $filename
1475	else
1476
1477	# CHECK, IF THE DIRECTORY WHERE FILE SHALL BE COPIED TO EXISTS
1478	# IF IT DOES NOT EXIST, TRY TO CREATE IT
1479	if [[ ! -d $catalogname ]]
1480	then
1481	if mkdir -p $catalogname
1482	then
1483	printf "\n\n *** directory:"
1484	printf "\n $catalogname"
1485	printf "\n was created\n"
1486	else
1487	printf "\n\n +++ OUTPUT-file:"
1488	printf "\n $filename"
1489	printf "\n cannot be created, because directory does not exist"
1490	printf "\n and cannot be created either"
1491	printf "\n"
1492	locat=output ; exit
1493	fi 2>/dev/null
1494	else
1495	printf "\n\n +++ OUTPUT-file:"
1496	printf "\n $filename"
1497	printf "\n cannot be created, although directory exists"
1498	printf "\n"
1499	locat=output ; exit
1500	fi
1501	fi 2>/dev/null
1502
1503	fi
1504
1505	fi
1506	done
1507
1508
1509	# DETERMINE THE NAME OF PALMRUN'S TEMPORARY WORKING DIRECTORY
1510	if [[ $running_in_batch_mode = false ]]
1511	then
1512	run_id_number=$RANDOM
1513	run_id=${run_identifier}.$run_id_number
1514
1515	tempdir=$fast_io_catalog/$run_id
1516
1517	# FOR COMPATIBILITY REASONS WITH OLDER VERSIONS SET JOB_ID
1518	export job_id=$run_id
1519
1520	fi
1521
1522
1523	# CHECK SETTINGS REQUIRED FOR BATCH JOBS
1524	if [[ $create_batch_job = true \|\| $create_remote_batch_job = true ]]
1525	then
1526
1527	# CHECK, IF JOB DIRECTIVES HAVE BEEN GIVEN IN CONFIGURATION FILE
1528	if [[ $ibd = 0 ]]
1529	then
1530	printf "\n"
1531	printf "\n +++ no batch directives found in configuration file"
1532	locat=config_file_batch_directives; (( iec = 0 )); exit
1533	fi
1534
1535	# CHECK IF CPUTIME IS GIVEN FOR JOB
1536	done=false
1537	cputime=$cpumax
1538	while [[ $done = false ]]
1539	do
1540	if (( cputime == 0 ))
1541	then
1542	printf "\n +++ cpu-time is undefined"
1543	printf "\n >>> Please type CPU-time in seconds as INTEGER:"
1544	printf "\n >>> "
1545	read cputime 1>/dev/null 2>&1
1546	else
1547	done=true
1548	fi
1549	done
1550	cpumax=$cputime
1551
1552	# CHECK THE MEMORY DEMAND
1553	done=false
1554	while [[ $done = false ]]
1555	do
1556	if (( memory == 0 ))
1557	then
1558	printf "\n +++ memory demand is undefined"
1559	printf "\n >>> Please type memory in MByte per process as INTEGER:"
1560	printf "\n >>> "
1561	read memory 1>/dev/null 2>&1
1562	else
1563	done=true
1564	fi
1565	done
1566
1567	# IN CASE OF REMOTE-JOBS CHECK, IF A USERNAME FOR THE REMOTE HOST IS GIVEN
1568	if [[ $create_remote_batch_job = true && -z $remote_username ]]
1569	then
1570	while [[ -z $remote_username ]]
1571	do
1572	printf "\n +++ username on remote host with IP \"$remote_ip\" is undefined"
1573	printf "\n >>> Please type username:"
1574	printf "\n >>> "
1575	read remote_username
1576	done
1577	fi
1578
1579	else
1580
1581	if [[ $running_in_batch_mode = false ]]
1582	then
1583	cputime=10000000 # NO LIMT FOR INTERACTIVE RUNS
1584	cpumax=$cputime
1585	else
1586	cputime=$cpumax
1587	fi
1588
1589	fi
1590
1591
1592	# CALCULATE HOURS/MINUTES/SECONDS, E.G. FOR BATCH-DIRECTIVES
1593	(( cpu_hours = cputime / 3600 ))
1594	(( resttime = cputime - cpu_hours * 3600 ))
1595	(( cpu_minutes = resttime / 60 ))
1596	(( cpu_seconds = resttime - cpu_minutes * 60 ))
1597	timestring=${cpu_hours}:${cpu_minutes}:${cpu_seconds}
1598
1599
1600	# OUTPUT OF THE PALMRUN-HEADER
1601	calltime=$(date)
1602	printf "\n"
1603	printf "#------------------------------------------------------------------------# \n"
1604	printf "\| %-35s%35s \| \n" "$version" "$calltime"
1605	printf "\| %-35s%35s \| \n" "PALM code $global_revision" " "
1606	printf "\| \| \n"
1607	column1="called on:"; column2=$(hostname)
1608	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
1609	if [[ $create_remote_batch_job = true ]]
1610	then
1611	column1="execution on:"; column2="$configuration_identifier (username: $remote_username)"
1612	else
1613	if [[ $running_on_remote = true ]]
1614	then
1615	column1="config. identifier:"; column2="$configuration_identifier (execute on IP: $remote_ip)"
1616	else
1617	column1="config. identifier:"; column2="$configuration_identifier (execute on IP: $local_ip)"
1618	fi
1619	fi
1620	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
1621
1622	column1="running in:"
1623	if [[ $running_in_batch_mode = true ]]
1624	then
1625	column2="batch job mode"
1626	else
1627	if [[ $create_batch_job = true \|\| $create_remote_batch_job = true ]]
1628	then
1629	column2="job creation mode"
1630	else
1631	column2="interactive run mode"
1632	fi
1633	fi
1634	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
1635
1636	if [[ $running_in_batch_mode = true \|\| $create_batch_job = true \|\| $create_remote_batch_job = true ]]
1637	then
1638	if [[ "$project_account" != "" ]]
1639	then
1640	column1="project account number:"
1641	column2="$project_account"
1642	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
1643	fi
1644	fi
1645
1646	if [[ -n $cores ]]
1647	then
1648	if [[ $run_coupled_model = false ]]
1649	then
1650	column1="number of cores:"; column2=$cores
1651	else
1652	column1="number of cores:"; column2="$cores (atmosphere: $cores_atmos, ocean: $cores_ocean)"
1653	fi
1654	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
1655	fi
1656	if [[ -n $tasks_per_node ]]
1657	then
1658	column1="tasks per node:"; column2="$tasks_per_node (number of nodes: $nodes)"
1659	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
1660	if (( remaining_cores > 0 ))
1661	then
1662	column1=" "; column2="one of the nodes only filled with $remaining_cores tasks"
1663	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
1664	fi
1665	fi
1666	if [[ $maximum_parallel_io_streams != $cores ]]
1667	then
1668	column1="max par io streams:"; column2="$maximum_parallel_io_streams"
1669	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
1670	fi
1671	if [[ $use_openmp = true ]]
1672	then
1673	column1="threads per task:"; column2="$threads_per_task"
1674	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
1675	fi
1676	if [[ $create_batch_job = true \|\| $create_remote_batch_job = true \|\| $running_in_batch_mode = true ]]
1677	then
1678	column1="memory demand / PE":; column2="$memory MB"
1679	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
1680	column1="job cpu time (h:m:s):"; column2="$timestring"
1681	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
1682	fi
1683	printf "\| \| \n"
1684	if [[ "$source_list" != "" ]]
1685	then
1686	if [[ "$make_options" != "" ]]
1687	then
1688	column1="make options:"; column2=$(echo "$make_options" \| cut -c-45)
1689	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
1690	line=$(echo "$make_options" \| cut -c46-)
1691	while [[ "$line" != "" ]]
1692	do
1693	column1=""
1694	column2=$(echo "$line" \| cut -c-45)
1695	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
1696	line=$(echo "$line" \| cut -c46-)
1697	done
1698	fi
1699	fi
1700
1701	column1="cpp directives:"; column2=$(echo "$cpp_options" \| cut -c-45)
1702	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
1703	line=$(echo "$cpp_options" \| cut -c46-)
1704	while [[ "$line" != "" ]]
1705	do
1706	column1=""
1707	column2=$(echo "$line" \| cut -c-45)
1708	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
1709	line=$(echo "$line" \| cut -c46-)
1710	done
1711
1712	column1="compiler options:"; column2=$(echo "$compiler_options" \| cut -c-45)
1713	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
1714	line=$(echo "$compiler_options" \| cut -c46-)
1715	while [[ "$line" != "" ]]
1716	do
1717	column1=""
1718	column2=$(echo "$line" \| cut -c-45)
1719	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
1720	line=$(echo "$line" \| cut -c46-)
1721	done
1722
1723	column1="linker options:"; column2=$(echo "$linker_options" \| cut -c-45)
1724	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
1725	line=$(echo "$linker_options" \| cut -c46-)
1726	while [[ "$line" != "" ]]
1727	do
1728	column1=""
1729	column2=$(echo "$line" \| cut -c-45)
1730	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
1731	line=$(echo "$line" \| cut -c46-)
1732	done
1733
1734	if [[ "$login_init_cmd" != "" ]]
1735	then
1736	column1="login init commands:"; column2=$(echo "$login_init_cmd" \| cut -c-45)
1737	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
1738	line=$(echo "$login_init_cmd" \| cut -c46-)
1739	while [[ "$line" != "" ]]
1740	do
1741	column1=""
1742	column2=$(echo "$line" \| cut -c-45)
1743	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
1744	line=$(echo "$line" \| cut -c46-)
1745	done
1746	fi
1747
1748	if [[ "$module_commands" != "" ]]
1749	then
1750	column1="module commands:"; column2=$(echo "$module_commands" \| cut -c-45)
1751	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
1752	line=$(echo "$module_commands" \| cut -c46-)
1753	while [[ "$line" != "" ]]
1754	do
1755	column1=""
1756	column2=$(echo "$line" \| cut -c-45)
1757	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
1758	line=$(echo "$line" \| cut -c46-)
1759	done
1760	fi
1761	printf "\| \| \n"
1762	column1="run identifier:"; column2=$run_identifier
1763	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
1764	column1="activation string list:"; column2=$(echo $activation_string_list)
1765	printf "\| %-25s%-45s \| \n" "$column1" "$column2"
1766
1767	if [[ "$ocean_file_appendix" = true ]]
1768	then
1769	printf "\| %-35s%-35s \| \n" "suffix \"_O\" is added to local files" " "
1770	fi
1771
1772	if [[ "$source_list" != "" ]]
1773	then
1774	printf "\| \| \n"
1775	printf "\| Files to be compiled: \| \n"
1776	line=$source_list
1777	while [[ "$line" != "" ]]
1778	do
1779	linestart=$(echo $line \| cut -c-70)
1780	printf "\| %-70s \| \n" "$linestart"
1781	line=$(echo "$line" \| cut -c71-)
1782	done
1783	fi
1784	printf "#------------------------------------------------------------------------#"
1785
1786
1787
1788	# OUTPUT OF FILE CONNECTIONS IN CASE OF TRACEBACK
1789	if [[ $do_trace = true ]]
1790	then
1791	(( i = 0 ))
1792	while (( i < nr_of_input_files ))
1793	do
1794	(( i = i + 1 ))
1795	if (( i == 1 ))
1796	then
1797	printf "\n\n >>> INPUT-file assignments:\n"
1798	fi
1799	printf "\n ${localin[$i]} : ${absnamein[$i]}"
1800	done
1801	(( i = 0 ))
1802	while (( i < iout ))
1803	do
1804	(( i = i + 1 ))
1805	if (( i == 1 ))
1806	then
1807	printf "\n\n >>> OUTPUT-file assignments:\n"
1808	fi
1809	printf "\n ${localout[$i]} : ${pathout[$i]}"
1810	done
1811	(( i = 0 ))
1812	while (( i < iic ))
1813	do
1814	(( i = i + 1 ))
1815	if (( i == 1 ))
1816	then
1817	printf "\n\n >>> INPUT-commands:\n"
1818	fi
1819	printf "\n ${in_command[$i]}"
1820	done
1821	(( i = 0 ))
1822	while (( i < ioc ))
1823	do
1824	(( i = i + 1 ))
1825	if (( i == 1 ))
1826	then
1827	printf "\n\n >>> OUTPUT-commands:\n"
1828	fi
1829	printf "\n ${out_command[$i]}"
1830	done
1831	fi
1832
1833	# QUERY FOR CONTINUE
1834	if [[ $silent = false && $running_in_batch_mode = false ]]
1835	then
1836	antwort=dummy
1837	printf "\n\n"
1838	printf " >>> everything o.k. (y/n) ? "
1839	while read antwort
1840	do
1841	if [[ "$antwort" != y && "$antwort" != Y && "$antwort" != n && "$antwort" != N ]]
1842	then
1843	printf " >>> everything o.k. (y/n) ? "
1844	else
1845	break
1846	fi
1847	done
1848	if [[ $antwort = n \|\| $antwort = N ]]
1849	then
1850	locat=user_abort; (( iec = 0 )); exit
1851	fi
1852	if [[ $create_batch_job = true \|\| $create_remote_batch_job = true ]]
1853	then
1854	printf "\n *** batch-job will be created and submitted"
1855	else
1856	printf "\n *** PALMRUN will now continue to execute on this machine"
1857	fi
1858	fi
1859
1860
1861
1862	# PROVIDE FILES TO EXECUTE PALM AND CREATE THE EXECUTABLE
1863	if [[ $restart_run = false && $running_in_batch_mode = false ]]
1864	then
1865
1866	if [[ $create_batch_job = true \|\| $create_remote_batch_job = true ]]
1867	then
1868	printf "\n\n *** creating executable and other sources for the remote host\n"
1869	else
1870	printf "\n\n *** creating executable and other sources for the local host\n"
1871	fi
1872
1873	# FIRST CHECK, IF A MAKE DEPOSITORY EXISTS, AND IF NOT, ASK THE USER IF
1874	# IT SHALL BE CREATED
1875	ask_for_make_depository=false
1876	if [[ $create_remote_batch_job = true ]]
1877	then
1878
1879	line=`grep %base_directory $config_file`
1880	make_depository=`echo $line \| cut -d" " -s -f2`/MAKE_DEPOSITORY_${configuration_identifier}
1881	ssh -q $ssh_key ${remote_username}@${remote_ip} "[[ ! -d ${make_depository} ]] && echo depository not found" 2>&1 \| tee ${configuration_identifier}_last_make_protokoll
1882
1883	if [[ $(grep -c "depository not found" ${configuration_identifier}_last_make_protokoll) != 0 ]]
1884	then
1885	printf "\n\n +++ make depository \"${make_depository}\""
1886	printf "\n on remote host not found!"
1887	ask_for_make_depository=true
1888	fi
1889	rm ${configuration_identifier}_last_make_protokoll
1890
1891	else
1892
1893	# CHECK FOR MAKE_DEPOSITORY ON THE LOCAL HOST
1894	make_depository=${base_directory}/MAKE_DEPOSITORY_${configuration_identifier}
1895	if [[ ! -d ${make_depository} ]]
1896	then
1897	printf "\n\n +++ make depository \"${make_depository}\""
1898	printf "\n on local host not found!"
1899	ask_for_make_depository=true
1900	fi
1901
1902	fi
1903
1904	if [[ $ask_for_make_depository = true ]]
1905	then
1906
1907	antwort=dummy
1908	printf "\n\n"
1909	printf " >>> Create a new one (y/n) ? "
1910	while read antwort
1911	do
1912	if [[ "$antwort" != y && "$antwort" != Y && "$antwort" != n && "$antwort" != N ]]
1913	then
1914	printf " >>> Create a new one (y/n) ? "
1915	else
1916	break
1917	fi
1918	done
1919	if [[ $antwort = n \|\| $antwort = N ]]
1920	then
1921	locat=user_abort; (( iec = 0 )); exit
1922	fi
1923
1924	if [[ $do_trace = true ]]
1925	then
1926	palmbuild -c $configuration_identifier
1927	else
1928	palmbuild -v -c $configuration_identifier
1929	fi
1930
1931	if [[ ${PIPESTATUS[0]} != 0 ]]
1932	then
1933
1934	# ABORT IN CASE OF COMPILATION PROBLEMS
1935	printf "\n +++ error while compiling for the MAKE_DEPOSITORY"
1936	locat=make_depository
1937	exit
1938	else
1939	echo " *** now continue with creating executable and other sources"
1940	fi
1941
1942	fi
1943
1944	# NOW CREATE THE SOURCES_FOR_RUN FOLDER
1945	palmbuild -v $use_existing_sources_folder -c $configuration_identifier -r $run_identifier
1946
1947	if [[ ${PIPESTATUS[0]} != 0 ]]
1948	then
1949
1950	# ABORT IN CASE OF COMPILATION PROBLEMS
1951	printf "\n +++ error while creating executable and/or other sources"
1952	locat=execution
1953	rm -rf ${base_directory}/${sources_for_run_catalog}
1954	exit
1955
1956	else
1957
1958	printf " *** executable and other sources created\n"
1959	rm -rf ${base_directory}/${sources_for_run_catalog}
1960
1961	fi
1962
1963	fi
1964
1965
1966	# WHEN CREATING A REMOTE BATCH JOB, THOSE INPUT FILES WITH TRANSFER-ATTRIBUT
1967	# WILL BE COPIED TO THE REMOTE HOST
1968	if [[ $create_remote_batch_job = true ]]
1969	then
1970	(( i = 0 ))
1971	while (( i < nr_of_input_files ))
1972	do
1973	(( i = i + 1 ))
1974	if [[ "${actionin[$i]}" = tr ]]
1975	then
1976	eval inputfile=${pathin[$i]}/${frelin[$i]}
1977	scp -q $ssh_key $PORTOPT $inputfile ${remote_username}@${remote_ip}:${fast_io_catalog}/${sources_for_run_catalog}/${frelin[$i]}
1978	fi
1979	done
1980	if (( i > 0 ))
1981	then
1982	printf "\n\n *** input files have been copied to the remote host\n"
1983	fi
1984	fi
1985
1986
1987	# NOW PERFORM THOSE ACTIONS REQUIRED TO EXECUTE THE PROGRAM (PALM) ON THIS MACHINE
1988	# (COMPILING/LINKING, EXECUTING, COPYING I/O FILES)
1989	if [[ $create_batch_job = false && $create_remote_batch_job = false ]]
1990	then
1991
1992	# CHANGE TO THE TEMPORARY WORKING DIRECTORY
1993	if [[ $running_in_batch_mode = false ]]
1994	then
1995
1996	# CREATE THE DIRECTORY
1997	mkdir -p $tempdir
1998	if [[ $? != 0 ]]
1999	then
2000	printf "\n +++ temporary working directory cannot be created."
2001	printf "\n Check setting of variable fast_io_catalog in your config file."
2002	locat=execution
2003	exit
2004	fi
2005	chmod go+rx $tempdir
2006	cd $tempdir
2007	printf "\n *** changed to temporary directory: $tempdir"
2008
2009	else
2010
2011	# IN BATCH MODE PALMRUN IS CALLED FROM TEMPDIR
2012	printf "\n *** running in temporary directory: $tempdir"
2013
2014	fi
2015
2016
2017	# PROVIDE THE INPUT FILES
2018	# LOOP OVER ALL ACTIVATED FILES (LISTED IN THE CONFIGURATION FILE)
2019	optional_files_missing=false
2020	(( i = 0 ))
2021	while (( i < nr_of_input_files ))
2022	do
2023	(( i = i + 1 ))
2024	if (( i == 1 ))
2025	then
2026	printf "\n\n *** providing INPUT-files:\n$dashes"
2027	fi
2028
2029
2030	# SKIP OPTIONAL FILES, IF THEY DO NOT EXIST
2031	if [[ "${actionin[$i]}" = unavailable ]]
2032	then
2033	optional_files_missing=true
2034	continue
2035	fi
2036
2037	# CHECK FOR SINGLE FILE (SERIAL RUN) OR DIRECTORY (ONE FILE PER CORE FOR PARELLEL EXECUTION)
2038	files_for_cores=false; filetype=file
2039	if [[ "${actionin[$i]}" = pe && -n $cores ]]
2040	then
2041	files_for_cores=true; filetype=files
2042	actionin[$i]=""
2043	elif [[ "${actionin[$i]}" = pe && ! -n $cores ]]
2044	then
2045	actionin[$i]=""
2046	elif [[ "${actionin[$i]}" = lnpe && -n $cores ]]
2047	then
2048	files_for_cores=true; filetype=files
2049	actionin[$i]="ln"
2050	# FILE HAS BEEN FOUND INSTEAD OF DIRECTORY
2051	if [[ -f "${absnamein[$i]}" ]]
2052	then
2053	files_for_cores=false; filetype=file
2054	fi
2055	elif [[ "${actionin[$i]}" = lnpe && ! -n $cores ]]
2056	then
2057	actionin[$i]="ln"
2058	fi
2059
2060	if [[ $files_for_cores = true ]]
2061	then
2062	printf "\n >>> INPUT: ${absnamein[$i]}/.... to ${localin[$i]}"
2063	else
2064	printf "\n >>> INPUT: ${absnamein[$i]} to ${localin[$i]}"
2065	fi
2066
2067	# INPUT-FILES TO BE LINKED
2068	if [[ "${actionin[$i]}" = ln ]]
2069	then
2070
2071	printf "\n $filetype will be linked"
2072	if [[ $files_for_cores = false ]]
2073	then
2074	if [[ -f "${absnamein[$i]}" ]]
2075	then
2076	ln ${absnamein[$i]} ${localin[$i]}
2077	got_tmp[$i]=true
2078	fi
2079	else
2080	if [[ -d "${absnamein[$i]}" ]]
2081	then
2082	mkdir -p ${localin[$i]}
2083	cd ${absnamein[$i]}
2084	for file in $(ls *)
2085	do
2086	ln $file $tempdir/${localin[$i]}
2087	done >\|/dev/null 2>&1
2088	cd $tempdir
2089	fi
2090
2091	# IF "ln -f" HAS FAILED DO A NORMAL COPY "cp -r"
2092	if [[ ! -f "${localin[$i]}/_000000" ]]
2093	then
2094	printf "\n --- WARNING: ln failed, using cp instead (might be time consuming...)"
2095	cp -r ${absnamein[$i]}/* ${localin[$i]}
2096	fi
2097
2098	got_tmp[$i]=true
2099	fi
2100	fi
2101
2102	# FILE IS STORED IN THE RESPECTIVE DIRECTORY GIVEN IN THE CONFIGURATION FILE
2103	if [[ "${actionin[$i]}" = "" \|\| "${actionin[$i]}" = "di" \|\| "${actionin[$i]}" = "tr" \|\| "${actionin[$i]}" = "npe" ]]
2104	then
2105
2106	if [[ "${actionin[$i]}" = "npe" && -n $cores ]]
2107	then
2108
2109	# FILE COPIES ARE PROVIDED FOR ALL CORES
2110	# EACH FILE GETS A UNIQUE FILENAME WITH A FOUR DIGIT NUMBER
2111	printf "\n file will be provided for $cores processors"
2112	mkdir -p ${localin[$i]}
2113	ival=$cores
2114	(( ii = 0 ))
2115	while (( ii <= ival-1 ))
2116	do
2117	if (( ii < 10 ))
2118	then
2119	cp ${absnamein[$i]} ${localin[$i]}/_000$ii
2120	elif (( ii < 100 ))
2121	then
2122	cp ${absnamein[$i]} ${localin[$i]}/_00$ii
2123	elif (( ii < 1000 ))
2124	then
2125	cp ${absnamein[$i]} ${localin[$i]}/_0$ii
2126	else
2127	cp ${absnamein[$i]} ${localin[$i]}/_$ii
2128	fi
2129	(( ii = ii + 1 ))
2130	done
2131
2132	else
2133
2134	if [[ $files_for_cores = true ]]
2135	then
2136
2137	# PROVIDE FILES FOR EACH CORE
2138	# FIRST CREATE THE LOCAL DIRECTORY, THEN COPY FILES
2139	# FROM THE PERMANENT DIRECTORY BY LINKING THEM TO THE LOCAL ONE
2140	printf "\n providing $cores files for the respective cores"
2141	mkdir -p ${localin[$i]}
2142	if [[ $link_local_input = true ]]
2143	then
2144	printf " files will be linked\n"
2145	cd ${absnamein[$i]}
2146	for file in $(ls *)
2147	do
2148	ln -f $file ${localin[$i]}
2149	done
2150	cd $tempdir
2151	fi
2152
2153	# IF "ln -f" FAILED OR IF "$link_local_input = false" DO A NORMAL "cp -r"
2154	if [[ ! -f "${localin[$i]}/_000000" ]]
2155	then
2156	if [[ $link_local_input = true ]]
2157	then
2158	printf "\n --- WARNING: ln failed, using cp instead (might be time consuming...)"
2159	fi
2160	cp -r ${absnamein[$i]}/* ${localin[$i]}
2161	fi
2162
2163	else
2164
2165	# PROVIDE FILE FOR RUNS ON A SINGLE CORE
2166	if [[ $link_local_input = true ]]
2167	then
2168	printf " file will be linked\n"
2169	ln -f ${absnamein[$i]} ${localin[$i]}
2170	fi
2171	# IF "ln -f" FAILS OR IF "$link_local_input = false" DO A NORMAL "cp"
2172	if [[ ! -f "${localin[$i]}" ]]
2173	then
2174	if [[ $link_local_input = true ]]
2175	then
2176	printf "\n --- WARNING: ln failed, using cp instead (might be time consuming...)"
2177	fi
2178	if [[ $running_on_remote = true && "${actionin[$i]}" = tr ]]
2179	then
2180	mv ${absnamein[$i]} ${localin[$i]}
2181	else
2182	cp ${absnamein[$i]} ${localin[$i]}
2183	fi
2184	fi
2185	fi
2186	fi
2187	fi
2188
2189	done
2190	if (( i != 0 ))
2191	then
2192	if [[ $optional_files_missing = true ]]
2193	then
2194	printf "\n *** INFORMATIVE: some optional INPUT-files are not present"
2195	fi
2196	printf "\n$dashes\n *** all INPUT-files provided \n"
2197	fi
2198
2199	# NOW COPY FILES (.f90, .o, config files, etc.) FROM SOURCES_FOR_RUN_... TO THE TEMPORARY
2200	# WORKING DIRECTORY
2201	cp ${fast_io_catalog}/${sources_for_run_catalog}/{,.[!.]} $tempdir
2202
2203
2204	# EXECUTE INPUT-COMMANDS GIVEN IN THE CONFIGURATION FILE
2205	(( i = 0 ))
2206	while (( i < iic ))
2207	do
2208	(( i = i + 1 ))
2209	if (( i == 1 ))
2210	then
2211	printf "\n\n *** execution of INPUT-commands:\n$dashes"
2212	fi
2213	printf "\n >>> ${in_command[$i]}"
2214	eval ${in_command[$i]}
2215	if (( i == iic ))
2216	then
2217	printf "\n$dashes\n"
2218	fi
2219	done
2220
2221
2222	# CHECK IF THE PROGRESS BAR NEEDS TO BE DISABLED
2223	if [[ $running_in_batch_mode = true \|\| $running_in_test_mode = true ]]
2224	then
2225	progress_bar_disabled=true
2226	else
2227	progress_bar_disabled=false
2228	fi
2229
2230
2231	# CREATE THE NAMELIST-FILE WITH VALUES OF ENVIRONMENT-VARIABLES REQUIRED BY PALM
2232	# (FILE ENVPAR WILL BE READ BY PALM)
2233	cat > ENVPAR << EOF
2234	&envpar run_identifier = '$run_identifier', host = '$configuration_identifier',
2235	write_svf = .${write_svf}., write_binary = .${write_binary}.,
2236	read_svf = .${read_svf}., tasks_per_node = $tasks_per_node,
2237	maximum_parallel_io_streams = $maximum_parallel_io_streams,
2238	maximum_cpu_time_allowed = ${cpumax}.,
2239	revision = '$global_revision',
2240	progress_bar_disabled = .${progress_bar_disabled}. /
2241
2242	EOF
2243
2244
2245	# STARTING THE EXECUTABLE
2246	printf "\n\n *** execution starts in directory\n \"`pwd`\"\n$dashes\n"
2247	PATH=$PATH:$tempdir
2248
2249
2250	# REPLACE PARAMETERS IN THE EXECUTION COMMAND WITH REAL VALUES
2251	line=`echo "${execute_command}" \| sed 's/{{/$/g' \| sed 's/}}//g'`
2252	line2=`echo "${execute_command}" \| sed 's/{{mpi_tasks}}/1/g' \| sed 's/{{tasks_per_node}}/1/g' \| sed 's/palm/combine_plot_fields.x/g'`
2253	eval line=\"$line\"
2254	execute_command="$line"
2255
2256
2257	# EXECUTION COMMAND FOR COMBINE_PLOT_FIELDS
2258	if [[ "$execute_command_for_combine" = "" ]]
2259	then
2260	eval line2=\"$line2\"
2261	execute_command_for_combine="$line2"
2262	fi
2263
2264
2265
2266	# PROVIDE A HOSTFILE, IF REQUIRED
2267	if [[ "$hostfile" != "" ]]
2268	then
2269
2270	if [[ $hostfile = auto ]]
2271	then
2272	# CREATE A NEW HOSTFILE
2273	(( ii = 1 ))
2274	while (( ii <= cores / threads_per_task ))
2275	do
2276	echo $(hostname) >> hostfile
2277	(( ii = ii + 1 ))
2278	done
2279	if (( cores / threads_per_task == 0 ))
2280	then
2281	echo $(hostname) >> hostfile
2282	fi
2283
2284	else
2285	cp $hostfile hostfile
2286	fi
2287	eval line=\"`head -n $ii hostfile`\"
2288	printf "\n *** running on: $line"
2289	fi
2290
2291
2292
2293	# SET THE NUMBER OF OPENMP-THREADS
2294	if [[ $use_openmp = true ]]
2295	then
2296	export OMP_NUM_THREADS=$threads_per_task
2297	printf "\n *** number of OpenMP threads per MPI-task: $OMP_NUM_THREADS"
2298	else
2299	export OMP_NUM_THREADS=1
2300	fi
2301
2302
2303	# PROVIDE DATA FOR ATMOSPHERE OCEAN COUPLING
2304	if [[ $run_coupled_model = false ]]
2305	then
2306	if [[ "$ocean_file_appendix" = true ]]
2307	then
2308	echo "precursor_ocean" > coupling_steering
2309	else
2310	echo "precursor_atmos" > coupling_steering
2311	fi
2312	else
2313	(( iia = $cores_atmos / $threads_per_task ))
2314	(( iio = $cores_ocean / $threads_per_task ))
2315	printf "\n coupled run ($iia atmosphere, $iio ocean)"
2316	printf "\n\n"
2317	echo "coupled_run $iia $iio" > coupling_steering
2318	fi
2319
2320	printf "\n *** execute command:"
2321	printf "\n \"$execute_command\" \n\n"
2322
2323
2324	if [[ $progress_bar_disabled = true ]]
2325	then
2326	$execute_command < coupling_steering &> >(grep -v --line-buffered -e '^STOP 1$' -e '^1$' &> >(tee STDOUT) )
2327	exit_code=${PIPESTATUS[0]}
2328	else
2329	$execute_command < coupling_steering &> >(tee STDOUT)
2330	exit_code=${PIPESTATUS[0]}
2331	fi
2332
2333	if [[ ${exit_code} != 0 ]]
2334	then
2335
2336	# ABORT IN CASE OF RUNTIME ERRORS
2337	printf "\n +++ runtime error occured"
2338	locat=execution
2339	exit
2340
2341	else
2342
2343	printf "\n$dashes\n *** execution finished \n"
2344
2345	fi
2346
2347
2348	# CALL OF combine_plot_fields IN ORDER TO MERGE SINGLE FILES WRITTEN
2349	# BY EACH CORE INTO ONE FILE
2350	if [[ ! -f combine_plot_fields.x ]]
2351	then
2352
2353	printf "\n\n\n +++ WARNING: no combine_plot_fields found"
2354	printf "\n 2d- and/or 3d-data may be incomplete!"
2355	printf "\n Your previous palmbuild may have failed. Please check.\n"
2356
2357	elif [[ "$combine_plot_fields" == true ]]
2358	then
2359
2360	printf "\n\n\n *** post-processing: now executing \"$execute_command_for_combine\" ..."
2361	$execute_command_for_combine
2362
2363	else
2364
2365	# TEMPORARY SOLUTION TO SKIP combine_plot_fields. THIS IS REQUIRED IN CASE OF HUGE AMOUNT OF
2366	# DATA OUTPUT
2367	printf "\n\n\n *** post-processing: skipping combine_plot_fields (-Z option set) ..."
2368	fi
2369
2370
2371
2372	# EXECUTE OUTPUT-COMMANDS GIVEN IN THE CONFIGURATION FILE
2373	(( i = 0 ))
2374	while (( i < ioc ))
2375	do
2376	(( i = i + 1 ))
2377	if (( i == 1 ))
2378	then
2379	printf "\n\n *** execution of OUTPUT-commands:\n$dashes"
2380	fi
2381
2382	# REPLACE PARAMETERS IN THE OUTPUT COMMAND WITH REAL VALUES
2383	out_command[$i]=`echo "${out_command[$i]}" \| sed 's/{{/$/g' \| sed 's/}}//g'`
2384
2385	printf "\n >>> ${out_command[$i]}"
2386	eval ${out_command[$i]}
2387	if (( i == ioc ))
2388	then
2389	printf "\n$dashes\n"
2390	fi
2391	done
2392
2393
2394	# IN A FIRST PASS, ADD ADDITIONAL OUTPUT FILE CONNECTIONS IN CASE OF
2395	# WILDCARDS
2396	(( i = 0 ))
2397	(( nr_of_output_files = 0 ))
2398
2399	while (( i < iout ))
2400	do
2401
2402	(( i = i + 1 ))
2403
2404	# FIRST CHECK FOR MULTIPLE NAMES WITH THE SAME LOCAL NAME AND
2405	# CREATE A LIST FOR THE DETECTED ENDINGS
2406	if [[ "${multout[$i]}" = true ]]
2407	then
2408	# DETERMINE THE EXISTING EXTENSIONS FROM THE LIST OF FILES
2409	ls -1 -d ${localout_pre[$i]} > filelist 2>/dev/null
2410	ls -1 -d ${localout_pre[$i]}_* >> filelist 2>/dev/null
2411
2412	endings="DEFAULT"
2413	while read line
2414	do
2415	# remove the local name from the beginning
2416	localnamestring="${localout_pre[$i]}"
2417	length_localname=${#localnamestring}
2418	ending=${line:${length_localname}}
2419
2420	if [[ "$ending" != "" ]]
2421	then
2422	endings="$endings $ending"
2423	fi
2424
2425	done <filelist
2426
2427	rm filelist
2428
2429	else
2430
2431	# SINGLE NAME
2432	endings=DEFAULT
2433
2434	fi
2435
2436	# FOR EACH BASENAME ENDING CREATE AN ENTRY IN THE FINAL OUTPUT FILE LIST
2437	for ending in $endings
2438	do
2439
2440	# DEFAULT MEANS THAT THE ENDING GIVEN IN .iofiles SHALL BE USED
2441	if [[ $ending = DEFAULT ]]
2442	then
2443	ending=""
2444	fi
2445
2446	# NEW ENTRY (ENDING IS ALSO ADDED TO LOCAL FILENAME READ BY PALM!)
2447	(( nr_of_output_files = nr_of_output_files + 1 ))
2448	localout[$nr_of_output_files]="${localout_pre[$i]}"$ending
2449	transout[$nr_of_output_files]="${transout_pre[$i]}"
2450	actionout[$nr_of_output_files]="${actionout_pre[$i]}"
2451	pathout[$nr_of_output_files]="${pathout_pre[$i]}"
2452	endout[$nr_of_output_files]="${endout_pre[$i]}"$ending
2453	extout[$nr_of_output_files]="${extout_pre[$i]}"
2454	warnout[$nr_of_output_files]="${warnout_pre[$i]}"
2455
2456	done
2457
2458	done
2459
2460
2461
2462
2463	# COPY LOCAL OUTPUT-FILES TO THEIR PERMANENT DESTINATIONS
2464	(( i = 0 ))
2465	while (( i < nr_of_output_files ))
2466	do
2467	(( i = i + 1 ))
2468	if (( i == 1 ))
2469	then
2470	timestring=`date +%H:%M:%S`
2471	printf "\n\n *** saving OUTPUT-files: local time: $timestring"
2472
2473	# GET RUN NUMBER ASSIGNED BY PALM
2474	if [[ -f RUN_NUMBER ]]
2475	then
2476	read run_number < RUN_NUMBER
2477	printf "\n *** PALM generated run_number = "$run_number" will be used as unified cycle number for all output files"
2478	usecycle_option="-U $run_number"
2479	else
2480	run_number=0
2481	usecycle_option=""
2482	fi
2483	if [[ $running_on_remote = true && "$remote_loginnode" != "" ]]
2484	then
2485	printf "\n *** in case of SCP transfers to local host"
2486	printf "\n they will be done via remote login-node \"$remote_loginnode\" "
2487	fi
2488	printf "\n$dashes"
2489	fi
2490
2491	if [[ ! ( $running_on_remote = true && ( "${actionout[$i]}" = tr \|\| "${actionout[$i]}" = tra \|\| "${actionout[$i]}" = trpe ) ) ]]
2492	then
2493
2494	eval filename=${pathout[$i]}/${run_identifier}${endout[$i]}
2495
2496	# DETERMINE THE CYCLE NUMBER
2497	ls -1 -d $filename > filelist 2>/dev/null
2498	ls -1 -d $filename.* >> filelist 2>/dev/null
2499	(( maxcycle = 0 ))
2500	while read line
2501	do
2502
2503	# filename without path (i.e. after the last "/")
2504	basefilename=$(basename ${line})
2505
2506	# check if there is an extension
2507	extension=${basefilename##*.}
2508	if [[ "$extension" = "${extout[$i]}" ]]
2509	then
2510	basefilename=${basefilename%.*}
2511	fi
2512
2513	# check for an existing cycle number
2514	cycle=${basefilename##*.}
2515	if [[ $cycle =~ ^-?[0-9]+$ ]]
2516	then
2517	# NUMBERS WITH LEADING ZEROS ARE INTERPRETED AS OCTAL NUMBERS
2518	# 10# EXPLICITLY SPECIFIES THE NUMBER BASE AS 10
2519	(( icycle = $((10#$cycle)) + 1 ))
2520	else
2521	(( icycle = 1 ))
2522	fi
2523
2524	if (( icycle > maxcycle ))
2525	then
2526	(( maxcycle = icycle ))
2527	fi
2528
2529	done <filelist
2530	rm filelist
2531
2532
2533	# SET THE CYCLE NUMBER
2534	# IN CASE OF FILE-APPEND, IT MUST BE THE HIGHEST EXISTING CYCLE NUMBER
2535	if [[ "${actionout[$i]}" = a ]]
2536	then
2537	if (( maxcycle > 0 ))
2538	then
2539	(( maxcycle = maxcycle - 1 ))
2540	fi
2541	fi
2542
2543	(( cycnum[$i] = maxcycle ))
2544	pathout[$i]=$filename
2545
2546
2547	# ADD CYCLE NUMBER TO FILENAME
2548	# IN APPEND MODE, FILES KEEP THEIR CURRENT CYCLE NUMBER
2549	if [[ "${actionout[$i]}" != "a" ]]
2550	then
2551	# SET RUN NUMBER AS CYCLE NUMBER, IF THERE IS NOT A CONFLICT
2552	# WITH AN EXISTING CYCLE NUMBER
2553	if (( run_number >= cycnum[$i] ))
2554	then
2555	(( cycnum[$i] = run_number ))
2556	else
2557	if (( run_number > 0 ))
2558	then
2559	printf "\n --- INFORMATIVE: The following file cannot get a unified cycle number"
2560	fi
2561	fi
2562	fi
2563	cyclestring=`printf "%03d" ${cycnum[$i]}`
2564	pathout[$i]=${pathout[$i]}.$cyclestring
2565	fi
2566
2567	# CHECK FOR SINGLE FILE (SERIAL RUN) OR DIRECTORY (ONE FILE PER CORE FOR PARELLEL EXECUTION)
2568	files_for_cores=false; filetype=file
2569	link_local_output=false
2570	if [[ "${actionout[$i]}" = pe && -n $cores ]]
2571	then
2572	files_for_cores=true; filetype=directory
2573	actionout[$i]=""
2574	elif [[ "${actionout[$i]}" = pe && ! -n $cores ]]
2575	then
2576	actionout[$i]=""
2577	elif [[ "${actionout[$i]}" = lnpe && -n $cores ]]
2578	then
2579	files_for_cores=true; filetype=directory
2580	link_local_output=true
2581	actionout[$i]=""
2582	# FILE HAS BEEN FOUND INSTEAD OF DIRECTORY
2583	if [[ -f ${localout[$i]} ]]
2584	then
2585	files_for_cores=false; filetype=file
2586	fi
2587	elif [[ "${actionout[$i]}" = lnpe && ! -n $cores ]]
2588	then
2589	link_local_output=true
2590	actionout[$i]=""
2591	elif [[ "${actionout[$i]}" = trpe && -n $cores ]]
2592	then
2593	files_for_cores=true; filetype=directory
2594	actionout[$i]="tr"
2595	elif [[ "${actionout[$i]}" = trpe && ! -n $cores ]]
2596	then
2597	actionout[$i]="tr"
2598	fi
2599
2600	if [[ ! -f ${localout[$i]} && $files_for_cores = false ]]
2601	then
2602	if [[ ${warnout[$i]} = true ]]
2603	then
2604	printf "\n +++ temporary OUTPUT-file ${localout[$i]} does not exist\n"
2605	fi
2606	elif [[ ! -d ${localout[$i]} && $files_for_cores = true ]]
2607	then
2608	if [[ ${warnout[$i]} = true ]]
2609	then
2610	printf "\n +++ temporary OUTPUT-file ${localout[$i]}/.... does not exist\n"
2611	fi
2612	else
2613
2614
2615	# COPY VIA SCP TO LOCAL HOST (ALWAYS IN BINARY MODE USING batch_scp option -m)
2616	# IF TARGET DIRECTORY DOES NOT EXISTS, TRY TO CREATE IT
2617	if [[ "${actionout[$i]}" = tr \|\| "${actionout[$i]}" = tra ]]
2618	then
2619	if [[ $running_on_remote = true ]]
2620	then
2621
2622	# SET OPTIONS FOR TRANSFER
2623	if [[ "${actionout[$i]}" = tr ]]
2624	then
2625	if [[ $files_for_cores = false ]]
2626	then
2627	catalog_option=""
2628	catalog_string=""
2629	else
2630	catalog_option="-c"
2631	catalog_string="/"
2632	fi
2633	append_option=""
2634	append_string=""
2635	else
2636	append_option="-A"
2637	append_string="append"
2638	fi
2639
2640	transfer_failed=false
2641	printf "\n >>> OUTPUT: ${localout[$i]}$catalog_string $append_string by SCP to"
2642	printf "\n ${pathout[$i]}/${configuration_identifier}_${run_identifier}${endout[$i]}$catalog_string\n"
2643
2644	# TRANSFER VIA SCP
2645	if [[ "$remote_loginnode" != "" ]]
2646	then
2647	ssh -q $remote_username@$remote_loginnode "cd $tempdir; ${fast_io_catalog}/${sources_for_run_catalog}/batch_scp $PORTOPT $catalog_option $append_option -b -m $usecycle_option -u $local_username $return_address ${localout[$i]} \"${pathout[$i]}\" ${configuration_identifier}_${run_identifier}${endout[$i]} ${extout[$i]}"
2648	else
2649	batch_scp $PORTOPT $catalog_option $append_option -b -m $usecycle_option -u $local_username $return_address ${localout[$i]} "${pathout[$i]}" ${configuration_identifier}_${run_identifier}${endout[$i]} ${extout[$i]}
2650	fi
2651	[[ ${PIPESTATUS[0]} != 0 ]] && transfer_failed=true
2652
2653
2654	# IF TRANSFER FAILED, CREATE BACKUP COPY ON THIS MACHINE
2655	if [[ $transfer_failed = true ]]
2656	then
2657	printf " +++ transfer failed. Trying to save a copy on this host under:\n"
2658	printf " ${pathout[$i]}/${configuration_identifier}_${run_identifier}${endout[$i]}_$run_id_number\n"
2659
2660	# FIRST CHECK, IF DIRECTORY EXISTS, AND CREATE IT, IF NECESSARY
2661	eval local_catalog=${pathout[$i]}
2662	if [[ ! -d $local_catalog ]]
2663	then
2664	printf " *** local directory does not exist. Trying to create:\n"
2665	printf " $local_catalog \n"
2666	mkdir -p $local_catalog
2667	fi
2668	eval cp ${localout[$i]} ${pathout[$i]}/${configuration_identifier}_${run_identifier}${endout[$i]}_$run_id_number
2669	transfer_problems=true
2670	fi
2671
2672	else
2673
2674	# UNSET actionout. DUE TO THIS SETTING, FILE WILL LATER JUST
2675	# BE COPIED OR APPENDED ON THIS MACHINE
2676	if [[ "${actionout[$i]}" = tr ]]
2677	then
2678	actionout[$i]=""
2679	else
2680	actionout[$i]="a"
2681	fi
2682	fi
2683	fi
2684
2685
2686	# APPEND ON THIS MACHINE
2687	if [[ "${actionout[$i]}" = "a" ]]
2688	then
2689	if [[ "${extout[$i]}" != " " && "${extout[$i]}" != "" ]]
2690	then
2691	printf "\n >>> OUTPUT: ${localout[$i]} append to"
2692	printf "\n ${pathout[$i]}.${extout[$i]}\n"
2693	cat ${localout[$i]} >> ${pathout[$i]}.${extout[$i]}
2694	else
2695	printf "\n >>> OUTPUT: ${localout[$i]} append to"
2696	printf "\n ${pathout[$i]}\n"
2697	cat ${localout[$i]} >> ${pathout[$i]}
2698	fi
2699	fi
2700
2701	# COPY ON THIS MACHINE
2702	# COPY HAS TO BE USED, BECAUSE MOVE DOES NOT WORK IF FILE-ORIGIN AND TARGET ARE
2703	# ON DIFFERENT FILE-SYSTEMS
2704	if [[ "${actionout[$i]}" = "" && $files_for_cores = false ]]
2705	then
2706
2707	# COPY IN CASE OF RUNS ON SINGLE CORES
2708	if [[ "${extout[$i]}" != " " && "${extout[$i]}" != "" ]]
2709	then
2710	printf "\n >>> OUTPUT: ${localout[$i]} to"
2711	printf "\n ${pathout[$i]}.${extout[$i]}\n"
2712	if [[ $link_local_output = true ]]
2713	then
2714	printf " file will be linked\n"
2715	ln -f ${localout[$i]} ${pathout[$i]}.${extout[$i]}
2716	fi
2717	# If "ln -f" fails of if "$link_local_output = false" do a normal "cp"
2718	if [[ ! -f "${pathout[$i]}.${extout[$i]}" ]]
2719	then
2720	if [[ $link_local_output = true ]]
2721	then
2722	printf " --- WARNING: ln failed, using cp instead (might be time consuming...)\n"
2723	fi
2724	cp ${localout[$i]} ${pathout[$i]}.${extout[$i]}
2725	fi
2726	else
2727	printf "\n >>> OUTPUT: ${localout[$i]} to"
2728	printf "\n ${pathout[$i]}\n"
2729	if [[ $link_local_output = true ]]
2730	then
2731	printf " file will be linked\n"
2732	ln -f ${localout[$i]} ${pathout[$i]}
2733	fi
2734	# If "ln -f" fails of if "$link_local_output = false" do a normal "cp"
2735	if [[ ! -f "${pathout[$i]}" ]]
2736	then
2737	if [[ $link_local_output = true ]]
2738	then
2739	printf " --- WARNING: ln failed, using cp instead (might be time consuming...)\n"
2740	fi
2741	cp ${localout[$i]} ${pathout[$i]}
2742	fi
2743	fi
2744
2745	elif [[ "${actionout[$i]}" = "" && $files_for_cores = true ]]
2746	then
2747
2748	# FILES FROM THE DIFFERENT CORES ARE MOVED WITH ln-COMMAND TO THE PERMANENT DIRECTORY
2749	# AS A FIRST STEP, THE PERMANENT DIRECTORY IS CREATED
2750	printf "\n >>> OUTPUT: ${localout[$i]}/_.... to"
2751	printf "\n ${pathout[$i]}\n"
2752	if [[ $link_local_output = true ]]
2753	then
2754	printf " files will be linked\n"
2755	mkdir -p ${pathout[$i]}
2756	cd ${localout[$i]}
2757	for file in $(ls *)
2758	do
2759	ln -f $file ${pathout[$i]}
2760	done >\|/dev/null 2>&1
2761	cd $tempdir
2762	fi
2763
2764	# IF "ln -f" HAS FAILED OR IF "$link_local_output = false" DO A NORMAL COPY "cp -r"
2765	if [[ ! -f "${pathout[$i]}/_000000" ]]
2766	then
2767	if [[ $link_local_output = true ]]
2768	then
2769	printf " --- WARNING: ln failed, using cp instead (might be time consuming...)\n"
2770	fi
2771	[[ ! -d "${pathout[$i]}" ]] && mkdir ${pathout[$i]}
2772	cp -r ${localout[$i]}/* ${pathout[$i]}
2773	fi
2774
2775	fi
2776	fi
2777	done
2778
2779	if (( i != 0 ))
2780	then
2781	timestring=`date +%H:%M:%S`
2782	if [[ $transfer_problems = true ]]
2783	then
2784	printf "\n$dashes\n *** OUTPUT-files saved local time: $timestring"
2785	printf "\n +++ WARNING: some data transfers failed! \n"
2786	else
2787	printf "\n$dashes\n *** all OUTPUT-files saved local time: $timestring \n"
2788	fi
2789	fi
2790
2791
2792	# IF REQUIRED, START A RESTART-JOB
2793	# FILE CONTINUE_RUN MUST HAVE BEEN CREATED BY THE EXECUTABLE (PALM)
2794	if [[ -f CONTINUE_RUN ]]
2795	then
2796
2797	# ADD RESTART-OPTIONS TO THE PALMRUN-CALL (IF THEY ARE NOT USED ALREADY):
2798	# -C TELLS PALMRUN THAT IT IS A RESTART-RUN
2799	# -v SILENT MODE WITHOUT INTERACTIVE QUERIES
2800	# -b START A BATCH JOB
2801	[[ $(echo $prc \| grep -c "\-C") = 0 ]] && prc="$prc -C"
2802	[[ $(echo $prc \| grep -c "\-v") = 0 ]] && prc="$prc -v"
2803	[[ $(echo $prc \| grep -c "\-b") = 0 && $running_on_remote = false ]] && prc="$prc -b"
2804
2805	# REPLACE SVFOUT IN THE ACTIVATION STRINGS (GIVEN WITH OPTION -a)
2806	# SO THAT RESTARTS USE SVF DATA CREATED BY THE INITIAL RUN
2807	if [[ $(echo $prc \| grep -c "svfout") != 0 ]]
2808	then
2809	prc=`echo $prc \| sed 's/svfout/svfin/g'`
2810	fi
2811
2812	# REPLACE THE HASH IN THE ACTIVATION STRINGS (GIVEN WITH OPTION -a)
2813	# SO THAT RESTARTS ACCESS DIFFERENT FILES THAN THE INITIAL RUN
2814	if [[ $(echo $prc \| grep -c "#") != 0 ]]
2815	then
2816	prc=`echo $prc \| sed 's/#/r/g'`
2817	fi
2818
2819
2820	# START THE RESTART-JOB
2821	printf "\n\n *** initiating restart-run on \"$local_ip\" using command:\n"
2822	echo " $prc"
2823	printf "\n$dashes\n"
2824	if [[ $running_on_remote = true ]]
2825	then
2826
2827	echo "*** ssh will be used to initiate restart-runs!"
2828	echo " return_address=\"$return_address\" "
2829	echo " return_username=\"$local_username\" "
2830
2831	if [[ "$remote_loginnode" != "" ]]
2832	then
2833	ssh -q $remote_username@$remote_loginnode "ssh -q $SSH_PORTOPT $local_username@$return_address \" PATH=\\\$PATH:$LOCAL_PALMRUN_PATH; cd $LOCAL_PWD; $prc\" " \| tee palmrun_restart.log
2834	else
2835	ssh -q $SSH_PORTOPT $local_username@$return_address " PATH=\\\$PATH:$LOCAL_PALMRUN_PATH; cd $LOCAL_PWD; $prc" \| tee palmrun_restart.log
2836	fi
2837
2838	# WAIT TO ALLOW THE RESTART-JOB TO BE QUEUED, BEFORE THE CURRENT JOB IS FINISHED
2839	sleep 30
2840
2841	else
2842
2843	# START THE RESTART JOB ON THE LOCAL HOST
2844	eval $prc \| tee palmrun_restart.log # THE ' MUST BE EVALUATED
2845
2846	fi
2847
2848	# CHECK, IF RESTART JOB HAS BEEN STARTED
2849	if [[ $( grep -c "+++ palmrun crashed" palmrun_restart.log ) != 0 ]]
2850	then
2851	printf "\n$dashes\n +++ creating restart run failed \n"
2852	locat=create_restart
2853	rm palmrun_restart.log
2854	exit
2855	elif [[ $( grep -c "*** palmrun finished" palmrun_restart.log ) != 1 ]]
2856	then
2857	printf "\n$dashes\n +++ creating restart run failed, probably due to network problems\n"
2858	locat=create_restart
2859	rm palmrun_restart.log
2860	exit
2861	else
2862	printf "\n$dashes\n *** restart run initiated \n"
2863	rm palmrun_restart.log
2864	fi
2865
2866
2867	# DELETE INPUT-(RESTART)FILES, WHICH HAVE BEEN FETCHED FROM THE TEMPORARY DATA
2868	# DIRECTORY, BACAUSE THEY ARE NOT REQUIRED BY THE RESTART-JOB.
2869	# THIS IS DONE IN ORDER TO AVOID EXCEEDING DISC QUOTAS OR DISC SPACE (RESTART-FILES
2870	# MAY BE VERY HUGE)
2871	(( i = 0 ))
2872	while (( i < nr_of_input_files ))
2873	do
2874	(( i = i + 1 ))
2875	if [[ "${got_tmp[$i]}" = true && $keep_data_from_previous_run = false ]]
2876	then
2877	rm -r ${absnamein[$i]}
2878	fi
2879	done
2880
2881	fi
2882
2883
2884	# ALL ACTIONS FINISHED, TEMPORARY WORKING-DIRECTORY CAN BE DELETED
2885	cd $HOME
2886	[[ $delete_temporary_catalog = true ]] && rm -rf $tempdir
2887
2888
2889	else
2890
2891
2892	# PREPARING ACTIONS,
2893	# IF A BATCH-JOB IS TO BE GENERATED AND TO BE STARTED ON A LOCAL OR REMOTE-MACHINE
2894
2895	# BUILD THE PALMRUN-COMMAND TO BE CALLED IN THE BATCH-JOB
2896	palmrun_com="$palmrun_script_name -r $run_identifier -c $configuration_identifier -m $memory -t $cpumax -q $queue -i $run_id_number -U $local_username"
2897	[[ "$activation_string_list" != "" ]] && palmrun_com=${palmrun_com}" -a \"$activation_string_list\""
2898	[[ "$global_revision" != "" ]] && palmrun_com=${palmrun_com}" -G \"$global_revision\""
2899	[[ $keep_data_from_previous_run = true ]] && palmrun_com=${palmrun_com}" -k"
2900	[[ $do_trace = true ]] && palmrun_com=${palmrun_com}" -x"
2901	[[ "$cores" != "" ]] && palmrun_com=${palmrun_com}" -X $cores"
2902	[[ $use_openmp = true ]] && palmrun_com=${palmrun_com}" -O $threads_per_task"
2903	[[ $tasks_per_node != 0 ]] && palmrun_com=${palmrun_com}" -T $tasks_per_node"
2904	[[ $delete_temporary_catalog = false ]] && palmrun_com=${palmrun_com}" -B"
2905	[[ "$ocean_file_appendix" = true ]] && palmrun_com=${palmrun_com}" -y"
2906	[[ $run_coupled_model = true ]] && palmrun_com=${palmrun_com}" -Y \"$coupled_dist\""
2907	[[ "$combine_plot_fields" = false ]] && palmrun_com=${palmrun_com}" -Z"
2908	[[ "$max_par_io_str" != "" ]] && palmrun_com=${palmrun_com}" -w $max_par_io_str"
2909	[[ "$project_account" != "" ]] && palmrun_com=${palmrun_com}" -A $project_account"
2910	if [[ $create_remote_batch_job = true ]]
2911	then
2912	palmrun_com=${palmrun_com}" -j -u $remote_username -R $local_ip"
2913	if [[ $do_trace = true ]]
2914	then
2915	printf "\n *** PALMRUN-command on remote host:\n $palmrun_com \n"
2916	fi
2917	elif [[ $create_batch_job = true ]]
2918	then
2919	palmrun_com=${palmrun_com}" -j"
2920	if [[ $do_trace = true ]]
2921	then
2922	printf "\n *** PALMRUN-command on local host:\n $palmrun_com \n"
2923	fi
2924	fi
2925
2926
2927	# DETERMINE THE FULL PATHS FOR THE JOB PROTOCOL FILES ON THE LOCAL AND
2928	# REMOTE HOST
2929	job_protocol_file_local=${local_jobcatalog}/${configuration_identifier}_${run_id}
2930	job_protocol_file=$job_protocol_file_local
2931	if [[ $create_remote_batch_job = true ]]
2932	then
2933	job_protocol_file_remote=${remote_jobcatalog}/${configuration_identifier}_${run_id}
2934	job_protocol_file=$job_protocol_file_remote
2935	job_transfer_protocol_file=${remote_jobcatalog}/last_job_transfer_protocol
2936	scpjob_file=${remote_jobcatalog}/scpjob.$run_id_number
2937	fi
2938
2939
2940	# BUILD THE JOB-SCRIPTS ON FILE jobfile
2941	jobfile=jobfile.$run_id_number
2942
2943
2944	# FIRST CREATE THE BATCH DIRECTIVES
2945	(( i = 0 ))
2946	while (( i < ibd ))
2947	do
2948	(( i = i + 1 ))
2949	line=`echo "${batch_directive[$i]}" \| sed 's/{{/$/g' \| sed 's/}}//g'`
2950	eval line=\"$line\"
2951	echo "$line" >> $jobfile
2952	done
2953	echo " " >> $jobfile
2954
2955
2956	# FOR BATCH JOBS ON REMOTE HOSTS, ADD THE JOBFILE TO SEND BACK THE JOB
2957	# PROTOCOL
2958	if [[ $create_remote_batch_job = true ]]
2959	then
2960	echo "set +vx" >> $jobfile
2961	echo "trap '" >> $jobfile
2962	echo "set +vx" >> $jobfile
2963	echo "cd ${remote_jobcatalog}" >> $jobfile
2964	echo "cat > scpjob.$run_id_number << %%END%%" >> $jobfile
2965
2966	# ADD THE BATCH DIRECTIVES
2967	(( i = 0 ))
2968	while (( i < ibdt ))
2969	do
2970	(( i = i + 1 ))
2971	line=`echo "${batch_directive_transfer[$i]}" \| sed 's/{{/$/g' \| sed 's/}}//g'`
2972	eval line=\"$line\"
2973	echo "$line" >> $jobfile
2974	done
2975	echo " " >> $jobfile
2976
2977	echo "set -x" >> $jobfile
2978	if [[ "$remote_loginnode" != "" ]]
2979	then
2980	echo "ssh -q $remote_username@$remote_loginnode \"cd ${remote_jobcatalog}; ${fast_io_catalog}/${sources_for_run_catalog}/batch_scp $PORTOPT -d -w 10 -u $local_username $local_ip $job_protocol_file_remote \\\"$local_jobcatalog\\\" ${configuration_identifier}_${run_identifier}\" " >> $jobfile
2981	else
2982	echo "${fast_io_catalog}/${sources_for_run_catalog}/batch_scp $PORTOPT -d -w 10 -u $local_username $local_ip $job_protocol_file_remote \"$local_jobcatalog\" ${configuration_identifier}_${run_identifier}" >> $jobfile
2983	fi
2984	echo "%%END%%" >> $jobfile
2985	echo "echo \" *** submitting job for transfering the job protocol file to $local_ip\" " >> $jobfile
2986	echo "$submit_command $scpjob_file" >> $jobfile
2987	echo "rm $scpjob_file" >> $jobfile
2988	echo "rm -rf $job_transfer_protocol_file" >> $jobfile
2989	echo "set -x" >> $jobfile
2990	echo " ' exit" >> $jobfile
2991	fi
2992
2993
2994	# ACTIVATE ERROR-TRACEBACK
2995	if [[ $do_trace = true ]]
2996	then
2997	echo "set -x" >> $jobfile
2998	else
2999	echo "set +vx" >> $jobfile
3000	fi
3001
3002
3003	# INITIALIZE THE ENVIRONMENT AND LOAD MODULES
3004	if [[ "$login_init_cmd" != "" ]]
3005	then
3006	echo "$login_init_cmd" >> $jobfile
3007	fi
3008	if [[ "$module_commands" != "" ]]
3009	then
3010	echo "$module_commands" >> $jobfile
3011	fi
3012
3013
3014	# CREATE TEMPORARY DIRECTORY AND SWITCH TO IT
3015	if [[ $create_remote_batch_job = true ]]
3016	then
3017	echo "mkdir $tempdir" >> $jobfile
3018	echo "if [[ \$? != 0 ]]" >> $jobfile
3019	echo "then" >> $jobfile
3020	echo " echo \"+++ temporary working directory cannot be created.\" " >> $jobfile
3021	echo " echo \" Check setting of variable fast_io_catalog in your config file.\" " >> $jobfile
3022	echo " exit" >> $jobfile
3023	echo "fi" >> $jobfile
3024	echo "chmod go+rx $tempdir" >> $jobfile
3025	else
3026	# DIRECTORY FOR LOCAL BATCH JOBS IS CREATED NOW, DUE TO A
3027	# REQUIREMENT OF THE GRID ENGINE BATCH SYSTEM (WORKING DIR IS GIVEN IN
3028	# BATCH DIRECTIVE -wd AND MUST ALREADY EXIST WHEN THE JOB IS SUBMITTED)
3029	mkdir $tempdir
3030	if [[ $? != 0 ]]
3031	then
3032	printf "\n +++ temporary working directory cannot be created."
3033	printf "\n Check setting of variable fast_io_catalog in your config file."
3034	locat=execution
3035	exit
3036	fi
3037	chmod go+rx $tempdir
3038	fi
3039	echo "cd $tempdir" >> $jobfile
3040	echo "export tempdir=$tempdir" >> $jobfile
3041	echo "cp ${fast_io_catalog}/${sources_for_run_catalog}/.[!.]* ." >> $jobfile
3042	echo "export PATH=.:${fast_io_catalog}/${sources_for_run_catalog}:\$PATH" >> $jobfile
3043	echo "export execute_palmrun=true" >> $jobfile
3044
3045
3046	# PROVIDE NAME OF THE CURRENT WORKING-DIRECTORY ON THE LOCAL MACHINE (FROM WHERE THE JOB IS
3047	# STARTED) BY SETTING AN ENVIRONMENT-VARIABLE. THIS INFORMATION IS USED IN THE JOB BY PALMRUN
3048	# IN CASE THAT RESTART-RUNS HAVE TO BE GENERATED
3049	echo "LOCAL_PWD=$working_directory" >> $jobfile
3050	echo "export LOCAL_PWD" >> $jobfile
3051
3052
3053	# PROVIDE THE PATH OF THE LOCAL PALMRUN-SCRIPT FOR THE SAME REASON
3054	echo "LOCAL_PALMRUN_PATH=${source_path}/../SCRIPTS" >> $jobfile
3055	echo "export LOCAL_PALMRUN_PATH" >> $jobfile
3056
3057
3058	# CALL PALMRUN WITHIN THE JOB
3059	# AS FINAL ACTION, REMOVE THE TEMPORARY DIRECTORY CREATED AT THE BEGINNING OF THE JOB
3060	echo "set -x" >> $jobfile
3061	echo "[[ \$execute_palmrun = true ]] && $palmrun_com" >> $jobfile
3062
3063
3064	# TRANSFER JOBFILE TO THE TARGET HOST
3065	if [[ $create_jobfile_only = false ]]
3066	then
3067
3068	# CREATE LOCAL JOB FOLDER
3069	eval local_jobcatalog=$local_jobcatalog
3070	if [[ ! -d $local_jobcatalog ]]
3071	then
3072	echo " "
3073	echo " *** local jobcatalog \"$local_jobcatalog\" does not exist and will be created now"
3074	mkdir $local_jobcatalog
3075	fi
3076
3077	if [[ $create_remote_batch_job = true ]]
3078	then
3079
3080	echo " "
3081	echo " *** transfer of job to remote host via scp"
3082
3083	# CREATE REMOTE JOB FOLDER, IF IT DOES NOT EXIST
3084	ssh -q $ssh_key $SSH_PORTOPT ${remote_username}@${remote_ip} "[[ ! -d $remote_jobcatalog ]] && mkdir $remote_jobcatalog" 2>&1
3085
3086	# COPY THE JOB FILE
3087	if [[ $do_trace = true ]]
3088	then
3089	echo " scp $ssh_key $PORTOPT $jobfile ${remote_username}@${remote_ip}:${remote_jobcatalog}/${configuration_identifier}_${run_id}"
3090	fi
3091	scp $ssh_key $PORTOPT $jobfile ${remote_username}@${remote_ip}:${remote_jobcatalog}/${configuration_identifier}_${run_id} > /dev/null
3092
3093	# SUBMIT THE JOB
3094	printf " *** submit the job (output of submit command, \n"
3095	printf " e.g. the job-id given by the batch-system, may follow) \n"
3096	if [[ $do_trace = true ]]
3097	then
3098	echo " ssh -q $ssh_key $SSH_PORTOPT ${remote_username}@${remote_ip} \"cd $remote_jobcatalog; $submit_command ${configuration_identifier}_${run_id}; rm ${configuration_identifier}_${run_id}\" "
3099	fi
3100	ssh -q $ssh_key $SSH_PORTOPT ${remote_username}@${remote_ip} "cd $remote_jobcatalog; $submit_command ${configuration_identifier}_${run_id}; rm ${configuration_identifier}_${run_id}"
3101
3102	elif [[ $create_batch_job = true ]]
3103	then
3104
3105	eval local_jobcatalog=$local_jobcatalog
3106	cp $jobfile ${local_jobcatalog}/${configuration_identifier}_${run_id}
3107	cd $local_jobcatalog
3108	echo " "
3109	echo " *** submit the job"
3110	if [[ $do_trace = true ]]
3111	then
3112	echo "$submit_command ${configuration_identifier}_${run_id}"
3113	fi
3114	$submit_command ${configuration_identifier}_${run_id}
3115	rm ${configuration_identifier}_${run_id}
3116	cd - > /dev/null
3117
3118	fi
3119
3120	rm -rf $jobfile
3121
3122	else
3123
3124	printf "\n *** jobfile created under name \"$jobfile\" "
3125	printf "\n no batch-job has been sent!"
3126
3127	fi
3128
3129	fi # END OF REMOTE-PART

Note: See TracBrowser for help on using the repository browser.

Download in other formats:

| Impressum | ©Leibniz Universität Hannover |