NAME
qsub - submit a batch job to Grid Engine.
qsh - submit an interactive X-windows session to Grid
Engine.
qlogin - submit an interactive login session to Grid
Engine.
qrsh - submit an interactive rsh session to Grid Engine.
qalter - modify a pending batch job of Grid Engine.
qresub - submit a copy of an existing Grid Engine job.
SYNTAX
qsub [ options ] [ command | -- [ command_args ]]
qsh [ options ] [ -- xterm_args ]
qlogin [ options ]
qrsh [ options ] [ command [ command_args ]]
qalter [ options ] wc_job_range_list [ -- [ command_args ]]
qalter [ options ] -u user_list | -uall [ -- [ command_args
]]
qresub [ options ] job_id_list
DESCRIPTION
Qsub submits batch jobs to the Grid Engine queuing system.
Grid Engine supports single- and multiple-node jobs. Command
can be a path to a binary or a script (see -b below) which
contains the commands to be run by the job using a shell
(for example, sh(1) or csh(1)). Arguments to the command
are given as command_args to qsub . If command is handled as
a script then it is possible to embed flags in the script.
If the first two characters of a script line either match
'#$' or are equal to the prefix string defined with the -C
option described below, the line is parsed for embedded com-
mand flags.
Qsh submits an interactive X-windows session to Grid Engine.
An xterm(1) is brought up from the executing machine with
the display directed either to the X-server indicated by the
DISPLAY environment variable or as specified with the
-display qsh option. Interactive jobs are not spooled if no
resource is available to execute them. They are either
dispatched to a suitable machine for execution immediately
or the user submitting the job is notified by qsh that
appropriate resources to execute the job are not available.
xterm_args are passed to the xterm(1) executable.
Qlogin is similar to qsh in that it submits an interactive
job to the queueing system. It does not open an xterm(1)
window on the X display, but uses the current terminal for
user I/O. Usually, qlogin establishes a telnet(1) connection
with the remote host, using standard client- and server-side
commands. These commands can be configured with the
qlogin_daemon (server-side, Grid Engine telnetd if not set,
otherwise something like /usr/sbin/in.telnetd) and
qlogin_command (client-side, Grid Engine telnet if not set,
otherwise something like /usr/bin/telnet) parameters in the
global and local configuration settings of sge_conf(5). The
client side command is automatically parameterized with the
remote host name and port number to which to connect,
resulting in an invocation like
/usr/bin/telnet my_exec_host 2442
for example. Qlogin is invoked exactly like qsh and its
jobs can only run on INTERACTIVE queues. Qlogin jobs can
only be used if the sge_execd(8) is running under the root
account.
Qrsh is similar to qlogin in that it submits an interactive
job to the queuing system. It uses the current terminal for
user I/O. Usually, qrsh establishes a rsh(1) connection with
the remote host. If no command is given to qrsh, an rlo-
gin(1) session is established. The server-side commands
used can be configured with the rsh_daemon and rlogin_daemon
parameters in the global and local configuration settings of
sge_conf(5). An Grid Engine rshd or rlogind is used if the
parameters are not set. If the parameters are set, they
should be set to something like /usr/sbin/in.rshd or
/usr/sbin/in.rlogind. On the client-side, the rsh_command
and rlogin_command parameters can be set in the global and
local configuration settings of sge_conf(5). If they are
not set, special Grid Engine rsh(1) and rlogin(1) binaries
delivered with Grid Engine are used. Use the cluster confi-
guration parameters to integrate mechanisms like ssh or the
rsh(1) and rlogin(1) facilities supplied with the operating
system.
Qrsh jobs can only run in INTERACTIVE queues unless the
option -now no is used (see below). They can also only be
run, if the sge_execd(8) is running under the root account.
Qrsh provides an additional useful feature for integrating
with interactive tools providing a specific command shell.
If the environment variable QRSH_WRAPPER is set when qrsh is
invoked, the command interpreter pointed to by QRSH_WRAPPER
will be executed to run qrsh commands instead of the users
login shell or any shell specified in the qrsh command-line.
Qalter can be used to change the attributes of pending jobs.
Once a job is executing, changes are no longer possible. For
array jobs, for which a part of the tasks can be pending and
another part can be running (see the -t option below),
modifications with qalter only affect the pending tasks.
Qalter can change most of the characteristics of a job (see
the corresponding statements in the OPTIONS section below),
including those which were defined as embedded flags in the
script file (see above).
Qresub allows the user to create jobs as copies of existing
pending or running jobs. The copied jobs will have exactly
the same attributes as the ones from which they were copied,
except for a new job ID. The only modification to the copied
jobs supported by qresub is assignment of a hold state with
the -h option. This can be used to first copy a job and then
change its attributes via qalter.
For qsub, qsh, qrsh, and qlogin the administrator and the
user may define default request files (see sge_request(5))
which can contain any of the options described below. If an
option in a default request file is understood by qsub and
qlogin but not by qsh the option is silently ignored if qsh
is invoked. Thus you can maintain shared default request
files for both qsub and qsh.
A cluster wide default request file may be placed under
$SGE_ROOT/$SGE_CELL/common/sge_request. User private
default request files are processed under the locations
$HOME/.sge_request and $cwd/.sge_request. The working direc-
tory local default request file has the highest precedence,
then the home directory located file and then the cluster
global file. The option arguments, the embedded script
flags and the options in the default request files are pro-
cessed in the following order:
left to right in the script line,
left to right in the default request files,
from top to bottom of the script file (qsub only),
from top to bottom of default request files,
from left to right of the command line.
In other words, the command line can be used to override the
embedded flags and the default request settings. The embed-
ded flags, however, will override the default settings.
Note, that the -clear option can be used to discard any pre-
vious settings at any time in a default request file, in the
embedded script flags, or in a command-line option. It is,
however, not available with qalter.
The options described below can be requested either hard or
soft. By default, all requests are considered hard until
the -soft option (see below) is encountered. The hard/soft
status remains in effect until its counterpart is encoun-
tered again. If all the hard requests for a job cannot be
met, the job will not be scheduled. Jobs which cannot be
run at the present time remain spooled.
OPTIONS
-@ optionfile
Forces qsub, qrsh, qsh, or qlogin to use the options
contained in optionfile. The indicated file may contain
all valid options. Comment lines must start with a "#"
sign.
-a date_time
Available for qsub and qalter only.
Defines or redefines the time and date at which a job
is eligible for execution. Date_time conforms to
[[CC]]YY]MMDDhhmm.[ss], where:
CC denotes the century in 2 digits.
YY denotes the year in 2 digits.
MM denotes the month in 2 digits.
DD denotes the day in 2 digits.
hh denotes the hour in 2 digits.
mm denotes the minute in 2 digits.
ss denotes the seconds in 2 digits (default 00).
If any of the optional date fields is omitted, the
corresponding value of the current date is assumed. If
CC is not specified, a YY of < 70 means 20YY.
Usage of this option may cause unexpected results if
the clocks of the hosts in the Grid Engine pool are out
of sync. Also, the proper behavior of this option very
much depends on the correct setting of the appropriate
timezone, e.g. in the TZ environment variable (see
date(1) for details), when the Grid Engine daemons
sge_qmaster(8) and sge_execd(8) are invoked.
Qalter allows changing this option even while the job
executes. The modified parameter will only be in effect
after a restart or migration of the job, however.
-ac variable[=value],...
Available for qsub, qsh, qrsh, qlogin and qalter only.
Adds the given name/value pair(s) to the job's context.
Value may be omitted. Grid Engine appends the given
argument to the list of context variables for the job.
Multiple -ac, -dc, and -sc options may be given. The
order is important here.
Qalter allows changing this option even while the job
executes.
-A account_string
Available for qsub, qsh, qrsh, qlogin and qalter only.
Identifies the account to which the resource consump-
tion of the job should be charged. The account_string
may be any arbitrary ASCII alphanumeric string but may
not contain "\n", "\t", "\r", "/", ":", "@", "\", "*",
or "?". In the absence of this parameter Grid Engine
will place the default account string "sge" in the
accounting record of the job.
Qalter allows changing this option even while the job
executes.
-b y[es]|n[o]
Available for qsub, qrsh only. Qalter does not allow
changing this option. This option cannot be embedded in
the script file itself.
Gives the user the possibility to indicate explicitly
whether command should be treated as binary or script.
If the value of -b is 'y', then command may be a
binary or script. The command might not be accessible
from the submission host. Nothing except the path of
the command will be transferred from the submission
host to the execution host. Path aliasing will be
applied to the path of command before command will be
executed.
If the value of -b is 'n' then command needs to be a
script and it will be handled as script. The script
file has to be accessible by the submission host. It
will be transferred to the execution host. qsub/qrsh
will search directive prefixes within script.
qsub will implicitly use -b n whereas qrsh will apply
the -b y option if nothing else is specified.
-c occasion_specifier
Available for qsub and qalter only.
Defines or redefines whether the job should be check-
pointed, and if so, under what circumstances. The
specification of the checkpointing occasions with this
option overwrites the definitions of the when parameter
in the checkpointing environment (see checkpoint(5))
referenced by the qsub -ckpt switch. Possible values
for occasion_specifier are
n no checkpoint is performed.
s checkpoint when batch server is shut down.
m checkpoint at minimum CPU interval.
x checkpoint when job gets suspended.
<interval> checkpoint in the specified time interval.
The minimum CPU interval is defined in the queue confi-
guration (see queue_conf(5) for details). <interval>
has to be specified in the format hh:mm:ss. The max-
imum of <interval> and the queue's minimum CPU interval
is used if <interval> is specified. This is done to
ensure that a machine is not overloaded by checkpoints
being generated too frequently.
-ckpt ckpt_name
Available for qsub and qalter only.
Selects the checkpointing environment (see check-
point(5)) to be used for checkpointing the job. Also
declares the job to be a checkpointing job.
-clear
Available for qsub, qsh, qrsh, and qlogin only.
Causes all elements of the job to be reset to the ini-
tial default status prior to applying any modifications
(if any) appearing in this specific command.
-cwd Available for qsub, qsh, qrsh and qalter only.
Execute the job from the current working directory.
This switch will activate Grid Engine's path aliasing
facility, if the corresponding configuration files are
present (see sge_aliases(5)).
In the case of qalter, the previous definition of the
current working directory will be overwritten if qalter
is executed from a different directory than the preced-
ing qsub or qalter.
Qalter allows changing this option even while the job
executes. The modified parameter will only be in effect
after a restart or migration of the job, however.
-C prefix_string
Available for qsub and qrsh with script submission (-b
n).
Prefix_string defines the prefix that declares a direc-
tive in the job's command. The prefix is not a job
attribute, but affects the behavior of qsub and qrsh.
If prefix is a null string, the command will not be
scanned for embedded directives.
The directive prefix consists of two ASCII characters
which, when appearing in the first two bytes of a
script line, indicate that what follows is an Grid
Engine command. The default is "#$".
The user should be aware that changing the first delim-
iting character can produce unforeseen side effects. If
the script file contains anything other than a "#"
character in the first byte position of the line, the
shell processor for the job will reject the line and
may exit the job prematurely.
If the -C option is present in the script file, it is
ignored.
-dc variable,...
Available for qsub, qsh, qrsh, qlogin and qalter only.
Removes the given variable(s) from the job's context.
Multiple -ac, -dc, and -sc options may be given. The
order is important.
Qalter allows changing this option even while the job
executes.
-display display_specifier
Available for qsh only.
Directs xterm(1) to use display_specifier in order to
contact the X server. The display_specifier has to
contain the hostname part of the display name (e.g.
myhost:1). Local display names (e.g. :0) cannot be used
in grid environments. Values set with the -display
option overwrite settings from the submission environ-
ment and from -v command line options.
-dl date_time
Available for qsub, qsh, qrsh, qlogin and qalter only.
Specifies the deadline initiation time in
[[CC]YY]MMDDhhmm[.SS] format (see -a option above). The
deadline initiation time is the time at which a dead-
line job has to reach top priority to be able to com-
plete within a given deadline. Before the deadline ini-
tiation time the priority of a deadline job will be
raised steadily until it reaches the maximum as config-
ured by the Grid Engine administrator.
This option is applicable only for users allowed to
submit deadline jobs.
-e [[hostname]:]path,...
Available for qsub and qalter only.
Defines or redefines the path used for the standard
error stream of the job. If the path constitutes an
absolute path name, the error-path attribute of the job
is set to path, including the hostname. If the path
name is relative, Grid Engine expands path either with
the current working directory path (if the -cwd switch
(see above) is also specified) or with the home direc-
tory path. If hostname is present, the standard error
stream will be placed in the corresponding location
only if the job runs on the specified host. If the path
contains a ":" without a hostname, a leading ":" has to
be specified.
By default the file name for standard error has the
form job_name.ejob_id and job_name.ejob_id.task_id for
array job tasks (see -t option below).
If path is a directory, the standard error stream of
the job will be put in this directory under the default
file name. If the pathname contains certain pseudo
environment variables, their value will be expanded at
runtime of the job and will be used to constitute the
standard error stream path name. The following pseudo
environment variables are supported currently:
$HOME home directory on execution machine
$USER user ID of job owner
$JOB_ID current job ID
$JOB_NAME current job name (see -N option)
$HOSTNAME name of the execution host
$TASK_ID array job task index number
Alternatively to $HOME the tilde sign "~" can be used
as common in csh(1) or ksh(1). Note, that the "~" sign
also works in combination with user names, so that
"~<user>" expands to the home directory of <user>.
Using another user ID than that of the job owner
requires corresponding permissions, of course.
Qalter allows changing this option even while the job
executes. The modified parameter will only be in effect
after a restart or migration of the job, however.
-hard
Available for qsub, qsh, qrsh, qlogin and qalter only.
Signifies that all resource requirements following in
the command line will be hard requirements and must be
satisfied in full before a job can be scheduled.
As Grid Engine scans the command line and script file
for Grid Engine options and parameters it builds a list
of resources required by a job. All such resource
requests are considered as absolutely essential for the
job to commence. If the -soft option (see below) is
encountered during the scan then all following
resources are designated as "soft requirements" for
execution, or "nice-to-have, but not essential". If the
-hard flag is encountered at a later stage of the scan,
all resource requests following it once again become
"essential". The -hard and -soft options in effect act
as "toggles" during the scan.
-h | -h {u|s|o|n|U|O|S}...
Available for qsub, qsh, qrsh, qlogin, qalter and
qresub.
List of holds to place on a job, a task or some tasks
of a job.
`u' denotes a user hold.
`s' denotes a system hold.
`o' denotes a operator hold.
`n' denotes no hold.
As long as any hold other than `n' is assigned to the
job the job is not eligible for execution. Holds can be
released via qalter and qrls(1). In case of qalter
this is supported by the following additional option
specifiers for the -h switch:
`U' removes a user hold.
`S' removes a system hold.
`O' removes a operator hold.
Grid Engine managers can assign and remove all hold
types, Grid Engine operators can assign and remove user
and operator holds, and users can only assign or remove
user holds.
In the case of qsub only user holds can be placed on a
job and thus only the first form of the option with the
-h switch alone is allowed. As opposed to this, qalter
requires the second form described above.
An alternate means to assign hold is provided by the
qhold(1) facility.
If the job is a array job (see the -t option below),
all tasks specified via -t are affected by the -h
operation simultaneously.
Qalter allows changing this option even while the job
executes. The modified parameter will only be in effect
after a restart or migration of the job, however.
-help
Prints a listing of all options.
-hold_jid wc_job_list
Available for qsub, qrsh, and qalter only.
Defines or redefines the job dependency list of the
submitted job. A reference by job name or regular
expression is only accepted if the referenced job is
owned by the same user as the referring job. The sub-
mitted job is not eligible for execution unless all
jobs referenced in the comma-separated job id and/or
job name list have completed successfully. With the
help of job names or regular expressions one can
specify a job dependency on multiple jobs satisfying
the regular expression or on all jobs with the
requested name. The name dependencies are resolved at
submit time and can only be changed via qalter. New
jobs or name changes of other jobs will not be taken
into account.
Qalter allows changing this option even while the job
executes. The modified parameter will only be in effect
after a restart or migration of the job, however.
-i [[hostname]:]file,...
Available for qsub, and qalter only.
Defines or redefines the file used for the standard
input stream of the job. If the file constitutes an
absolute filename, the input-path attribute of the job
is set to path, including the hostname. If the path
name is relative, Grid Engine expands path either with
the current working directory path (if the -cwd switch
(see above) is also specified) or with the home direc-
tory path. If hostname is present, the standard input
stream will be placed in the corresponding location
only if the job runs on the specified host. If the path
contains a ":" without a hostname, a leading ":" has to
be specified.
By default /dev/null is the input stream for the job.
It is possible to use certain pseudo variables, whose
values will be expanded at runtime of the job and will
be used to express the standard input stream as
described in the -e option for the standard error
stream.
Qalter allows changing this option even while the job
executes. The modified parameter will only be in effect
after a restart or migration of the job, however.
-inherit
Available only for qrsh and qmake(1).
qrsh allows the user to start a task in an already
scheduled parallel job. The option -inherit tells qrsh
to read a job id from the environment variable JOB_ID
and start the specified command as a task in this job.
Please note that in this case, the hostname of the host
where the command will be executed must precede the
command to execute; the syntax changes to
qrsh -inherit [ other options ] hostname command [
command_args ]
Note also, that in combination with -inherit, most
other command line options will be ignored. Only the
options -verbose, -v and -V will be interpreted. As a
replacement to option -cwd please use -v PWD.
Usually a task should have the same environment
(including the current working directory) as the
corresponding job, so specifying the option -V should
be suitable for most applications.
Note: If in your system the qmaster tcp port is not
configured as a service, but rather via the environment
variable SGE_QMASTER_PORT, make sure that this variable
is set in the environment when calling qrsh or qmake
with the -inherit option. If you call qrsh or qmake
with the -inherit option from within a job script,
export SGE_QMASTER_PORT with the option "-v
SGE_QMASTER_PORT" either as a command argument or an
embedded directive.
-j y[es]|n[o]
Available for qsub and qalter only.
Specifies whether or not the standard error stream of
the job is merged into the standard output stream.
If both the -j y and the -e options are present, Grid
Engine sets but ignores the error-path attribute.
Qalter allows changing this option even while the job
executes. The modified parameter will only be in effect
after a restart or migration of the job, however.
-js job_share
Available for qsub, qsh, qrsh, qlogin and qalter only.
Defines or redefines the job share of the job relative
to other jobs. Job share is an unsigned integer value.
The default job share value for jobs is 0.
The job share influences the Share Tree Policy and the
Functional Policy. It has no effect on the Urgency and
Override Policies (see share_tree(5), sched_conf(5) and
the Grid Engine Installation and Administration Guide
for further information on the resource management pol-
icies supported by Grid Engine).
In case of the Share Tree Policy, users can distribute
the tickets to which they are currently entitled among
their jobs using different shares assigned via -js. If
all jobs have the same job share value, the tickets are
distributed evenly. Otherwise, jobs receive tickets
relative to the different job shares. Job shares are
treated like an additional level in the share tree in
the latter case.
In connection with the Functional Policy, the job share
can be used to weight jobs within the functional job
category. Tickets are distributed relative to any
uneven job share distribution treated as a virtual
share distribution level underneath the functional job
category.
If both the Share Tree and the Functional Policy are
active, the job shares will have an effect in both pol-
icies, and the tickets independently derived in each of
them are added to the total number of tickets for each
job.
-l resource=value,...
Available for qsub, qsh, qrsh, qlogin and qalter only.
Launch the job in a Grid Engine queue meeting the given
resource request list. In case of qalter the previous
definition is replaced by the specified one.
complex(5) describes how a list of available resources
and their associated valid value specifiers can be
obtained.
There may be multiple -l switches in a single command.
You may request multiple -l options to be soft or hard
both in the same command line. In case of a serial job
multiple -l switches refine the definition for the
sought queue.
Qalter allows changing this option even while the job
executes. The modified parameter will only be in effect
after a restart or migration of the job, however.
-m b|e|a|s|n,...
Available for qsub, qsh, qrsh, qlogin and qalter only.
Defines or redefines under which circumstances mail is
to be sent to the job owner or to the users defined
with the -M option described below. The option argu-
ments have the following meaning:
`b' Mail is sent at the beginning of the job.
`e' Mail is sent at the end of the job.
`a' Mail is sent when the job is aborted or
rescheduled.
`s' Mail is sent when the job is suspended.
`n' No mail is sent.
Currently no mail is sent when a job is suspended.
Qalter allows changing the b, e, and a option arguments
even while the job executes. The modification of the b
option argument will only be in effect after a restart
or migration of the job, however.
-M user[@host],...
Available for qsub, qsh, qrsh, qlogin and qalter only.
Defines or redefines the list of users to which the
server that executes the job has to send mail, if the
server sends mail about the job. Default is the job
owner at the originating host.
Qalter allows changing this option even while the job
executes.
-masterq wc_queue_list
Available for qsub, qrsh, qsh, qlogin and qalter. Only
meaningful for parallel jobs, i.e. together with the
-pe option.
Defines or redefines a list of cluster queues, queue
domains and queue instances which may be used to become
the so called master queue of this parallel job. A more
detailed description of wc_queue_list can be found in
sge_types(1). The master queue is defined as the queue
where the parallel job is started. The other queues to
which the parallel job spawns tasks are called slave
queues. A parallel job only has one master queue.
This parameter has all the properties of a resource
request and will be merged with requirements derived
from the -l option described above.
Qalter allows changing this option even while the job
executes. The modified parameter will only be in effect
after a restart or migration of the job, however.
-notify
Available for qsub, qrsh (with command) and qalter
only.
This flag, when set causes Grid Engine to send "warn-
ing" signals to a running job prior to sending the sig-
nals themselves. If a SIGSTOP is pending, the job will
receive a SIGUSR1 several seconds before the SIGSTOP.
If a SIGKILL is pending, the job will receive a SIGUSR2
several seconds before the SIGKILL. The amount of time
delay is controlled by the notify parameter in each
queue configuration (see queue_conf(5)).
Note that the Linux operating system "misused" the user
signals SIGUSR1 and SIGUSR2 in some early Posix thread
implementations. You might not want to use the -notify
option if you are running multi-threaded applications
in your jobs under Linux, particularly on 2.0 or ear-
lier kernels.
Qalter allows changing this option even while the job
executes.
-now y[es]|n[o]
Available for qsub, qsh, qlogin and qrsh.
-now y tries to start the job immediately or not at
all. The command returns 0 on success, or 1 on failure
(also if the job could not be scheduled immediately).
For array jobs submitted with the -now option, if all
tasks cannot be immediately scheduled, no tasks are
scheduled. -now y is default for qsh, qlogin and qrsh
With the -now n option, the job will be put into the
pending queue if it cannot be executed immediately.
-now n is default for qsub.
-N name
Available for qsub, qsh, qrsh, qlogin and qalter only.
The name of the job. The name can be any printable set
of characters except "\n", "\t", "\r", "/", ":", "@",
"\", "*", and "?", and it has to start with an alpha-
betic character.
If the -N option is not present, Grid Engine assigns
the name of the job script to the job after any direc-
tory pathname has been removed from the script-name. If
the script is read from standard input, the job name
defaults to STDIN.
In the case of qsh or qlogin with the -N option is
absent, the string `INTERACT' is assigned to the job.
Qalter allows changing this option even while the job
executes.
-noshell
Available only for qrsh with a command line.
Do not start the command line given to qrsh in a user's
login shell, i.e. execute it without the wrapping
shell.
This option can be used to speed up execution as some
overhead, like the shell startup and sourcing the shell
resource files, is avoided.
This option can only be used if no shell-specific com-
mand line parsing is required. If the command line con-
tains shell syntax like environment variable substitu-
tion or (back) quoting, a shell must be started. In
this case, either do not use the -noshell option or
include the shell call in the command line.
Example:
qrsh echo '$HOSTNAME'
Alternative call with the -noshell option
qrsh -noshell /bin/tcsh -f -c 'echo $HOSTNAME'
-nostdin
Available only for qrsh.
Suppress the input stream STDIN - qrsh will pass the
option -n to the rsh(1) command. This is especially
useful, if multiple tasks are executed in parallel
using qrsh, e.g. in a make(1) process - it would be
undefined, which process would get the input.
-o [[hostname]:]path,...
Available for qsub, qsh, qrsh, qlogin and qalter only.
The path used for the standard output stream of the
job. The path is handled as described in the -e option
for the standard error stream.
By default the file name for standard output has the
form job_name.ojob_id and job_name.ojob_id.task_id for
array job tasks (see -t option below).
Qalter allows changing this option even while the job
executes. The modified parameter will only be in effect
after a restart or migration of the job, however.
-ot override_tickets
Available for qalter only.
Changes the number of override tickets for the speci-
fied job. Requires manager/operator privileges.
-P project_name
Available for qsub, qsh, qrsh, qlogin and qalter only.
Specifies the project to which this job is assigned.
The administrator needs to give permission to indivi-
dual users to submit jobs to a specific project. (see
-aprj option to qconf(1)).
-p priority
Available for qsub, qsh, qrsh, qlogin and qalter only.
Defines or redefines the priority of the job relative
to other jobs. Priority is an integer in the range
-1023 to 1024. The default priority value for jobs is
0.
Users may only decrease the priority of their jobs.
Grid Engine managers and administrators may also
increase the priority associated with jobs. If a pend-
ing job has higher priority, it is earlier eligible for
being dispatched by the Grid Engine scheduler.
-pe parallel_environment n[-[m]]|[-]m,...
Available for qsub, qsh, qrsh, qlogin and qalter only.
Parallel programming environment (PE) to instantiate.
The range descriptor behind the PE name specifies the
number of parallel processes to be run. Grid Engine
will allocate the appropriate resources as available.
The sge_pe(5) manual page contains information about
the definition of PEs and about how to obtain a list of
currently valid PEs.
You can specify a PE name which uses the wildcard char-
acter, "*". Thus the request "pvm*" will match any
parallel environment with a name starting with the
string "pvm". In the case of multiple parallel environ-
ments whose names match the name string, the parallel
environment with the most available slots is chosen.
The range specification is a list of range expressions
of the form "n-m", where n and m are positive, non-zero
integers. The form "n" is equivalent to "n-n". The
form "-m" is equivalent to "1-m". The form "n-" is
equivalent to "n-infinity". The range specification is
processed as follows: The largest number of queues
requested is checked first. If enough queues meeting
the specified attribute list are available, all are
allocated. If not, the next smaller number of queues is
checked, and so forth.
If additional -l options are present, they restrict the
set of eligible queues for the parallel job.
Qalter allows changing this option even while the job
executes. The modified parameter will only be in effect
after a restart or migration of the job, however.
-q wc_queue_list
Available for qsub, qrsh, qsh, qlogin and qalter.
Defines or redefines a list of cluster queues, queue
domains or queue instances which may be used to execute
this job. Please find a description of wc_queue_list in
sge_types(1). This parameter has all the properties of
a resource request and will be merged with requirements
derived from the -l option described above.
Qalter allows changing this option even while the job
executes. The modified parameter will only be in effect
after a restart or migration of the job, however.
-R y[es]|n[o]
Available for qsub, qrsh, qsh, qlogin and qalter.
Indicates whether a reservation for this job should be
done. Reservation is never done for immediate jobs,
i.e. jobs submitted using the -now yes option. Please
note that regardless of the reservation request, job
reservation might be disabled using max_reservation in
sched_conf(5) and might be limited only to a certain
number of high priority jobs.
By default jobs are submitted with the -R n option.
-r y[es]|n[o]
Available for qsub and qalter only.
Identifies the ability of a job to be rerun or not. If
the value of -r is 'yes', the job will be rerun if the
job was aborted without leaving a consistent exit
state. (This is typically the case if the node on
which the job is running crashes). If -r is 'no', the
job will not be rerun under any circumstances.
Interactive jobs submitted with qsh, qrsh or qlogin are
not rerunnable.
Qalter allows changing this option even while the job
executes.
-sc variable[=value],...
Available for qsub, qsh, qrsh, qlogin and qalter only.
Sets the given name/value pairs as the job's context.
Value may be omitted. Grid Engine replaces the job's
previously defined context with the one given as the
argument. Multiple -ac, -dc, and -sc options may be
given. The order is important.
Contexts provide a way to dynamically attach and remove
meta-information to and from a job. The context vari-
ables are not passed to the job's execution context in
its environment.
Qalter allows changing this option even while the job
executes.
-shell y[es]|n[o]
Available only for qsub.
-shell n causes qsub to execute the command line
directly, as if by exec(2). No command shell will be
executed for the job. This option only applies when -b
y is also used. Without -b y, -shell n has no effect.
This option can be used to speed up execution as some
overhead, like the shell startup and sourcing the shell
resource files is avoided.
This option can only be used if no shell-specific com-
mand line parsing is required. If the command line con-
tains shell syntax, like environment variable substitu-
tion or (back) quoting, a shell must be started. In
this case either do not use the -shell n option or exe-
cute the shell as the command line and pass the path to
the executable as a parameter.
If a job executed with the -shell n option fails due to
a user error, such as an invalid path to the execut-
able, the job will enter the error state.
-shell y cancels the effect of a previous -shell n.
Otherwise, it has no effect.
See -b and -noshell for more information.
-soft
Available for qsub, qsh, qrsh, qlogin and qalter only.
Signifies that all resource requirements following in
the command line will be soft requirements and are to
be filled on an "as available" basis.
As Grid Engine scans the command line and script file
for Grid Engine options and parameters, it builds a
list of resources required by the job. All such
resource requests are considered as absolutely essen-
tial for the job to commence. If the -soft option is
encountered during the scan then all following
resources are designated as "soft requirements" for
execution, or "nice-to-have, but not essential". If the
-hard flag (see above) is encountered at a later stage
of the scan, all resource requests following it once
again become "essential". The -hard and -soft options
in effect act as "toggles" during the scan.
-sync y[es]|n[o]
Available for qsub.
-sync y causes qsub to wait for the job to complete
before exiting. If the job completes successfully,
qsub's exit code will be that of the completed job. If
the job fails to complete successfully, qsub will print
out a error message indicating why the job failed and
will have an exit code of 1. If qsub is interrupted,
e.g. with CTRL-C, before the job completes, the job
will be canceled.
With the -sync n option, qsub will exit with an exit
code of 0 as soon as the job is submitted successfully.
-sync n is default for qsub.
If -sync y is used in conjunction with -now y, qsub
will behave as though only -now y were given until the
job has been successfully scheduled, after which time
qsub will behave as though only -sync y were given.
If -sync y is used in conjunction with -t n[-m[:i]],
qsub will wait for all the job's tasks to complete
before exiting. If all the job's tasks complete suc-
cessfully, qsub's exit code will be that of the first
completed job tasks with a non-zero exit code, or 0 if
all job tasks exited with an exit code of 0. If any of
the job's tasks fail to complete successfully, qsub
will print out a error message indicating why the job
task(s) failed and will have an exit code of 1. If
qsub is interrupted, e.g. with CTRL-C, before the job
completes, all of the job's tasks will be canceled.
-S [[hostname]:]pathname,...
Available for qsub, qsh and qalter.
Specifies the interpreting shell for the job. Only one
pathname component without a host specifier is valid
and only one path name for a given host is allowed.
Shell paths with host assignments define the interpret-
ing shell for the job if the host is the execution
host. The shell path without host specification is used
if the execution host matches none of the hosts in the
list.
Furthermore, the pathname can be constructed with
pseudo environment variables as described for the -e
option above.
In the case of qsh the specified shell path is used to
execute the corresponding command interpreter in the
xterm(1) (via its -e option) started on behalf of the
interactive job.
Qalter allows changing this option even while the job
executes. The modified parameter will only be in effect
after a restart or migration of the job, however.
-t n[-m[:s]]
Available for qsub and qalter only.
Submits a so called Array Job, i.e. an array of identi-
cal tasks being differentiated only by an index number
and being treated by Grid Engine almost like a series
of jobs. The option argument to -t specifies the number
of array job tasks and the index number which will be
associated with the tasks. The index numbers will be
exported to the job tasks via the environment variable
SGE_TASK_ID. The option arguments n, m and s will be
available through the environment variables
SGE_TASK_FIRST, SGE_TASK_LAST and SGE_TASK_STEPSIZE.
Following restrictions apply to the values n and m:
1 <= n <= MIN(2^31-1, max_aj_tasks)
1 <= m <= MIN(2^31-1, max_aj_tasks)
n <= m
max_aj_tasks is defined in the cluster configuration
(see sge_conf(5))
The task id range specified in the option argument may
be a single number, a simple range of the form n-m or a
range with a step size. Hence, the task id range speci-
fied by 2-10:2 would result in the task id indexes 2,
4, 6, 8, and 10, for a total of 5 identical tasks, each
with the environment variable SGE_TASK_ID containing
one of the 5 index numbers.
All array job tasks inherit the same resource requests
and attribute definitions as specified in the qsub or
qalter command line, except for the -t option. The
tasks are scheduled independently and, provided enough
resources exist, concurrently, very much like separate
jobs. However, an array job or a sub-array there of
can be accessed as a single unit by commands like
qmod(1) or qdel(1). See the corresponding manual pages
for further detail.
Array jobs are commonly used to execute the same type
of operation on varying input data sets correlated with
the task index number. The number of tasks in a array
job is unlimited.
STDOUT and STDERR of array job tasks will be written
into different files with the default location
<jobname>.['e'|'o']<job_id>'.'<task_id>
In order to change this default, the -e and -o options
(see above) can be used together with the pseudo
environment variables $HOME, $USER, $JOB_ID, $JOB_NAME,
$HOSTNAME, and $SGE_TASK_ID.
Note, that you can use the output redirection to divert
the output of all tasks into the same file, but the
result of this is undefined.
-u username,...
Available for qalter only. Changes are only made on
those jobs which were submitted by users specified in
the list of usernames. For managers it is possible to
use the qalter -u '*' command to modify all jobs of all
users.
If you use the -u switch it is not permitted to specify
an additional wc_job_range_list.
-v variable[=value],...
Available for qsub, qsh, qrsh, qalter and qresub.
Defines or redefines the environment variables to be
exported to the execution context of the job. If the
-v option is present Grid Engine will add the environ-
ment variables defined as arguments to the switch and,
optionally, values of specified variables, to the exe-
cution context of the job.
Qalter allows changing this option even while the job
executes. The modified parameter will only be in effect
after a restart or migration of the job, however.
-verbose
Available only for qrsh and qmake(1).
Unlike qsh and qlogin, qrsh does not output any infor-
mational messages while establishing the session, com-
pliant with the standard rsh(1) and rlogin(1) system
calls. If the option -verbose is set, qrsh behaves
like the qsh and qlogin commands, printing information
about the process of establishing the rsh(1) or rlo-
gin(1) session.
-verify
Available for qsub, qsh, qrsh, qlogin, qalter and
qresub.
Instead of submitting a job, prints detailed informa-
tion about the would-be job as though qstat(1) -j were
used, including the effects of command-line parameters
and the external environment.
-V Available for qsub, qsh, qrsh, qalter and qresub.
Specifies that all environment variables active within
the qsub utility be exported to the context of the job.
-w e|w|n|v
Available for qsub, qsh, qrsh, qlogin, qalter and
qresub.
Specifies a validation level applied to the job to be
submitted (qsub, qlogin, and qsh) or the specified
queued job (qalter). The information displayed indi-
cates whether the job can possibly be scheduled assum-
ing an empty system with no other jobs. Resource
requests exceeding the configured maximal thresholds or
requesting unavailable resource attributes are possible
causes for jobs to fail this validation.
The specifiers e, w, n and v define the following vali-
dation modes:
`e' error - jobs with invalid requests will be
rejected; the default for qrsh, qsh
and qlogin.
`w' warning - only a warning will be displayed
for invalid requests.
`n' none - switches off validation; the default for
qsub and qalter.
`v' verify - does not submit the job but prints an
extensive validation report.
Note, that the necessary checks are performance
consuming and hence the checking is switched off by
default.
Note also, that the reasons for job requirements being
invalid with respect to resource availability of queues
are displayed in the "-w v" case using the format as
described for the qstat(1) -F option (see description
of Full Format in section OUTPUT FORMATS of the
qstat(1) manual page.
command
Available for qsub and qrsh only.
The job's scriptfile or binary. If not present or if
the operand is the single-character string '-', qsub
reads the script from standard input.
command_args
Available for qsub, qrsh and qalter only.
Arguments to the job. Not valid if the script is
entered from standard input.
Qalter allows changing this option even while the job
executes. The modified parameter will only be in effect
after a restart or migration of the job, however.
xterm_args
Available for qsh only.
Arguments to the xterm(1) executable, as defined in the
configuration. For details, refer to sge_conf(5)).
ENVIRONMENTAL VARIABLES
SGE_ROOT Specifies the location of the Grid Engine
standard configuration files.
SGE_CELL If set, specifies the default Grid Engine
cell. To address a Grid Engine cell qsub,
qsh, qlogin or qalter use (in the order of
precedence):
The name of the cell specified in the
environment variable SGE_CELL, if it is
set.
The name of the default cell, i.e.
default.
SGE_DEBUG_LEVEL
If set, specifies that debug information
should be written to stderr. In addition the
level of detail in which debug information is
generated is defined.
SGE_QMASTER_PORT
If set, specifies the tcp port on which
sge_qmaster(8) is expected to listen for com-
munication requests. Most installations will
use a services map entry for the service
"sge_qmaster" instead to define that port.
DISPLAY For qsh jobs the DISPLAY has to be specified
at job submission. If the DISPLAY is not set
by using the -display or the -v switch, the
contents of the DISPLAY environment variable
are used as default.
In addition to those environment variables specified to be
exported to the job via the -v or the -V option (see above)
qsub, qsh, and qlogin add the following variables with the
indicated values to the variable list:
SGE_O_HOME the home directory of the submitting client.
SGE_O_HOST the name of the host on which the submitting
client is running.
SGE_O_LOGNAME the LOGNAME of the submitting client.
SGE_O_MAIL the MAIL of the submitting client. This is
the mail directory of the submitting client.
SGE_O_PATH the executable search path of the submitting
client.
SGE_O_SHELL the SHELL of the submitting client.
SGE_O_TZ the time zone of the submitting client.
SGE_O_WORKDIR the absolute path of the current working
directory of the submitting client.
Furthermore, Grid Engine sets additional variables into the
job's environment, as listed below.
ARC
SGE_ARCH The Grid Engine architecture name of the node
on which the job is running. The name is
compiled-in into the sge_execd(8) binary.
SGE_CKPT_ENV Specifies the checkpointing environment (as
selected with the -ckpt option) under which a
checkpointing job executes. Only set for
checkpointing jobs.
SGE_CKPT_DIR Only set for checkpointing jobs. Contains
path ckpt_dir (see checkpoint(5) ) of the
checkpoint interface.
SGE_STDERR_PATH
the pathname of the file to which the stan-
dard error stream of the job is diverted.
Commonly used for enhancing the output with
error messages from prolog, epilog, parallel
environment start/stop or checkpointing
scripts.
SGE_STDOUT_PATH
the pathname of the file to which the stan-
dard output stream of the job is diverted.
Commonly used for enhancing the output with
messages from prolog, epilog, parallel
environment start/stop or checkpointing
scripts.
SGE_STDIN_PATH the pathname of the file from which the stan-
dard input stream of the job is taken. This
variable might be used in combination with
SGE_O_HOST in prolog/epilog scripts to
transfer the input file from the submit to
the execution host.
SGE_JOB_SPOOL_DIR
The directory used by sge_shepherd(8) to
store job related data during job execution.
This directory is owned by root or by a Grid
Engine administrative account and commonly is
not open for read or write access to regular
users.
SGE_TASK_ID The index number of the current array job
task (see -t option above). This is an unique
number in each array job and can be used to
reference different input data records, for
example. This environment variable is set to
"undefined" for non-array jobs. It is possi-
ble to change the predefined value of this
variable with -v or -V (see options above).
SGE_TASK_FIRST The index number of the first array job task
(see -t option above). It is possible to
change the predefined value of this variable
with -v or -V (see options above).
SGE_TASK_LAST The index number of the last array job task
(see -t option above). It is possible to
change the predefined value of this variable
with -v or -V (see options above).
SGE_TASK_STEPSIZE
The stepsize of the array job specification
(see -t option above). It is possible to
change the predefined value of this variable
with -v or -V (see options above).
ENVIRONMENT The ENVIRONMENT variable is set to BATCH to
identify that the job is being executed under
Grid Engine control.
HOME The user's home directory path from the
passwd(5) file.
HOSTNAME The hostname of the node on which the job is
running.
JOB_ID A unique identifier assigned by the
sge_qmaster(8) when the job was submitted.
The job ID is a decimal integer in the range
1 to 99999.
JOB_NAME The job name. For batch jobs or jobs submit-
ted by qrsh with a command, the job name is
built as basename of the qsub script filename
resp. the qrsh command. For interactive jobs
it is set to `INTERACTIVE' for qsh jobs,
`QLOGIN' for qlogin jobs and `QRLOGIN' for
qrsh jobs without a command.
This default may be overwritten by the -N.
option.
LOGNAME The user's login name from the passwd(5)
file.
NHOSTS The number of hosts in use by a parallel job.
NQUEUES The number of queues allocated for the job
(always 1 for serial jobs).
NSLOTS The number of queue slots in use by a paral-
lel job.
PATH A default shell search path of:
/usr/local/bin:/usr/ucb:/bin:/usr/bin
SGE_BINARY_PATH
The path where the Grid Engine binaries are
installed. The value is the concatenation of
the cluster configuration value binary_path
and the architecture name $SGE_ARCH environ-
ment variable.
PE The parallel environment under which the job
executes (for parallel jobs only).
PE_HOSTFILE The path of a file containing the definition
of the virtual parallel machine assigned to a
parallel job by Grid Engine. See the descrip-
tion of the $pe_hostfile parameter in
sge_pe(5) for details on the format of this
file. The environment variable is only avail-
able for parallel jobs.
QUEUE The name of the cluster queue in which the
job is running.
REQUEST Available for batch jobs only.
The request name of a job as specified with
the -N switch (see above) or taken as the
name of the job script file.
RESTARTED This variable is set to 1 if a job was res-
tarted either after a system crash or after a
migration in case of a checkpointing job. The
variable has the value 0 otherwise.
SHELL The user's login shell from the passwd(5)
file. Note: This is not necessarily the shell
in use for the job.
TMPDIR The absolute path to the job's temporary
working directory.
TMP The same as TMPDIR; provided for compatibil-
ity with NQS.
TZ The time zone variable imported from
sge_execd(8) if set.
USER The user's login name from the passwd(5)
file.
RESTRICTIONS
There is no controlling terminal for batch jobs under Grid
Engine, and any tests or actions on a controlling terminal
will fail. If these operations are in your .login or .cshrc
file, they may cause your job to abort.
Insert the following test before any commands that are not
pertinent to batch jobs in your .login:
if ( $?JOB_NAME) then
echo "Grid Engine spooled job"
exit 0
endif
Don't forget to set your shell's search path in your shell
start-up before this code.
EXIT STATUS
The following exit values are returned:
0 Operation was executed successfully.
25 It was not possible to register a new job according to
the configured max_u_jobs or max_jobs limit. Additional
information may be found in sge_conf(5)
>0 Error occurred.
EXAMPLES
The following is the simplest form of a Grid Engine script
file.
=====================================================
#!/bin/csh
a.out
=====================================================
The next example is a more complex Grid Engine script.
=====================================================
#!/bin/csh
# Which account to be charged cpu time
#$ -A santa_claus
# date-time to run, format [[CC]yy]MMDDhhmm[.SS]
#$ -a 12241200
# to run I want 6 or more parallel processes
# under the PE pvm. the processes require
# 128M of memory
#$ -pe pvm 6- -l mem=128
# If I run on dec_x put stderr in /tmp/foo, if I
# run on sun_y, put stderr in /usr/me/foo
#$ -e dec_x:/tmp/foo,sun_y:/usr/me/foo
# Send mail to these users
#$ -M santa@nothpole,claus@northpole
# Mail at beginning/end/on suspension
#$ -m bes
# Export these environmental variables
#$ -v PVM_ROOT,FOOBAR=BAR
# The job is located in the current
# working directory.
#$ -cwd
a.out
==========================================================
FILES
$REQUEST.oJID[.TASKID] STDOUT of job #JID
$REQUEST.eJID[.TASKID] STDERR of job
$REQUEST.poJID[.TASKID] STDOUT of par. env. of job
$REQUEST.peJID[.TASKID] STDERR of par. env. of job
$REQUEST.hostsJID[.TASKID] hosts file of par. env. of job
$cwd/.sge_aliases cwd path aliases
$cwd/.sge_request cwd default request
$HOME/.sge_aliases user path aliases
$HOME/.sge_request user default request
<sge_root>/<cell>/common/sge_aliases
cluster path aliases
<sge_root>/<cell>/common/sge_request
cluster default request
<sge_root>/<cell>/common/act_qmaster
Grid Engine master host file
SEE ALSO
sge_intro(1), qconf(1), qdel(1), qhold(1), qmod(1), qrls(1),
qstat(1), accounting(5), sge_aliases(5), sge_conf(5),
sge_request(5), sge_pe(5), complex(5).
COPYRIGHT
If configured correspondingly, qrsh and qlogin contain por-
tions of the rsh, rshd, telnet and telnetd code copyrighted
by The Regents of the University of California. Therefore,
the following note applies with respect to qrsh and qlogin:
This product includes software developed by the University
of California, Berkeley and its contributors.
See sge_intro(1) as well as the information provided in
<sge_root>/3rd_party/qrsh and <sge_root>/3rd_party/qlogin
for a statement of further rights and permissions.
Man(1) output converted with
man2html