A batchtools SGE backend resolves futures in parallel via a SGE job scheduler

Usage

batchtools_sge(..., template = "sge", scheduler.latency = 1, fs.latency = 65)

Arguments

template: (optional) Name of job-script template to be searched for by batchtools::findTemplateFile(). If not found, it defaults to the templates/sge.tmpl part of this package (see below).
scheduler.latency: [numeric(1)]
Time to sleep after important interactions with the scheduler to ensure a sane state. Currently only triggered after calling submitJobs.
fs.latency: [numeric(1)]
Expected maximum latency of the file system, in seconds. Set to a positive number for network file systems like NFS which enables more robust (but also more expensive) mechanisms to access files and directories. Usually safe to set to 0 to disable the heuristic, e.g. if you are working on a local file system.
...: Not used.

Details

Batchtools sge futures use batchtools cluster functions created by batchtools::makeClusterFunctionsSGE(), which requires that SGE commands qsub, qstat, and qdel are installed on the current machine.

The default template script templates/sge.tmpl can be found in:

system.file("templates", "sge.tmpl", package = "future.batchtools")

and comprise:

#!/bin/bash
######################################################################
# A batchtools launch script template for SGE
#
# Author: Henrik Bengtsson
######################################################################
## Shell:
#$ -S /bin/bash         # Run this as a bash script (required)

## Job name:
#$ -N <%= job.name %>

## Direct streams to logfile:
#$ -o <%= log.file %>

## Merge standard error and output:
#$ -j y

## Tell the queue system to use the current directory
## as the working directory
#$ -cwd

## Use environment variables
#$ -V

## Resources needed:
<% if (length(resources) > 0) {
  ## As-is resource specifications, e.g.
  ## sge_options <- c("-pe smp 2", "-R yes")
  ## plan(batchtools_sge, resources = list(asis = sge_options))
  if ("asis" %in% names(resources)) {
    cat(sprintf("#$ %s\n", resources[["asis"]]))
    resources <- resources[names(resources) != "asis"]
  }

  ## Remaining resources are assumed to be of type '-l', e.g.
  ## plan(batchtools_sge, resources = list(mem_free = "1G", h_rt="00:20:00"))
  opts <- unlist(resources, use.names = TRUE)
  opts <- sprintf("-l %s=%s", names(opts), opts)
  cat(sprintf("#$ %s\n", opts))
} %>

echo "Batchtools job name: '<%= job.name %>'"

echo "Session information:"
date
hostname
which Rscript
Rscript --version
Rscript -e ".libPaths()"

## Launch R and evaluate the batchtools R job
echo "Command: Rscript -e 'batchtools::doJobCollection("<%= uri %>")' ..."
Rscript -e 'batchtools::doJobCollection("<%= uri %>")'
res=$?
echo " - exit code: ${res}"
echo "Command: Rscript -e 'batchtools::doJobCollection("<%= uri %>")' ... done"

## End-of-job summary
qstat -j "${JOB_ID}"

## Relay the exit code from Rscript
exit "${res}"

References

https://en.wikipedia.org/wiki/Oracle_Grid_Engine

Examples

if (FALSE) { # interactive()
# Limit runtime to 3 minutes and memory to 200 MiB per future
plan(batchtools_sge, resources = list(h_rt = "00:03:00", mem_free = "200M"))

message("Main process ID: ", Sys.getpid())

f <- future(Sys.getpid())
pid <- value(f)
message("Worker process ID: ", pid)
}