A batchtools LSF backend resolves futures in parallel via a Load Sharing Facility (LSF) job scheduler

Usage

batchtools_lsf(..., template = "lsf", scheduler.latency = 1, fs.latency = 65)

Arguments

template: (optional) Name of job-script template to be searched for by batchtools::findTemplateFile(). If not found, it defaults to the templates/lsf.tmpl part of this package (see below).
scheduler.latency: [numeric(1)]
Time to sleep after important interactions with the scheduler to ensure a sane state. Currently only triggered after calling submitJobs.
fs.latency: [numeric(1)]
Expected maximum latency of the file system, in seconds. Set to a positive number for network file systems like NFS which enables more robust (but also more expensive) mechanisms to access files and directories. Usually safe to set to 0 to disable the heuristic, e.g. if you are working on a local file system.
...: Not used.

Details

Batchtools lsf futures use batchtools cluster functions created by batchtools::makeClusterFunctionsLSF(), which requires that LSF commands bsub, bjobs, and bkill are installed on the current machine.

The default template script templates/lsf.tmpl can be found in:

system.file("templates", "lsf.tmpl", package = "future.batchtools")

and comprise:

#!/bin/bash
######################################################################
# A batchtools launch script template for LSF
#
# Author: Henrik Bengtsson
######################################################################

## Job name:
#BSUB -J <%= job.name %>

## Direct streams to logfile:
#BSUB -o <%= log.file %>

## Resources needed:
<% if (length(resources) > 0) {
  opts <- unlist(resources, use.names = TRUE)
  opts <- sprintf("-%s=%s", names(opts), opts)
  cat(sprintf("#BSUB %s\n", opts))
} %>

echo "Batchtools job name: '<%= job.name %>'"

echo "Session information:"
date
hostname
which Rscript
Rscript --version
Rscript -e ".libPaths()"

## Launch R and evaluate the batchtools R job
echo "Command: Rscript -e 'batchtools::doJobCollection("<%= uri %>")' ..."
Rscript -e 'batchtools::doJobCollection("<%= uri %>")'
res=$?
echo " - exit code: ${res}"
echo "Command: Rscript -e 'batchtools::doJobCollection("<%= uri %>")' ... done"

## End-of-job summary
bjobs -l "${LSB_JOBID}"

## Relay the exit code from Rscript
exit "${res}"

References

https://en.wikipedia.org/wiki/IBM_Spectrum_LSF

Examples

if (FALSE) { # interactive()
# Limit runtime to 3 minutes and memory to 200 MiB per future
plan(batchtools_lsf, resources = list(W = "00:03:00", M = "200"))

message("Main process ID: ", Sys.getpid())

f <- future(Sys.getpid())
pid <- value(f)
message("Worker process ID: ", pid)
}