A batchtools slurm backend resolves futures in parallel via a Slurm job scheduler

Usage

batchtools_slurm(
  ...,
  template = "slurm",
  scheduler.latency = 1,
  fs.latency = 65
)

Arguments

template: (optional) Name of job-script template to be searched for by batchtools::findTemplateFile(). If not found, it defaults to the templates/slurm.tmpl part of this package (see below).
scheduler.latency: [numeric(1)]
Time to sleep after important interactions with the scheduler to ensure a sane state. Currently only triggered after calling submitJobs.
fs.latency: [numeric(1)]
Expected maximum latency of the file system, in seconds. Set to a positive number for network file systems like NFS which enables more robust (but also more expensive) mechanisms to access files and directories. Usually safe to set to 0 to disable the heuristic, e.g. if you are working on a local file system.
...: Not used.

Details

Batchtools slurm futures use batchtools cluster functions created by batchtools::makeClusterFunctionsSlurm(), which requires that Slurm commands sbatch, squeue, and scancel are installed on the current machine.

The default template script templates/slurm.tmpl can be found in:

system.file("templates", "slurm.tmpl", package = "future.batchtools")

and comprise:

#!/bin/bash
######################################################################
# A batchtools launch script template for Slurm
#
# Author: Henrik Bengtsson
######################################################################


#SBATCH --job-name=<%= job.name %>
#SBATCH --output=<%= log.file %>
<%
defaults <- list(
  nodes = 1,         # single-host processing
  time = "00:05:00", # 5-min runtime
  mem  = "100M"      # 100 MiB memory
)
resources <- c(resources, defaults[setdiff(names(defaults), names(resources))])
opts <- unlist(resources, use.names = TRUE)
opts <- sprintf("--%s=%s", names(opts), opts)
cat(sprintf("#SBATCH %s\n", opts))
%>

echo "Batchtools job name: '<%= job.name %>'"

echo "Session information:"
date
hostname
which Rscript
Rscript --version
Rscript -e ".libPaths()"

## Launch R and evaluate the batchtools R job
echo "Command: Rscript -e 'batchtools::doJobCollection("<%= uri %>")' ..."
Rscript -e 'batchtools::doJobCollection("<%= uri %>")'
res=$?
echo " - exit code: ${res}"
echo "Command: Rscript -e 'batchtools::doJobCollection("<%= uri %>")' ... done"

## End-of-job summary
sstat --format="JobID,AveCPU,MaxRSS,MaxPages,MaxDiskRead,MaxDiskWrite" --allsteps --jobs="${SLURM_JOB_ID}"

## Relay the exit code from Rscript
exit "${res}"

References

https://en.wikipedia.org/wiki/Slurm_Workload_Manager

Examples

if (FALSE) { # interactive()
# Limit runtime to 3 minutes and memory to 200 MiB per future
plan(batchtools_slurm, resources = list(time = "00:03:00", mem = "200M"))

message("Main process ID: ", Sys.getpid())

f <- future(Sys.getpid())
pid <- value(f)
message("Worker process ID: ", pid)
}