Skip to contents

A batchtools bash backend that resolves futures sequentially via a Bash template script

Source: R/batchtools_bash.R
batchtools_bash.Rd

The batchtools_bash backend was added to illustrate how to write a custom future.batchtools backend that uses a templated job script. Please see the source code, for details.

Usage

batchtools_bash(
  ...,
  cluster.functions = makeClusterFunctionsBash(template = "bash", fs.latency =
    fs.latency),
  fs.latency = 0,
  template = "bash",
  registry = list(),
  conf.file = findConfFile(),
  resources = list(),
  finalize = getOption("future.finalize", TRUE)
)

makeClusterFunctionsBash(template = "bash", fs.latency = 0)

Arguments

cluster.functions

(optional) Assigned as-is to the each future's batchtools Registry.

fs.latency

[numeric(1)]
Expected maximum latency of the file system, in seconds. Set to a positive number for network file systems like NFS which enables more robust (but also more expensive) mechanisms to access files and directories. Usually safe to set to 0 to disable the heuristic, e.g. if you are working on a local file system.

template

(optional) Name of job-script template to be searched for by batchtools::findTemplateFile(). If not found, it defaults to the templates/bash.tmpl part of this package (see below).

registry

(optional) A named list of settings applied to each future's batchtools Registry. This is a more convenient alternative to using argument conf.file.

conf.file

(optional) A "batchtools-configuration" R script, which is sourced when each future's batchtools Registry is created. Any variables created by this script is assigned to the registry. The default file is the one found by batchtools::findConfFile(), if any.

resources

(optional) A named list passed to the batchtools job-script template as variable resources. See Section 'Resources' in batchtools::submitJobs() more details.

finalize

If TRUE, a future's batchtools Registry is automatically deleted when the future is garbage collected, otherwise not.

...

Not used.

Value

makeClusterFunctionsBash() returns a ClusterFunctions object.

Details

Batchtools bash futures use batchtools cluster functions created by makeClusterFunctionsBash() and requires that bash is installed on the current machine and the timeout command is available.

The default template script templates/bash.tmpl can be found in:

system.file("templates", "bash.tmpl", package = "future.batchtools")

and comprise:

#!/bin/bash
######################################################################
# A batchtools launch script template
#
# Author: Henrik Bengtsson
######################################################################

## Maximum runtime?
<%
## The default maximum runtime is one hours
runtime <- resources[["runtime"]]
if (is.null(runtime)) runtime <- 3600
%>

# Launch R and evaluate the batchtools R job
timeout <%= runtime %> Rscript -e 'batchtools::doJobCollection("<%= uri %>", output = "<%= log.file %>")'

Examples

if (FALSE) { # interactive()
# Limit runtime to 30 seconds per future
plan(batchtools_bash, resources = list(runtime = 30))

message("Main process ID: ", Sys.getpid())

f <- future(Sys.getpid())
pid <- value(f)
message("Worker process ID: ", pid)
}