A batchtools future is a future whose value will be resolved via batchtools — BatchtoolsFutureBackend • future.batchtools

A batchtools local future is an synchronous uniprocess future that will be evaluated in a background R session. A batchtools interactive future is an synchronous uniprocess future that will be evaluated in the current R session (and variables will be assigned to the calling environment rather than to a local one). Both types of futures will block until the futures are resolved.

A batchtools multicore future is an asynchronous multiprocess future that will be evaluated in a background R session.

We highly recommend using future::multisession (sic!) futures of the future package instead of multicore batchtools futures.

A batchtools SSH future is an asynchronous multiprocess future that will be evaluated in a background R session.

We highly recommend using future::multisession (sic!) futures of the future package instead of SSH batchtools futures.

Usage

BatchtoolsFutureBackend(
  workers = NULL,
  resources = list(),
  finalize = getOption("future.finalize", TRUE),
  cluster.functions = NULL,
  registry = list(),
  conf.file = findConfFile(),
  interrupts = TRUE,
  ...
)

BatchtoolsBashFutureBackend(
  ...,
  cluster.functions = makeClusterFunctionsBash(template = template, fs.latency =
    fs.latency),
  fs.latency = 0,
  template = "bash"
)

BatchtoolsCustomFutureBackend(...)

BatchtoolsInteractiveFutureBackend(fs.latency = 0, ...)

BatchtoolsLocalFutureBackend(fs.latency = 0, ...)

BatchtoolsMulticoreFutureBackend(
  workers = availableCores(constraints = "multicore"),
  fs.latency = 0,
  ...
)

BatchtoolsSSHFutureBackend(workers = availableWorkers(), fs.latency = 65, ...)

Arguments

workers: The number of multicore processes to be available for concurrent batchtools multicore futures.
resources: (optional) A named list passed to the batchtools job-script template as variable resources. See Section 'Resources' in batchtools::submitJobs() more details.
finalize: If TRUE, a future's batchtools Registry is automatically deleted when the future is garbage collected, otherwise not.
cluster.functions: (optional) Assigned as-is to the each future's batchtools Registry.
registry: (optional) A named list of settings applied to each future's batchtools Registry. This is a more convenient alternative to using argument conf.file.
conf.file: (optional) A "batchtools-configuration" R script, which is sourced when each future's batchtools Registry is created. Any variables created by this script is assigned to the registry. The default file is the one found by batchtools::findConfFile(), if any.
interrupts: If FALSE, attempts to interrupt futures will not take place on this backend, even if the backend supports it. This is useful when, for instance, it takes a long time to interrupt a future.
fs.latency: [numeric(1)]
Expected maximum latency of the file system, in seconds. Set to a positive number for network file systems like NFS which enables more robust (but also more expensive) mechanisms to access files and directories. Usually safe to set to 0 to disable the heuristic, e.g. if you are working on a local file system.
template: (optional) A batchtools template file or a template string (in brew format). If not specified, it is left to the batchtools package to locate such file using its search rules.
...: Additional arguments passed to BatchtoolsFutureBackend().

Value

A future::FutureBackend object of class BatchtoolsFutureBackend

An object of class BatchtoolsFuture.

An object of class BatchtoolsUniprocessFuture.

An object of class BatchtoolsMulticoreFuture.

Details

batchtools local futures rely on the batchtools backend set up by batchtools::makeClusterFunctionsInteractive(external = TRUE) and batchtools interactive futures on the one sQet up by batchtools::makeClusterFunctionsInteractive(). These are supported by all operating systems.

An alternative to batchtools local futures is to use cluster futures of the future package with a single local background session, i.e. plan(cluster, workers = "localhost").

An alternative to batchtools interactive futures is to use plan(sequential, split = TRUE) futures of the future package.

batchtools multicore futures rely on the batchtools backend set up by batchtools::makeClusterFunctionsMulticore(). The batchtools multicore backend only works on operating systems supporting the ps command-line tool, e.g. Linux and macOS.

batchtools SSH futures rely on the batchtools backend set up by batchtools::makeClusterFunctionsSSH(). The batchtools SSH backend only works on operating systems supporting the ssh and ps command-line tool, e.g. Linux and macOS.

Examples

options(error = function(...) {
  print(traceback())
})

cf <- batchtools::makeClusterFunctionsInteractive(external = TRUE)
print(cf)
#> ClusterFunctions for mode: Interactive
#>   List queued Jobs : FALSE
#>   List running Jobs: FALSE
#>   Kill Jobs        : FALSE
#>   Hooks            : -
str(cf)
#> List of 11
#>  $ name                : chr "Interactive"
#>  $ submitJob           :function (reg, jc)  
#>  $ killJob             : NULL
#>  $ listJobsQueued      : NULL
#>  $ listJobsRunning     : NULL
#>  $ array.var           : chr NA
#>  $ store.job.collection: logi TRUE
#>  $ store.job.files     : logi FALSE
#>  $ scheduler.latency   : num 0
#>  $ fs.latency          : num 0
#>  $ hooks               : list()
#>  - attr(*, "class")= chr "ClusterFunctions"
plan(batchtools_custom, cluster.functions = cf)
print(plan())
#> batchtools_custom:
#> - args: function (..., cluster.functions = list(name = "Interactive", submitJob = function (reg, jc) { assertRegistry(reg, writeable = TRUE) assertClass(jc, "JobCollection") if (external) { runOSCommand(Rscript(), sprintf("-e \"batchtools::doJobCollection('%s', output = '%s')\"", jc$uri, jc$log.file)) } else { doJobCollection(jc, output = jc$log.file) } makeSubmitJobResult(status = 0L, batch.id = "cfInteractive") }, killJob = NULL, listJobsQueued = NULL, listJobsRunning = NULL, array.var = NA_character_, store.job.collection = TRUE, store.job.files = FALSE, scheduler.latency = 0, fs.latency = 0, hooks = list()), workers = "<NULL>")
#> - tweaked: TRUE
#> - call: plan(batchtools_custom, cluster.functions = cf)
#> BatchtoolsCustomFutureBackend:
#> Inherits: BatchtoolsMultiprocessFutureBackend, BatchtoolsFutureBackend, MultiprocessFutureBackend, FutureBackend
#> UUID: d07fa81c418631e9275dc14107dc7910
#> Number of workers: 100
#> Number of free workers: 100
#> Available cores: 8
#> Automatic garbage collection: FALSE
#> Early signaling: FALSE
#> Interrupts are enabled: TRUE
#> Maximum total size of globals: +Inf
#> Maximum total size of value: +Inf
#> Number of active futures: 0
#> Number of futures since start: 0 (0 created, 0 launched, 0 finished)
#> Total runtime of futures: 0 secs (NaN secs/finished future)
#> batchtools configuration file: <NA>
#> batchtools cluster functions: ‘Interactive’
#> batchtools cluster functions template: <NA>
#> batchtools resources:
#>  list()
#> Cache directory: ‘/tmp/henrik/RtmpW5FTaG/future.batchtools/docs/reference/.future/20250720_003156-CJGJmo’ (0 folders)
print(nbrOfWorkers())
#> [1] 100

## Create explicit future
f <- future({
  cat("PID:", Sys.getpid(), "\n")
  42L
})
print(f)
#> BatchtoolsCustomFuture:
#> Label: <unnamed-1>
#> Expression:
#> {
#>     cat("PID:", Sys.getpid(), "\n")
#>     42L
#> }
#> Globals: <none>
#> Packages: <none>
#> L'Ecuyer-CMRG RNG seed: <none> (seed = FALSE)
#> Capture standard output: TRUE
#> Capture condition classes: ‘condition’ (excluding ‘<none>’)
#> Immediate condition classes: ‘immediateCondition’
#> Lazy evaluation: FALSE
#> Local evaluation: TRUE
#> Asynchronous evaluation: TRUE
#> Early signaling: FALSE
#> Environment: 0x5c62e3c496e8
#> State: ‘running’
#> Resolved: TRUE
#> Unique identifier: 02f4e02cd17ac6609959be37ddf29b36-1
#> Owner process: 02f4e02cd17ac6609959be37ddf29b36
#> Class: ‘BatchtoolsCustomFuture’, ‘BatchtoolsFuture’, ‘Future’
#> Value: <not collected>
#> Conditions captured: <none>
#> batchtools configuration file: <NA>
#> batchtools cluster functions: ‘Interactive’
#> batchtools cluster functions template: <NA>
#> batchtools status: ‘defined’, ‘finished’, ‘started’, ‘submitted’
#> batchtools Registry:
#>   File dir exists: TRUE
#>   Work dir exists: TRUE
#> Job Registry
#>   Backend  : Interactive
#>   File dir : /tmp/henrik/RtmpW5FTaG/future.batchtools/docs/reference/.future/20250720_003156-CJGJmo/batchtools_263761858
#>   Work dir : /tmp/henrik/RtmpW5FTaG/future.batchtools/docs/reference
#>   Jobs     : 1
#>   Seed     : 17538
#>   Writeable: TRUE
v <- value(f)
#> PID: 621224 
print(v)
#> [1] 42

options(error = NULL)


## Create explicit future
f <- future({
  cat("PID:", Sys.getpid(), "\n")
  42L
})
print(f)
#> BatchtoolsCustomFuture:
#> Label: <unnamed-2>
#> Expression:
#> {
#>     cat("PID:", Sys.getpid(), "\n")
#>     42L
#> }
#> Globals: <none>
#> Packages: <none>
#> L'Ecuyer-CMRG RNG seed: <none> (seed = FALSE)
#> Capture standard output: TRUE
#> Capture condition classes: ‘condition’ (excluding ‘<none>’)
#> Immediate condition classes: ‘immediateCondition’
#> Lazy evaluation: FALSE
#> Local evaluation: TRUE
#> Asynchronous evaluation: TRUE
#> Early signaling: FALSE
#> Environment: 0x5c62e3c496e8
#> State: ‘running’
#> Resolved: TRUE
#> Unique identifier: 02f4e02cd17ac6609959be37ddf29b36-2
#> Owner process: 02f4e02cd17ac6609959be37ddf29b36
#> Class: ‘BatchtoolsCustomFuture’, ‘BatchtoolsFuture’, ‘Future’
#> Value: <not collected>
#> Conditions captured: <none>
#> batchtools configuration file: <NA>
#> batchtools cluster functions: ‘Interactive’
#> batchtools cluster functions template: <NA>
#> batchtools status: ‘defined’, ‘finished’, ‘started’, ‘submitted’
#> batchtools Registry:
#>   File dir exists: TRUE
#>   Work dir exists: TRUE
#> Job Registry
#>   Backend  : Interactive
#>   File dir : /tmp/henrik/RtmpW5FTaG/future.batchtools/docs/reference/.future/20250720_003156-CJGJmo/batchtools_1622339445
#>   Work dir : /tmp/henrik/RtmpW5FTaG/future.batchtools/docs/reference
#>   Jobs     : 1
#>   Seed     : 29963
#>   Writeable: TRUE
v <- value(f)
#> PID: 621244 
print(v)
#> [1] 42



## Create explicit future
f <- future({
  cat("PID:", Sys.getpid(), "\n")
  42L
})
v <- value(f)
#> PID: 621266 
print(v)
#> [1] 42