Skip to contents

Introduction

The future.batchtools package is a backend for the future framework, also known as the Futureverse. The Futureverse makes it easy to parallelize existing R code - often with only a minor change of code. Most users interact with it via higher-level packages such as futurize, future.apply, furrr, and doFuture, and you choose where and how the code runs by setting a future backend with plan(). This package adds backends that run futures as jobs on a high-performance compute (HPC) cluster, via job schedulers such as Slurm, SGE, LSF, and TORQUE/PBS, using the batchtools package.

library(futurize)
plan(future.batchtools::batchtools_slurm)  # parallelize via Slurm jobs

# A slow function
slow_fcn <- function(x) {
  Sys.sleep(1.0)
  x^2
}

# Example data
xs <- 1:10

# Sequential and parallel base R apply
ys <- lapply(xs, slow_fcn)
ys <- lapply(xs, slow_fcn) |> futurize()

# Sequential and parallel purrr map
library(purrr)
ys <- xs |>  map(slow_fcn)
ys <- xs |>  map(slow_fcn) |> futurize()

# Sequential and parallel foreach calls
library(foreach)
ys <- foreach(x = xs) %do% slow_fcn(x)
ys <- foreach(x = xs) %do% slow_fcn(x) |> futurize()

The only thing that differs from using any other future backend is the plan() call. The code does not have to change, it gives identical results, and it behaves exactly the same.

The batchtools backends

Backend Description Alternative in future package
batchtools_lsf Futures are evaluated via a Load Sharing Facility (LSF) job scheduler N/A
batchtools_openlava Futures are evaluated via an OpenLava job scheduler N/A
batchtools_sge Futures are evaluated via a Sun/Son of/Oracle/Univa/Altair Grid Engine (SGE) job scheduler N/A
batchtools_slurm Futures are evaluated via a Slurm job scheduler N/A
batchtools_torque Futures are evaluated via a TORQUE / PBS job scheduler N/A
batchtools_custom Futures are evaluated via a custom batchtools configuration R script or via a set of cluster functions N/A
batchtools_multicore parallel evaluation by forking the current R process plan(multicore)
batchtools_local sequential evaluation in a separate R process (on current machine) plan(cluster, workers = I(1))

Each future is submitted as a job to the job scheduler. You can request the resources each job needs via argument resources, e.g.

# Limit runtime to 10 minutes and memory to 400 MiB per future, and
# load environment module 'r/4.5' in the job script
plan(future.batchtools::batchtools_slurm, resources = list(
  time = "00:10:00", mem = "400M", modules = c("r/4.5")
))

Advantages

  • You can harness the compute power of an HPC cluster from within R, without having to write job scripts or change any code.

  • You can specify the resources that the jobs need, e.g. runtime, memory, CPU cores, and environment modules, via plan().

  • All of the common job schedulers are supported, and others can be supported via a custom batchtools configuration.

Limitations

  • Each future is a separate job, which means it has to wait in the job queue before it starts. This makes the batchtools backends best suited for futures that take minutes or longer to run, rather than many small futures.

  • The batchtools registry folder, which is .future/ in the current working directory by default, must be on a file system that is shared between the machine submitting the jobs and the compute nodes.

Installation

R package future.batchtools is available on CRAN and can be installed in R as:

install.packages("future.batchtools")

Pre-release version

To install the pre-release version, use:

install.packages("future.batchtools", repos = c("https://futureverse-devel.r-universe.dev", "https://cloud.r-project.org"))