# future.batchtools: A Future API for Parallel and Distributed Processing using ‘batchtools’ ## TL;DR Here is an example of how to evaluate R expressions on a Slurm high-performance computing (HPC) cluster from within R. ``` r library(future) # Limit runtime to 10 minutes and memory to 400 MiB per future, # request a parallel environment with four slots on a single host. # On this system, R is available via environment module 'r'. By # specifying 'r/4.5', 'module load r/4.5' will be added to # the submitted job script. plan(future.batchtools::batchtools_slurm, resources = list( time = "00:10:00", mem = "400M", nodes = 1, ntasks = 4, modules = c("r/4.5") )) # Give it a spin f <- future({ data.frame( hostname = Sys.info()[["nodename"]], os = Sys.info()[["sysname"]], cores = unname(parallelly::availableCores()), modules = Sys.getenv("LOADEDMODULES") ) }) info <- value(f) print(info) #> hostname os cores modules #> 1 n12 Linux 4 r/4.5.2 ``` ## Introduction The **[future](https://cran.r-project.org/package=future)** package provides a generic API for using futures in R. A future is a simple yet powerful mechanism to evaluate an R expression and retrieve its value at some point in time. Futures can be resolved in many different ways depending on which strategy is used. There are various types of synchronous and asynchronous futures to choose from in the **[future](https://cran.r-project.org/package=future)** package. This package, **[future.batchtools](https://cran.r-project.org/package=future.batchtools)**, provides a type of futures that utilizes the **[batchtools](https://cran.r-project.org/package=batchtools)** package. This means that *any* type of backend that the **batchtools** package supports can be used as a future. More specifically, **future.batchtools** will allow you or users of your package to leverage the compute power of high-performance computing (HPC) clusters via a simple switch in settings - without having to change any code at all. For instance, the following two future R expressions will be processed by parallel R workers launched on different compute nodes by the specified job scheduler: ``` r library(future) plan(future.batchtools::batchtools_slurm) f_x <- future({ Sys.sleep(5); 3.14 }) f_y <- future({ Sys.sleep(5); 2.71 }) x <- value(f_x) y <- value(f_y) x + y #> [1] 5.85 ``` This is just a toy example to illustrate what futures look like and how to work with them. For an introduction as well as full details on how to use futures, please see or consult the package vignettes of the **[future](https://cran.r-project.org/package=future)** package. ## Demos The **[future](https://cran.r-project.org/package=future)** package provides a demo using futures for calculating a set of Mandelbrot planes. The demo does not assume anything about what type of futures are used. *The user has full control of how futures are evaluated*. For instance, to use local batchtools futures, run the demo as: ``` r library(future) plan(future.batchtools::batchtools_local) demo("mandelbrot", package = "future", ask = FALSE) ``` ## Available batchtools backends The **future.batchtools** package implements a generic future wrapper for all batchtools backends. Below are the most common types of batchtools backends. For other types of parallel and distributed backends, please see . | Backend | Description | Alternative in future package | |:---|:---|:---| | `batchtools_lsf` | Futures are evaluated via a [Load Sharing Facility (LSF)](https://en.wikipedia.org/wiki/Platform_LSF) job scheduler | N/A | | `batchtools_openlava` | Futures are evaluated via an [OpenLava](https://en.wikipedia.org/wiki/OpenLava) job scheduler | N/A | | `batchtools_sge` | Futures are evaluated via a [Sun/Son of/Oracle/Univa/Altair Grid Engine (SGE)](https://en.wikipedia.org/wiki/Oracle_Grid_Engine) job scheduler | N/A | | `batchtools_slurm` | Futures are evaluated via a [Slurm](https://en.wikipedia.org/wiki/Slurm_Workload_Manager) job scheduler | N/A | | `batchtools_torque` | Futures are evaluated via a [TORQUE](https://en.wikipedia.org/wiki/TORQUE) / PBS job scheduler | N/A | | `batchtools_custom` | Futures are evaluated via a custom batchtools configuration R script or via a set of cluster functions | N/A | | `batchtools_multicore` | parallel evaluation by forking the current R process | `plan(multicore)` | | `batchtools_local` | sequential evaluation in a separate R process (on current machine) | `plan(cluster, workers = I(1))` | ## Installation R package future.batchtools is available on [CRAN](https://cran.r-project.org/package=future.batchtools) and can be installed in R as: ``` r install.packages("future.batchtools") ``` ### Pre-release version To install the pre-release version that is available in Git branch `develop` on GitHub, use: ``` r remotes::install_github("futureverse/future.batchtools", ref="develop") ``` This will install the package from source. # Package index ## All functions - [`batchtools_bash()`](https://future.batchtools.futureverse.org/reference/batchtools_bash.md) [`makeClusterFunctionsBash()`](https://future.batchtools.futureverse.org/reference/batchtools_bash.md) : A batchtools Bash backend that resolves futures sequentially via a Bash template script - [`batchtools_interactive()`](https://future.batchtools.futureverse.org/reference/batchtools_interactive.md) : A batchtools backend that resolves futures sequentially in the current R session - [`batchtools_local()`](https://future.batchtools.futureverse.org/reference/batchtools_local.md) : A batchtools backend that resolves futures sequentially in transient background R sessions - [`batchtools_lsf()`](https://future.batchtools.futureverse.org/reference/batchtools_lsf.md) : A batchtools LSF backend resolves futures in parallel via a Load Sharing Facility (LSF) job scheduler - [`batchtools_multicore()`](https://future.batchtools.futureverse.org/reference/batchtools_multicore.md) : A batchtools backend that resolves futures in parallel via forked background R processes - [`batchtools_openlava()`](https://future.batchtools.futureverse.org/reference/batchtools_openlava.md) : A batchtools OpenLava backend resolves futures in parallel via an OpenLava job scheduler - [`batchtools_sge()`](https://future.batchtools.futureverse.org/reference/batchtools_sge.md) : A batchtools SGE backend resolves futures in parallel via a Sun/Son of/Oracle/Univa/Altair Grid Engine job scheduler - [`batchtools_slurm()`](https://future.batchtools.futureverse.org/reference/batchtools_slurm.md) : A batchtools Slurm backend resolves futures in parallel via a Slurm job scheduler - [`batchtools_torque()`](https://future.batchtools.futureverse.org/reference/batchtools_torque.md) : A batchtools TORQUE backend resolves futures in parallel via a TORQUE/PBS job scheduler - [`future.batchtools`](https://future.batchtools.futureverse.org/reference/future.batchtools.md) [`future.batchtools-package`](https://future.batchtools.futureverse.org/reference/future.batchtools.md) : future.batchtools: A Future for batchtools - [`makeClusterFunctionsSlurm2()`](https://future.batchtools.futureverse.org/reference/makeClusterFunctionsSlurm2.md) : ClusterFunctions for Slurm Systems (patched) - [`zzz-future.batchtools.options`](https://future.batchtools.futureverse.org/reference/zzz-future.batchtools.options.md) [`future.batchtools.options`](https://future.batchtools.futureverse.org/reference/zzz-future.batchtools.options.md) [`future.batchtools.delete`](https://future.batchtools.futureverse.org/reference/zzz-future.batchtools.options.md) [`future.batchtools.expiration.tail`](https://future.batchtools.futureverse.org/reference/zzz-future.batchtools.options.md) [`future.batchtools.output`](https://future.batchtools.futureverse.org/reference/zzz-future.batchtools.options.md) [`future.batchtools.workers`](https://future.batchtools.futureverse.org/reference/zzz-future.batchtools.options.md) [`R_FUTURE_CACHE_PATH`](https://future.batchtools.futureverse.org/reference/zzz-future.batchtools.options.md) [`R_FUTURE_BATCHTOOLS_DELETE`](https://future.batchtools.futureverse.org/reference/zzz-future.batchtools.options.md) [`R_FUTURE_BATCHTOOLS_EXPIRATION_TAIL`](https://future.batchtools.futureverse.org/reference/zzz-future.batchtools.options.md) [`R_FUTURE_BATCHTOOLS_OUTPUT`](https://future.batchtools.futureverse.org/reference/zzz-future.batchtools.options.md) [`R_FUTURE_BATCHTOOLS_WORKERS`](https://future.batchtools.futureverse.org/reference/zzz-future.batchtools.options.md) : Options used for batchtools futures # Articles ### All vignettes - [A Future for batchtools](https://future.batchtools.futureverse.org/articles/future.batchtools.md):