Lazy batchtools futures only creates the internal batchtools registry when the future is launched.
Removed S3 generic functions await(), finished(), and status(), which were functions that were used for internal purposes.
It is now possible to configure the default number of workers on the job queue of an HPC scheduler via either R option ‘future.batchtools.workers’ or environment variable ‘R_FUTURE_BATCHTOOLS_WORKERS’.
It is now possible to configure the batchtools registries that are used by batchtools futures via new argument ‘registry’ to plan(). This argument should be a named list of parameters recognized by the ‘batchtools’ package, e.g. plan(batchtools_sge, registry = list(…)). For notable example, see below news entries.
The default working directory for batchtools futures is the current working directory of R _when_ the batchtools future is created. This corresponds to specifying plan(batchtools_nnn, registry = list(work.dir = NULL). Sometimes it is useful to use a explicit working directory that is guaranteed to be available on all workers on a shared file system, e.g. plan(batchtools_nnn, registry = list(work.dir = “~”)).
It is possible to control if and how batchtools should use file compression for exported globals and results by specifying batchtools registry parameter ‘compress’. For example, to turn off file compression, use plan(batchtools_nnn, registry = list(compress = FALSE)).
The default location of the ‘.future’ folder can be controlled by R option ‘future.cache.path’ or environment variable ‘R_FUTURE_CACHE_PATH’.
batchtools_custom() and BatchtoolsFuture gained argument ‘conf.file’. Using plan(batchtools_custom) will now use any batchtools configuration file (an R script) found on the batchtools::findConfFile() search path.
Setting option ‘future.delete’ to FALSE will now prevent removal of the batchtools registry folders.
When a batchtools job expires, for instance when the scheduler terminates it because the job was running out of its allocated resources, then a BatchtoolsFutureError is produced which by default outputs the tail of the output logged by batchtools. The default number of lines displayed from the end is now increased from six to 48 - a number which now can be set via option ‘future.batchtools.expiration.tail’.
Now a more informative error message is produced if a batchtools *.tmpl template file was not found.
Debug messages are now prepended with a timestamp.
If the built-in attempts of batchtools for finding a default template file fails, then system(“templates”, package = “future.batchtools”) is searched for template files as well. Currently, there exists a `torque.tmpl` file.
A job’s name in the scheduler is now set as the future’s label (requires batchtools 0.9.4 or newer). If no label is specified, the default job name is controlled by batchtools.
The period between each poll of the scheduler to check whether a future (job) is finished or not now increases geometrically as a function of number of polls. This lowers the load on the scheduler for long running jobs.
The error message for expired batchtools futures now include the last few lines of the logged output, which sometimes includes clues on why the future expired. For instance, if a TORQUE/PBS job use more than the allocated amount of memory it might be terminated by the scheduler leaving the message “PBS: job killed: vmem 1234000 exceeded limit 1048576” in the output.
print() for BatchtoolsFuture returns the object invisibly.
Calling future_lapply() with functions containing globals part of non-default packages would when using batchtools futures give an error complaining that the global is missing. This was due to updates in future (>= 1.4.0) that broke this package.
loggedOutput() for BatchtoolsFuture would always return NULL unless an error had occurred.
Added batchtools_custom() for specifying batchtools futures using any type of batchtools cluster functions.
batchtools_template(pathname = NULL, type = <type>) now relies on the batchtools package for locating the <type> template file.
nbrOfWorkers() for batchtools futures now defaults to +Inf unless the evaluator’s ‘workers’ or ‘cluster.functions’ specify something else.
Renamed argument ‘pathname’ to ‘template’ for batchtools_<tmpl>() functions.
TESTS: Added test of future_lapply() for batchtools backends.
TESTS: Added optional tests for batchtools_* HPC schedulers listed in environment variable ‘R_FUTURE_TESTS_STRATEGIES’.