---
title: "Quick Start"
output:
  rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Quick Start}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{css echo=FALSE}
img {
    border: 0px !important;
    margin: 2em 2em 2em 2em !important;
}
code {
    border: 0px !important;
}
```

```{r echo=FALSE, results="hide"}
options(clustermq.scheduler = "local")
knitr::opts_chunk$set(
    cache = FALSE,
    echo = TRUE,
    collapse = TRUE,
    comment = "#>"
)
```

This package will allow you to send function calls as jobs on a computing
cluster with a minimal interface provided by the `Q` function:

```{r}
# load the library and create a simple function
library(clustermq)
fx = function(x) x * 2

# queue the function call on your scheduler
Q(fx, x=1:3, n_jobs=1)
```

Computations are done [entirely on the network](https://zeromq.org/)
and without any temporary files on network-mounted storage, so there is no
strain on the file system apart from starting up R once per job. All
calculations are load-balanced, i.e. workers that get their jobs done faster
will also receive more function calls to work on. This is especially useful if
not all calls return after the same time, or one worker has a high load.

## Installation

First, we need the [ZeroMQ](https://github.com/zeromq/libzmq)
system library. This is probably already installed on your system. If not, your
package manager will provide it:

```{sh, eval=FALSE}
# You can skip this step on Windows and macOS, the package binary has it
# On a computing cluster, we recommend to use Conda or Linuxbrew
brew install zeromq # Linuxbrew, Homebrew on macOS
conda install zeromq # Conda, Miniconda
sudo apt-get install libzmq3-dev # Ubuntu
sudo yum install zeromq-devel # Fedora
pacman -S zeromq # Arch Linux
```

Then install the `clustermq` package in R from CRAN:

```{r, eval=FALSE}
install.packages('clustermq')
```

Alternatively you can use the `remotes` package to install directly from Github:

```{r, eval=FALSE}
# install.packages('remotes')
remotes::install_github('mschubert/clustermq')
# remotes::install_github('mschubert/clustermq', ref="develop") # dev version
```

You should be good to go!

By default, `clustermq` will look for `sbatch` (SLURM), `bsub` (LSF), or `qsub`
(SGE) in your `$PATH` and use the scheduler that is available. If the examples
don't run out of the box, you might need to set your scheduler explicitly.

## Setting up the scheduler explicitly

An HPC cluster's scheduler ensures that computing jobs are distributed to
available worker nodes. Hence, this is what clustermq interfaces with in order
to do computations.

We currently support the [following
schedulers](https://mschubert.github.io/clustermq/articles/userguide.html#setting-up-the-scheduler)
(either locally or via SSH):

* [Multiprocess](https://mschubert.github.io/clustermq/articles/userguide.html#local-parallelization) -
  *test your calls and parallelize on cores using* `options(clustermq.scheduler="multiprocess")`
* [LSF](https://mschubert.github.io/clustermq/articles/userguide.html#lsf) - *should work without setup*
* [SGE](https://mschubert.github.io/clustermq/articles/userguide.html#sge) - *should work without setup*
* [SLURM](https://mschubert.github.io/clustermq/articles/userguide.html#slurm) - *should work without setup*
* [PBS](https://mschubert.github.io/clustermq/articles/userguide.html#pbs)/[Torque](https://mschubert.github.io/clustermq/articles/userguide.html#torque) - *needs* `options(clustermq.scheduler="PBS"/"Torque")`
* via [SSH](https://mschubert.github.io/clustermq/articles/userguide.html#ssh-connector) -
*needs* `options(clustermq.scheduler="ssh", clustermq.ssh.host=<yourhost>)`

Default submission templates [are
provided](https://github.com/mschubert/clustermq/tree/master/inst) and [can be
customized](https://mschubert.github.io/clustermq/articles/userguide.html#configuration),
e.g. to activate [compute environments or
containers](https://mschubert.github.io/clustermq/articles/userguide.html#environments).

## Examples

The package is designed to distribute arbitrary function calls on HPC worker
nodes. There are, however, a couple of caveats to observe as the R session
running on a worker does not share your local memory.

The simplest example is to a function call that is completely self-sufficient,
and there is one argument (`x`) that we iterate through:

```{r}
fx = function(x) x * 2
Q(fx, x=1:3, n_jobs=1)
```

Non-iterated arguments are supported by the `const` argument:

```{r}
fx = function(x, y) x * 2 + y
Q(fx, x=1:3, const=list(y=10), n_jobs=1)
```

If a function relies on objects in its environment that are not passed as
arguments, they can be exported using the `export` argument:

```{r}
fx = function(x) x * 2 + y
Q(fx, x=1:3, export=list(y=10), n_jobs=1)
```

If we want to use a package function we need to load it on the worker using a
`library()` call or referencing it with `package_name::`:

```{r}
fx = function(x) {
    `%>%` = dplyr::`%>%`
    x %>%
        dplyr::mutate(area = Sepal.Length * Sepal.Width) %>%
        head()
}
Q(fx, x=list(iris), n_jobs=1)
```

`clustermq` can also be used as a parallel backend for
[`foreach`](https://cran.r-project.org/package=foreach). As this is also
used by [`BiocParallel`](http://bioconductor.org/packages/release/bioc/html/BiocParallel.html),
we can run those packages on the cluster as well:

```{r}
library(foreach)
register_dopar_cmq(n_jobs=2, memory=1024) # accepts same arguments as `workers`
foreach(i=1:3) %dopar% sqrt(i) # this will be executed as jobs
```

```{r eval=FALSE}
library(BiocParallel)
register(DoparParam()) # after register_dopar_cmq(...)
bplapply(1:3, sqrt)
```

More examples are available in [the
user guide](https://mschubert.github.io/clustermq/articles/userguide.html).

## Usage

The following arguments are supported by `Q`:

 * `fun` - The function to call. This needs to be self-sufficient (because it
        will not have access to the `master` environment)
 * `...` - All iterated arguments passed to the function. If there is more than
        one, all of them need to be named
 * `const` - A named list of non-iterated arguments passed to `fun`
 * `export` - A named list of objects to export to the worker environment

Behavior can further be fine-tuned using the options below:

 * `fail_on_error` - Whether to stop if one of the calls returns an error
 * `seed` - A common seed that is combined with job number for reproducible results
 * `memory` - Amount of memory to request for the job (`bsub -M`)
 * `n_jobs` - Number of jobs to submit for all the function calls
 * `job_size` - Number of function calls per job. If used in combination with
        `n_jobs` the latter will be overall limit
 * `chunk_size` - How many calls a worker should process before reporting back
        to the master. Default: every worker will report back 100 times total

The full documentation is available by typing `?Q`.

## Comparison to other packages

There are some packages that provide high-level parallelization of R function calls
on a computing cluster. A thorough comparison of features and performance is available
[on the wiki](https://github.com/mschubert/clustermq/wiki#comparison-to-other-packages).

Briefly, we compare how long it takes different HPC scheduler tools to submit, run
and collect function calls of negligible processing time (multiplying a numeric
value by 2). This serves to quantify the maximum throughput we can reach with
`BatchJobs`, `batchtools` and `clustermq`.

We find that `BatchJobs` is unable to process 10<sup>6</sup> calls or more but
produces a reproducible `RSQLite` error. `batchtools` is able to process more
function calls, but the file system practically limits it at about
10<sup>6</sup> calls. `clustermq` has no problems processing 10<sup>9</sup>
calls, and is still faster than `batchtools` at 10<sup>6</sup> calls.

![](http://image.ibb.co/cRgYNR/plot.png)

In short, use `ClusterMQ` if you want:

* a one-line solution to run cluster jobs with minimal setup
* access cluster functions from your local Rstudio via SSH
* fast processing of many function calls without network storage I/O

Use [`batchtools`](https://github.com/mllg/batchtools) if:

* want to use a mature and well-tested package
* don't mind that arguments to every call are written to/read from disc
* don't mind there's no load-balancing at run-time

Use [Snakemake](https://snakemake.readthedocs.io/en/latest/) (or
[`flowr`](https://github.com/sahilseth/flowr),
[`remake`](https://github.com/richfitz/remake),
[`drake`](https://github.com/ropensci/drake)) if:

* you want to design and run a pipeline of different tools

Don't use [`batch`](https://cran.r-project.org/package=batch)
(last updated 2013) or [`BatchJobs`](https://github.com/tudo-r/BatchJobs)
(issues with SQLite on network-mounted storage).
