Getting help

The best way to seek user support is the nimble-users list. Information on how to join can be found at https://r-nimble.org.

Illustration: A simple occupancy model

Occupancy models are used for data from repeated visits to a set of sites, where the detection (1) or non-detection (0) of a species of interest is recorded on each visit. Define y[i, j] as the observation from site i on visit j. y[i, j] is 1 if the species was seen and 0 if not.

Typical code for for an occupancy model would be as follows. Naturally, this is written for nimble, but the same code should work for JAGS or BUGS (WinBUGS or OpenBUGS) when used as needed for those packages.

library(nimble)
#> nimble version 0.9.0 is loaded.
#> For more information on NIMBLE and a User Manual,
#> please visit http://R-nimble.org.
#> 
#> Attaching package: 'nimble'
#> The following object is masked from 'package:stats':
#> 
#>     simulate
library(nimbleEcology)
#> Loading nimbleEcology. 
#> Registering the following user-defined functions: 
#> dOcc, dDynOcc, dCJS, dHMM, dDHMM, dNmixture

occupancy_code <- nimbleCode({
  psi ~ dunif(0,1)
  p ~ dunif (0,1)
  for(i in 1:nSites) {
    z[i] ~ dbern(psi)
    for(j in 1:nVisits) {
      y[i, j] ~ dbern(z[i] * p)
    }
  }
})

In this code:

• psi is occupancy probability;
• p is detection probability;
• z[i] is the latent state of whether a site is really occupied (z[i] = 1) or not (z[i] = 0);
• nSites is the number of sites; and
• nVisits is the number of sampling visits to each site.

The new version of this model using nimbleEcology’s specialized occupancy distribution will only work in nimble (not JAGS or BUGS). It is:

occupancy_code_new <- nimbleCode({
  psi ~ dunif(0,1)
  p ~ dunif (0,1)
  for(i in 1:nSites) {
    y[i, 1:nVisits] ~ dOcc_s(probOcc = psi, probDetect = p, len = nVisits)
  }
})

In the new code, the vector of data from all visits to site i, namely y[i, 1:nVisits], has its likelihood contribution calculated in one step, dOcc_s. This occupancy distribution calculates the total probability of the data by summing over the cases that the site is occupied or unoccupied. That means that z[i] is not needed in the model, and MCMC will not need to sample over z[i]. Details of all calculations, and discussion of the pros and cons of changing models in this way, are given later this vignette.

The _s part of dOcc_s means that p is a scalar, i.e. it does not vary with visit. If it should vary with visit, a condition sometimes described as being time-dependent, it would need to be provided as a vector, and the distribution function should be dOcc_v.

MCMC with both versions of the example occupancy model.

We can run an MCMC for this model in the following steps:

1. Build the model.
2. Configure the MCMC.
3. Build the MCMC.
4. Compile the model and MCMC.
5. Run the MCMC.
6. Extract the samples.

The function nimbleMCMC does all of these steps for you. The function runMCMC does steps 5-6 for you, with convenient management of options such as discarding burn-in samples. The full set of steps allows great control over how you use a model and configure and use an MCMC. We will go through the steps 1-4 and then use runMCMC for steps 5-6.

In this example, we also need simulated data. We can use the same model to create simulated data, rather than writing separate R code for that purpose.

occupancy_model <- nimbleModel(occupancy_code, 
                               constants = list(nSites = 50, nVisits = 5))
#> defining model...
#> building model...
#> running calculate on model (any error reports that follow may simply reflect missing values in model variables) ... 
#> checking model sizes and dimensions... This model is not fully initialized. This is not an error. To see which variables are not initialized, use model$initializeInfo(). For more information on model initialization, see help(modelInitialization).
#> model building finished.

occupancy_model$psi <- 0.7
occupancy_model$p <- 0.15
simNodes <- occupancy_model$getDependencies(c("psi", "p"), self = FALSE)
occupancy_model$simulate(simNodes)
occupancy_model$z
#>  [1] 1 1 0 1 1 1 0 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 1 1 0 0 0 1 1 1 1 1 1 1
#> [36] 1 0 1 1 0 1 1 1 1 1 1 1 1 1 1
head(occupancy_model$y, 10) ## first 10 rows
#>       [,1] [,2] [,3] [,4] [,5]
#>  [1,]    0    0    0    0    1
#>  [2,]    0    0    0    0    0
#>  [3,]    0    0    0    0    0
#>  [4,]    0    0    0    0    0
#>  [5,]    0    0    0    0    1
#>  [6,]    0    0    0    0    0
#>  [7,]    0    0    0    0    0
#>  [8,]    0    1    0    0    0
#>  [9,]    0    0    0    0    0
#> [10,]    0    0    0    0    0
occupancy_model$setData('y') ## set "y" as data

MCMCconf <- configureMCMC(occupancy_model)
MCMC <- buildMCMC(occupancy_model)

## These can be done in one step, but many people
## find it convenient to do it in two steps.
Coccupancy_model <- compileNimble(occupancy_model)
#> compiling... this may take a minute. Use 'showCompilerOutput = TRUE' to see C++ compilation details.
#> compilation finished.
CMCMC <- compileNimble(MCMC, project = occupancy_model)
#> compiling... this may take a minute. Use 'showCompilerOutput = TRUE' to see C++ compilation details.
#> compilation finished.

samples <- runMCMC(CMCMC, niter = 10000, nburnin = 500, thin = 10)
#> runMCMC's handling of nburnin changed in nimble version 0.6-11. Previously, nburnin samples were discarded *post-thinning*.  Now nburnin samples are discarded *pre-thinning*.  The number of samples returned will be floor((niter-nburnin)/thin).
#> running chain 1...

occupancy_model_new <- nimbleModel(occupancy_code_new, 
                                   constants = list(nSites = 50, nVisits = 5),
                                   data = list(y = occupancy_model$y),
                                   inits = list(psi = 0.7, p = 0.15))
#> defining model...
#> building model...
#> setting data and initial values...
#> running calculate on model (any error reports that follow may simply reflect missing values in model variables) ... 
#> checking model sizes and dimensions...
#> model building finished.
MCMC_new <- buildMCMC(occupancy_model_new) ## This will use default call to configureMCMC.
Coccupancy_model_new <- compileNimble(occupancy_model_new)
#> compiling... this may take a minute. Use 'showCompilerOutput = TRUE' to see C++ compilation details.
#> compilation finished.
CMCMC_new <- compileNimble(MCMC_new, project = occupancy_model_new)
#> compiling... this may take a minute. Use 'showCompilerOutput = TRUE' to see C++ compilation details.
#> compilation finished.
samples_new <- runMCMC(CMCMC_new, niter = 10000, nburnin = 500, thin = 10)
#> runMCMC's handling of nburnin changed in nimble version 0.6-11. Previously, nburnin samples were discarded *post-thinning*.  Now nburnin samples are discarded *pre-thinning*.  The number of samples returned will be floor((niter-nburnin)/thin).
#> running chain 1...

#> Registered S3 methods overwritten by 'ggplot2':
#>   method         from 
#>   [.quosures     rlang
#>   c.quosures     rlang
#>   print.quosures rlang
#> ── Attaching packages ─────────────────────────────────────────────────────────────────────────────────────────────────────────────── tidyverse 1.2.1 ──
#> ✔ ggplot2 3.1.1     ✔ purrr   0.3.2
#> ✔ tibble  2.1.2     ✔ dplyr   0.8.1
#> ✔ tidyr   0.8.3     ✔ stringr 1.4.0
#> ✔ readr   1.3.1     ✔ forcats 0.4.0
#> ── Conflicts ────────────────────────────────────────────────────────────────────────────────────────────────────────────────── tidyverse_conflicts() ──
#> ✖ dplyr::filter() masks stats::filter()
#> ✖ dplyr::lag()    masks stats::lag()

Use the new version for something different: maximum likelihood or maximum a posteriori estimation.

It is useful that the new way to write the model does not have discrete latent states. Since this example also does not have other latent states or random effects, we can use it simply as a likelihood or posterior density calculator. More about how to do so can be found in the nimble User Manual. Here we illustrate making a compiled nimbleFunction for likelihood calculations for parameters psi and p and maximizing the likelihood using R’s optim.

CalcLogLik <- nimbleFunction(
  setup = function(model, nodes)
    calcNodes <- model$getDependencies(nodes, self = FALSE),
  run = function(v = double(1)) {
    values(model, nodes) <<- v
    return(model$calculate(calcNodes))
    returnType(double(0))
  }
)
OccLogLik <- CalcLogLik(occupancy_model_new, c("psi", "p"))
COccLogLik <- compileNimble(OccLogLik, project = occupancy_model_new)
#> compiling... this may take a minute. Use 'showCompilerOutput = TRUE' to see C++ compilation details.
#> compilation finished.
optim(c(0.5, 0.5), COccLogLik$run, control = list(fnscale = -1))$par
#> [1] 0.7483715 0.1229434

A note on static typing

Some distribution names are followed by a suffix indicating the type of some parameters, for example the _s in dOcc_s. NIMBLE uses a static typing system, meaning that a function must know in advance if a certain argument will be a scalar, vector, or matrix. There may be notation such as sv or svm if there are two or three parameters that can be time-independent (s) or time-dependent (v or m) in one or more dimensions. In general, s corresponds to a scalar argument, v to a vector argument, and m to a matrix argument. The order of these type indicators will correspond to the order of the relevant parameters, but always check the documentation when using a new distribution with the R function ? (e.g., both ?dOcc and ?dOcc_s work).

The static typing requirement may be relaxed somewhat in the future.

Capture-recapture (dCJS)

Cormack-Jolly-Seber models give the probability of a capture history for each of many individuals, conditional on first capture, based on parameters for survival and detection probability. nimbleEcology provides a distribution for individual capture histories, with variants for time-independent and time-dependent survival and/or detection probability. Of course, the survival and detection parameters for the CJS probability may themselves depend on other parameters and/or random effects. The rest of this summary will focus on one individual’s capture history.

Define \(\phi_t\) as survival from time \(t\) to \(t+1\) and \(\mathbf{\phi} = (\phi_1, \ldots, \phi_T)\), where \(T\) is the length of the series. We use “time” and “sampling occasion” as synonyms in this context, so \(T\) is the number of sampling occasions. (Be careful with indexing. Sometimes you might see \(\phi_t\) defined as survival from time \(t-1\) to \(t\).) Define \(p_t\) as detection probability at time \(t\) and \(\mathbf{p} = (p_1, \ldots, p_T)\). Define the capture history as \(\mathbf{y} = y_{1:T} = (y_1, \ldots, y_T)\), where each \(y_t\) is 0 or 1. The notation \(y_{i:j}\) means the sequence of observations from time \(i\) to time \(j\). The first observation, \(y_0 = 1\), is included in \(\mathbf{y}\).

There are multiple ways to write the CJS probability. We will do so in a state-space format because that leads to the more general DHMM case next. The probability of observations given parameters, \(P(\mathbf{y} | \mathbf{\phi}, \mathbf{p})\), is factored as: \[ P(\mathbf{y} | \mathbf{\phi}, \mathbf{p}) = \prod_{t = 1}^T P(y_{t} | y_{1:t-1}, \mathbf{\phi}, \mathbf{p}) \]

Each factor \(P(y_{t} | y_{1:t-1}, \mathbf{\phi}, \mathbf{p})\) is calculated as: \[ P(y_{t} | y_{1:t-1}, \mathbf{\phi}, \mathbf{p}) = I(y_t = 1) (A_t p_t) + I(y_t = 0) (A_t (1-p_t) + (1-A_t)) \] The indicator function \(I(y_t = 1)\) is 1 if it \(y_t\) is 1, 0 otherwise, and vice versa for \(I(y_t = 0)\). Here \(A_t\) is the probability that the individual is alive at time \(t\) given \(y_{1:t-1}\), the data up to the previous time. This is calculated as: \[ A_t = G_{t-1} \phi_{t-1} \] where \(G_{t}\) is the probability that the individual is alive at time \(t\) given \(y_{1:t}\), the data up to the current time. This is calculated as: \[ G_{t} = I(y_t = 1) 1 + I(y_t = 0) \frac{A_t (1-p_t)}{A_t (1-p_t) + (1-A_t)} \] The sequential calculation is initialized with \(G_0 = 1\). For time step \(t\), we calculate \(A_t\), then \(P(y_t | y_{1:t-1}, \mathbf{\phi}, \mathbf{p})\), then \(G_t\), leaving us ready for time step \(t+1\). This is a trivial hidden Markov model where the latent state, alive or dead, is not written explicitly.

In the cases with time-independent survival or capture probability, we simply drop the time indexing for the corresponding parameter.

Hidden Markov models (HMMs) and Dynamic hidden Markov models (DHMMs)

Hidden Markov models give the probability of detection history that can handle:

• Detections in different states, such as locations or breeding states,
• Incorrect state information, such as if breeding state might not be observed correctly,
• Probabilities of transitions between states.

In a HMM, “dead” is simply another state, and “unobserved” is a possible observation. Thus, HMMs are generalizations of the CJS model. In capture-recapture, HMMs encompass multi-state and multi-event models.

We again use the factorization \[ P(\mathbf{y} | \mathbf{\phi}, \mathbf{p}) = \prod_{t = 1}^T P(y_{t} | y_{1:t-1}, \mathbf{\phi}, \mathbf{p}) \] In the case of a HMM, \(y_t\) is the recorded state at time \(t\), taking an integer value, with the possibility that one possible value corresponds to “unobserved”. Define \(S\) as the number of possible states and \(K\) as the number of possible observed states. Observed states need not correspond to real states. For example, often there is an observed state for “unobserved”. Another example is that two real states might never be distinguishable in observations, so they may correpond to only one observed state.

Define \(A_{i, t}\) as the probability that the individual is in state \(i\) at time \(t\), given \(y_{1:t-1}\), the data up to the previous time. Define \(p_{i,j,t}\) as the probability that, at time \(t\), an individual in state \(i\) is observed in state \(j\). Then \(P(y_{t} | y_{1:t-1}, \mathbf{\phi}, \mathbf{p})\) is calculated as: \[ P(y_{t} | y_{1:t-1}, \mathbf{\phi}, \mathbf{p}) = \sum_{i=1}^S A_{i, t} p_{i, y_t, t} \] where \(j = y_t\) is the observed state of the individual.

Define \(G_{i, t}\) as the probability that the individual is in state \(i\) at time \(t\) given \(y_{1:t}\), the data up to the current time. Define \(\psi_{i, j, t}\) as the probability that, from time \(t\) to \(t+1\), an individual in state \(i\) transitions to state \(j\). Note that the time-step is different here (\(t\) to \(t+1\)) than for the CJS model above (which uses \(t-1\) to \(t\)). Then the calculation of \(A_{j,t}\) for each possible state \(j\) is given by: \[ A_{j, t} = \sum_{i=1}^S G_{i, t-1} \psi_{i, j, t-1}, \mbox{ for } j=1,\ldots,S \]

Finally, calculation of \(G_{i, t}\) for each possible state \(i\) is done by conditioning on \(y_t\) as follows: \[ G_{i, t} = \frac{A_{i, t} p_{i, y_t, t}}{P(y_{t} | y_{1:t-1}, \mathbf{\phi}, \mathbf{p})}, \mbox{ for } j=1,\ldots,S \]

The calculation of the total probability of a detection history with uncertain state information starts by initializing \(A_{i, 1}\) to some choice of initial (or prior) probability of being in each state before any observations are made. Then, starting with \(t = 1\), we calculate \(P(y_{t} | y_{1:t-1}, \mathbf{\phi}, \mathbf{p})\). If \(t < T\), we prepare for time step \(t+1\) by calculating \(G_{i, t}\) for each \(i\) followed by \(A_{i, t+1}\) for each \(i\).

If the transition probabilities, \(\psi_{i, j, t}\), actually depend on \(t\), we refer to the model as “Dynamic”, namely a DHMM instead of an HMM. It may also be the case that observation probabilities \(p_{i, j, t}\) depend on \(t\) or not.

HMM and DHMM distributions in nimbleEcology

HMMs and DHMMs are available in four distributions in nimbleEcology. These differ only in whether transition and/or observation probabilities are time-dependent or time-independent, yielding four combinations:

• dHMM: Both are time-independent.
• dDHMM: State transitions are time-dependent (dynamic). Observation probabilities are time-independent.
• dHMMo: Observation probabilities are time-dependent. State transitions are time-independent (not dynamic).
• dDHMMo: Both are time-dependent.

In this notation, the leading D is for “dynamic” (time-dependent state transitions), while the trailing “o” is for “observations” being time-dependent.

The usage for each is similar. An example for dDHMM is:

y[i, 1:T] ~ dDHMM(init = initial_probs[i, 1:T], obsProb = p[1:nStates, 1:nCat], transProb = Trans[i, 1:nStates, 1:nStates, 1:(T-1)], len = T)

Note the following points:

• As above, this is written as if i indexes individuals in the model, but this is arbitrary as an example.
• nStates is \(S\) above.
• nCat is \(K\) above.
• init[i] is the initial probability of being in state i, namely \(A_{i, 1}\) above.
• obsProb[i, j] (i.e., p[i, j] in this example) is probability of observing an individual who is truly in state i as being in observation state j. This is \(p_{i, j}\) above if indexing by \(t\) is not needed. If observation probabilities are time-dependent (in dHMMo and dDHMMo), then obsProb[i, j, t] is \(p_{i, j, t}\) above.
• transProb[i, j, t] (i.e., Trans[i, j, t] in this example) is the probability that an individual changes from state i to state j during time step t to t+1. This is \(\psi_{i,j,t}\) above.
  
• len is the length of the observation record, T in this example.
• len must match the length of the data, y[i, 1:T] in this example.
• The dimensions of obsProb must be \(K \times S\) in the time-independent case (dHMM or dDHMM) or \(K \times S \times T\) in the time-dependent case (dHMMo or dDHMMo).
• The dimensions of transProb must be \(S \times S\) in the time-independent case (dHMM or dHMMo) or \(S \times S \times (T-1)\) in the time-dependent case (dDHMM or dDHMMo). The last dimension is one less than \(T\) because no transition to time \(T+1\) is needed.

Occupancy

An occupancy model gives the probability of a series of detection/non-detection records for a species during multiple visits to each of multiple site. The occupancy distributions in nimbleEcology give the probability of the detection history for one site, so this summary focuses on data from one site.

Define \(y_t\) to be the observation at time \(t\), with \(y_t = 1\) for a detection and \(y_t = 0\) for a non-detection. Again, we use “time” as a synonym for “sampling occasion”. Again, define the vector of observations as \(\mathbf{y} = (y_1, \ldots, y_T)\), where \(T\) is the number of sampling occasions.

Define \(\psi\) as the probability that a site is occupied. Define \(p_t\) as the probability of a detection on sampling occasion \(t\) if the site is occupied, and \(\mathbf{p} = (p_1, \ldots, p_T)\). Then the probability of the data given the parameters is: \[ P(\mathbf{y} | \psi, \mathbf{p}) = \psi \prod_{t = 1}^T p_t^{y_t} (1-p_t)^{1-y_t} + (1-\psi) I\left(\sum_{t=1}^T y_t= 0 \right) \] The indicator function usage in the last term, \(I(\cdot)\), is 1 if the given summation is 0, i.e. if no detections were made. Otherwise it is 0.

Dynamic occupancy

Dynamic occupancy models give the probability of detection records from multiple seasons (primary periods) in each of which there were multiple sampling occasions (secondary periods) at each of multiple sites. The dynamic occupancy distribution in nimbleEcology provides probability calculations for data from one site at a time.

We will use “year” for primary periods and “time” or “sampling occasion” as above for secondary periods. Define \(y_{r, t}\) as the observation (1 or 0) on sampling occasion \(t\) of year \(r\). Define \(\mathbf{y}_r\) as the detection history in year \(r\), i.e. \(\mathbf{y}_r = (y_{r, 1}, \ldots, y_{r, T})\) . Define \(\phi_t\) as the probability of being occupied at time \(t+1\) given the site was occupied at time \(t\), called “persistence”. Define \(\gamma_t\) as the probability of being occupied at time \(t+1\) given the site was unoccupied at time \(t\), called “colonization”. Define \(p_{r, t}\) as the detection probability on sampling occasion \(t\) of year \(r\) given the site is occupied.

The probability of all the data given parameters is: \[ P(\mathbf{y} | \mathbf{\phi}, \mathbf{\gamma}, \mathbf{p}) = \prod_{r = 1}^R P(\mathbf{y}_{r} | \mathbf{y}_{1:r-1}, \mathbf{\phi}, \mathbf{\gamma}, \mathbf{p}) \] Each factor \(P(\mathbf{y}_{r} | \mathbf{y}_{1:r-1}, \mathbf{\phi}, \mathbf{\gamma}, \mathbf{p})\) is calculated as: \[ P(\mathbf{y}_{r} | \mathbf{y}_{1:r-1}, \mathbf{\phi}, \mathbf{\gamma}, \mathbf{p}) = A_{r} \prod_{t = 1}^T p_{r,t}^{y_{r,t}} (1-p_{r,t})^{1-y_{r,t}} + (1-A_{r}) I\left(\sum_{t=1}^T y_{r,t} = 0 \right) \] Here \(A_r\) is the probability that the site is occupied in year \(r\) given observations up to the previous year \(\mathbf{y}_{1:r-1}\). Otherwise, this equation is just like the occupancy model above, except there are indices for year \(r\) in many places. \(A_r\) is calculated as: \[ A_r = G_{r-1} \phi_{r-1} + (1-G_{r-1}) \gamma_{r-1} \] Here \(G_r\) is the probability that the site is occupied given the data up to time \(r\), \(\mathbf{y}_{1:r}\). This is calculated as \[ G_r = \frac{A_{r} \prod_{t = 1}^T p_{r,t}^{y_{r,t}} (1-p_{r,t})^{1-y_{r,t}}}{\mathbf{y}_{r} | \mathbf{y}_{1:r-1}, \mathbf{\phi}, \mathbf{\gamma}, \mathbf{p})} \]

The sequential calculation is initiated with \(A_1\), which is natural to think of as “\(\psi_1\)”, probability of occupancy in the first year. Then for year \(r\), starting with \(r = 1\), we calculate \(P(\mathbf{y}_{r} | \mathbf{y}_{1:r-1}, \mathbf{\phi}, \mathbf{\gamma}, \mathbf{p})\). If \(r < R\), we calculate \(G_r\) and then \(A_{r+1}\), leaving us ready to increment \(r\) and iterate.

N-mixture

An N-mixture model gives the probability of a set of counts from repeated visits to each of multiple sites. The N-mixture distribution in nimbleEcology gives probability calculations for data from one site.

Define \(y_t\) as the number of individuals counted at the site on sampling occasion (time) \(t\). Define \(\mathbf{y} = (y_1, \ldots, y_t)\). Define \(\lambda\) as the average density of individuals, such that the true number of individuals, \(N\), follows a Poisson distribution with mean \(\lambda\). Define \(p_t\) to be the detection probability for each individual at time \(t\), and \(\mathbf{p} = (p_1, \ldots, p_t)\).

The probability of the data given the parameters is: \[ P(\mathbf{y} | \lambda, \mathbf{p}) = \sum_{N = 1}^\infty P(N | \lambda) \prod_{t = 1}^T P(y_t | N) \] where \(P(N | \lambda)\) is a Poisson probability and \(P(y_t | N)\) is a binomial probability. That is, \(y_t \sim \mbox{binomial}(N, p_t)\), and the \(y_t\)s are independent.

In practice, the summation over \(N\) can start at a value greater than 0 and must be truncated at some value \(\lt \infty\). Two options are provided for the range of summation:

1. Start the summation at the largest value of \(y_t\) (there must be at least this many individuals) and truncate it at a value of \(maxN\) provided by the user.
2. The following heuristic can be used.

If we consider a single \(y_t\), then \(N - y_t | y_t \sim \mbox{Poisson}(\lambda (1-p_t))\) (See opening example of Royle and Dorazio). Thus, a natural upper end for the summation range of \(N\) would be \(y_t\) plus a very high quantile of The \(\mbox{Poisson}(\lambda (1-p_t))\) distribution. For a set of observations, a natural choice would be the maximum of such values across the observation times. We use the 0.99999 quantile to be conservative.

Correspondingly, the summation can begin at smallest of the 0.00001 quantiles of \(N | y_t\). If \(p_t\) is small, this can be considerably larger than the maximum value of \(y_t\), allowing more efficient computation.

Describe structural zeros.