Leave-one-detector-out jackknife for a union call density estimate

Refits the union call density estimate once per contributing detector, each time leaving that detector out, and reports how far each leave-one-out estimate moves from the full-set estimate. Answers the question "how much does my answer depend on any single detector being in the pool?" – computable on real data, with no ground truth required.

Usage

jackknifeDetectors(
  capHistTab,
  observerCols,
  signalCols,
  noiseCols,
  NcTable = capHistTab,
  groundTruthCol = "groundTruth",
  fullCoverageConfirmed = FALSE,
  ...
)

Arguments

capHistTab

Capture history table used for fitting and for the false discovery rate, as passed to cde. Must contain groundTruthCol, every column named in observerCols, signalCols and noiseCols, and an SNR column.

observerCols

Character vector of the detector columns making up the union, in the same order as signalCols/noiseCols.

signalCols, noiseCols

Per-detector signal and noise level columns, in the same order as observerCols – these are subset alongside it as each detector is dropped.

NcTable

Table from which each leave-one-out Nc is recomputed via unionDetections. This should be the full, non-adjudicated capture history table (all detections from every contributing detector), matching how Nc is derived for an ordinary union estimate. Defaults to capHistTab, which is only correct if that already is the full table.

groundTruthCol

Ground truth column, default 'groundTruth'.

fullCoverageConfirmed

Passed to unionDetections; see there for why this must be confirmed rather than assumed.

...

Further arguments passed to cde (SL, TL, A, T, k, outerloop, ...). These are held identical across every refit, so differences in the result come only from the detector set.

Value

A data.frame with one row per dropped detector: dropped (the omitted column name), Dc, pa, c, CV.Dc for that leave-one-out fit, and shift (that fit's Dc minus the full-set Dc). Two attributes carry the summary: attr(, "full") is the full-set cde() result, and attr(, "jackknifeCV") is the jackknife coefficient of variation across the leave-one-out estimates, sqrt(((n-1)/n) * sum((Dc_i - mean(Dc_i))^2)) / mean(Dc_i).

Details

Structurally this is leave-one-out refitting, like leave-one-out cross-validation, but the purpose is different. Cross-validation estimates prediction error by scoring a model on held-out data. The jackknife estimates an estimator's own variance and sensitivity by watching how much it moves as each unit is removed. Here the unit removed is an entire detector rather than a data point, so what comes back is detector-set sensitivity rather than predictive accuracy.

This sees a source of uncertainty that cde's own reported CV.Dc structurally cannot. That CV comes from a parametric bootstrap over the coefficients of one fitted detection function, so it propagates uncertainty conditional on the detector set and the fitted curve. It has no way to represent how differently the analysis would have gone with a different mix of detectors. The jackknife spread reported here typically comes out larger than CV.Dc for exactly that reason, and the two are complementary rather than alternatives.

A large shift for one detector is not automatically a problem, and does not by itself mean that detector is bad or should be dropped – a detector contributing genuinely distinct information should move the estimate when removed. It flags where to look, not what to conclude.

Each leave-one-out refit rebuilds its own adjudicated fitting set as "ground-truth-positive and flagged by at least one remaining detector", rather than reusing the full set's. This is necessary, not cosmetic: an event caught only by the dropped detector would otherwise become an all-zero capture history for the remaining ones, which VGAM::posbernoulli.t cannot represent (it conditions on capture on at least one occasion).

See also

cde, unionDetections