grassr

Context-Conditioned Reporting for Binary Rater Reliability

Generates a Report Card for rater reliability on binary outcomes from an N x k subject-by-rater rating matrix, on both the inter-rater and intra-rater axes. Each panel coefficient is positioned on a data-generating-process-calibrated reference surface conditioned on the study's rater count, sample size, and prevalence, yielding a pooled percentile (the coefficient's position within the design's achievable agreement range) together with a consistency band on panel quality: the quality levels whose sampling distributions are consistent with the observed value at that design. The panel coefficients are the prevalence-adjusted bias-adjusted kappa (PABAK) of Byrt, Bishop, and Carlin (1993) <doi:10.1016/0895-4356(93)90018-V>, the first-order agreement coefficient (AC1) of Gwet (2008) <doi:10.1348/000711006X126600>, the multi-rater kappa of Fleiss (1971) <doi:10.1037/h0031619>, and the observed intraclass correlation. A cross-coefficient discordance diagnostic (delta-hat) reports the spread of the coefficients' implied panel qualities and flags panels for which no single coefficient is a stable summary by the spread's percentile on a matched null distribution; for such divergent panels the report routes to a pairwise PABAK matrix and per-rater sensitivity and specificity recovered from the latent-class model of Dawid and Skene (1979) <doi:10.2307/2346806>, with the two-rater bounds of Hui and Walter (1980) <doi:10.2307/2530508>.

Versions across snapshots

Dependencies (latest)