American Statistical Association
Back in the day, researchers routinely found that promising new markers added little to the discrimination of existing models. This was largely because a) the models were pretty good; b) no model is going to be anywhere close to perfect due to inherent randomness in predicting future events; c) the markers were correlated with existing predictors. Nonetheless, Pencina et al. proposed the net reclassification improvement (NRI) statistic as an alternative to discrimination metrics. NRI was found to be grossly anti-conservative, which a) explained its popularity amongst clinical investigators and b) led to two subsequent developments, recommendation of the Brier score as a metric, and tweaks to the NRI to improve its properties.
In joint work with Melissa Assel and Dan Sjoberg, we show that the Brier score is prevalence dependent in such a way that the rank ordering of tests or models may inappropriately vary by prevalence. We explored four common clinical scenarios in which the Brier score gave an inappropriate rank ordering of the tests and models. Conversely, net benefit, a decision-analytic measure, gave results that always favored the preferable test or model. We also show that the novel versions of NRI is only valid under restrictive assumptions and even then, is equivalent to net benefit.
|Date:||Wednesday, February 28, 2018|
|Time:||4:00 - 5:00 P.M.|
Memorial Sloan Kettering Cancer Center
Department of Epidemiology and Biostatistics
485 Lexington Avenue
(Between 46th & 47th Streets)
2nd Floor, Conference Room B
New York, New York
**Outside visitors please email email@example.com for building access. You must be on the security list to enter the floor.