Abstract
The Net Reclassification Index (NRI) is a very popular measure for evaluating the improvement in prediction performance gained by adding a marker to a set of baseline predictors. However, the statistical properties of this novel measure have not been explored in depth. We demonstrate the alarming result that the NRI statistic calculated on a large test dataset using risk models derived from a training set is likely to be positive even when the new marker has no predictive information. A related theoretical example is provided in which a miscalibrated risk model that includes an uninformative marker is proven to erroneously yield a positive NRI. Some insight into this phenomenon is derived from Hilden and Gerds (2013) who noted that the NRI statistic does not function as a proper scoring rule. Since large values for the NRI statistic may simply be due to use of miscalibrated risk models we suggest caution in using the NRI as the basis for marker evaluation. Other measures of prediction performance improvement, such as measures derived from the ROC curve, the net benefit function and the Brier score, cannot be large due to model miscalibration and may be preferred for that reason.
Disciplines
Categorical Data Analysis | Clinical Epidemiology
Suggested Citation
Pepe, Margaret; Fang, Jin; Feng, Ziding; Gerds, Thomas; and Hilden, Jorgen, "The Net Reclassification Index (NRI): a Misleading Measure of Prediction Improvement with Miscalibrated or Overfit Models" (March 2013). UW Biostatistics Working Paper Series. Working Paper 392.
https://biostats.bepress.com/uwbiostat/paper392