Assessing improvement in disease prediction using net reclassification improvement: Impact of risk cut-offs and number of risk categories

Kristin Mühlenbruch, Alexandros Heraclides, Ewout W. Steyerberg, Hans Georg Joost, Heiner Boeing, Matthias B. Schulze

Research output: Contribution to journalArticle

24 Citations (Scopus)

Abstract

Net reclassification improvement (NRI) has received much attention for comparing risk prediction models, and might be preferable over the area under the receiver operating characteristics (ROC) curve to indicate changes in predictive ability. We investigated the influence of the choice of risk cut-offs and number of risk categories on the NRI. Using data of the European Prospective Investigation into Cancer and Nutrition-Potsdam study, three diabetes prediction models were compared according to ROC area and NRI with varying cut-offs for two and three risk categories and varying numbers of risk categories. When compared with a basic model, including age, anthropometry, and hypertension status, a model extension by waist circumference improved discrimination from 0.720 to 0.831 (0.111 [0.097-0.125]) while increase in ROC-AUC from 0.831 to 0.836 (0.006 [0.002-0.009]) indicated moderate improvement when additionally considering diet and physical activity. However, NRI based on these two model comparisons varied with varying cut-offs for two (range: 5.59-23.20 %; -0.79 to 4.09 %) and three risk categories (20.37-40.15 %; 1.22-4.34 %). This variation was more pronounced in the model extension showing a larger difference in ROC-AUC. NRI increased with increasing numbers of categories from minimum NRIs of 18.41 and 0.46 % to approximately category-free NRIs of 79.61 and 19.22 %, but not monotonically. There was a similar pattern for this increase in both model comparisons. In conclusion, the choice of risk cut-offs and number of categories has a substantial impact on NRI. A limited number of categories should only be used if categories have strong clinical importance.

Original languageEnglish
Pages (from-to)25-33
Number of pages9
JournalEuropean Journal of Epidemiology
Volume28
Issue number1
DOIs
Publication statusPublished - Jan 2013

Keywords

  • Diabetes mellitus, type 2
  • Model comparison
  • Model discrimination
  • Models
  • Risk assessment
  • Risk model
  • Statistical

Fingerprint Dive into the research topics of 'Assessing improvement in disease prediction using net reclassification improvement: Impact of risk cut-offs and number of risk categories'. Together they form a unique fingerprint.

  • Cite this