Abstract
Provider profiling (ranking, "league tables") is prevalent in health services research. Similarly, comparing educational institutions and identifying differentially expressed genes depend on ranking. Effective ranking procedures must be structured by a hierarchical (Bayesian) model and guided by a ranking-specific loss function, however even optimal methods can perform poorly and estimates must be accompanied by uncertainty assessments. We use the 1998-2001 Standardized Mortality Ratio (SMR) data from United States Renal Data System (USRDS) as a platform to identify issues and approaches. Our analyses extend Liu et al. (2004) by combining evidence over multiple years via an AR(1) model; by considering estimates that minimize errors in classifying providers above or below a percentile cutpoint in addition to those that minimize rank-based, squared-error loss; by considering ranks based on the posterior probability that a provider's SMR exceeds a threshold; by comparing these ranks to those produced by ranking MLEs and ranking P-values associated with testing whether a provider's SMR = 1; by comparing results for a parametric and a non-parametric prior; by reporting on a suite of uncertainty measures.
Results show that MLE-based and hypothesis test based ranks are far from optimal, that uncertainty measures effectively calibrate performance; that in the USRDS context ranks based on single-year data perform poorly, but that performance improves substantially when using the AR(1) model; that ranks based on posterior probabilities of exceeding a properly chosen SMR threshold are essentially identical to those produced by minimizing classification loss. These findings highlight areas requiring additional research and the need to educate stakeholders on the uses and abuses of ranks; on their proper role in science and policy; on the absolute necessity of accompanying estimated ranks with uncertainty assessments and ensuring that these uncertainties influence decisions.
Disciplines
Health Services Research | Statistical Models | Vital and Health Statistics
Suggested Citation
Lin, Rongheng; Louis, Thomas A.; Paddock, Susan M.; and Ridgeway, Greg, "Ranking USRDS Provider-Specific SMRs from 1998-2001" (December 2004). Johns Hopkins University, Dept. of Biostatistics Working Papers. Working Paper 67.
https://biostats.bepress.com/jhubiostat/paper67
Included in
Health Services Research Commons, Statistical Models Commons, Vital and Health Statistics Commons