UW Biostatistics Working Paper Series

The Net Reclassification Index (NRI): a Misleading Measure of Prediction Improvement with Miscalibrated or Overfit Models

Margaret Pepe et al. — Tue, 19 Mar 2013 09:25:13 PDT

The Net Reclassification Index (NRI) is a very popular measure for evaluating the improvement in prediction performance gained by adding a marker to a set of baseline predictors. However, the statistical properties of this novel measure have not been explored in depth. We demonstrate the alarming result that the NRI statistic calculated on a large test dataset using risk models derived from a training set is likely to be positive even when the new marker has no predictive information. A related theoretical example is provided in which a miscalibrated risk model that includes an uninformative marker is proven to erroneously yield a positive NRI. Some insight into this phenomenon is derived from Hilden and Gerds (2013) who noted that the NRI statistic does not function as a proper scoring rule. Since large values for the NRI statistic may simply be due to use of miscalibrated risk models we suggest caution in using the NRI as the basis for marker evaluation. Other measures of prediction performance improvement, such as measures derived from the ROC curve, the net benefit function and the Brier score, cannot be large due to model miscalibration and may be preferred for that reason.

Asymptotic and Finite Sample Behavior of Net Reclassification Indices

Zheyu Wang — Thu, 28 Feb 2013 08:48:08 PST

The Net Reclassification Index (NRI) introduced by Pencina and colleagues [1, 2] is designed to quantify the prediction increment provided by a new biomarker. It has become popular for evaluating and selecting novel markers. The published variance formulae for NRI statistics do not account for the fact that risks are estimated based on risk models fit to data, and thus are not valid in practice when estimated risks are used [3]. Kerr and colleagues [4] showed that the confidence intervals constructed based on a bootstrap estimate of the variance and Normal approximation had the best performance among various methods they examined, including the one based on bootstrap quantiles. This paper establishes asymptotic Normality of NRI statistics when true risks are unknown and are estimated. Our results provide theoretical support for constructing confidence intervals for NRI statistics based on a Normal approximation. We also derive explicit variance formulae for NRI statistics that are calculated based on estimated risks. In addition, we examine finite sample distributional behavior of NRI statistics in a simulation study. These results provide some guidance on the sample size required for adopting a Normal approximation for NRI inference in practice.

Methods for Dealing with Death and Missing Data, and for Standardizing Different Health Variables in Longitudinal Datasets: The Cardiovascular Health Study

Paula Diehr — Wed, 20 Feb 2013 16:02:34 PST

Longitudinal studies of older adults usually need to account for deaths and missing data. The databases often include multiple health-related variables, which are hard to compare because they were measured on different scales. Here we present the unified approach to these three problems, developed and used in the Cardiovascular Health Study. Data were first transformed to a new scale that had integer/ratio properties, and on which “dead” takes the value zero. Missing data were then imputed on this new scale, using each person’s own data over time. Imputation could thus be informed by impending death. The new transformed and imputed variable has a value for every person at every potential time, accounts for death, and can also be considered as a measure of “standardized health” that permits comparison of variables that were originally measured on different scales. The new variable can also be transformed back to the original scale, where it differs from the original data in that missing values have been imputed. Each observation is labeled as to whether it was observed, imputed (and how), or the person was dead at the time. An example using real is CHS data is given. The resulting “tidy” dataset can be considered complete, but is flexible enough to permit analysts to handle missing data and deaths in other ways. This approach may be useful for other longitudinal studies as well as for the Cardiovascular Health Study.

Statistical Methods for Evaluating and Comparing Biomarkers for Patient Treatment Selection

Holly Janes et al. — Wed, 09 Jan 2013 12:07:16 PST

Despite the heightened interest in developing biomarkers predicting treatment response that are used to optimize patient treatment decisions, there has been relatively little development of statistical methodology to evaluate these markers. There is currently no unified statistical framework for marker evaluation. This paper proposes a suite of descriptive and inferential methods designed to evaluate individual markers and to compare candidate markers. An R software package has been developed which implements these methods. Their utility is illustrated in the breast cancer treatment context, where candidate markers are evaluated for their ability to identify a subset of women who do not benefit from adjuvant chemotherapy and can therefore avoid its toxicity.

An Evaluation of Inferential Procedures for Adaptive Clinical Trial Designs with Pre-specified Rules for Modifying the Sample Size

Greg P. Levin et al. — Tue, 08 Jan 2013 12:37:37 PST

Many papers have introduced adaptive clinical trial methods that allow modifications to the sample size based on interim estimates of treatment effect. There has been extensive commentary on type I error control and efficiency considerations, but little research on estimation after an adaptive hypothesis test. We evaluate the reliability and precision of different inferential procedures in the presence of an adaptive design with pre-specified rules for modifying the sampling plan. We extend group sequential orderings of the outcome space based on the stage at stopping, likelihood ratio test statistic, and sample mean to the adaptive setting in order to compute median-unbiased point estimates, exact confidence intervals, and P-values uniformly distributed under the null hypothesis. The likelihood ratio ordering is found to average shorter confidence intervals and produce higher probabilities of P-values below important thresholds than alternative approaches. The bias adjusted mean demonstrates the lowest mean squared error among candidate point estimates. A conditional error-based approach in the literature has the benefit of being the only method that accommodates unplanned adaptations. We compare the performance of this and other methods in order to quantify the cost of failing to plan ahead in settings where adaptations could realistically be pre-specified at the design stage. We find the cost to be meaningful for all designs and treatment effects considered, and to be substantial for designs frequently proposed in the literature.

A Regionalized National Universal Kriging Model Using Partial Least Squares Regression for Estimating Annual PM2.5 Concentrations in Epidemiology

Paul D. Sampson et al. — Tue, 11 Dec 2012 08:59:09 PST

Many cohort studies in environmental epidemiology require accurate modeling and prediction of fine scale spatial variation in ambient air quality across the U.S. This modeling requires the use of small spatial scale geographic or “land use” regression covariates and some degree of spatial smoothing. Furthermore, the details of the prediction of air quality by land use regression and the spatial variation in ambient air quality not explained by this regression should be allowed to vary across the continent due to the large scale heterogeneity in topography, climate, and sources of air pollution. This paper introduces a regionalized national universal kriging model for annual average fine particulate matter (PM2.5) monitoring data across the U.S. To take full advantage of an extensive database of land use covariates we chose to use the method of Partial Least Squares, rather than variable selection, for the regression component of the model (the “universal” in “universal kriging”) with regression coefficients and residual variogram models allowed to vary across three regions defined as West Coast, Mountain West, and East. We demonstrate a very high level of cross-validated accuracy of prediction with an overall R2 of 0.88 and well-calibrated predictive intervals. In accord with the spatially varying characteristics of PM2.5 on a national scale and differing kriging smoothness parameters, the accuracy of the prediction varies by region with predictive intervals being notably wider in the West Coast and Mountain West in contrast to the East.

A National Model Built with Partial Least Squares and Universal Kriging and Bootstrap-based Measurement Error Correction Techniques: An Application to the Multi-Ethnic Study of Atherosclerosis

Silas Bergen et al. — Wed, 05 Dec 2012 15:39:49 PST

Studies estimating health effects of long-term air pollution exposure often use a two-stage approach, building exposure models to assign individual-level exposures which are then used in regression analyses. This requires accurate exposure modeling and careful treatment of exposure measurement error. To illustrate the importance of carefully accounting for exposure model characteristics in two-stage air pollution studies, we consider a case study based on data from the Multi-Ethnic Study of Atherosclerosis (MESA). We present national spatial exposure models that use partial least squares and universal kriging to estimate annual average concentrations of four PM2.5 components: elemental carbon (EC), organic carbon (OC), sulfur (S), and silicon (Si). Our models perform well, with cross-validated R2s ranging from 0.62 to 0.95. We predict PM2.5 component exposures for the MESA cohort and estimate cross-sectional associations with carotid intima-media thickness (CIMT), adjusting for subject-specific covariates. In naïve analyses that do not account for measurement error, we find statistically significant associations between CIMT and increased exposure to OC, S, and Si. We correct for measurement error using recently developed methods that account for the spatial structure of predicted exposures. OC exhibits little spatial correlation, and the corrected inference is unchanged from the naïve analysis. The S and Si exposure surfaces display notable spatial correlation, resulting in corrected confidence intervals (CIs) that are 50% wider than the naïve CIs, but that are still statistically significant. The impact on health effect inference is concordant with the degree of spatial correlation in the exposure surfaces.

Decline in Health for Older Adults: 5-Year Change in 13 Key Measures of Standardized Health

Paula H. Diehr et al. — Mon, 22 Oct 2012 14:43:04 PDT

Introduction

The health of older adults declines over time, but there are many ways of measuring health. We examined whether all measures declined at the same rate, or whether some aspects of health were less sensitive to aging than others.

Methods

We compared the decline in 13 measures of physical, mental, and functional health from the Cardiovascular Health Study: hospitalization, bed days, cognition, extremity strength, feelings about life as a whole, satisfaction with the purpose of life, self-rated health, depression, digit symbol substitution test, grip strength, ADLs, IADLs, and gait speed. Each measure was standardized against self-rated health. We compared the 5-year change to see which of the 13 measures declined the fastest and the slowest.

Results

The 5-year change in standardized health varied from a decline of 12 points (out of 100) for hospitalization to a decline of 17 points for gait speed. In most comparisons, standardized health from hospitalization and bed days declined the least while health measured by ADLs, IADLs, and gait speed declined the most. These rankings were independent of age, sex, mortality patterns, and the method of standardization.

Discussion

All of the health variables declined, on average, with advancing age, but at significantly different rates. Standardized measures of mental health, cognition, quality of life and hospital utilization did not decline as fast as gait speed, ADLs, and IADLs. Public health interventions to address problems with gait speed, ADLs, and IADLs may help older adults to remain healthier in all dimensions.

Methods for Evaluating Prediction Performance of Biomarkers and Tests

Margaret Pepe et al. — Wed, 17 Oct 2012 14:35:19 PDT

This chapter describes and critiques methods for evaluating the performance of markers to predict risk of a current or future clinical outcome. We consider three criteria that are important for evaluating a risk model: calibration, benefit for decision making and accurate classification. We also describe and discuss a variety of summary measures in common use for quantifying predictive information such as the area under the ROC curve and R-squared. The roles and problems with recently proposed risk reclassification approaches are discussed in detail.

Transitions Among Health States Using 12 Measures of Successful Aging: Results from the Cardiovascular Health Study

Stephen Thielke et al. — Fri, 03 Aug 2012 09:44:28 PDT

Introduction

Successful aging has many dimensions, which may manifest differently in men and women and at different ages. We sought to characterize one-year transitions in 12 measures of successful aging among a large cohort of older adults.

Methods

We analyzed twelve different measures of health in the Cardiovascular Health Study: self-rated health, ADLs, IADLs, depression, cognition, timed walk, number of days spent in bed, number of blocks walked, extremity strength, recent hospitalizations, feelings about life as a whole, and life satisfaction. We dichotomized responses for each variable into “healthy” or “sick”, and estimated the prevalence of the healthy state and the probability of transitioning from one state to another, or dying, during yearly intervals. We compared men and women, and three age groups (65-74, 75-84, and 85-94).

Findings

All measures of successful aging showed similar results, except for hospitalizations and cognition. Most participants remained healthy even into advanced ages, although health declined for all measures. Men had a higher death rate than women, regardless of health status, and were also more likely to be healthy.

Discussion

The results suggest a qualitatively different experience of successful aging between men and women, with men showing a more "square" mortality curve. Men did not simply "age faster" than women.

Conclusion

Men and women age differently with regard to health status, with consistency among various health measures.

Fitting and Interpreting Continuous-Time Latent Markov Models for Panel Data

Jane M. Lange et al. — Tue, 03 Jul 2012 09:38:52 PDT

Multistate models are used to characterize disease processes within an individual. Clinical studies often observe the disease status of individuals at discrete time points, making exact times of transitions between disease states unknown. Such panel data pose considerable modeling challenges. Assuming the disease process progresses according a standard continuous-time Markov chain (CTMC) yields tractable likelihoods, but the assumption of exponential sojourn time distributions is typically unrealistic. More flexible semi-Markov models permit generic sojourn distributions yet yield intractable likelihoods for panel data in the presence of reversible transitions. One attractive alternative is to assume that the disease process is characterized by an underlying latent CTMC, with multiple latent states mapping to each disease state. These models retain analytic tractability due to the CTMC framework but allow for flexible, duration-dependent disease state sojourn distributions. We have developed a robust and efficient expectation-maximization (EM) algorithm in this context. Our complete data state space consists of the observed data and the underlying latent trajectory, yielding computationally efficient expectation and maximization steps. Our algorithm outperforms alternative methods measured in terms of time to convergence and robustness. We also examine the frequentist performance of latent CTMC point and interval estimates of disease process functionals based on simulated data. The performance of estimates depends on time, functional, and data-generating scenario. Finally, we illustrate the interpretive power of latent CTMC models for describing disease processes on a data-set of lung transplant patients. We hope our work will encourage wider use of these models in the biomedical setting.

The Importance of Statistical Theory in Outlier Detection

Sarah C. Emerson et al. — Wed, 17 Aug 2011 11:01:36 PDT

We explore the performance of the outlier-sum statistic (Tibshirani and Hastie, Biostatistics 2007 8:2--8), a proposed method for identifying genes for which only a subset of a group of samples or patients exhibits differential expression levels. Our discussion focuses on this method as an example of how inattention to standard statistical theory can lead to approaches that exhibit some serious drawbacks. In contrast to the results presented by those authors, when comparing this method to several variations of the $t$-test, we find that the proposed method offers little benefit even in the most idealized scenarios, and suffers from a number of limitations including difficulty of calibration, high false positive rates owing to its asymmetric treatment of groups, poor power or discriminatory ability under many alternatives, and poorly defined application to one-sample settings. Further issues in the Tibshirani and Hastie paper concern the presentation and accuracy of their simulation results; we were unable to reproduce their findings, and we discuss several undesirable and implausible aspects of their results.

Some Observations on the Wilcoxon Rank Sum Test

Scott S. Emerson — Tue, 16 Aug 2011 13:49:04 PDT

This manuscript presents some general comments about the Wilcoxon rank sum test. Even the most casual reader will gather that I am not too impressed with the scientific usefulness of the Wilcoxon test. However, the actual motivation is more to illustrate differences between parametric, semiparametric, and nonparametric (distribution-free) inference, and to use this example to illustrate how many misconceptions have been propagated through a focus on (semi)parametric probability models as the basis for evaluating commonly used statistical analysis models. The document itself arose as a teaching tool for courses aimed at graduate students in biostatistics and statistics, with parts of the document originally written for applied biostatistics classes and parts written for a course in mathematical statistics. Hence, some of the material is also meant to provide an illustration of common methods of deriving moments of distributions, etc.

Testing for improvement in prediction model performance

Margaret S. Pepe PhD et al. — Fri, 08 Jul 2011 14:01:12 PDT

New methodology has been proposed in recent years for evaluating the improvement in prediction performance gained by adding a new predictor, Y, to a risk model containing a set of baseline predictors, X, for a binary outcome D. We prove theoretically that null hypotheses concerning no improvement in performance are equivalent to the simple null hypothesis that the coefficient for Y is zero in the risk model, P(D = 1|X, Y ). Therefore, testing for improvement in prediction performance is redundant if Y has already been shown to be a risk factor. We investigate properties of tests through simulation studies, focusing on the change in the area under the ROC curve (AUC). An unexpected finding is that standard testing procedures that do not adjust for variability in estimated regression coefficients are extremely conservative. This may explain why the AUC is widely considered insensitive to improvements in prediction performance and suggests that the problem of insensitivity has to do with use of invalid procedures for inference rather than with the measure itself. To avoid redundant testing and use of potentially problematic methods for inference, we recommend that hypothesis testing for no improvement be limited to evaluation of Y as a risk factor, for which methods are well developed and widely available. Analyses of measures of prediction performance should focus on estimation rather than on testing.

When Does Combining Markers Improve Classification Performance and What Are Implications for Practice?

Aasthaa Bansal et al. — Mon, 27 Jun 2011 09:44:32 PDT

When an existing standard marker does not have sufficient classification accuracy on its own, new markers are sought with the goal of yielding a combination with better performance. The primary criterion for selecting new markers is that they have good performance on their own and preferably be uncorrelated with the standard. Most often linear combinations are considered. In this paper we investigate the increment in performance that is possible by combining a novel continuous marker with a moderately performing standard continuous marker under a variety of biologically motivated models for their joint distribution. We find that an uncorrelated continuous marker with moderate performance on its own usually yields only minimally improved performance. We identify other settings that lead to large improvements, including a novel marker that has very poor performance on its own but is highly correlated with the standard and a novel marker with poor to moderate performance that is highly correlated with the standard but only in one class category. These results suggest changing current strategies for identifying markers to be included in panels for possible combination. Using simulated and real datasets we examine the merits of a broadened strategy compared with the standard strategy that selects panels of markers as candidates based on their marginal performance. We find that a broadened strategy can be fruitful but necessitates using studies with large numbers of subjects.

Adaptive Clinical Trial Designs with Pre-specified Rules for Modifying the Sample Size: Understanding Efficient Types of Adaptation

Gregory P. Levin et al. — Mon, 09 May 2011 10:21:57 PDT

Methods allowing unplanned adaptations to the sample size based on the interim estimate of treatment effect do not base inference on the minimal sufficient statistic and suffer losses in efficiency when compared to group sequential designs [1, 2, 3]. However, when adaptive sampling plans are completely pre-specified at the design stage of the trial, investigators can proceed with frequentist inference based on the minimal sufficient statistic at the analysis stage. In the context of two general settings where different optimality criteria govern the choice of clinical trial design, we quantify the relative costs and benefits of a variety of fixed sample, group sequential, and pre-specified adaptive designs with respect to standard operating characteristics. We find pre-specified symmetric adaptive designs that are ``optimal" in the sense that they minimize the expected sample size at the design alternatives. Our results build on others' prior research [1, 4, 5, 6] by demonstrating in realistic settings that simple and easily implemented pre-specified adaptive designs provide only very small efficiency gains over group sequential designs with the same number of analyses. In addition, we describe optimal rules for modifying the sample size, providing efficient adaptation boundaries on a variety of scales for the interim test statistic for adaptation analyses occurring at several different stages of the trial. These findings provide insight into what are good and bad choices of adaptive sampling plans and suggest that adaptive designs proposed in the literature are often based on inefficient rules for modifying the sample size.

BATE Curve in Assessment of Clinical Utility of Predictive Biomarkers

Xiao-Hua Zhou et al. — Tue, 22 Feb 2011 11:21:30 PST

In this paper, for time-to-event data, we propose a new statistical framework for casual inference in evaluating clinical utility of predictive biomarkers and in selecting an optimal treatment for a particular patient. This new casual framework is based on a new concept, called Biomarker Adjusted Treatment Effect (BATE) curve, which can be used to represent the clinical utility of a predictive biomarker and select an optimal treatment for one particular patient. We then propose semi-parametric methods for estimating the BATE curves of biomarkers and establish asymptotic results of the proposed estimators for the BATE curves. We also conduct extensive simulation studies to evaluate finite-sample properties of the proposed estimation methods. Finally, we illustrate the application of the proposed method in a real-world data set.

Evaluating Markers for Treatment Selection Based on Survival Time

Xiao Song et al. — Fri, 21 Jan 2011 15:41:51 PST

For many medical conditions there are several treatment options available to patients. We consider evaluating markers based on a simple treatment selection policy that incorporates information on the patient's marker value exceeding a threshold. Although traditional regression methods may assess the effect of the marker and treatment on outcomes, it is appealing to quantify more directly the potential impact on the population of using the marker to select treatment. A useful tool is the selection impact (SI) curve proposed by Song and Pepe (2004, \textit{Biometrics} \textbf{60}, 874--883) for binary outcomes. However, this approach does not deal with continuous outcomes, nor does it adjust for other covariates that are important for treatment selection. In this paper, we propose the SI curve for general outcomes, with specific focus on the survival time. We further propose the covariate specific SI curve to incorporate covariate information in treatment selection. Nonparametric and semiparametric estimators are developed accordingly. We show that the proposed estimators are consistent and asymptotically normal. Simulation studies demonstrate that these estimators work well with realistic sample sizes. We illustrate the SI curve and the statistical inference for it with data from an AIDS clinical trial.

Semiparametric Estimation of the Covariate-Specific ROC Curve in Presence of Ignorable Verification Bias

Danping Liu et al. — Fri, 21 Jan 2011 15:20:51 PST

Covariate-specific ROC curves are often used to evaluate the classification accuracy of a medical diagnostic test or a biomarker, when the accuracy of the test is associated with certain covariates. In many large-scale screening tests, the gold standard is subject to missingness due to high cost or harmfulness to the patient. In this paper, we propose a semiparametric estimation method for the covariate-specific ROC curves with a partial missing gold standard. A location-scale model is constructed for the test result to model the covariates' effect, but the residual distributions are left unspecified. Thus the baseline and link functions of the ROC curve both have flexible shapes. With the gold standard missing at random (MAR) assumption, we consider weighted estimating equations for the location-scale parameters, and weighted kernel estimating equations for the residual distributions. Three ROC curve estimators are proposed and compared, namely, imputation-based, inverse probability weighted and doubly robust estimators. We derive the asymptotic normality of the estimated ROC curve, as well as the analytical form the standard error estimator. The proposed method is motivated and applied to the data in an Alzheimer's disease study.

Doubly Robust Estimates for Binary Longitudinal Data Analysis with Missing Response and Missing Covariates

Baojiang Chen et al. — Fri, 21 Jan 2011 15:12:56 PST

Longitudinal studies often feature incomplete response and covariate data. Likelihood-based methods such as the EM algorithm give consistent estimators for model parameters when data are missing at random provided that the response model and the missing covariate model are correctly specified; but we do not need to specify the missing data mechanism. An alternative method is the weighted estimating equation which gives consistent estimators if the missing data and response models are correctly specified; but we do not need to specify the distribution of the covariates that have missing values. In this paper we develop a doubly robust estimation method for longitudinal data with missing response and missing covariate when data are missing at random. This method is appealing in that it can provide consistent estimators if either the missing data model or the missing covariate model is correctly specified. Simulation studies demonstrate that this method performs well in a variety of situations.