Assessing performance of counterfactual prediction models for time-to-event outcomes

A lecture of Biostatistical Center will take place on Tuesday 6/7/2022, at 10:00 at IBMI. The lecture shall be given by prof. Nan van Geloven from Leiden University, Netherlands.

Abstract:

Assessing performance of counterfactual prediction models for time-to-event outcomes

Nan van Geloven¹, Ruth Keogh²

¹ Department of Biomedical Data Sciences, Leiden University Medical Centre, Netherlands

² Department of Medical Statistics, London School of Hygiene & Tropical Medicine, UK

Counterfactual prediction models provide estimates of absolute risks under a particular treatment pattern while conditioning on other patient characteristics that are predictive of the outcome. Counterfactual predictions model can inform treatment decisions by providing estimates of a patient’s risk if they were to be given the treatment and their risk if they were not to be given the treatment. This is an important improvement over ‘factual’ prediction models that typically ignore treatments during model development and then can only generate risks that apply to the mix of treated and untreated patients as observed in the development dataset.

While the importance of counterfactual predictions is recognized, concerns have been raised that counterfactual prediction models cannot be validated in ‘factual’, i.e. observational, datasets. Thorough validation is pivotal for any prediction model before it is advocated for use in medical practice, including assessment of calibration, discrimination and prediction error. In this work we propose extended versions of several popular performance measures: the c-index, the cumulative/dynamic area under the receiving operator characteristic curve (C/D AUC_t). Our extensions allow validation of counterfactual prediction models.

Our focus is on time-to-event outcomes and on a binary time-dependent treatment. We assume a validation dataset exists in which patients are observed with different treatment patterns over time, and allow for time-dependent confounders. Our aim is to assess discrimination of counterfactual outcome predictions under a particular fixed treatment pattern. We artificially censor patients when they first deviate from this treatment pattern and construct time-dependent inverse probability of censoring weighted versions of the performance measures. Validity of the proposed performance measures is shown through simulation of counterfactual outcomes.