Assessing performance of counterfactual prediction models for time-to-event outcomes

Event Date: 
Wednesday, 8 June, 2022 - 10:00
Location: 
IBMI
Lecturer: 
Prof Nan van Geloven

A lecture of Biostatistical Center will take place on Tuesday 6/7/2022, at 10:00 at IBMI. The lecture shall be given by prof. Nan van Geloven from Leiden University, Netherlands.

Abstract:

Assessing performance of counterfactual prediction models for time-to-event outcomes

Nan van Geloven1, Ruth Keogh2

1 Department of Biomedical Data Sciences, Leiden University Medical Centre, Netherlands

2 Department of Medical Statistics, London School of Hygiene & Tropical Medicine, UK

 

Counterfactual prediction models provide estimates of absolute risks under a particular treatment pattern while conditioning on other patient characteristics that are predictive of the outcome. Counterfactual predictions model can inform treatment decisions by providing estimates of a patient’s risk if they were to be given the treatment and their risk if they were not to be given the treatment. This is an important improvement over ‘factual’ prediction models that typically ignore treatments during model development and then can only generate risks that apply to the mix of treated and untreated patients as observed in the development dataset.

While the importance of counterfactual predictions is recognized, concerns have been raised that counterfactual prediction models cannot be validated in ‘factual’, i.e. observational, datasets. Thorough validation is pivotal for any prediction model before it is advocated for use in medical practice, including assessment of calibration, discrimination and prediction error. In this work we propose extended versions of several popular performance measures: the c-index, the cumulative/dynamic area under the receiving operator characteristic curve (C/D AUCt). Our extensions allow validation of counterfactual prediction models.

Our focus is on time-to-event outcomes and on a binary time-dependent treatment. We assume a validation dataset exists in which patients are observed with different treatment patterns over time, and allow for time-dependent confounders. Our aim is to assess discrimination of counterfactual outcome predictions under a particular fixed treatment pattern. We artificially censor patients when they first deviate from this treatment pattern and construct time-dependent inverse probability of censoring weighted versions of the performance measures. Validity of the proposed performance measures is shown through simulation of counterfactual outcomes.

About IBMI

Institute for Biostatistics and Medical Informatics (IBMI), formerly Institute for BioMedical Informatics (so still IBMI) was founded by the Faculty of Medicine as a result of a need for a unit which would perform, or coordinate, tasks related to data analysis and providing information, relevant for research in medicine. The programme of the institute, and its development, have been adjusting thorugh time to changes in financing and technological progress, but the basic aim remain the same: to support research in medicine. This is achieved through the following tasks:

Contact

Institute for Biostatistics and Medical Informatics
University of Ljubljana, Faculty of Medicine
Vrazov trg 2, 1000 Ljubljana
Slovenia

tel: +386 1 543-77-70
fax: +386 1 543-77-71
email: ibmi (at) mf.uni-lj.si