Webinar Series: Mathematical Foundations of Data Science

On Nearly Assumption-Free Tests of Nominal Confidence Interval Coverage for Causal Parameters Estimated by Machine Learning

Abstract

For many causal effect parameters of interest, doubly robust machine learning (DRML) estimators are the state-of-the-art, incorporating the good prediction performance of machine learning; the decreased bias of doubly robust estimators; and the analytic tractability and bias reduction of sample splitting with cross fitting. Nonetheless, even in the absence of confounding by unmeasured factors, the nominal Wald confidence interval may still undercover even in large samples, because the bias may be of the same or even larger order than its standard error.

In this paper, we introduce essentially assumption-free tests that (i) can falsify the null hypothesis that the bias is of smaller order than its standard error, (ii) can provide a upper confidence bound on the true coverage of the Wald interval, and (iii) are valid under the null under no smoothness/sparsity assumptions on the nuisance parameters. The tests, which we refer to as Assumption Free Empirical Coverage Tests (AFECTs), are based on a U-statistic that estimates part of the bias.

Our claims need to be tempered in several important ways. First no test, including ours, of the null hypothesis that the ratio of the bias to its standard error is smaller than some threshold can be consistent [with-out additional assumptions (e.g. smoothness or sparsity) that may be incorrect]. Second the above claims only apply to certain parameters in a particular class. For most of the others, our results are unavoidably less sharp.

About the Speaker

James M. Robins is an epidemiologist and biostatistician best known for advancing methods for drawing causal inferences from complex observational studies and randomized trials, particularly those in which the treatment varies with time. He is the 2013 recipient of the Nathan Mantel Award for lifetime achievement in statistics and epidemiology.

He graduated in medicine from Washington University in St. Louis in 1976. He is currently Mitchell L. and Robin LaFoley Dong Professor of Epidemiology at Harvard T.H. Chan School of Public Health. He has published over 100 papers in academic journals and is an ISI highly cited researcher.