The questions remain the same for the Ingram Olkin Forum on Unplanned Disruptions of Clinical Trials.
What are the implications of unplanned disruptions of clinical trials? What strategies can be put into place to ensure the integrity of statistics using data that have already or will not be collected? Are there changes to methods or plans that researchers can put into place moving forward?
For Day Four of this series the focus was on randomization tests.
Day Four speakers discussed the use and practice of randomization-based inference. For randomized clinical trials randomization based inference, both testing and confidence intervals procedures, offer a robust analysis framework that is still valid even if the independence and/or exchangeability of the responses is called into question. The focus of discussion for Day 4 was to determine if the disruptive challenges from COVID-19 prompt future research in the use of randomization tests.
During Day One of the forum, experts from various agencies within the National Institutes for Health presented the impact of the COVID-19 pandemic on clinical trials in progress, about to start, and even those being planned. Day Two brought together three more experts from the FDA, the CDC the European Medicines Agency to share their perspectives. On Day Three, more than two dozen specialists with specific interest met online as a working group to investigate the first topic – estimands and missing data.
Similarly, Day Four brought together a working group of experts to focus on randomization tests. The first speaker was William F (Bill) Rosenberger, (George Mason University). Bill provided a robust overview of the development and use of randomization as a basis of inference since the early use of clinical trials and its current ease of computation with high speed computers. He also emphasized the importance of proper design of the randomized trial. In summary Bill asked the question, “If a trial has an unplanned interruption, when it is restarted will the clinical trials population have the same characteristics?” His reply specified that while the most critical aspect is the introduction of heterogeneity, randomization-based inference can help because it retains operating characteristics under heterogeneity. In short, “Randomization Matters!”
The second speaker of the day was Martin Posch (Medical University of Vienna). His talk posed the question, “Salvaging Type 1 Error Rate Control by Randomization Tests When Your Trial is Affected by Covid?” After indicating the various ways that clinical trials have been affected by COVID-19, primarily that the estimand tested in the trial may have changed, he stressed how the primary endpoint cannot be assessed as planned. Martin concluded that whereas permutation tests might not be the complete solution, by restricting the variables under consideration, they provide a valid test based on the closed testing procedure under specific parameters.
The final speaker of the day was Michael Proschan (National Institute of Allergy and Infectious Diseases). He began his talk with a pre-test. The two questions were: “Randomization tests do not allow generalization to the population, whereas t tests do.” True or False? And, “If I look at blinded data and see a bimodal distribution, that will unblind me and cause alpha inflation and have other negative consequences.” Again, True or False? From there he walked through a number of examples or “fables” as he called them that allowed him to reveal the important caveats involved in the use of randomization tests. This lead up to his thesis that “Randomization tests are known to be asymptotically the same as t tests under reasonable assumptions.” And, in conclusion he showed that “Because randomization tests are valid and are asymptotically equivalent to t tests, t tests are approximately valid after blinded changes if sample sizes are large, (but use the randomization variance).”
Immediately following these presentations the meeting participants moved into two breakout rooms in Zoom. Both rooms were given the task to identify and formulate open questions and issues related to COVID-19 driven disruptions. These two groups were moderated by Diane Uschner (George Washington University) and Ralf-Dieter Hilgers (RWTH Aachen) who reported back to the forum at the conclusion of the event. One immediate outcome was the mention of the Randomization Working Group (https://randomization-working-group.rwth-aachen.de/rwg/). Kerstine Carter, (Boehringer-Ingelheim, USA) mentioned that they are actively welcoming new members interested in joining this group of biostatisticians and others from industry and academia looking at innovative designs related to randomization. A small working group of Day Four participants will be organized for the purpose of synthesizing recommendations based on these discussions.
Days Five and Six on aspects of Adaptive Designs in the context of unplanned disruptions are in the planning stage. The organizing committee is aiming for October 20, 2020 for Day 5. So, hold this date (10 am -1 pm ET) and watch for future announcements! Visit the NISS website for more information and to register and pay to become a participant in these future working groups! Join the working group!
By the way, how did you do on Michael Proschan’s pre-test? Below you can review the recording of the session speakers and the slides that they used. Scoring via the honor system!
Recording of the Session and Slides Used by the Speakers
William Rosenberger, (George Mason University)
"Randomization Tests: The Forgotten Component of the Randomized Clinical Trial"
Martin Posch, (Medical University of Vienna)
"Salvaging type 1 error rate control by randomization tests when your trial is affected by Covid?"
Michael Proschan, (NIAID, NIH)
"Re-randomization Tests in the Context of Trial Disruptions”