Ingram Olkin Forum Focuses on Data and Models for Opening and Closing Schools and Learning Loss

Stephen Sawchuk (Education Week) reviews that work that his publication has been involved in.

Ravi Goyal (Mathematica) reviews the categorization of parameters used in his model.

Dina Mistry (IDM) provides an overview of their model called Covasim.

Megan Kuhfeld (Northwest Evaluation Association) presents the research questions that led her study.

Julia Kaufman (RAND) compares Spring and Fall of 2020 in terms of teacher reports of student learning.

Claude Messan Setodji (RAND) responds to a question about weighting from the audience.

To honor the work and actions of the late Ingram Olkin, an eminent statistician, mentor and contributor to social causes, NISS brings together experts and researchers from a variety of fields and backgrounds into forums. The focus of these forums is to bring forward a current societal issue that might benefit from attention from the statistical community and others.

The pandemic has impelled school administrators to make urgent and critical choices. Should schools be open? Closed? Under what circumstances? What criteria are important, indicative and/or informative? Furthermore, how has student learning been impacted by all of these disruptions?

The NISS Ingram Olkin Forum session organizer Betsy Becker (Florida State University) brought together a number of individuals who have distinguished themselves through their work in this area. The speakers at this session included journalist Stephen Sawchuk from Education Week, Senior Fellow Brian Gill and Senior Statistician Ravi Goyal from Mathematica, Post-doctoral Research Scientist Dina Mistry and Senior Research Scientist Cliff Kerr from the Institute for Disease Modeling (IDM), Senior Research Scientist Megan Kuhfeld from the Northwest Evaluation Association, and Senior Policy Researcher Julia Kaufman and Senior Statistician Claude Setodji from RAND. As you might imagine, these speakers addressed these questions from a variety of perspectives.

Stephen Sawchuk (Education Week) set the stage by providing a compelling overview of how those in the schooling context now dealing with the pandemic. His remarks were based on his comprehensive work as a journalist reporting, creating interactive maps, analyzing district re-opening plans and surveying educators. He described the variability of the composition of schools, the confused messages and lack of federal guidance, along with the types of health and non-health data that are being used by administrators. It quickly became evident how difficult it has been to make sense out of what is clearly a very unstructured problem space that surrounds these decisions.

Brian Gill and Ravi Goyal (Mathematica) described their work evaluating the options for school operations using agent-based models (ABM) – computational simulations to predict COVID-19 spread. While admittedly, there is no “crystal ball,” the thousands of simulations that they have run suggest that with precautions such as masking, social distancing and hybrid operations, many schools can operate without substantially increasing infection spread. Central to these simulations has been developing a better understanding of disease progression including the parameters for transmission probabilities and contact networks.

Similarly, IDM representatives Dina Mistry and Cliff Kerr shared what they have learned through Covasim an agent-based COVID-19 model built to explore different reopening scenarios. The parameters of interest include numbers of infections, the nature of person-to-person contact and the presence or absence of various interventions. Dina explored an example where the simulation was applied to a known school network and included a number of different reopening and diagnostic screening scenarios. The implications made it clear that there are no “risk free” scenarios. While certain countermeasures are effective in schools, results continue to underscore the importance of decreasing transmission within the community.

Megan Kuhfeld (Northwest Evaluation Association) turned the focus of the session. Her research involves the annual analysis of the assessment of learning. However, given the fact that disruptions seriously curtailed learning in schools in different ways for different children, Megan’s research is trying to understand how sensitive results on students’ academic progress during the pandemic are to “missing” students and other disruptions. She reviewed different scenarios to better understand the data on missing students and presented how results had changed from the Fall 2019 to Fall 2020.

The final speaker of the session was Julia Kaufman (RAND). Her research focuses on the results of national surveys of educators. She described a number of very telling findings. Among these are that teachers indicated that students’ learning experiences were limited by COVID 19, more so in the spring than the fall. In fact, in the spring, half of secondary teachers and teachers in schools with more poverty and students of color could not contact all their students. Furthermore, later in October, average daily student attendance was still down from what is typical, particularly in high poverty schools. Together, these and many other findings derived from RAND surveys provide a strong foundation for informing policy makers.

The forum kept the presentations short allowing a generous time for audience questions and a discussion of the challenging issues that have arisen during the pandemic. Moderator Betsy Becker (Florida State University) fielded these and also threw in a couple of questions of her own. Some of these questions which included: “If the National Center for Education Statistics (NCES) had the resources to do new surveys, what would you see as the most important questions to be asked to inform not only the rest of the school year but the return to full school operations?,” “How willing are people to share information about the data and the factors they consider?,” “How critical is access to technology and internet access?,” and more directly “Is your software or data available to the public?” However, the question that got perhaps the most interesting reaction from panelists was a challenge from Betsy: “What is the most dubious assumption that you have made in the modeling that you have done?!”

The back and forth of this session was wonderful, which makes the discussion around these topics definitely worth reviewing!

Below you can find a recording of this session along with copies of the slides that the speakers used.