Analysis, Exploration and Inference in Large Educational Data Sets

Educational policy at the national, state, school district and even school levels depends on the large education data bases that are publicly available, especially those of the National Center for Education Statistics (NCES). Making sure that interpretations of these data are accurate and that the methodology used to report and to draw inferences do so correctly is essential to a solid basis for setting educational policy and standards. Making sure that appropriate methodology is available to the users of these data is critical.

A research collaboration amongst statisticians, psychometricians, education researchers tackled key areas related to analysis of NCES data in general and test results from NAEP (National Assessment of Education Progress) as a specific case. They analyzed the strengths and weaknesses of general methods often used for education data; and then they focused on development of new, correct statistical methods for analyzing complex surveys in the field of education. Problems considered included: How to compare in a fair way NAEP results for school districts with differing populations and widely different demographics? How many times can the same data be used for analysis of different objectives? How to improve upon a simple statistic like an overall mean when some population subgroups are disproportionately represented or are under-represented in the data? How to distinguish simultaneous occurrence from causality in large education data sets? How do analytic methods differ when seeking to test (in order to either confirm or reject) a hypothesis and when exploring to generate new conjectures (hypotheses)?

Answering these questions and providing new methodology for correctly making inferences gave education data analysts new tools for answering basic questions facing education policy makers.