Citing Significance in NCES Data Reporting

The vigorous national dialogue about how to describe the “significance” of research findings that an NCES-NISS Expert Panel addressed in 2018 has shifted. No longer is it about whether to move away from dichotomizing results into “significant” or “non-significant” (e.g., p < 0.05) but rather about how to do it and what information must now be supplied. That earlier panel determined that whether findings are reported as a data summary or as in-depth analyses, their importance must reflect both the magnitude and the associated uncertainty. In addition, interpretation depends on the subject matter context and the intended use of the information.

Therefore the specific charge to the current (second) panel was to recommend how to report the importance of results clearly and accurately in a manner that moves away from the p-value < 0.05 standard but is understandable to the public and acceptable to academic institutions. The panel responded that substituting alternative language could not be sufficient because the significance of a research finding must be judged in its substantive context. Magnitude and uncertainty address the questions: “What is the best estimate? What are the possible alternatives and how likely are they?” But the driving question is substantive: How much do any of these matter (in a substantive sense)?

The overall objective is clear, accurate, complete and transparent reporting of findings from NCES data. Accomplishing this requires providing more complete information (magnitude and uncertainty, at least) than a threshold.

Publication of NCES reports is unique in two important aspects that present specific challenges for NCES data reports. The first challenge arises from the breadth of the NCES readership. The second challenge for NCES reports is to make additional, more detailed information available about some of these complexities based on the finer scale data that NCES maintains in restricted files. Meeting these challenges requires accessible reporting that is credible at multiple technical levels from non-technical for the general public to technically clear for researchers in academia and outside.

Primary Recommendations

• Lead with Magnitude and its associated Uncertainty.
• Represent both magnitude and uncertainty everywhere, in every format: text, table, graph, figure, other visualization.
• Formulate an analysis plan in substantive terms with a narrative that drives selection of variables and factors selected, defines populations included/excluded.
• Support the analysis plan with appropriate statistical approach and methods.
• Publish complete results from all analyses corresponding to the analysis plan and give transparent access to statistical analysis process (since there is no direct access to restricted data for confirmation).
• Distinguish secondary and exploratory analyses clearly from planned analyses, also noting that uncertainty measures and p-values cannot be assumed to apply accurately.
• Present reports with equal depth and equal clarity in non-technical and technical language, with links to the underlying statistical analyses to permit validation.

Specific Recommendations for Implementation

• Revise NCES Standards and Guidelines to update and expand statistical methodology and to include modern data visualization.
• Educate NCES Staff and Contractors.
• Involve substantive and statistical experts in analysis plan and in report development.
• Align Review Process to new Standards and Guidelines.