Effect Size, Uncertainty, Completeness

The Task Force was convened at ESSI in Washington, DC on December 11, 2006. Also present were NCES Chief Statistician Marilyn Seastrom and Special Assistant to the Commissioner Andrew White. Presentations to the Task Force were made by NCES staff members Chris Chapman, Bill Tirre and John Wirt. Following the meeting the Task Force formulated an initial set of recommendations, which were refined and finalized through a series of e-mail interchanges. These appear in final form in this report.

Summary

The Task Force strongly supports reporting of effect sizes by NCES while recognizing that in some instances this may be inappropriate, ineffective or even impossible. Also, the Task Force acknowledges that effect sizes are attractive because they are dimensionless, this can also raise issues of interpretability that cannot be dismissed lightly.

Recommendation

NCES should routinely report effect sizes for differences in means and for categorical comparisons, unless there is compelling reason not to.

Recommendation

A chief exception for reporting is when the absolute difference is not statistically significant or when it is below an a priori designated detection threshold. In such cases, effect sizes should not be reported.

Recommendation

NCES tables that contain effect sizes should not also contain the actual differences in means or proportions (categorical comparisons).

Recommendation

NCES should identify the circumstances in which reporting uncertainties associated with effect sizes improves quality and usability and implement this practice.

Recommendation

NCES should evaluate the feasibility of developing and implementing a sensible, consistent mechanism for calling attention to large effect sizes.