Full Population Estimates for NAEP

The goal of NAEP is to provide high-quality indicators of performance for well-defined populations of students enrolled in selected grades of U.S. schools. Under current NAEP protocols, some students with disabilities (SD) and some English language learners (ELL) may be excluded from assessment, and inclusion rates differ across states.

The National Center for Education Statistics (NCES) charged a panel to technical experts, under the auspices of the NAEP Education Statistics Services Institute (NESSI), to “recommend to NCES whether and how NAEP should construct and report full population estimates (FPEs),” by addressing such questions as:

• Are FPEs a valid scientific construct? Are students with disabilities (SD) and English language learners (ELL) conceptually different issues for FPEs?
• Should FPEs be reported at all, and at what levels of resolution? How should associated uncertainties be reported?
• Is sound statistical methodology available to calculate FPEs, which can also address issues such as adjustment of weights?
• Should FPEs be reported in addition to or instead of estimates based only on students who actually took the test?
• If FPEs are reported, what interpretations or warnings should accompany them?
• If FPEs are reported in addition to current estimates, how should the relationship between them be portrayed? In particular, would one be presented as subsidiary to the other? How should inconsistencies be presented and interpreted? What are the policy implications?
• Should FPEs be used at all or only some levels of resolution (geographical, subpopulations, ...)?

Primary Recommendations

The task force believes that as a result of these exclusions that are not uniform across states, the goal of NAEP is difficult to meet under current protocols, and they will become increasingly difficult to meet in the future. The task force further believes that NCES must ultimately choose between two alternatives:

1. Adjust reported NAEP findings to include estimates of the performance of SD and ELL students who were not tested, but reasonably could have been.
2. Redefine the population that NAEP claims to cover so that it does not include some SD and ELL students.

The task force recommends Alternative 1 - construction and publication of adjusted estimates. It is recommended that these adjusted estimates be named expanded population estimates (EPEs). The task force further recommends that NCES set as its goal to report EPEs as the primary (or only) measure of NAEP performance. Finally, the task force recommends that NCES move as rapidly as possible to conduct studies that sharpen understanding of the statistical issues described in the report.

The task force strongly supports NCES’ continuing to take a proactive approach to the problem of variable inclusion practices. Otherwise, important comparisons (state-to-state, year-to-year and subgroup-to-subgroup) may be distorted by differences and changes in inclusion practices.