NISS/NCES Technical Panel on Configuration and Data Integration for Longitudinal Studies

This is the final report of the National Institute of Statistical Sciences (NISS) Technical Panel on Configuration and Data Integration for Longitudinal Studies (hereafter, CDI).

The principal recommendations regarding configuration are as follows:

1. The National Center for Education Statistics (NCES) should configure grades K-12 studies as a series of three studies: (1) a grades K-5 study, followed immediately by (2) a grades 6-8 study, followed immediately by (3) a grades 9-12 study. One round of such studies, ignoring postsecondary follow-up to the grades 9-12 study, requires 13 years to complete.
2. Budget permitting, NCES should initiate a new round of grades K-12 studies every 10 years. This can be done in a way that minimizes the number of years in which multiple major assessments occur.

The technical panel finds that there is no universal strategy by means of which NCES can institutionalize data integration across studies. One strategy was examined in detail: continuation of students from one study to the next. Based on experiments conducted by NISS, the technical panel finds as follows:

• The case for continuation on the basis that it supports cross-study statistical inference is weak. Use of high-quality retrospective data that are either currently available or are likely to be available in the future can accomplish nearly as much at lower cost.
• Continuation is problematic in at least two other senses. First, principled methods for constructing weights may not exist. Second, no matter how much NCES might advise to the contrary, researchers are likely to attempt what is likely to be invalid or uninformative inference on the basis of continuation cases alone.

The technical panel stops short of a categorical recommendation against continuation. If the continuation group was a representative sample, then it might provide meaningful results, albeit with large variability. The technical panel urges that, as an alternative means of addressing specific issues that cross studies, NCES consider the expense and benefit of small, targeted studies that target specific components of students’ trajectories.

The technical panel was not charged to examine in detail the articulation between grades K-12 and postsecondary studies. It does, however, note that the current once-every-4-years frequency of the National Postsecondary Student Aid Study (NPSAS) is not congruent with the 10-year cycle of recommendation 1. By contrast, a 5-year frequency for NPSAS would allow every other NPSAS to follow immediately after a grades 9-12 study.