Study Design Decisions for Postsecondary Sample Surveys

Redesigning a survey presents a rare opportunity to consider all aspects of the survey from the perspectives of both the producer and user. The National Center for Education Statistics (NCES) charged the National Institute of Statistical Sciences (NISS) with convening a panel of technical experts to consider the issues for design of the National Postsecondary Student Aid Study. In particular, the panel was asked to address the sample design strategies currently in use for NPSAS, strategies including responsive designs employed by other recent NCES surveys, and applicable innovations, methodological advances and also alternative design approaches from other large-scale federal or governmental agency surveys and assessments. Broad topics to be considered included: i) Sample Design, Sampling Error and Design Effects, ii) Measurement Error, and iii) Imputation and Administrative Data.

The panel met via teleconferences with an in-person meeting at NCES on 31 August -1 September 2016.

Recommendations – I. A Perspective and a Framework

1. In considering redesign of the NPSAS to improve data quality, frame the design from a total error perspective that seeks to minimize the error from the primary error sources within specified costs and scheduling constraints.
2. In addition to accuracy of the NPSAS data, consider the so-called user dimensions of total survey quality shown in Figure 1.
3. Recommendations – II. Sample Design, Sampling Error and Design Effects
4. Evaluate precision goals. Determine whether all 1100+ precision targets are necessary to meet the major goals of the survey and whether reduction of the number of targets could lead to important cost savings. Ascertain whether all goals were met for NPSAS:12 and NPSAS:16.
5. Evaluate allocation to institutional and student strata. Utilize variance components and update the NPSAS cost model to determine the most efficient allocation of institutions and students to strata.
6. Explore use of a dual frame design with a special frame for veterans (using a VA list of veterans who receive educational benefits) to control the veterans’ sample size better and reduce some under-coverage.
7. Explore rotation of sample institutions.
8. Evaluate composite measures of size. Evaluate how well the composite MOS’s are related to actual student counts and how often within-institution sampling rates have to be adjusted to account for inaccurate MOS’s.
9. Evaluate weighting steps. Devise a study to investigate the contributions of each step to bias reduction, variance increase or decrease, and the contribution to mean squared error for a set of important estimates.

Recommendations- III. Measurement Error

1. Repeat the 1997 measurement error report.
2. Evaluate two additional aspects of measurement quality: interviewer effect and mode effect.
3. Utilize NCES’s excellent assessment of measurement error to improve questions and reduce measurement error going forward. Institute a routine system to review, revise, and/or eliminate questions that are subject to large amounts of measurement error.

Recommendations – IV. Imputation and Administrative Data

1. Implement imputation methods that account for variance induced by imputation, for example fractional imputation or multiple imputation.
2. Make trumping rules explicit and provide an evaluation of their utility.
3. Include in the imputation documentation for NPSAS the list of class covariates used for each variable and the order in which the variables were imputed.
4. Assess and document the quality of the various administrative data in NPSAS, specifically addressing what biases, if any, are present.
5. To the extent possible, determine whether the students who do not match to any administrative data are comparable to those who match to administrative data; and evaluate the effect of any differences with respect to variables of interest on imputation and subsequent analyses.
6. Examine the variance implications when direct substitutions are treated as response data as differences in the distributions of the administrative data from those of the response data can result in bias (in either direction) of the calculated variance.

Further Recommendations

In addition to the specific findings, the panel is concerned that NCES faces a structural conundrum in regard to the statistical elements of study or survey design. The technical statistical expertise to continue to assure best statistical practices is not currently present in-house, nor is it represented on current survey oversight committees. While contractors do possess this expertise, there is some possibility for conflict of interest as innovation may or may not be in the contractor’s own interest. The panel notes that NCES statistical leadership is well aware of this issue and has encouraged innovation and exploration of new statistical methodology for design, imputation, estimation and all phases of analysis.

Additional specifically focused recommendations are included in each section of this report.