Non-Response Bias Analysis

The purpose for convening this task force was to assist NCES in understanding the range of methods used currently for nonresponse bias analyses in its data collections, the criteria by which such methods are selected, and other available techniques for Non-Response Bias Analysis (NRBA), including Bayesian methods. Experiments were also conducted to inform the Task Force in reaching its recommendations. (See Appendices A and B). Ultimately, NCES may revise its statistical standards - specifically, Standard 4.4 (http://nces.ed.gov/statprog/2002/std4_4.asp) in light of recommendations by the Task Force.

SPECIFIC RECOMMENDATIONS

The Task Force recommends that NCES employ the NRBA process described in section 1, and specifically that:

1. A minimal NRBA report be required for NCES data collections prior to the main data collection, which compares respondents and nonrespondents in terms of available - frame and other) - variables identified on the basis of domain knowledge or other data collections to be related to responses and outcomes of interest in the data. In general, such comparisons can and should be performed using multivariate methods.
2. From the minimal report, NCES determine whether the response bias is unimportant or important. 1
3. In the case of unimportant nonresponse bias, base weights of respondents be adjusted, and the data collection and subsequent reporting proceed.
4. In the case of important response bias, two mandatory additional steps be performed: (1) Finegrained analysis, in the same as in the minimal report, for subgroups of importance; and (2) A benchmarking comparison to similar studies.
5. Also, in the case of important response bias, as many as four optional additional steps be performed, chosen from (1) Sensitivity analysis; (2) Level-of-effort analysis; (3) Identification of additional predictors of the responses and outcomes of interest, and further comparison of respondents and nonrespondents using them; (4) Follow-up of nonrespondents.
6. Following completion of the mandatory and optional additional steps, NCES decide whether to “discard” or “employ” the data, and if the data are employed, whether to label nonrespondent values as missing, or to reconstruct these values using multiple imputation.

The Task Force further recommends that:

1. Frame and other information pertaining to nonrespondents not be removed from restricted or publicly released databases.
2. The minimal report and the results of mandatory and optional additional steps be made available by NCES to users of the data, possibly in redacted form.