Constructing Full Sample and Replicate Weights for NAEP Teacher Data

Data weights are constructed in order for statistical analyses of data to correctly represent results presented on a national scale, to accurately reflect the composition of the national population and to provide estimated standard errors for all reported statistics. The goal of this study was to explore the feasibility and utility of constructing full sample and replicate weights for the set of teachers whose data is collected by the National Assessment of Educational Progress (NAEP).

Two sources of data for fourth grade mathematics teachers were compared with respect to national averages for selected teacher characteristics: NAEP (using the reconstructed weights) and the 2010–11 Schools and Staffing Survey (SASS).¹ The selected characteristics were both compared marginally and jointly, using estimated standard errors calculated employing NAEP replicate weights and SASS replicate weights.

There are two principal findings:

1. Using NAEP school weights as teacher weights, and with a straightforward, national calibration of the NAEP weights to the SASS weights, the two sets of national estimates for five teacher characteristics common to both the NAEP and SASS datasets are essentially indistinguishable. In other words, the procedure to create teacher weights from these two sources works.
2. In general, estimated NAEP standard errors are smaller than estimated SASS standard errors. We believe that this is largely the result of the larger sample size in NAEP.

The implications for the National Center for Education Statistics (NCES) will require careful investigation. If, as these results suggest, NAEP has the potential to be superior to SASS as a mechanism for collecting data about teachers, then the NAEP teacher questionnaire will require re-design, because NAEP collects only limited information about the teachers themselves. Further discussion appears in the full text of this report.

The potential for using NAEP to explore relationships between teacher characteristics and student performance is addressed only tangentially in this study. How such exploration might be done is illustrated in the full text using the NAEP weights as constructed here. However, it must be stressed that the results presented only address the question of whether teachers with different characteristics teach student populations with different characteristics.
_________________________
¹ Such characteristics must, of course, be present in both data sets; see §2.3.