As schools in the United States are using more and more digital technology for teaching assessment a new window is opening into students’ actual processes of arriving at responses to each item. The National Center for Education Statistics (NCES) charged NISS with convening two panels to consider how best to organize and provide access to nationally administered tests – starting with released items from NAEP (National Assessment of Educational Progress, i.e., “the nation’s report card”).
Digitally-based testing records each student’s progress through a test, click by click with each action time-stamped. In addition digitally-based assessments provide greater opportunity for the use assistive technology and can be adapted to provide enhanced accommodations for students with special needs. Most importantly, collecting these new types of data, can serve to enhance understandings of what students know and can do as they engage with technology to problem solve.
Test process data is now, for the first time, being made available to researchers – with consulting expertise from NISS!
These new data are the test process data that reveal in detail the paths students follow as they progress through an online test. For example, some students go from start to finish in an orderly way, even more follow different paths. Some jump forwards and backwards, some return to items to change answers even while they are proceeding through the test and some students return to check or change answers before submitting; some students never look back. Since each click is time-stamped, it is also easy to learn how long a student spent on a single question and what response the student chooses each time the student visits that question.
“Researchers have been waiting for this kind of data that chronicles the click-by-click path of each individual student as the student responds to the items on a test. Now testing can be studied by analyzing more than just correctness of responses. This has implications for constructing better tests as well as for understanding students’ thought processes.”
Nell Sedransk, NISS Director-DC
NISS was commissioned by NCES to convene panels of technical experts to advise on how to most efficiently and safely release the data in a useful form. Panelists included leading researchers in the technical fields of technology-based assessment, computational psychometrics, psychology of testing, machine learning and natural language processing, federal statistical data systems, and statistical and data sciences.
The first panel focused the experts’ attention to both the data base and data extraction methods used so that data could be released effectively and comprehensively would maintain strict privacy and confidentiality requirements avoiding unacceptable security or disclosure risks. (read the full report: Release of Process Data to Researchers (2019))
The second panel’s attention was focused on the definitions used for calculated variables so that these data could also be made directly available in addition to the raw “click data”. This second charge required substantial IT/data base collaborative expertise to efficiently extract these data into a usable, standardized data file that allows researchers to access and work with this complex, high-dimensional data. (read the full report: Release of Process Data to Researchers, (2020))
Who needs these data? Not just test developers! More importantly, this is data that those who study how students solve problems and what differentiates different ways of processing information are very interested in being able to review. For example, as a result of having access to this deeper set of student behavior data in an online test-taking environment, researchers can analyze students’ strategies, their attitudes and abilities to focus in responding to the challenge of a test. Researchers who study special groups of students such as English language learners or students with particular disabilities can look at differences in response paths compared to mainstream students and may be able to identify the potential for biased assessment from particular kinds of questions or presentations of item stem information.
For more on this see “The Nation’s Report Card”: Response Process Data From the 2017 NAEP Grade 8 Mathematics Assessment.
NISS is proud to have become a partner with NCES and others in tackling challenges where information and quantitative analysis are keys to solutions and decisions. NISS is always looking for ways to leverage the rich expertise of its staff with that of its base of affiliated organizations in academia, industry and government to provide high-impact research in science and in public policy.