Research Project
The Internet is rapidly becoming the main way people read and process information. The Internet requires additional reading, comprehension skills, beyond those needed for printed text. One dilemma is how do students discern what information is correct and what is incorrect?
NISS participated in a $2.8 million research project with the University of Connecticut, the Pennsylvania State University and the University of Rhode Island funded by the Institute of Education Sciences (IES) of the US Department of Education to study just this subject. Assessing Online Reading Comprehension: The ORCA Project directly involves the states of North Carolina, Connecticut and Maine. The Regional Educational Laboratory for Northeast and Islands will also be involved in the project to consider the practicalities of online assessment. The project is being carried out by a research team with expertise in multiple areas essential to developing online reading comprehension assessments. The team members are at work on theory development, research design, measurement and statistical analysis to develop valid, reliable and practical assessments of online reading comprehension. NISS' role focuses on the design of the studies and on the statistical analysis of the data gathered from the three states.
The study has four main goals. The first is to develop three different types of valid and reliable assessments to measure online reading comprehension. The second goal is to evaluate the internal assessment characteristics for each instrument type to inform decisions about which is most useful and practical for schools. Next, the research team will evaluate the extent to which performance on each format is associated with four measures of contextual validity. Finally, the utility and practicality of each instrument in the eyes of key decision makers were estimated.
When the study was being developed in 2008, many school systems were not in a position to adequately teach online reading skills in the United States. The schools lacked a valid, reliable assessment of online reading comprehension. By evaluating reading comprehension in terms of four parts of online reading (locating information, evaluating credibility, synthesizing information and communicating knowledge and inferences), the study also aimed to see whether or how Internet reading skills go beyond simple text comprehension and if so, whether traditionally measured reading skills are accelerated or impeded.
NISS was a key participant in this project from its inception, bringing statistical principles to bear on the problem of designing this study to ensure unbiased results. NISS also developed the sampling plans that assured each participating state that the classrooms participating are fully representative of the state.
Create an assessment that would measure both overall student capabilities in using the internet as a source for learning and would isolate and measure specific skills required for success. Understand how students perform differently on a performance based, interactive ORCA compared to traditionally formulated test, and identify student characteristics that create advantages or disadvantages in this new testing framework.
Research Team: Nell Sedransk, NISS; Don Leu, University of Connecticut; Jonna Kulikowich, Pennsylvania State University; and Julie Coiro, University of Rhode Island.
Postdoctoral Fellow(s): Wiewei Cui (NISS postdoc)