Summary:
ORCA I is the first in a series of large studies to evaluate a performance-based and a comparable task-based assessment of internet literacy. Over 1300 7th graders at 43 schools in two states participated; of these 1079 completed all background information, an offline reading measure and two assessments. Each student was assigned to one of three formats. Two were performance-based assessments defined by their internet access: ORCA-Open with access to the ORCA-Open, ORCA-Closed with access to a synthetic internet built for this project with a search engine and an extensive set of resources. The third format was a task-based multiple choice (MC) assessment that was administered on a computer screen but following a traditional approach; the items themselves mimicked the items in the performance-based assessment.
The overall goal of this study was to investigate the differences between performancebased online assessment and task-based traditional multiple choice assessment of online reading comprehension. The first analyses compared performance-based and task-based assessments (internet, including both ORCA-Open and ORCA-Closed compared to Multiple Choice) and then compared the real-world context of ORCA-Open with the synthetic but stable context of ORCA-Closed. The second set of analyses, reported here, examined the students’ performance profiles across the four components (Locate, Evaluate, Synthesize, Communicate) that make up the performance-based assessment. Then the important factors in students’ performance on these assessments were identified and subgroups of students who shared common performance profiles where characterized in terms of these factors.
tr192.pdf