ATC21S Case Study

Assessing & Teaching 21st Century Skills

World Class Arena Limited led the development of collaborative problem solving assessments for the ATC21S project, as well as contributing to a literature review of curriculum frameworks for 21st Century skills.

The ATC21S project was established to find means of effecting change in the skills that are taught and assessed in high schools. The project was funded by Cisco, Intel and Microsoft and was led by Professor Barry McGaw and the University of Melbourne.

Background

The ATC21S project was created by Cisco, Intel and Microsoft; it was based on a belief that students were not being well-enough prepared for work and life in the 21st Century and that school curricula and assessments could be redefined to address the problem.

World Class Arena's expertise is specifically in the area of innovative design of assessments.

The ATC21S project reasoned that high stakes assessment has, in many school systems, become a major barrier to curriculum change.

Assessments signal priorities for curriculum and instruction; high visibility tests serve to focus the content of instruction. School administrators and teachers pay attention to what is tested, analyse test results, and adapt curriculum and teaching accordingly.
Teachers tend to model the pedagogical approach reflected by high-visibility tests.
Curriculum developers, particularly because of their commercial interests, respond to important tests. They modify existing textbooks and other instructional materials and/or develop and market new ones to address test expectations.
Schools and teachers tend to focus on what is tested, rather than on the underlying standards or learning goals, and to ignore what is not tested.
Instructional/teaching time is diverted into specific test preparation activities.

In terms of the longer-term impact on a student's learning, focusing on the test, rather than on the underlying learning, may encourage a one-time performance orientation and transmission-type teaching.

The project team's more detailed analysis can be found in the Springer publication Assessment and Teaching of 21st Century Skills.

The Challenge

Based on a review of existing curriculum and assessment frameworks for 21st Century Skills, the ATC21S project team commissioned World Class Arena Limited (WCAL) to design and develop assessments of Collaborative Problem Solving (CPS).

Although assessments of problem solving already existed, the ATC21S project team determined that the majority of these assessments were based on cognitive problem solving, where the challenge and demand of the assessments derived from the content matter. What the team sought was an approach where the problem itself would be unfamiliar to students and would require them to identify an effective strategy for solving it.

Furthermore, the brief from the ATC21S team to WCAL included a requirement that the problems should be designed so as to require two students to work collaboratively to arrive at a solution. Although the gaming industry had much experience of multi-player games, this approach had not previously been adapted successfully to high stakes assessments.

The challenge for WCAL was to:

define problem solving
define collaboration
design and implement assessments of CPS
find an IT delivery solution that would technically work in classrooms in a range of countries.

Our Solution

World Class Arena designed and developed prototype assessments of Collaborative Problem Solving. An early dimension to this was agreement with the ATC21S team that the assessments would consist of problem tasks for two students to solve together, working through an on-line chat interface as the means by which they would collaborate. The students would not be in the same classroom nor necessarily in the same school.

Initial ideas for the problem tasks were to be drawn from WCAL's database of problem solving questions. It was further agreed that each task would be designed to make collaboration central - for example, the two students could each be given different but complementary data needed to understand the problem, or the tasks could be designed to provide the two students with different functionality to use within the task.

This involved agreeing a working model (or definition) for problem solving. In this project, WCAL focused on the following dimensions of problem solving:

collecting and sharing information in order to develop an understanding of the problem
organising and sorting information to provide some structure and to devise a solution approach
a trial and error process to determine whether the approach would provide a solution
testing hypothesis to understand 'what if' scenarios.

WCAL also worked with the ATC21S project team to derive a definition of 'collaboration' in the context of assessment of problem solving capabilities.

Action
Interaction
Task Completion
Responsiveness
Audience Awareness
Self-Evaluation
Transitive Memory
Negotiation
Responsibility

For each of these dimensions , a 3-level grading schema was designed. The grading approach was designed to identify the contribution of each student within each pairing, providing an assessment of each student’s CPS skills.

Based on the above frameworks, WCAL created a team to design and code working prototypes of a small number of collaborative problem solving assessments. The ATC21S team conducted cognitive labs, involving students in reviewing draft assessments. Once finalised, the assessments were translated to other languages.

The Result

WCAL developed a number of tasks that the ATC21S project continued to refine and pilot in Australia, Costa Rica, Finland, Netherlands, Singapore and the United States.

One such task was Game of 20. In this task, the two student players combine to form a team competing against the computer. The student-team and the computer alternated turns to add a score of 1-5 points to a running total. The objective was to be the team to make the total 20; in order to do this successfully, the students needed to discuss how the game works and to figure out a winning strategy.

The students each select a number (1-5), which is combined to represent the students' move. As the students progress through the game, they are presented with a series of increasingly challenging questions designed to probe their understanding of how the game works as well as their ability to collaborate in formulating a winning strategy.

A full description of the ATC21S Collaborative Problem Solving Tasks can be found here.

In addition to designing and developing these tasks, World Class Arena Limited worked as part of global research team, reviewing existing frameworks of 21st Century Skills and the ways in which educators in different countries have conceptualised these skills. This work led to the development of a meta-framework, identifying the following five learning pillars:

Knowledge

Skills

Attitudes

Values

Ethics

This has become represented as the KSAVE framework.

This meta-analysis and other descriptions of the ATC21S project have been published in Assessment and Teaching of 21st Century Skills, available here.