Our Client Case Studies
​
Our Client Case Studies
​
PISA & the OECD
The OECD’s Programme for International Student Assessment, running since 2000 and now in its sixth tri-annual global survey, was first delivered on computer in 73 countries in 2015.
​
The Case Study below describes WCAL's role in providing analysis and technical support to inform PISA Governing Board's decisions about whether to transition PISA to computer-based delivery, as well as WCAL's role in writing the technical specification for the required global delivery platform.
Background
​
The tri-annual international PISA student survey includes tests to be completed by 15 year-old students. These tests cover mathematics, reading and science plus usually one additional domain. When PISA was introduced in 2000 (and until 2015) these main tests were administered in participating countries on paper. An essential component of the PISA surveys is that all countries should complete the assessments under identical conditions, and this requirement had limited the scope of any computer-based tests. There had been some 'pilot' activity, but this had been restricted to optional and non-core aspects of the surveys and had involved only a few volunteer countries.
​
In 2011, the countries that together comprise the PISA Governing Board commissioned a Cost-Benefit study to investigate and report on issues relating to the administration of PISA tests on computer. WCAL was commissioned to undertake this investigation which was reported to the Governing Board early in 2012. The case study below provides more information about this investigation and about the subsequent development of a technical specification for the global CBT delivery system requirements for PISA tests.
The Issues
​
The PISA Governing Board was seeking data and information that would help their discussions and decision-making regarding the potential transition of the tri-annual assessments to a computer-based platform. Lacking specific data and information, discussion risked entrenching attitudes and perceptions of CBT.
​
The PISA Governing Board had identified a number of issues:
​
-
some countries perceived that CBT would involve additional expenses and would be more clostly than paper-based administration. There was a need for more specific data about how CBT would affect countries' PISA-related cost profiles;
​
-
there was a wish to understand better how the measurement of international educational trends (which had begun in 2000) would be affected by any switch to CBT. There was debate about whether standards would be affected by such a switch; and there was a parallel debate about whether the trend measurement was sufficiently important to warrant sticking with paper;
​
-
there was a need to understand the benefits of CBT in the context of the PISA programme and to understand how the speed of data-processing might be affected by CBT, whether the quality of data would be affected, and what changes there would be to data-flows;
​
-
there was a need to understand more about each country's experience of using CBT in school assessment and to build an understanding of the support and training that might be required;
​
-
finally, the Governing Board wished to understand the arguments regarding validity of the PISA surveys in the context of CBT and was seeking more specific information regarding the potential need for assessments of maths, reading and science to better reflect the skills that are needed in 21st century economies.
​
​Based on the above data and information, the Governing Board wished to be provided with proposed models and options for migrating the PISA assessments to a computer-based platform.
The Solution
​
World Class Arena Limited (WCAL) designed a number of strands within the Cost-Benefit study. A small number of PISA-participating countries were invited to participate in the study, ensuring there was a range of countries in terms of educational and technology backgrounds, perceptions of CBT and economic development.
​
A first study strand focused on collecting background information regarding the profile of costs incurred by each country in administering paper-based tests, beliefs about the benefits of CBT and experience of using CBT in educational contexts, including any participation in previous PISA CBT pilots.
​
This first strand was to be followed by a more detailed survey of CBT, conducted with the same group of countries. This detailed survey provided countries with descriptions of a CBT model, making clear the main changes and data flows. The survey asked countries to assess the model, identifying major cost changes, data processing changes that might affect timescales and data quality, as well as to identify aspects of the CBT model that would require additional training or controls.
​
A final fieldwork strand included telephone interviews and discussion with the study countries to probe their responses in more detail.
​
The study report was designed to provide data and metrics to support debate and decision-making at the PISA Governing Board about whether to transition the student assessments to CBT. The report covered:
​
-
Forecasts of the financial effects on in-country costs of running a PISA survey across the full 3-year cycle of each assessment.
​
-
Benefits of CBT and opportunities for CBT to deliver improvements in the timetable for conducting key operations in a PISA cycle and to improve the quality of data captured.
​
-
Management implications, with a summary of the skills and knowledge that would be required within each PISA-participating country team, as well as the programme risks that would need to be managed.
​
-
Thumbnail sketches of each country that contributed to the Cost-Benefit survey, summarising their overall approach, experience of using computers in national testing, and participation in optional PISA CBT surveys and pilots.
High-level PISA assessment and data processing map:
Paper-based administration, with associated timing requirements.
The Result
​
An important function of the Cost-Benefit survey was to support the countries involved in the study in coming to a firm view regarding whether the PISA assessments should be transitioned to CBT. In the event, by the time WCAL's report was delivered to the PISA Governing Body, all but one of the participating countries was supportive of the transition.
​
The report was presented in 2012, as a result of which the PISA Governing Body made the decision to transition to CBT.
​
World Class Arena was subsequently commissioned to write the technical specification of requirements for the global test delivery platform. Once this specification was published by the OECD as part of an international RFP, WCAL also sat on the selection panel, reviewing proposals received in response to the RFP and advising the OECD and PISA Governing Board on the selection of a vendor to provide the computer platform.
​
The 2015 PISA survey was successfully delivered on computer. The 2018 survey has now also been delivered on the same platform.
