Science Improvement Program, Malaysia | WCAL

Science Improvement Program (SIP), Malaysia

World Class Arena supported Prestariang and TWIG in designing and delivering an evaluation of the effectiveness of a

Science Improvement Program (SIP). SIP was designed to support the Ministry of Education in Malaysia in its aim of raising standards of performance in its secondary schools following publication of international PISA results in 2009 and 2012.

Background

TWIG was looking to provide evidence to Malaysia's Ministry of Education of the efficacy of its science learning resources in improving learning outcomes. The Ministry of Education was looking to find ways of improving the country's learning outcomes, as measured through PISA assessments.

TWIG partnered with Prestariang in Malaysia to design a 'proof-of-concept' evaluation study. World Class Arena's role was to author the learning resources, create pre-study and post-study student assessments, and to provide teacher training. The Institute of Education in London was appointed to provide independent advice on the design of the study and to provide the quantitative analyses.

The Challenge

TWIG and Prestariang wanted to provide evidence, based on an empirical study, of the effectiveness of their learning resources and Science Intervention Program.

World Class Arena was asked to design a classroom intervention study that could be conducted in schools across Malaysia. A major drive for the Ministry of Education was to improve the science education of secondary school students and their performance in PISA studies; it was therefore a requirement of the study that the effectiveness of the intervention could be related to PISA outcomes. The intervention would need to be based in the science curriculum and would need to reflect closely the national science curriculum, paralleling the content that would be being taught in schools at the time of the intervention.

It was further required that teachers should be trained in the use of the learning resources.

The intervention needed to be evaluated independently of TWIG and Prestariang.

Our Solution

World Class Arena Limited (WCAL) designed an approach based on the evidence of the effectiveness of classroom interventions. Educational literature showed clearly that where teacher training was placed at the heart of an intervention, larger improvements in learning were obtained.

WCAL designed an eight-week intervention. The intervention programme involved the creation of a short course, focused on Physics and the topic of friction. WCAL worked with the Institute of Education in London to agree a study design. This involved the creation of two samples of schools; the schools in the Treatment Group attended the Teacher Training and implemented the lessons provided by WCAL. Schools in the second sample, the Control Group, did not attend training or have access to the lesson materials. Students in both groups completed a science test prior to the intervention as well as completing a second, parallel test, at the end of the intervention.

The intervention program included the following elements:

Lessons designed by WCAL. Each lesson included student resources and an accompanying Teacher Handbook. All resources were translated to Malaysian by Prestariang.

Comprehensive Teacher Training, designed and delivered by WCAL, Prestariang and TWIG.

The pre-intervention and post-intervention assessments were authored and calibrated by WCAL. These assessments included a sample of published PISA science items in order to provide some referencing to the PISA international standards.

Student questionnaires and teacher questionnaires were also developed by WCAL.

A sample of schools was drawn. This involved 52 classes and 60 teachers, with a total of 1575 students. Of the 60 teachers, 30 were in the Treatment Group, following the intervention. The other 30 were in the Control Group.

The Result

WCAL created a team of experts in science curriculum design and in assessment to develop the intervention materials.

Creation of Friction Lessons

WCAL authored five lessons, focused on friction, for grade 10 students, The five lessons included: an investigation, an experiment, friction in a novel context (bobsledding) and a lesson that involved calculations. WCAL created student resources and teacher guidance notes. These notes included background information, pedagogical guidance and suggestions for 'further research' for students to explore the topic more deeply.

Where helpful, lessons embedded TWIG science video resources.

All lessons included examples of assessments, some of which were presented as sample test questions.

Provision of Teacher Training

WCAL supported TWIG and Prestariang in providing teacher training prior to the intervention. The training focused on the science content of the lessons, as well as providing opportunities to develop pedagogical understanding.

WCAL's training included an overview and background briefing on PISA and science improvement.

Tests and PISA

WCAL designed and authored tests to measure the effectiveness of the intervention. Two tests were created - one used prior to the intervention, the second used after teachers had delivered the science lessons. The tests were anchored to each other. The tests also included a small number of published PISA science questions; it was deemed unlikely that any of the study teachers would have accessed those sample questions.

WCAL also developed the scoring rubrics for these assessments. Scoring was completed by Prestariang.

Questionnaires

WCAL designed a Teacher Questionnaire and a Student Questionnaire. The Student Questionnaire focused on student's attitudes towards science, their views of their performance and achievements in science, as well as on their future career aspirations.

Study Findings

The main data used to assess the effectiveness of the intervention was the test performance data. Comparing the Control Group of classrooms with the Treatment Group, the analysis completed by the Institute of Education in London found highly significant improvements, as measured by the tests, as a result of the intervention. The graphic below provides a summary view of these classroom-level data. The overall outcomes demonstrated improvements in student performance in around one-third of the sample of schools involved in the study.

The intervention also met a primary purpose of identifying students performing at the higher levels of the PISA science scale.

Teachers rated the lessons highly and responded well to the training.

The study resulted in a recommendation to the Ministry of Education to implement a national science intervention, based on the principles that had informed the design of the SIP study.