Abstract
INTRODUCTION
The OECD PISA results have been used as a catalyst for educational reform, even though the evidence for invariance across languages is weak. Scores obtained from tests that are subsequently adapted cannot be assumed comparable unless scalar equivalence across languages is present.
The use of PISA across nations, cultures and languages has been criticized. The key criticisms point to the linguistic and cultural biases potentially underlying the design of reading comprehension tests and which raise doubts about the legitimacy of comparisons across language boundaries.
OBJECTIVES
Our research question was: What level of invariance is seen in the PISA Reading Comprehension test by language and culture relative to performance by the Australian English speaking reference group?
METHODOLOGY
To ensure equivalent sample sizes 500 students were randomly selected from PISA 2009 dataset for each country. MG-CFA based on MACS was used to examine levels of measurement invariance (MI). Changes in CFI ≤.01 were used to determine invariance. A scatterplot of ∆CFI against ∆Chi-square was used to identify patterns with respect to language and country.
RESULTS
Measurement model had good fit for each country. Only Ireland, New Zealand, UK, English Canada and USA were invariant to Australia. Scatterplot with respect to language clarifies that language alone is not sufficient for invariance. For example, Trinidad and Tobago uses English but is not a high-wealth society. Spanish in Spain is relatively close to Australia compared to poorer countries of Latin America.
CONCLUSIONS
Results indicate that complex factors to do with educational practice and socio-economic resourcing of education, rather than language per se do interfere with the MI of the PISA reading comprehension results. These results suggest that PISA reading comprehension tests are very much a product of a wealthy, Anglo-Commonwealth society and educational approaches to reading instruction.
The OECD PISA results have been used as a catalyst for educational reform, even though the evidence for invariance across languages is weak. Scores obtained from tests that are subsequently adapted cannot be assumed comparable unless scalar equivalence across languages is present.
The use of PISA across nations, cultures and languages has been criticized. The key criticisms point to the linguistic and cultural biases potentially underlying the design of reading comprehension tests and which raise doubts about the legitimacy of comparisons across language boundaries.
OBJECTIVES
Our research question was: What level of invariance is seen in the PISA Reading Comprehension test by language and culture relative to performance by the Australian English speaking reference group?
METHODOLOGY
To ensure equivalent sample sizes 500 students were randomly selected from PISA 2009 dataset for each country. MG-CFA based on MACS was used to examine levels of measurement invariance (MI). Changes in CFI ≤.01 were used to determine invariance. A scatterplot of ∆CFI against ∆Chi-square was used to identify patterns with respect to language and country.
RESULTS
Measurement model had good fit for each country. Only Ireland, New Zealand, UK, English Canada and USA were invariant to Australia. Scatterplot with respect to language clarifies that language alone is not sufficient for invariance. For example, Trinidad and Tobago uses English but is not a high-wealth society. Spanish in Spain is relatively close to Australia compared to poorer countries of Latin America.
CONCLUSIONS
Results indicate that complex factors to do with educational practice and socio-economic resourcing of education, rather than language per se do interfere with the MI of the PISA reading comprehension results. These results suggest that PISA reading comprehension tests are very much a product of a wealthy, Anglo-Commonwealth society and educational approaches to reading instruction.
| Original language | English |
|---|---|
| Pages | 128-128 |
| Number of pages | 1 |
| Publication status | Published - 2014 |
| Externally published | Yes |
| Event | The 9th Conference of the International Test Commission:Global and Local Challenges for Best Practices in Assessment - Miramar Palace in San Sebastian, Sen Sebastian, Spain Duration: 2 Jul 2014 → 5 Jul 2014 https://www.intestcom.org/news/1 |
Conference
| Conference | The 9th Conference of the International Test Commission:Global and Local Challenges for Best Practices in Assessment |
|---|---|
| Country/Territory | Spain |
| City | Sen Sebastian |
| Period | 2/07/14 → 5/07/14 |
| Other | The 9th Conference of the International Test Commission which was organized in San Sebastian, Spain, 2-5 July 2014, has just come to an end. Excellently organized and a biiiig success: congratulations to the local organizer, Prof. Paula Elosua, and thank you to all attendants and sponsors who made such a great event possible. Under the theme Global and Local Challenges for Best Practices in Assessment, ITC2014 has showcased new frontiers of assessment as a means for improving and developing psychological and educational tests and test uses. ITC2014 was held at the Miramar Palace in San Sebastian, Spain. The Miramar Palace was built in 1893, and is located on the beautiful La Concha Bay. Miramar offers one of the most spectacular views of San Sebastian, which itself is an amazing city in the North-East part of Spain. |
| Internet address |