Originally published on Brooking Institution's Brown Center Chalkboard, January 9, 2013
Last month, the latest results were released from the 2011 Trends in International Mathematics and Science Study (TIMSS). The scores dispelled the myth that all international tests are the same.
TIMSS and Program for International Student Assessment (PISA) are quite different. TIMSS is curriculum-based, reflecting the skills and knowledge taught in schools. PISA assesses whether students can apply what they’ve learned to solve “real world” problems.PISA tests an age-based sample (15 year olds). PIRLS and TIMSS are grade-based (4th and 8th graders). PISA is overseen by representatives from participating governments who meet under the auspices of the Organization for Economic Cooperation and Development (OECD). TIMSS and PIRLS are governed by a consortium of researchers and government representatives known as the International Association for the Evaluation of Educational Achievement (IEA).
It is true that scores from the two tests are highly correlated at the national level. But the two tests do not measure the same learning. Scores from NAEP’s math and reading tests are also highly correlated when aggregated to the state level. No one would argue that NAEP’s reading and math tests measure the same learning.
Discrepant results from PISA and TIMSS are revealing. When releasing the latest PISA results in 2010, Angel Gurría, Secretary-General of the OECD called New Zealand a “high flier.” New Zealand is definitely not a high flyer on the TIMSS math tests, scoring only 486 in 4th grade and 488 in 8th grade–significantly below the international means of 500.
The PISA math assessment is based on a philosophy known as Real Mathematics Education (RME), championed by the Freudenthal Institute in the Netherlands. Jan deLange of the Freudenthal Institute chairs the PISA expert group in mathematics. RME’s constructivist, problem solving orientation is controversial among mathematicians. In the U.S. in the 1990s, a coalition of mathematicians, parents, and local educators opposed similar types of curricula in what became known as the “math wars.” In 2010, New Zealand implemented national standards that are compatible with RME and PISA—and less compatible with TIMSS.
The contrast is stark between constructivist countries and those favoring more traditional math curricula. On PISA, New Zealand scores within 27 points of Korea (519 vs. 546). On TIMSS, New Zealand and Korea are separated by a whopping 125 points (488 vs. 613), a difference of nearly one full standard deviation between the two tests! Chinese Taipei outscores Finland by only 2 points on PISA (543 vs. 541, the scores are statistically indistinguishable)—but by 95 points on TIMSS (609 vs. 514).
Finland is a great country and makes a wonderful travel destination. It also has fine schools. But its reputation in education is a bit overblown, based primarily on high PISA scores and an aggressive educational tourism industry. Like New Zealand, Finland also has a math curriculum compatible with PISA. In 2011, Finland participated in TIMSS for the first time since 1999. Finland’s 2011 math scores are statistically indistinguishable from the U.S. at both 4th and 8th grades. While U.S. scores have improved since 1999, Finland’s have declined (both changes are statistically significant). TIMSS is normally given to 8th graders, but to help monitor progress, Finland gave TIMSS to a random sample of 7th graders in both 1999 and 2011. Those scores fell from 520 to 482, a huge decline (see Exhibit 1.6 in the TIMSS International Report).
Here’s another way to compare Finland to the U.S. Nine of our states took part in the 2011 TIMSS 8th grade math assessment. Four states scored statistically significantly higher than Finland (MA, MN, NC, IN), three about the same (CO, CT, and FL), and two lower (CA and AL). Based on this distribution of scores, had Finland participated in the 2011 NAEP as a U.S. state, its 8th grade math score probably would have been in the middle of the pack.
The U.S. press has not told the entire story about Finland. Not everyone in Finland has applauded the emphasis on real world math. In 2005, a petition signed by more than 200 university mathematicians complained that, despite the country’s high PISA scores,students were increasingly showing up for college unprepared in mathematics. An analysis of items on a Finnish matriculation exam revealed a sharp fall off in computation skills,particularly with problems involving fractions and exponents. The concern in Finland about the math curriculum mirrors that expressed in the 1990s about math reform in the U.S.
Next December, when the latest PISA scores are released, New Zealand will undoubtedly be a high flyer again. So will Finland. Please remember that PISA is not a curriculum-based test, and whether we want schools to teach a mathematics curriculum compatible with PISA is a proposition that many people find objectionable.