Peter Schryvers is the Calgary-based author of Bad Data: Why We Measure the Wrong Things and Often Miss the Metrics That Matter.
On Thursday, Alberta’s United Conservative Party (UCP) released its second budget, and there has been much attention paid to its details. And fair enough: Alberta is struggling with a stagnant economy, lower resource revenues and high-cost services, all of which has led to a large deficit.
The UCP won a majority government after promising to rein in spending in order to bring the province’s finances back in line, and a great deal of focus has revolved around the precise amounts laid out for public-sector employees; while it will hold the line on health and education spending, plans for more public-sector job cuts will present a major challenge for Premier Jason Kenney.
But while journalists, academics and the opposition invariably comb through the several-hundred page budget document for the new numbers that will dictate the direction of health care, education and municipalities in the province, that’s not the best approach. Instead, we should focus on the metrics.
I recognize the word metrics may cause eyes to glaze over, but from congestion in our cities, to how well our children are learning in school, metrics define much of the world around us. We live in a society that’s drowning in data – and we drown, indeed, because we don’t understand what the numbers mean or how they change us.
The problem with any performance metrics is that people find unanticipated ways to meet them, often with unintended consequences. Measure call centres on the time they spend on the phone with customers, and they’ll just hang up. Measure surgeons on the success rate of their surgeries and they’ll avoid taking on complex patients. Measure train conductors if they’re on time and they will skip stations, leaving passengers behind. What you measure isn’t always what you get.
And even before the budget was released, the UCP had indicated it will use two performance metrics to set standards around public-sector education spending – and those metrics look like they’re not going to deliver the outcomes the government wants.
Take the UCP’s proposal to use labour market outcomes (such as graduate employment) as a criterion for postsecondary funding. Universities may respond to this metric in perverse ways. They may focus only on programs in stable fields, such as education, health care or law.
In a province that is struggling to diversify its economy, evaluating schools on the short-term employability of their graduates may be at odds with imparting skills that may not result in jobs right out of school, but will be needed to create and sustain new industries in the long term. Additionally, doesn’t this metric simply punish schools when the province is in a recession? Wouldn’t a recession be a good time to focus on new skills and training? The metric can actually work against the goal.
Earlier this year, the much-anticipated MacKinnon Report on Alberta’s Finances proposed that some levels of funding at the K-12 level of education should also be tied to performance, through standardized test results. Thursday’s budget followed up on those recommendations – achieving acceptable standards on language arts and math will be performance measures for K-12 education.
Never mind that standardized tests are an incredibly poor indicator of a student’s ability to succeed in postsecondary education or in the job market, focusing on tests just results in good test takers, not good employees, entrepreneurs or citizens. The UCP adoption of this proposal could cause schools to simply become test factories as a result – teaching students how to write tests at the expense of critical thinking and creativity.
Tying funding to test results could even cause teachers and principals to take desperate actions to improve test scores. Indeed, we just need to look south of the border to see how poorly such initiatives worked out. The high-stakes testing that occurred under George W. Bush’s No Child Left Behind program and Barack Obama’s Race for the Top initiative didn’t just fail to improve education outcomes – both fuelled some of the largest cheating scandals in the history of U.S. education.
And it wasn’t the students who were cheating – it was the teachers. In Atlanta, nearly 180 teachers, principals and superintendents were charged with racketeering and other crimes for running a cheating scheme, changing students’ answers in order to qualify for extra funding, or sometimes simply in order to keep their jobs and their school open. Such pressure leads to desperate measures.
In both circumstances, there are better ways to improve education. By using the wrong metrics, we’re led to bad data and bad results. The UCP cannot forget that what we measure affects what we do – and that the metrics matter just as much as the raw numbers themselves.
Keep your Opinions sharp and informed. Get the Opinion newsletter. Sign up today.