"In the Treasury in 1979, they were very, very good at essays. We had wonderful essays that went to the chancellor about monetary policy,” said Lord O’Donnell, recalling his first days in the civil service. Having begun his career as an economist and academic, he’d found some HMT practices very odd: “These essays didn’t have any numbers in them, which was pretty staggering to my mind.”
The former cabinet secretary was speaking at the Royal Statistical Society last month, on the topic of evidence-based policymaking: those first experiences, he said, sparked a real passion for this agenda. “For me, trying to get the voice of evidence-based policy into policymaking has been a lifelong struggle from that day.”
Though O’Donnell praised the UK Statistics Authority, Office of National Statistics and Major Projects Authority for greatly improving the quality of available evidence, he warned that policymakers must select their data carefully. People don’t always tell the truth to pollsters, he said, pointing to the Scottish referendum; O’Donnell places much more trust in betting data showing where people risked their cash. Sales data from Betfair showing the “implied percentage chance of a ‘no’ vote” suggests that it “hardly ever goes below 70% – so the message is that it was never close.”
Then the former head of the civil service displayed his mischievous side – a side frequently evident during his informal, upbeat presentation. “We almost in that last week clutched defeat out of the jaws of victory,” he commented. “And we’re going to have to live with the consequences of some of the strange things said towards the end.”
The ‘Big Data’ phenomenon should also be treated with caution, he said – it’s “often over-hyped by people impressed by large sample sizes, without any consideration of information content” – and policymakers must consider whether the metrics they’re using provide a good measure of value. In particular, the former mandarin and economist raised concerns over the use of GDP statistics, pointing out that they “assume a pound received by a pauper is worth the same as a pound received by a millionaire.”
What’s more, policymakers must be wary of using historical evidence to predict the future. “Structural breaks happen,” O’Donnell pointed out. “The future isn’t necessarily like the past.” Take the credit crunch: “The models used by the banks to work out how much capital they needed were estimated on the basis of the past – what else do you estimate it on? – but the past was very benign. And if you estimate on a benign past, you’ll forecast a benign future. Sometimes we need to think very hard about how the world might be different in the future.”
O’Donnell concluded by setting out three ideas for improving the operation of government. Five-year spending plans should be introduced to match the newly-fixed duration of a Parliament, he said. The government should produce five-year infrastructure and major project plans, appointing an independent scrutineer to assess their feasibility and monitor progress. And cross-departmental collaboration should be enhanced by incentivising departments to invest in work that benefits other parts of government. “Many, many successful policies involve investment by one department, but the gains are spread to others,” he noted.
Answering questions after his speech, the peer agreed that ministers could benefit from training in the use of evidence. “Ideally, they would have some real training on policy work,” he said. “That would be about understanding what questions to ask and how to interpret the evidence. What worries me most is that someone can come up with an anecdote and that can have huge amounts of power, as opposed to boring old statistics which tell you that the anecdote is an outlier.” Whilst O’Donnell didn’t name any such anecdotes, readers can no doubt think of some; one good example is the home secretary’s story about the Human Rights Act and – as she put it – “the illegal immigrant who cannot be deported because, and I am not making this up, he had a pet cat.”
Statistical data would, no doubt, show that pet cats do not play a big part in deciding immigration cases. But no matter how much evidence policymakers gather, the reality is that senior civil servants and ministers rarely have as much robust data as they would wish. It’s often the case, O’Donnell pointed out, that all of the potential courses of action “have nasty outcomes”, and that “you haven’t got as much information as you’d like – but at the end of the day you’re going to have to decide.”
“When you’re cabinet secretary, all the easy decisions get made by someone else; you get all the tough ones,” he added. “And you never have perfect evidence: it’s always decision-making under uncertainty. That’s the world we live in.”