“It was like going into a sweet shop,” Jonathan Simcock says, recalling his first impressions of the project portfolio being run from the civil service in 2007. “What struck me immediately was the huge scale and diversity of the biggest projects in government. There were so many extraordinary things happening.”
He was to lead what was then the Major Projects Directorate – now incorporated into the National Infrastructure and Service Transformation Authority – in the Treasury. The projects he remembers from those early days include the first iteration of identity cards being introduced and setting up the Rural Payments Agency. There was also a big project led by the Home Office for vetting and barring people who were going to work with children and vulnerable adults.
There was one thing that these enormous projects had in common, according to Simcock: none of them were on track to succeed. “It wasn’t a secret, really, that not everything was going well,” he says. The Rural Payments Agency “was kind of a poster child for disastrous projects”. And the Home Office project, which grew into what we now know as the Disclosure and Barring Service? “That wasn’t going well either.”
Simcock is talking to CSW a few weeks after the publication of his book, The Delivery Gap: Why Government Projects Really Fail and What Can be Done About It. After 18 years involved in government work, he is ready to be candid about the significant government projects he’s seen at close quarters – from Crossrail to smart meters to HS2 – and why so many of them are seen to have failed.
A qualified mechanical engineer, he arrived at the civil service in his 40s, “mid-career”, following 25 years at Shell, where he oversaw a wide range of projects. A lot of people who make the move from the world of business “either sink without trace or leave”, he says. “They come in thinking, ‘all we have to do is to run government as though it’s a private sector organisation’”.
But this view is flawed, he says: government is far too complex to run as a corporation, “and that’s one of the reasons I enjoyed it so much. I like working on complex problems, and there’s nowhere in the economy really where things are more interesting and complex.”
Complexity is a double-edged sword, according to Simcock. He argues that the largest government projects can’t be measured against those carried out in the private sector. In fact, government has to address the challenges that are too complex for the market, and it can’t always turn down projects just because they look too difficult – the problems are often too important to walk away from.
“A major international oil company might have a portfolio of 30 or 40 mega-projects, of which they end up doing 10. The government wants to do them all,” he says. “Societal problems, from defence of the nation to bringing people out of poverty, are going to be highly complex. And really complex endeavours never end up where you think they’re going to end up. They’re intrinsically kind of chaotic.”
And here is his key point, the one he returns to several times in his book: it is “essentially impossible” to deliver highly complex projects to a predictable timeframe and cost, especially when they are determined long before the scope of the work has been finalised.
Despite his own love of complex challenges, Simcock warns that the civil service is too much in thrall to complexity, and resistant to the idea of breaking complex problems down into simplified, manageable projects until they are deliverable. “The Cabinet Office manual on setting up projects for success has well over a hundred references to complexity but no mention of simplification,” he says in his book. And recruitment into the Fast Stream promises “projects of unparalleled complexity”.
The response to a complex social issue should not be a highly complex project, he explains. “In fact, if you make the project highly complex, then you shouldn’t expect it to do what you think it’s going to do.”
When CSW asks Simcock for his headline advice to senior civil servants preparing to green-light a major project, he says: “If what you are looking at is huge, unique and highly complex – don’t.”
The fate of the problem-plagued HS2 project, which he had seen evolve over the years as a member of the Major Projects Review Group and a review team leader, is what prompted Simcock to investigate why government projects fail. Then-prime minister Rishi Sunak’s announcement in 2023 that phase two of HS2 – the northern leg of the project – was being cancelled felt like “proof that I was living in ‘can’t do’ Britain”, he says in the opening chapter of the book. “The fate of this railway amplified an unacknowledged notion that was already resonating in my head. A growing feeling that failure of huge government projects is somehow predestined. Even worse than this depressing idea was my feeling of complicity.”
When did this feeling of complicity begin? Simcock is reluctant to name a specific project. But he describes being “in the situation where the plan is on the table halfway through delivering something, and it no longer looks like you can do what you said you would be able to do”.
He says he felt “a genuine tension” in that moment, between “do we re-baseline into something that we could genuinely deliver, or do we keep the pressure on?” While re-baselining would be the honest response to the problem, he says this runs the risk that “everyone will relax… and we will be back in exactly the same position in a year and a half”. All the incentives in this situation drive a lack of candour about the problem, he says.
“If your choice is going upstairs to explain a delay to the minister now, or putting it off for 18 months, by which time he’ll be gone and you’ll be gone… the temptation to leave it to the next person is really strong.” Why would you take the hit to your reputation, he adds, if you don’t have to; if there’s a chance that a forthcoming election might mean a change of direction? “So it’s these cumulative incentives that drive us collectively to not bite the bullet.”
A significant factor, Simcock suggests, is the way in which projects are announced by politicians, with an apparent certainty about what is to be achieved, by when, and at what cost. Announcements are “always over-optimistic”, he says. Some government projects would not even get off the ground without what he calls a “systemic but unspoken conspiracy to get projects sanctioned by collective self-deception”.
“Crossrail could have happened two years earlier, and been £2bn cheaper, if they’d faced up to reality”
This discrepancy between what gets announced and what is achievable is what Simcock means by the “delivery gap”. He cites the Crossrail project as a good example. “Large numbers of people bought homes in the anticipation that they would be able to commute to work on this new line in 2018,” he says. “But they were not able to do so until 2022. That’s the delivery gap.”
But it’s wrong to simply blame politicians for the problem, he says, despite the fact that over-optimistic announcements often come from them in the first instance. He points out that in 2018, Crossrail Limited was still saying it was going to open the railway by December of that year. The delivery body would have known at this point that the project would be undeliverable in that timeframe. “That wasn’t because a minister was forcing them to do that, or politics was forcing them to do that. So it’s too easy to just blame the politicians.”
“Success and failure are pretty easy to define in the private sector,” Simcock says. “Either you’re driving the share price up or you’re not.” And even the most challenging private sector projects are unlikely to be a completely new endeavour: “A deep sea oil platform might be bigger and more difficult than the last one you did, but you did one before.”
Public sector projects, he says, tend to be unique. This means it’s “almost impossible” to accurately predict how they will play out. Ultimately, he says, being more courageous earlier would allow government to redefine success and failure. Sometimes a project appears to be failing when in fact, had its problems been acknowledged in a realistic way, it might have been seen as a great success.
The Elizabeth Line, as Crossrail is now known, is one example among many, Simcock says: “It is a magnificent piece of infrastructure. And in 30 years’ time, when it’s still a magnificent piece of infrastructure, people aren’t going to look back and say, ‘Well, you know, it cost a few billion more than they said it would.’”
A much less obvious example is the Palace of Westminster Restoration and Renewal Programme. “Viewed through one lens, you could say that of all the projects I describe [in the book], this is the most successful,” Simcock says.
This is hard to fathom. The restoration is an example of a project that urgently needs to happen – it’s commonly understood to be just a matter of time before someone is seriously hurt by falling masonry, or the whole place goes up in flames.
But fixing the problem would be astronomically expensive – estimated to cost £7-13bn and to take 19-28 years. Announcing such a project seems politically impossible. And so the work to make the Houses of Parliament safe appears to be indefinitely stalled.
But, Simcock argues, the decision not to go ahead is “a bizarre kind of success”. The nation’s elected representatives listened to what the Delivery Authority told them it would cost… and decided not to do it. “Maybe that’s better than pretending it will be cheaper and quicker than it really will, and starting a project that’s doomed to fail. If there is no economically or politically acceptable project available… then maybe the earlier you reach this conclusion, the better.”
He also accepts that the decision not to proceed with the project is far from ideal. He describes the problem in poetic terms in his book: “The nature of our democracy might just cost us the home of our democracy.”
In conversation, he puts it more bluntly: “It won’t feel like a success if the Palace burns down tomorrow.”
Does the delivery gap really matter, if we will ultimately come to see projects such as Crossrail as great achievements?
“It absolutely matters,” Simcock says, “because of the real impact on real people.” Crossrail had a huge impact on those thousands of people who moved house to be within commuting distance of a station that didn’t materialise for several more years. It also left Transport for London with a huge hole in its finances.
“TfL had to pay the extra £4bn to finish the project, and didn’t receive the fares they had been expecting until years later... So the impact of the delivery gap on Transport for London is absolutely real.
“A lot of excellent people had their careers damaged by the Crossrail experience,” Simcock continues.
“But the problems with Crossrail should have been evident – and in some cases they were evident – to people much earlier. And had they been acknowledged several years earlier, then, yes, there would have been a bit of a political storm. You know, the mayor would have had to stand up and say, ‘I know we said we were going to do it by 2018, it’s going to take a bit longer.’ But that storm would have passed.”
“Really complex endeavours never end up where you think they’re going to end up. They’re intrinsically kind of chaotic”
He provides an estimate of the size of the Crossrail delivery gap: “A general view is that [Crossrail] would have happened probably two years earlier than it actually did – because the project wouldn’t have been in crisis, re-baselining would have been a planned process – and it would probably have been £2bn cheaper, if they’d faced up to reality earlier.”
“In very complex situations, it is human nature to underestimate uncertainty and risk,” Simcock writes. Asked to expand, he says that this leads to a “reluctance to face reality”. He prefers not to call this dishonesty, feeling that the term is too judgemental of people who are part of a wider problem requiring systemic change. He refers to civil servants as “excellent people who are doing their best” but who, in some cases, “make decisions that afterwards they probably really regret”.
An enforced duty of candour – such as the statutory obligation in the forthcoming Hillsborough Law – is one of the proposals he puts forward in the book, to correct this problem. “Nobody should think of my solutions as the last word,” he says of these proposals. “What I’m trying to do here is to raise our level of disquiet with the status quo and to advance the debate.”
He accepts that the civil service code already requires civil servants to be honest, and says the duty of candour would need to be backed up with other measures, such as stronger accountability for decisions. “If I am the chief financing officer of an arm’s-length body and I say that, in my professional opinion, an appropriate cost and timeframe for this project is X, I should be expected to be held to account for that, and I should expect the system to have my back. And if I do succumb to the pressure to say that X is three when it’s really six, then I should expect to get found out, and there will be consequences for me,” he says.
Reflecting on the experience of getting a book published, Simcock says, “In a funny sort of way, I don’t really care how many people read this book. If I could choose the 100 people I really want to read it, then I wouldn’t care if nobody else read it. And of those 100 people, 60 would be readers of Civil Service World.”
Does he have a final word for those readers? “Don’t let yourself slip into cynicism,” he says. He characterises this as a perception that “nothing ever works”. He believes civil servants should remain excited by the vitally important, high-profile projects that government undertakes. “Scepticism and realism are really valuable, but cynicism is a killer. You know, it’s an absolute guaranteed way of making sure nothing works.”