Policymakers must provide more "robust" evidence for claims about the productivity and efficiency benefits of artificial intelligence innovations, the Ada Lovelace Institute has warned.
The independent research institute has published a briefing designed to help policymakers and others to critically evaluate AI productivity claims in the UK public sector.
The briefing warns that claims that AI will deliver significant productivity benefits and savings in the public sector are shaping government policy and spending decisions, but a "more robust and rigorous approach to assessing these claims is needed”.
"A single productivity estimate or methodology can contribute to decisions to spend billions, reshape workforce planning for thousands of civil servants and lock in technology choices for years to come," the paper says.
"As AI adoption moves quicker than evidence, single studies can become powerfully relied on."
The paper, Measuring up: Evaluating claims about AI and productivity in the UK public sector, also warns that "the ripple effects of single studies have been amplified by the development of tools that are designed to automate and scale the current approaches to productivity estimates within the civil service".
The paper picks out several "recurring weaknesses" in AI evidence-building.
It finds that most studies focus on time savings or cost reductions but do not examine the impact on other public benefit outcomes, such as better services, greater equity, improved citizen experiences, enhanced institutional capacity, or improved worker wellbeing.
The research also finds that “many value-for-money or savings estimates do not appropriately account for the ‘lifetime cost’ or opportunity cost of public spending on AI technologies”.
The briefing paper also highlights problems with industry involvement and influence in research, selective use of positive findings, adoption of flawed methodologies, a lack of research following AI use over time, and limited participation from workers and the public in research design.
Funded by the Nuffield Foundation, the Ada Lovelace Institute works to ensure that the benefits of data and AI are justly and equitably distributed, and enhance individual and social wellbeing.
Sumedha Deshmukh, lead author of the briefing paper and affiliate fellow at the institute, said: “When we dig into the evidence on AI and productivity in the public sector, we find a much more mixed picture than the headline figures suggest. Numbers for numbers’ sake are not enough – we need a sharper focus on whether productivity claims actually translate into public benefit.
“There are also legitimate grounds for scrutiny, including industry involvement in research design, selective presentation of positive findings, and a lack of long-term studies tracking adoption over time. Policymakers need to understand what sits behind the topline figures before relying on them in critical decisions. This briefing gives them the tools to do that.”
The paper makes several recommendations for studies into public sector AI. It says research should:
- Reflect uncertainty rather than "project false precision" – reporting ranges and variation, not just headline numbers, and "making a holistic assessment of where AI helps, where it hinders and where effects are negligible, rather than assuming positive impacts"
-
Measure what matters, not just what is easy to quantify – tracking service quality, error rates, user satisfaction, equity of access and institutional capacity alongside time and cost metrics and actively considering how costs and benefits are distributed societally
-
Embrace methodological pluralism rather than rely on a single method
-
Acknowledge context specificity: that productivity effects are not universal properties of AI tools but depend heavily on the setting in which those tools are deployed
Matt Davies, social and economic policy lead at the Ada Lovelace Institute, said: “The UK government sees AI as a transformative force for public sector productivity, but policy and investment decisions must be informed by rigorous research and clear-eyed assessments of public benefit and value-for-money.
“The stakes are high. Claims about the impact of AI on public sector productivity are influencing major government decisions. But policymakers must not lose sight of the public value that AI adoption is meant to deliver, and consider whether current approaches to measuring the potential productivity benefits are fit for purpose."