Watchdog MPs have called on the next government to drive reforms to enable better cross-departmental data-sharing, with costs for the shift borne centrally rather than by individual bodies.
Members of parliament's Public Administration and Constitutional Affairs Committee said departments and public bodies choose not to share data because they are not incentivised to do so.
They said that seven years since the passage of the Digital Economy Act, the UK had failed to bring its "disparate datasets" together to enrich its public evidence base. "Instead, data withers in silos across countless government bodies," they said.
MPs said that despite having a statistical system that is highly regarded internationally, broader opportunities to benefit from joined-up data were being missed because of long-standing issues with cross-departmental data sharing.
PACAC members also cautioned against proposals floated by the Office for National Statistics to stop the 10-yearly census, in favour of drawing on other data sources. They questioned whether it had yet been demonstrated that administrative data could deliver the information census users needed and called on the ONS to undertake further reviews.
"Concerns remain about whether data will be available at the granularity required on topics such as ethnicity, and whether data on topics such sexual orientation will be available at all," they said.
The committee’s report found "significant data gaps" on the UK's different health services and on issues such as school attendance.
They added that good evidence is not yet being used effectively in decision-making across government, and found that official communications too frequently disregard evidence, damaging trust in democracy and public support for policies.
MPs said the government needs to establish a framework for identifying and prioritising demands for data that is "rooted in the understanding that evidence is for all, rather than for the government of the day".
The report said the Cabinet Office’s existing initiatives for improving data sharing are "self-evidently insufficient" and called on the department to work with the ONS to develop a comprehensive new programme aimed at improving data sharing for statistical and research purposes.
"The programme must clearly define deliverables and timelines, and must be owned by a senior responsible officer at an appropriately high level," they said. "We also recommend that HM Treasury establish mechanisms so that the costs are not borne by individual departments, but rather centrally."
Under the proposals, the Cabinet Office would prepare and publish an annual progress report on delivery against the programme.
PACAC chair Dame Jackie Doyle-Price said the quantity and variety of data available to the nation was a source of "great opportunities" that needed to be tapped more effectively.
"Data remains locked in departmental silos and there are also concerns over significant data gaps," she said.
"Our report recommends that we should proceed with caution when it comes to replacing the census, until we are certain that officials have resolved issues around data sharing."
Elsewhere, the report found a "mismatch" between the ambition of the government analysis function to "deliver better outcomes for the public by providing the best analysis to inform decision-making" and the "very limited funding" provided for its vision.
MPs noted: "Ironically, for a group of people dedicated to the cause of informed decision-making, analysts appear to have done little by way of evaluating the function’s success in delivering its vision. What we did hear anecdotally, however, suggests that there might be significant room for improvement."
They called on the next government to reaffirm its commitment to the analysis function and for HM Treasury to review options for its future funding.
The UK Statistics Authority said it welcomed PACAC's report and would respond fully after the general election.
"Working in partnership to support administrative data to reach their full potential is fundamental to the vision outlined by the national statistician," a spokesperson said.
"We will publish our recommendation on the future of population and migration statistics in due course."