As it probably had to be, the government's consultation on open data has proved for the most part to be, well, open. In all, 247 government departments, councils, companies, campaigners and other interested groups collectively submitted nearly 500 responses last autumn, providing plenty of material to inform the government's development of its open data strategy. Tim Kelsey, whose job as the government's transparency director was recently made permanent, told CSW last year that his team would "use the responses to develop proposals for a white paper" (CSW 5 October 2011, p3) - and Cabinet Office officials are now trying to fashion a broad consensus around a strategy that will lead to the publication of much larger volumes of public data.
Overall, respondents were largely supportive of an enhanced public right to data, but it's clear that the consultation - 'Making Open Data Real' - has prompted concerns around issues such as privacy and where the additional costs of publication will fall, and highlighted tensions over how much data should be made public.
Behind the project is the coalition's over-arching belief that greater publication of public data will both enable private companies to boost economic growth, and make services more accountable to their users. To this end, more than 100 businesses were invited to contribute to the consultation. The government has also proposed the creation of a Public Data Corporation (itself the subject of a separate consultation, for which a summary of responses is yet to be published) and is deliberating on whether the corporation should sell public data or give it away. This question is complicated by the fact that some trading funds - such as the Met Office and Ordnance Survey - earn vital revenue from selling their data. These two bodies are overseen by the Department of Business, Innovation and Skills, which is understood to be less keen on open access, and chose not to reply to the open data consultation.
One thorny matter is the question of whether contractors to government - set to expand as the coalition contracts out more public work - should be subject to the same transparency requirements as public sector bodies. "Large amounts of traditional public functions - such as NHS services - are being contracted out," says Maurice Frankel, director of the Campaign for Freedom of Information. "It's not a great leap of principle to say that the open data requirement should follow that function." However, many private companies argue that their data should be protected by commercial confidentiality - leaving voluntary organisations concerned that their operations might be made transparent to private competitors, giving businesses an unfair advantage (see news, p3).
Generally, respondents felt that the regulation of data transparency would be best handled at the centre, with many arguing that the Information Commissioner's Office (ICO) should oversee compliance. As the Association of School and College Leaders (ASCL) said in its submission: "25,000 schools and colleges, each inventing its own system to do this, would not be cost effective or provide consistency." Frankel argues that the ICO will need a clear role, underpinned by legal powers. "If open data is not a legal requirement and you are working on the assumption that people will simply do it - well, the reality is that they won't," he says.
The potential for open data to compromise personal privacy was a recurring theme, particularly in the form of 'jigsaw' re-identification - the piecing together of identities from disparate data published by various organisations. Health and education bodies were particularly cautious here: "Does making the dataset open, either in isolation or linked to other data, risk putting personal data in the public domain?", asked AstraZeneca, the pharmaceutical giant.
The publication process is likely to be fraught with the risk of inadvertently identifying children, according to Malkiat Thiarai, head of corporate information management at Birmingham City Council. "We could publish transaction data from the vulnerable children's database, but there's a risk that separate data could combine with that and disclose an identity," he warns. "It would be tortuous to cross-reference such data. You may only realise it has happened with hindsight."
Data on individual members of staff can also be plucked out of datasets, and the ASCL stressed that a balance must be struck: "Helping people to find the best doctor or teacher for their needs," it said, "should not mean giving every onlooker unfettered access to all records of individual doctors and teachers."
The Ministry of Defence (MoD), of course, is well aware of the need for privacy and security - but a spokeswoman told CSW that she believes that such quandaries will be overcome. "This will always be an issue. There will be occasions when it is not in the public interest to publish, particularly if this could jeopardise safety or compromise national security," she says. "The challenge is to foster a culture where people are naturally disposed towards openness."
That hardy perennial, doubts about the ability of government IT systems to cope with large projects, loomed large, as did an almost universal anxiety that public service personnel may be overwhelmed by demand for data. "The type of requests you get will be as varied as for Freedom of Information," warns Thiarai.
One potential solution, he suggests, is for Kelsey to set out guidelines as to how requests for data should be prioritised. As Thiarai says of his council: "We have up to 450 different data systems - human resources, finance and so on - and there could be a lot of work that is unnecessary" if he's required to publish them all. "If there was a way for government to identify which data sets people are more interested in - say transport, housing - then we could focus on these areas rather than, for example, what time staff clock on and off."
One way of coping with the additional demand, says National Council for Voluntary Organisations research officer David Kane, would be to invest in training that would "bring the staff and the geeks, the technical people, together. This is as much about skills as it is about culture." And most departments were relatively relaxed about the additional workload: the MoD, which answers more than 3,000 FoI requests a year, believes the resource challenge can be met. "It is an additional task for a reducing number of staff to deal with, but it is not optional," the spokeswoman says.
The environment department, ironically enough, successfully asked for three parts of its submission to be redacted before publication. "It would not be in the national interest to release some data relating to rare species and nesting sites, protection of critical national infrastructure and Defra's role in responding to civil emergencies," a spokesman explains.
Taken as a whole, the submissions clearly identify some areas where the Cabinet Office will have to make crucial - and, no doubt, controversial - decisions. And in some cases there is little consensus about the solutions: for example, many views emerged over the sanctions required to harry recalcitrant organisations. Some favour celebrating best practice, while others want to name and shame those who fail to publish or who put out sketchy data. Backing the latter approach, for example, the Nuffield Trust pointed out how the shaming of health providers who publish low-quality hospital A&E datasets has raised reporting standards.
"There were mixed opinions on how the agenda should be implemented," acknowledges a spokesman for the Cabinet Office - which, in its summary of the consultation, raised the spectre of further consultations to iron out a final consensus. Open data has some way to travel before the floodgates can be opened for the forthcoming wave of public information.