The qualifications regulator Ofqual turned down an offer of independent expert help with the algorithm used to award this year’s A-Level results because statisticians refused to sign non-disclosure agreements, it has emerged.
The Royal Statistical Society, which has since criticised a lack of independent expertise on the technical panel set up to advise Ofqual on the algorithm, wrote to the exam regulator in April with a warning about some of the challenges involved in estimating student grades.
The royal society called for an advisory panel to be created and suggested independent statisticians sit on it – but was told they would need to sign an NDA to be considered.
The revelation comes amid a furore over A-Level results being calculated using a statistical model after exams were cancelled because of Covid-19. Some students received much lower grades than expected, and the row has led to calls for education secretary Gavin Williamson to be sacked.
The letter from Sharon Witherspoon, RSS vice president, education and statistical literacy, said Ofqual faced two major issues in deciding how to calculate results: data collection and quality; and the subsequent statistical modelling of anticipated grades.
"These are objectively difficult decisions by any standards," she said. “The Royal Statistical Society is ready to help if that would be useful. We believe that having an engaged expert advisory group would be of use as you make decisions, and would be preferable to a purely internal process,” Witherspoon wrote.
She put forward two RSS fellows: Guy Nason, a stats professor at Imperial College who already sits on a Department for Education advisory group on higher-education figures; and Paula Williamson, a professor of medical statistics at the University of Liverpool.
But in a letter to the Office for Statistics Regulation last week, Witherspoon and RSS president Deborah Ashby said Ofqual had insisted the two sign an NDA “that gave us real concern”.
The letter said advisers should not give a running commentary nor divulge confidential information, and that Witherspoon had said as much to Ofqual. “But the proposed confidentiality agreement would, on our reading, have precluded these fellows (who were suggested precisely because of their relevant statistical expertise, and lack of ties to qualification regulators or exam-awarding bodies) from commenting in any way on the final choice of the model for some years after this year’s results were released."
Witherspoon said she had written to Ofqual outlining her concerns with the NDA “and restated our willingness to help if a more suitable agreement could be reached”.
But the letter, dated 14 August, added: “We did not get an official response to those questions, and our offer to help was not taken up.”
A spokesperson for the qualifications regulator said it was “routine" for Ofqual to use NDAs when handling "sensitive information which it considers requires protection for reasons in the public interest”.
They said: "We published the precise details of the standardisation model, as we said we would do, in addition to a number of technical reports.
"Throughout the process we have had an expert advisory group in place, first meeting with them in early April. The group includes independent members drawn from the statistical and assessment communities. The advisory group has provided advice, guidance, insight and expertise as we developed the detail of our standardisation approach.”
Formal review 'essential'
Witherspoon and Ashby wrote to the OSR last week asking the regulator to conduct a “measured review” of the statistical models used by qualifications bodies.
The review should also examine the process behind putting those models in place, and should be used to adjust this year’s predicted exam results, they said.
They sent the letter after the OSR published a statement last week saying it would not be reviewing the implications for individual results of the model used by Ofqual and other bodies. “It is not within OSR’s role to regulate the operational use of models by government and other public bodies,” the regulator said.
“We want to be clear that the RSS is not seeking that. Given where we are in all the four nations of the United Kingdom, and the tight timetable for appeals and so on, we do not believe that would serve either individuals’ interests or the wider public good,” Ashby and Witherspoon wrote, adding that issues of individual grades, appeals and admissions to higher and further education institutions would be “dealt with in other ways”.
But they said it was “essential” that the OSR conduct a formal review to tackle the two issues they had raised in the April letter and on several occasions since then.
The RSS leaders said it was “not clear” that Ofqual’s modelling had taken into account issues they had raised previously, including the likelihood of “systematic upward bias” in teacher-assessed grades and variability in performance by exam centres.
They also questioned the trustworthiness of the statistics because of the limited transparency with which A-Level stats have been set out and considered. “We have concerns about the approach of the qualification regulators, which we have expressed increasingly clearly throughout the months since the decision was taken to cancel exams,” Ashby and Witherspoon said.
Their letter said the technical advisory group, while welcome, did not have enough independent external members who were neither government employees nor current or former employees of qualifications regulators.
“We believe that a review is essential to address the issue of the extent to which the qualifications regulators did indeed adhere to their obligation to serve the public good,” it added.
“As you have acknowledged, they have a duty to act in accordance with the principles set out in the Code of Practice for Statistics, to achieve ‘trustworthiness, quality and value’. We ask in particular whether the models and processes adopted by the qualification regulators did in fact achieve quality and trustworthiness. This is linked to the two issues the RSS has raised since April.”
CSW has approached the OSR for a comment.