A trial of Microsoft 365 Copilot by the Department for Business and Trade has found that those undertaking written work made the largest time savings, but that other tasks took longer than without the tool.
An evaluation by the Department for Business and Trade of Microsoft Copilot, involving 1,000 staff, found that it saved most users time but did not find evidence that the large language model artificial intelligence tool improved productivity.
"Small time savings were observed across most use cases, with written tasks presenting the largest time savings," the report said.
But it found that "those scheduling and generating images spent longer than they would have otherwise".
This was usually because Copilot was not able to produce high-quality outputs or because the task was only taken on because users had access to the tool, the report said.
The report also said that the evaluation "did not find evidence that time savings have led to improved productivity, and control group participants had not observed productivity improvements from colleagues taking part in the M365 Copilot pilot".
However, it said "many pilot participants reported noticing time savings in their own roles due to M365 Copilot".
The trial also revealed quality issues, with participants observing AI-generated hallucinations in Copilot’s output, as well as some limiting their usage due to concerns over the environmental impact of generative AI. However, it was found to be beneficial for neurodiverse staff and non-native English speakers.
“Further evaluation is needed to assess the value for money and environmental impact of M365 Copilot usage in the department,” the evaluation said.
More than two-thirds of those participating in the three-month trial reported being satisfied or highly satisfied with Copilot. Participants who focused on written tasks, including reviewing, summarising information and drafting, tended to be more satisfied, while those trying to use it to produce PowerPoint slide decks or schedule meetings were less happy.
Training had a significant impact, with those who undertook training at their own pace being more satisfied than those who participated in formal training sessions.
The department provided 1,000 staff members with Microsoft 365 Copilot licences from October to December 2024, with 70% of the volunteers and 30% selected randomly to ensure a representative sample. It based its assessment on diary studies filled in by 32% of participants, which asked users to record tasks undertaken with Copilot and answer questions about it.
The most popular use for Copilot was transcribing or summarising a meeting, with 234 cases, followed by writing an email (167), summarising written communications such as emails (153) and asking questions (124).
Diary study respondents were asked to describe any changes they made to Copilot output. Many said they edited grammar, spelling, including American English and tone. Several also identified inaccurate information which they had to correct.
The study also drew on Microsoft usage data, which showed that participants took an average of 1.14 actions using Copilot each working day. Additionally, 19 qualitative interviews were conducted with 13 participants and six people in a control group.
This is a rewritten version of a story that originally appeared on PublicTechnology