The Department of Health and Social Care is facing a legal challenge over his department's handling of patients with learning disabilities and autism, following action by the equalities watchdog.
The Equalities and Human Rights Commission said it had sent the pre-action letter to health secretary Matt Hancock amid growing concerns for patients with learning difficulties who have been left stuck in "inappropriate" care facilities.
The watchdog said the legal proceedings would challenge the government's handling of around 2,000 patients who they claimed were currently being detained in secure hospitals, often far from their homes.
In a statement, the EHRC said ministers may have breached the European Convention of Human Rights for failing to meet their own targets in improving care for patients, accusing them of presiding over a "systemic failure to protect the right to a private and family life, and right to live free from inhuman or degrading treatment or punishment".
It comes after an BBC investigation found vulnerable patients at the NHS-funded Whorlton Hall were being badly mistreated by staff.
Undercover filming by the broadcaster's Panorama programme in 2019 saw care staff taunting and intimidating patients with learning difficulties despite the Care Quality Commission handing the privately-run facility a "good" rating in an earlier inspection.
The EHRC said the department would have 14 days to respond to the letter, or they would be willing to delay the action for up to three months if Hancock could produce a detailed timetable setting out how he would tackle the problem.
EHRC chief executive Rebecca Hilsenrath said: "We cannot afford to miss more deadlines. We cannot afford any more Winterbourne Views or Whorlton Halls. We cannot afford to risk further abuse being inflicted on even a single more person at the distressing and horrific levels we have seen. We need the DHSC to act now.
"There are people who deserve our support and compassion, not abuse and brutality. Inhumane and degrading treatment in place of adequate healthcare cannot be the hallmark of our society. One scandal should have been one too many."
A Department of Health and Social Care spokesperson said: “We are committed to protecting the rights of everyone with a learning disability or autism, and are determined to continue reducing the number of people with these conditions in mental health hospitals.
“Abuse of any kind against patients in care is abhorrent and we take any allegations very seriously.
“We have received the pre-action letter from the ECHR today and will respond in due course."