More than 160,000 people have told HM Revenue and Customs to delete biometric data it gathered during phone calls.
The tax authority captured millions of callers' voice data after launching its Voice ID system in 2017, but originally gave people no easy way to opt out.
Privacy group Big Brother Watch said people had been "railroaded into a mass ID scheme by the back door".
HMRC now lets people opt out or decline Voice ID and delete captured "voiceprints".
But it has kept millions of Voice ID records in a database held by a third party.
Big Brother Watch said the tax authority had "created one of the largest known state-held voice databases in the world".
About 7 million people are currently enrolled in the database, which lets taxpayers use their voice instead of a password when calling the helpline.
A Freedom of Information request revealed that 162,185 people had "opted out" and had their voice data deleted by December 2018.
Prior to that, people had to get in touch with HMRC separately and ask to be removed from the Voice ID system. About 80 people did so.
Big Brother Watch said it had reported HMRC to the Information Commissioner's Office (ICO), suggesting it had broken data protection law.
"It is down to the ICO to take robust action and show that the government isn't above the law," Big Brother Watch said in a statement.
An HMRC spokesman said: "Our Voice ID system is very popular with millions of customers as it gives a quick route to access accounts by phone.
"All our data is stored securely and customers can opt out of Voice ID or delete their records any time they want."
The organisation is cooperating with the ICO, which is investigating to see whether the system breached data protection law.