"It's inexcusable and Americans are right to be angry about it and I am angry about it," the president said. "I will not tolerate this kind of behavior in any agency, but particularly the IRS given the power that it has and the reach that it has in all of our lives."
Miller had previously been set to leave his position in early June for reasons unrelated to the controversy, a source close to Miller told Fox News.
While much media attention has focused on audits of tea party groups, some concerns have also been raised by pro-life groups and those working to defend marriage.
The Chicago-based Thomas More Society has said that the Texas group Christian Voices for Life received IRS letters demanding to know whether the group educates "on both sides of the issues" and whether its members try to block those attempting to enter an abortion clinic or try to talk to them. The society's executive director, Peter Breen, said the requests suggest the IRS could be denying or delaying the group's tax exempt status because of its pro-life views.
The prominent Protestant minister Rev. Franklin Graham has said the IRS selected for auditing two non-profits he heads, the Billy Graham Evangelistic Association and Samaritan's Purse. Graham voiced concern that it was not a "coincidence" that the audit came after the organizations took out ads in support of the North Carolina marriage amendment and others which encouraged voters to choose candidates using "biblical principles."
The National Organization for Marriage has also charged that an IRS employee illegally leaked confidential donor information to its opponents.
Hendershott said she thinks the new focus on possible wrongdoing at the IRS should lessen the worries of those concerned that they might be targeted for audits for their advocacy.
"I think it is safe now," she said. "If you asked me that last month, I would have advised them to use caution."