Culture Secretary Nadine Dorries claimed growing calls for the PM to resign were the result of a "co-ordinated campaign" by backbenchers.
Criticism of Mr Johnson among Tory MPs has increased since Sue Gray's Partygate report last week.
But Ms Dorries said the "overwhelming" majority still backed him.
Twelve Conservative MPs have called on Mr Johnson to quit since the release of Ms Gray's report laid bare the scale of Covid rule-breaking in No 10.
It takes the number now openly calling on him to resign to 28. Most of have written letters of no confidence in him, although the total number formally calling for a contest may be higher.
Mr Johnson has dismissed calls for him to quit, adding it would not be "responsible right now given everything that's going on".
Under Tory party rules, 54 letters are required to set up a vote on the PM's position, a move which could eventually lead to a contest to replace him.
Only Sir Graham Brady, chairman of the backbench committee that organises leadership contests, knows exactly how many have been submitted.
Speaking on BBC Radio 4's The World at One, Ms Dorries said the debate over Partygate had become a "Westminster bubble issue" and the public was "ready to move on".
"There is obviously, I think probably led by one or two individuals, a campaign behind the scenes to try, attempt to remove the prime minister for individual reasons to do with personal ambition or other reasons," she said.
She added: "The people who most want to get rid of Boris Johnson are Keir Starmer and the SNP".
"I would just ask my colleagues to reflect on that, and do we really want to do the opposition's work - and do we really believe the public will vote for a party that they think is divided?"
Despite the uncertainty over the number of no-confidence letters, there has been mounting speculation Mr Johnson could face a no-confidence vote soon.
On Monday, former Tory leader Lord Hague said the PM was in "real trouble" and predicted a vote could take place as early as next week, when MPs return from their short early summer recess.
But speaking earlier, deputy PM Dominic Raab told Sky News he doubted the number of letters could be "that high".
There is no consensus among Tory MPs about who should be their next leader, but Foreign Secretary Liz Truss, Trade Minister Penny Mordaunt, former Health Secretary Jeremy Hunt and senior backbencher Tom Tugendhat are among those being discussed as possible challengers.
Chancellor Rishi Sunak was widely seen as a potential leadership rival to the PM, but his prospects are thought to have waned after he picked up his own Partygate fine and was embroiled in a controversy over his wife's finances earlier this year.
There's no obvious thread linking the MPs that have publicly called on the prime minister to go.
Some of the calls have come in quick succession, others months apart.
They've come from MPs elected in different years, with different ideologies. Some of them backed Boris Johnson at the last Tory leadership contest, others backed his rivals.
Some Tory MPs are convinced there is a degree of co-ordination behind closed doors.
One senior Tory told me he believes supporters of Jeremy Hunt are among those being particularly critical publicly.
Others attribute the steady drip-drip of letters in recent days to a ripple effect. As soon as a few critical MPs came out publicly, they argue, others felt emboldened to do the same.
Co-ordination or not, there is no doubt MPs do talk to each other - there are Whatsapp groups and sub-Whatsapp groups of Tory MPs.
Even before Sue Gray's report came out, I was told the tea rooms of Westminster were alive with chatter about who was "ready to move".
Some of that chatter has been more muted as MPs are out of office for recess - with some arguing the Jubilee weekend is not the time for "internal warfare".
But that could all liven up again quite quickly when they return next week.
Meanwhile, the debate over the PM's future has been given fresh impetus after his standards adviser Lord Geidt called on him to explain how his own Partygate fine tallied with the rulebook for ministers, which says they have a duty to comply with the law.
Along with his wife Carrie and Chancellor Rishi Sunak, Mr Johnson was fined for attending a birthday party for him in the Cabinet Room in June 2020.
In a report on Monday, Lord Geidt wrote there was a "legitimate question" over whether this constituted a breach of the ministerial code.
In a written reply, Mr Johnson said he had not fallen foul of the code because he had not broken Covid laws on purpose.
In an interview with Mumsnet on Tuesday, the prime minister insisted he thought the gathering in the Cabinet Room counted as a work event, and he was "very, very surprised" and "taken aback" to have received a fine.
Labour's deputy leader Angela Rayner accused the prime minister of ignoring Lord Geidt's advice, and finding himself innocent of breaching ministerial rules in "his own courtroom".
Adding that Mr Johnson was "dodging questions", she called for him to "make a statement" on his "apparent breach" of the rules.
For the SNP, MP Mhairi Black called on Scotland's six Tory MPs to "step up and do the right thing" and submit no-confidence letters in "this Partygate prime minister".