Business group leader Tony Danker has urged ministers to "grip the emerging crisis", arguing it "made no sense to wait" for the next PM to arrive.
And Nicola Sturgeon, Scotland's first minister, has called for an urgent meeting with the devolved governments.
Downing Street said it recognised the public were facing "challenging times".
However, Mr Johnson's spokesman argued it would be up to his successor to make any decisions on further help.
Mr Johnson is due to leave office in early September, and No 10 said "by convention it is not for this prime minister to make major fiscal interventions during this period".
Cabinet Office minister Kit Malthouse said the government had already and was continuing to do work on cost-of-living problems.
"My job is to make sure that when the new prime minister arrives, the plan is in place for them to put their foot on the accelerator," he told the BBC's PM programme.
He also dismissed calls from ex-Prime Minister Gordon Brown for the government to hold a meeting of the government's emergency response committee Cobra, arguing that would be "a performative political act".
Earlier this year, the government announced a £15bn package of support to ease the cost-of-living crisis.
This included giving households a £400 discount on energy bills and £650 for the poorest households to help with wider living costs.
However, the government has come under pressure to do more, following worsening economic warnings.
Last week, the Bank of England forecast that the UK was heading into a recession, with inflation set to hit 13%.
Meanwhile, the energy price cap will go up in October, with industry analysts estimating average domestic energy bills could hit more than £3,600 a year.
Mr Danker, head of the Confederation of British Industry (CBI), said the prime minister and the candidates for the Tory leadership, Rishi Sunak and Liz Truss, "should come together to agree a common pledge to support people and help quell fears".
"The economic situation people and businesses are facing requires all hands to the pump this summer," he said.
"We simply cannot afford a summer of government inactivity while the leadership contest plays out followed by a slow start from a new prime minister and cabinet."
The CBI - which represents British businesses - said the prime minister should direct the civil service to draw up options on helping the most vulnerable people - and that these options should be made available to both leadership candidates.
The organisation also said Mr Johnson, Mr Sunak and Ms Truss should agree "a way forward to support people and businesses with energy bills" ahead of 26 August, when Ofgem is due to announce the next energy price cap.
Elsewhere, Ms Sturgeon has called for an emergency meeting with Mr Johnson and the leaders of devolved governments, to agree steps to help people.
In a letter to the prime minister, she said the SNP-led government was "committed to doing all we can" but added: "We cannot respond on the scale required without action by your government."
Writing in the Daily Record, Mr Brown - who was the Labour prime minister between 2007 and 2010 - said the UK faced "a financial timebomb" and urged Mr Johnson, Mr Sunak and Ms Truss to agree "an immediate Budget to prevent a winter heating catastrophe".
He also said the emergency committee Cobra should be in "permanent session to deal with the coming fuel and energy crisis".
Speaking to the BBC's World This Weekend programme on Sunday, Mr Brown said there was "a vacuum" in government because "the prime minister is on holiday, the chancellor is on holiday, the two leadership candidates for prime minister are on the campaign trail."
Asked if Mr Johnson - who has returned from a holiday in Slovenia - would summon a meeting of Cobra, No 10 said it would be up to Mr Johnson's successor to make any decisions on further support for households.
The spokesman also said there were no plans to recall Parliament, but that Mr Johnson and his Chancellor Nadhim Zahawi are meeting this week to discuss progress on the support measures already announced.
Responding to No 10's comments, Labour's shadow chancellor Rachel Reeves said "people are worried sick about how they'll pay their bills... and all this Tory prime minister does is shrug his shoulders".
She said the two leadership candidates could only "offer more of the same", adding: "Labour would start by scrapping tax breaks on oil and gas producers and providing more help to people who are struggling to pay their energy bills."
Meanwhile, the two MPs hoping to take over the leadership have been criticising each other's plans to address mounting living costs.
Mr Sunak has warned that the plans of his rival Ms Truss "won't touch the sides".
Supporters of Ms Truss say her proposed tax cuts would help people, but did not rule out offering further support.