Mr Raab said he instead prioritised "security" at Kabul airport and "delegated" the call to a junior minister.
But that call did not happen due to the "deteriorating situation", he added.
The foreign secretary has rejected demands from the opposition to resign.
Speaking after a meeting of the government's emergency Cobra committee on Afghanistan, the prime minister dismissed suggestions the UK was uninterested in the issue, saying: "The whole of the government has been working virtually round the clock."
He added that the situation at the airport in Kabul was "getting slightly better".
He also said that the UK's commitment to Afghanistan was "lasting", adding: "What I want to assure people is that our political and diplomatic efforts to find a solution for Afghanistan - working with the Taliban, of course, if necessary - will go on."
Since Saturday, the UK has evacuated 1,615 people, including 399 British nationals and their dependants, 320 embassy staff, and 402 Afghan nationals who worked for the UK government in Afghanistan.
Meanwhile, the government has announced £5m for local councils in England, Scotland and Wales offering to house Afghans who have arrived under the Afghanistan Relocations and Assistance Policy (ARAP) programme.
Under the scheme, Afghans who face threats from the Taliban having work for the UK in Afghanistan, have been allowed to come to the UK.
A separate resettlement scheme has also been set up to relocate up to 20,000 Afghans over the long term with Home Secretary Priti Patel emphasising that the majority of those should be women and girls and persecuted minorities.
New arrivals from Afghanistan will be offered a first dose of a Covid vaccine and "rapid access to support for mental wellbeing".
The government has said it will also provide "appropriate clothing" and toys for the children to keep them entertained.
Earlier this week it emerged, in a report by The Daily Mail, that Dominic Raab had been on holiday in Crete last Friday and unavailable to make the phone call to the Afghan foreign minister, as the Taliban advanced towards Kabul, Afghanistan's capital.
Labour's shadow foreign secretary Lisa Nandy said Mr Raab's defence "simply doesn't add up".
"It should have been an absolute priority to speak to the Afghan government and set out the immediate actions necessary to ensure the safe evacuation of Britons and Afghans," she added.
On Friday, Mr Raab attended a virtual Nato meeting where the members emphasised the need for an "inclusive" government in Afghanistan and urged the Taliban to uphold human rights.
Mr Raab said: "The UK's immediate priorities are working alongside our Nato allies to prevent Afghanistan from becoming a safe-haven for terrorists, ensuring that life-saving humanitarian aid can reach those who need it most, and working with international partners to safeguard stability in the region."