With cases on the increase, he said he was "not persuaded that this is a moment for such an approach".
The UK government has faced criticism from some scientists over its plans for a "freedom day" on 19 July.
However, Mr Drakeford said the four nations were moving in "broadly" the same direction.
The comments in the Senedd came after Welsh government advisers warned of a risk of a wave of deaths despite vaccines "holding up" in the race against the Delta variant.
Mr Drakeford's cabinet will make the final decision on Wednesday morning, with a statement to the Welsh Parliament expected that afternoon.
Nearly all restrictions in England will be scrapped on 19 July, with no legal requirement for people to wear face masks and with limits on social distancing to end.
Public Health Wales reported that the case rate stood at 138.2 per 100,000 people on Tuesday, a slight rise.
Although cases are still rising, the doubling time is now about 12 days, whereas cases were doubling every seven days towards the end of June.
Face masks are expected to be kept in some settings in Wales and in Scotland - where Nicola Sturgeon has announced an easing of restrictions, with advice on working from home and rules on social distancing while outside staying in place.
Mr Drakeford told the Senedd he hoped he would be able to take steps towards moving "beyond" the Covid rules.
"What it will not be will be a wholesale abandonment of the collective actions that we have taken during the pandemic, and which continue to keep Wales safe."
"The mood of Welsh people is not a mood of thirsting for some spurious freedom day," he later added.
The Welsh government has already said face masks would continue to be required by law on public transport and in health and social care settings, but a decision on whether that will be the case in shops - as called for by Plaid Cymru and trade unions - is yet to be made.
On Wednesday ministers will announce whether restrictions can be eased to allow more to take place indoors, including whether the rules on social contact in people's private homes may be relaxed.
They are also expected to publish plans on how Wales will move to a new "alert level zero", where there will be fewer legal restrictions.
Following a meeting with officials from the four UK governments, Mr Drakeford said: "We may be doing it at slightly different speeds or in different ways, but the essential direction of travel is the same between us."
However, he did accuse Boris Johnson of not making it clear enough that his announcement on Monday was only for England: "It was disappointing again to find that the prime minister finds that such a difficult thing to do, because it would have helped with clarity."
He also said he agreed with the Senedd Welsh Conservative leader Andrew RT Davies in being opposed to "the mandation of vaccine passports".
That was despite comments in March when Mr Drakeford said there were "prizes to be won through domestic vaccine certification".
Welsh Conservative Senedd leader Andrew RT Davies said: "Scotland has a plan, England has a plan, and for the sake of Welsh jobs and our economic recovery, it's now time Labour ministers published a detailed plan for the restoration of freedoms in Wales.
"We have to learn to live with the virus and with the summer kicking in, and the school term coming to its conclusion, many families, workers and businesses will be asking if not now, when?"
Vaccines are "holding up" in the race against the Delta variant, which originated in India, but there is still a risk of a wave in hospital admissions and deaths as cases grow rapidly, Welsh government advisers have said.
The Technical Advisory Group (TAG) said the ratio of confirmed infections to hospital admissions was falling as fewer people have become seriously ill.
TAG said this could be because the Delta variant was spreading among young people, who are less likely to end up in hospital, or it could be a result of the vaccine.
Modelling from Swansea University presents a range of scenarios, with the most pessimistic showing a peak in hospital admissions and deaths this summer similar to the second wave of the pandemic.
But other scenarios estimate a lower peak and the report suggests they are more likely.
New modelling from Warwick University also shows a lower peak, happening around March 2022.
The ratio of cases to deaths is falling in all age groups. Among the over-70s it fell from 24% in December to 2% on 1 July.
"This is likely due to the effects of the Covid-19 vaccines," the report says.