More than 300,000 cases were passed to investigators between March and May, according to figures seen by the BBC.
The government was not able to say how many of these were found to have broken the rules or could not be traced.
The Home Office has said it aims to pay home visits to all travellers suspected of not following the rules.
But Labour's shadow home secretary Nick Thomas-Symonds said the figures obtained by the BBC "confirm our worst fears" about the government's "lax border policy".
And he accused the Home Office of "gross negligence".
It comes as the UK's four chief medical officers recommend vaccinating healthy children aged between 12 and 15 - as part of a series of measures to control the virus this winter.
It will now be up to ministers whether to accept the recommendation of the four CMOs.
Earlier this year, the government introduced new rules for people arriving from abroad in an effort to slow the spread of coronavirus and stop new variants arriving in the country.
People arriving from high-risk countries - deemed "red list" - had to quarantine in a hotel.
Those coming from "amber list" - or medium-risk areas - were required to self-isolate for 10 days and provide evidence of negative Covid tests.
From 17 March to 31 May more than a million people arrived in England and Northern Ireland from amber list countries.
Figures for this period obtained under Freedom of Information laws show a total of 301,076 cases were referred to investigators for checks on whether they were self-isolating.
During this time, the highly contagious Delta variant of coronavirus - first detected in India - was spreading rapidly through the country.
Call handlers employed by the Department of Health and Social Care were tasked with contacting arrivals to check they were obeying the self-isolation and testing rules.
Cases where the contact ended the call, refused to co-operate, indicated they would break the quarantine or testing rules, or could not be contacted after three attempts were referred to investigators at the Border Force Criminal Justice Unit and the police.
Officers would then attempt to visit the contact at home to check they were following the rules.
After 26 April, the Home Office hired private contractor Mitie to carry out home visits to international travellers required to isolate, from contacts supplied by NHS Test and Trace.
"We visit over 99% of the cases referred to this service by NHS Test and Trace," a government spokesman said.
In Wales, the Arriving Travellers Team is responsible for monitoring compliance with the rules - including escalating cases to regional teams to undertake doorstep visits if deemed necessary.
And in Scotland, quarantine measures are monitored by Public Health Scotland through the National Contact Tracing Centre. PHS has the ability to refer to cases to Police Scotland.
On 1 April, Henrik Pakula began 10 days of isolation at home after returning to England from Poland, where he had been visiting relatives.
"On day one I received a phone call just asking me if I knew the rules on self-isolation, and informing me that there could be random phone calls over the next 10 days just to check I was still at home, but none of those calls materialised," he said.
"I could have nipped out to the supermarket if I had so desired - no-one would have checked, no-one would have known.
"I got a text message saying these were the rules but there was no contact with anybody at all.
"I knew I had to isolate for 10 days, which I did, but it was taken on trust."
Lucy Moreton, of the Immigration Services Union, said the quarantine system "very much relied on the honesty of people to do the right thing, rather than any type of meaningful enforcement".
"There's limited point in putting rules in place if you don't enforce them," she added.
Yvette Cooper, Labour MP and chair of the Home Affairs Committee, said: "We've been concerned throughout that there just weren't proper follow-ups on the checks that were done, and as a result you just had these huge gaps in the home quarantine system and that's what made it easier for the Delta variant to spread."
But Conservative MP Huw Merriman, chair of the Transport Committee, said: "There was a balancing act from government... would we have had the resources to check every single person who was quarantining at home? Of course not - so we always knew those risks were in the system".