Minister says the 10% of population unjabbed ‘must think about the damage they are doing to society’