Sheku Bayoh, 31, died after being restrained by officers in Kirkcaldy, Fife, in 2015.
At a public inquiry into his death last week, his sister Kadi Johnson said she no longer feels safe in Scotland.
She told BBC One Scotland's The Sunday Show that she now regrets encouraging her brother to move from London.
Mr Bayoh's family believe that race played a part in his death.
"Scotland is a place that I once loved and I felt safe in, but since my brother's death I don't feel safe anymore," Ms Johnson said.
Asked if Scotland was a racist country, she replied: "For me I will say yes, I'm afraid I'll say yes because the way we have been treated."
She said she had lost trust in the police since the death of her brother and accused officers of failing to tell her the truth about his death.
But she added that racism was also evident in other walks of life.
"Scotland still has a lot of work to do around racism," Ms Johnson said.
"We who are in it, feel it. We know what we are going through on a daily basis. It is different from a white person and we have seen it in our workplaces, our schools."
Research by charity Show Racism the Red Card published last year revealed there were 2,251 racist incidents reports in Scotland's schools in the previous three years.
In one incident a pupil shouted "Isis" to a group of girls wearing headscarves; in another a child wrote KKK on a black pupil's locker, according to the charity.
The Scottish government has since provided a new package of support materials for teachers to "embed anti-racism and race equality" in school life.
Ms Johnson said the issue should be tackled first in schools.
"For this to change, it starts from the grassroots, which is the schools," she said. "Start from the schools and work your way up because it's happening there.
"Black and ethnic minority children are experiencing that and it goes up to us, the adults."
Last week the public inquiry into Mr Bayoh's death heard that Chief Constable Iain Livingstone believes that Police Scotland needs to be "anti-racist".
His QC Maria Maguire told the hearing: "The chief constable is aware that it is not enough to be alert to racism and deal with it on a case-by-case basis.
"Nor is it enough simply to be non-racist. Police Scotland needs to be anti-racist."
The public inquiry opened a year after the Kenmure Street protests, when hundreds of people surrounded an immigration enforcement van in which two Indian nationals were detained.
Lawyer Aamer Anwar, who also appeared on the programme, said that despite incidents like that there remained a "significant minority" of people in Scotland who are racist.
He said little had changed for people of colour in Scotland in the last 20 years.
"I stood on the steps of the high court 20-odd years ago after the murder Surjit Singh Chhokar - a racist murder that took 17 years to get justice - and asked how many black judges, how many black senior prosecutors, how many black senior police officers, how many black senior executives in public bodies?
"And the answer still today is pretty much zero. It hasn't changed that much in reality for the black or Asian minority ethnic community."
A spokesman for the Scottish government said it would await the findings of the independent inquiry before making further comment.
"Our thoughts are with the family and friends of Mr Bayoh, who have shown remarkable dignity and perseverance during their five-year wait for an inquiry into his death," he added.
Police were called out in the early hours of 3 May 2015 after Sheku Bayoh was seen behaving erratically with a knife in a Kirkcaldy street. He had earlier taken drugs which friends said altered his behaviour.
According to police statements, when officers arrived he no longer had the knife, but failed to obey instructions to get down on the ground.
The officers used force on Mr Bayoh, including CS Spray and batons. He then punched PC Nicole Short, who fell to the ground.
Two officers, PC Craig Walker and PC Ashley Tomlinson, later told investigators that Mr Bayoh carried out a violent stamping attack on PC Short. However, evidence obtained by the BBC's Panorama programme suggested these accounts may have been false.
Mr Bayoh was restrained for five minutes before falling unconscious. He was pronounced dead in hospital a short time later.