The Strath Taieri Community Board chairman was censured by the Dunedin City Council which called on him to resign over the slur.
At a community board meeting held at Hyde this afternoon, Mr Williams apologised to the board and the community and said he had apologised to the victim.
However, he said he would not be resigning.
He had thought it over and decided to continue.
Board deputy chairwoman Anna Wilson said the board was disappointed by Mr Williams' comments.
About 30 members of the public turned out for the meeting in the Hyde Community Hall.
Board council representative Bill Acklin acted as chair for the meeting in place of Mr Williams.
Council chief executive Sandy Graham was also present.
Shane Loader, presenting in the public forum, said the area had become a symbol of the racist underbelly of New Zealand.
"It's going to be hard to shake off that reputation."
The board was now a lame duck incapable of representing the community.
"Some on the board believe Barry should continue as chairman because he got the most votes," Mr Loader said.
He rejected that outlook and said Mr Williams was not a team player.
He had started his presentation talking about tourism and Cr Acklin asked him to stay on topic.
Mr Loader said the area should be one people would want to visit.
Andrea Bosshard decried bullying, racism and misogyny and said people in the community were fearful of speaking out.
Some people had a misplaced conception Mr Williams was a victim.
"The victim is the woman who has left town," Ms Bosshard said.
"In order to bridge political divides, leadership is crucial. If he is a courageous man, he will step down with grace."
The council had called on Mr Williams to resign and in a letter penned by Dunedin Mayor Jules Radich said: "Council is appalled by your behaviour, which is a clear breach of the code of conduct and a serious failure to uphold the standard of behaviour expected of an elected community representative."
The fallout from the complaint has spread to Mr Radich who faces censure for minimising the racial slur in a radio interview.
Following the RNZ interview, deputy mayor Sophie Barker and Cr Jim O’Malley laid a code of conduct complaint about Mr Radich to the Dunedin City Council last Friday.
Cr Barker has since resigned as deputy mayor and filed a code of conduct complaint against Cr Carmen Houlahan.
Before she stepped down as deputy mayor she disclosed she had received "17 awful texts" and eight unanswered phone calls from Cr Houlahan.
Cr Houlahan said she was shocked by Cr Barker and Cr O’Malley making the complaint about the mayor and had not realised she had sent so many text messages.
Cr Barker was furious on Monday when Cr Houlahan accused her of having a political agenda and starting campaigning for the mayoralty two years out from the election.
