The Speaker will not be taking action on a request from the Green Party that New Zealand First leader Winston Peters be asked to withdraw and apologise for his comments to their MP Lawrence Xu-Nan.

Gerry Brownlee told RNZ a request for someone to withdraw and apologise needed to take place in the moment, and by the person who had taken offence.

Brownlee, who made it clear he disapproves of the comments, said he would not consider a request nearly a week later.

The Greens are disappointed at the lack of action.

Peters, the Foreign Affairs Minister, has been criticised for telling Xu-Nan in the House to "go back to your own country," with Prime Minister Christopher Luxon calling the statement racist.

The Greens wrote to the Speaker requesting he encourage Peters to withdraw and apologise.

"Should this apology not be forthcoming, we will pursue this as a breach of privilege," the letter read.

The Speaker said if he was asked to consider whether it was a breach of privilege, then he wouldn't be.

Brownlee said there is free speech in New Zealand and the public should be able to hear what politicians are thinking.

He suggested the public would have "certainly learnt a lot" from the remarks.

The Greens chief musterer, Ricardo Menendez March, told RNZ they were disappointed and the lack of action showed the Speaker and Prime Minister were scared of holding Peters accountable.

He said they had always been clear this was a matter of Luxon "not showing leadership".

"The Speaker may choose to let this behaviour slide or not take action on it, but ultimately, Luxon needs to face the facts that he's simply too scared to do anything about Winston Peters."

Menendez March said Peters was "walking all over him" and "running circles around him".

He said the issue was the Greens' concern of "double standards" from the Speaker, and that Luxon refuses to take action against Peters because he was worried "it will result in consequences" for National's electoral chances.

The election will held on November 7.

On Tuesday, Greens' co-leader Chlöe Swarbrick was kicked out of the House by Brownlee after using a point of order to say the Speaker had "double standards".

"Empty words just won't cut it for a community that is seeing one of the most senior figures in government actively scapegoating them," said Menendez March, who was concerned about how the remarks might encourage discrimination toward migrant communities.

RNZ asked Brownlee if he regretted not intervening in the moment during the debate last week.

He said he had asked himself this numerous times since, but noted that it was a "boisterous" debate, with a lot of interrupting going on.

Brownlee said Peters' comment came as a result of multiple interjections, more than just what was recorded on Hansard.

He also made the point that, had the comments been withdrawn, the matter would not be able to be discussed in the House.

Brownlee said he had made it clear he believes Peters' remarks were "offensive".