The meeting was at Te Papa in Wellington on Thursday, which comes ahead of further strike action next week.
In his speech to the Association of Salaried Medical Specialists, he called on doctors to call off their strike and put patients before politics.
There were audible cries of disbelief when he said the action - while their legal right - had "crossed an ethical line".
Brown said Health NZ estimates the four-hour strike on 23 October would see 6300 procedures cancelled and almost 13,000 assessments delayed.
The room erupted in laughter when he said he had been advised by Health NZ that the average salary for a senior doctor was $325,000, and the employer's offer represented a $160 million pay rise.
"No-one disputes that doctors need to be recognised for the job you do, but New Zealanders also deserve a health system that works for them," Brown said.
He said patients were paying the price for the strike action, and accused the union of walking away from negotiations.
"Patients should never be collateral damage in disputes between management and unions."
The union said its officials had twice this month asked for bargaining but that offer had been rejected by Health NZ.
