Association of Salaried Medical Specialists executive director Ian Powell said the resignations were a serious issue for the Southern District Health Board and a big loss for the department.
Mr Powell blamed a "petty and pious'' attitude which resulted in the radiologists saying "blow this, we work our butts off'' and quitting.
The SDHB has played down the resignations, saying it recently appointed two new radiologists, one of whom would replace a fifth radiologist who retired.
Two of the radiologists resigned in March, and left last week, and another two resigned recently, and would leave in September, the board confirmed.
Mr Powell said an issue in the department acted as a "catalyst'' on top of concerns over heavy workloads, but he declined to comment further, saying it was an employment matter.
The resignations were discussed as a "serious issue'' at a regular joint meeting with health board management last Friday, he said.
Mr Powell said he believed commissioner Kathy Grant had not addressed problems with the senior reaches of management at the SDHB.
The board had a "negative leadership culture'', he said.
"When you treat people badly, something will snap, especially when they have options.''
A source close to the issue, who did not want to be named, told the Otago Daily Times the dispute involved concern over radiologists potentially using public health time to process chores from private work.
It affected two radiologists, one of who was among the four who resigned, and an internal review effectively cleared them of the suggestion they used time inappropriately, the source said.
In a statement, chief medical officer Dr Nigel Millar said the two radiologists who left last week were Dunedin-based and would stay in the city to work in private practice.
"We have recently appointed two full-time radiologists who will join us in August and October.
"One appointment is to replace the first two radiologists who resigned (both part-time) and the other is replacing a [fifth] radiologist who recently retired.''
Dr Millar said the SDHB had 16 radiologists in total, 13 of them based at Dunedin Hospital, in a mix of part-time and fulltime positions.
Dr Millar said the resignations would not affect radiology services.
He declined to comment on "other staff matters'' concerning departmental issues.
In a one-line statement, Mrs Grant said she supported Dr Millar's comments about the "operational matter''.
