A police spokeswoman said they were called to the scene at Cape Saunders just before 4.10pm on Saturday, along with a rescue helicopter.
Another man suffered serious injuries.
A witness said he was on a fishing trip after work with others, fishing on some rocks, when one of the men was swept off by a "freak wave" and into the water.
A work colleague grabbed a life ring - installed at Cape Saunders in late 2010 - and went in the water to support the man, who was still alive, he said.

Two helicopters - the Otago Regional Rescue Helicopter and an Otago Rescue Helicopter - were with the men.
Both were taken separately by helicopter to Dunedin Hospital, he said.
Sea conditions at the time were fairly flat with a gentle swell, and not a lot of wind, the photographer said.
The death will be referred to the Coroner.
Te Runanga o Otakou announced on Sunday morning that a rahui - or temporary closure - would be in place at Cape Saunders following the drowning.
The rahui covered in area from Taiaroa Head to Tow Rock, north of Hoopers Inlet, and would remain in place for 24 hours.











