Dunedin Marine Search and Rescue spokesman Scott Weatherall said the man was walking to Taieri Island at low tide to fish about midday when he was swept off his feet.
A combination of rising tide and high flows from the Taieri and Clutha Rivers, made conditions "inappropriate" for such a venture, Mr Weatherall said.
The man drifted about 1km south before clinging to rocks off Taieri Beach. He stayed there for some time before being swept into the sea, where he was rescued about 20 minutes later, at 1.50pm, by the Otago Regional Rescue Helicopter.
Marine search and rescue provided backup for the helicopter, heading into the water as a stand-by measure.
The man was 75m to 100m from shore when he was winched to safety, Mr Weatherall said.
He was "very lucky". The water was cold and "bloody rough", Mr Weatherall said.
Senior Sergeant Brian Benn said an "astute" local had raised the alarm when he saw the man clinging to the rocks.
"It was the first step in saving this guy's life."
Brighton Surf Lifesaving Club, Taieri Mouth First Response Group, and St John also attended.
Rescue helicopter pilot Greig Shirley said although sea conditions were rough, the rescue was straightforward.
Rescue swimmer Paul Sharp was winched down to the sea, where he hooked the man on to him.
The man was "not too bad", but cold and tired and had lost skin off his knuckles, Mr Shirley said.
Taken to Dunedin Hospital with suspected hypothermia, the man was discharged later in the afternoon, a Dunedin Hospital spokesman said.
