Police and search and rescue personnel were called after the man got into difficulty about 150m off shore, Constable Matt Sheat said.
Using an IRB, rescuers hauled the man from the water and brought him back to shore at Ocean View Recreation Reserve, from where he was taken to Dunedin Hospital by St John.
The man was conscious and talking but appeared shaken and showed signs of hypothermia, police said.