Emergency services were notified at 12.20pm and a rescue operation involving police, coastguard and ambulance staff was launched immediately, police said.
The rescue operation was carried out in the waters between Bluff Port and Tiwai Wharf.
It was believed there were four people on board, police said in an earlier statement.
About 2.25pm a St John spokeswoman said two people had been taken to Southland Hospital in critical condition.
Maritime New Zealand have been advised and Police are making enquiries into the deaths on behalf of the Coroner.
Earlier. the St John spokeswoman said three ambulances and a first responder had been sent to Bluff, with the manager also going to South Port.
A reporter at the scene said a helicopter could be seen hovering over the port.
The MetService maritime sea forecast for Bluff rates the boating conditions as bad, with wind at 18 knots and gusts of up to 23 knots, a chop height of 0.6m and a swell of 0.8m.
Police would like to hear from anyone who saw a small boat in Bluff Harbour today or who has CCTV footage of the harbour.
