Swann (51) will be released from prison to live at Mr Cassels' Governors Bay home on July 31.
The property is reportedly worth about $1.3 million and includes an indoor pool and harbour views.
But Mr Cassels, whose business interests include The Cassels and Sons Brewery, described his house as ''average''.
''I don't think he [Swann] will find it very comfortable.''
Mr Cassels said he had known Swann and his family for between 15 and 20 years and was happy to accommodate the former Otago District Health Board employee.
He recently received a phone call asking if Swann could reside with him when released from jail.
''The parole board had a problem with where he was going to stay. He needed another address so I said yes, and if I hadn't he probably wouldn't have got parole,'' Mr Cassels said.
He did not expect Swann to live with him until his parole ended in May 2018.
''It's just until he finds his feet, really.''
Mr Cassels said Swann had served his time and was trying to get his life in order.
''What would you do if you knew someone and then they wanted a place to stay for a month or two - what would anybody say?''
When released on July 31, Swann will have spent four years and eight months in jail for his part in defrauding the health board of $16.9 million.
He was sentenced in March 2009 to nine years and six months' jail, with a minimum non-parole period of four years and six months.
News of his release sparked anger from some people yesterday.
Comments posted on the Otago Daily Times website referred to Swann profiting from his crime, on the basis that most of what he stole was still outstanding.
''And they say that crime doesn't pay. It certainly does, and this guy is laughing all the way to the bank,'' a reader said.
To date, police have recovered about $3.6 million from Swann, mostly through the sale of assets seized under the Proceeds of Crime Act.
Millions of dollars more remain unaccounted for, despite extensive investigations by police and the Serious Fraud Office.
A retired Dunedin nurse hoped more money could be recovered from the seizure and sale of Swann's assets.
The nurse, who asked to remain anonymous, contacted the ODT yesterday, concerned Dunedin police had not acted on information about an alleged cache of Swann's cars stored near Dunedin airport.
She said a patient told her in April last year he and his wife had visited a property about 2km east of the airport with a group of people, and there they had seen 10 or 12 vehicles owned by Swann stored in a large shed.
The vehicles were in good condition.
The nurse said she contacted Dunedin police and left messages for the detective in charge of the Swann case.
But she never heard back and wondered whether police had investigated the claim.
''He [Swann] has done the hospital board out of so much money and if they could recover some more . . . it really annoys me,'' she said. ''Why should he get away with it?''
A police spokeswoman said many lines of inquiry were followed but it could not be confirmed yesterday whether the nurse's particular claim had been acted on because officers involved in the case were away.
