The 37-year-old, who was not expected to be selected for the fifth and final test at the Oval after being replaced by Peter Nevill, was returning for personal reasons, Melbourne's The Age newspaper reported.
England have already regained the Ashes after a comprehensive innings and 78-run victory in the fourth test at Trent Bridge gave them a 3-1 lead in the five-match series.
Haddin had been granted leave earlier in the tour and missed the second test because his daughter was ill in a London hospital.
He was not selected for the third test when he returned to the squad due to an impressive performance by Nevill, who made his debut in the second test at Lord's - the only game Michael Clarke's side have won in the series.
The Age reported that a Cricket Australia spokesperson had asked the media to respect Haddin and his family's privacy when they returned to Australia.
Haddin had started the tour as first-choice wicketkeeper, but has been usurped by Nevill after a lean spell with the bat over the last 12 months.
Australia are due to play Northants in a three-day match starting on Friday before the final test begins on Aug. 20.
They then play Ireland in a one-day game before meeting England in a six-match limited overs series that includes one Twenty20 fixture and five ODIs.
