Australian federal and state police, the Australian Security Intelligence Organisation, and other agencies that form part of the Joint Counter Terrorism Team carried out the arrests on Tuesday morning.
Police said three men, aged 30, 26 and 21, were taken into custody after they allegedly sought to acquire a semi-automatic gun to carry out an attack.
All were Australian citizens and their passports had been cancelled earlier this year.
"We now have sufficient evidence to act in relation to preventing a terrorist attack," Graham Ashton, Chief Commissioner of Victoria Police, told reporters.
Police said the suspects had yet to decide on the site of their planned attack but they believed the act was imminent.
"They were certainly looking at a place of mass gathering, where there would be crowds," Ashton said. "They were trying to focus on trying to have a place where they could kill as many people as possible."
Police said they believed the arrests had nullified any threat from the group.
Australia, a staunch US ally that sent troops to Afghanistan and Iraq, has been on heightened alert since 2014 for attacks by home-grown militants returning from fighting in the Middle East or their supporters.
Police said the three men were known to authorities and their passports were cancelled because of concerns they would travel to a conflict zone overseas.
The arrests come less than two weeks after a man set fire to a pickup truck laden with gas cylinders in the centre of Melbourne and stabbed three people, killing one, before he was shot by police.
As in the case of that attack, police said on Tuesday the three men had been inspired by Islamic State rather than being directed by the militant group.
