The group would potentially draw on the troops and resources of both Australian and New Zealand, although the details were yet to be worked out.
Mr Rudd and his New Zealand counterpart, John Key, announced the plan today at Canberra's Parliament House, one of several stops in Mr Key's five-day Australian tour.
"We're both willing to explore the possibility of forming, over time, an Anzac contingent...which might be commonly deployed to various security environments in the future," Mr Rudd told reporters.
"We believe, given the bonds that already exist between our armed forces, their common training, the compatibility of so much of their equipment, (that) this is actually a useful thing for us to do together."
Defence chiefs of both countries would work on the detail.
"The precise operational scope of any such agreement would be properly crafted by our respective defence chiefs," Mr Rudd said.
While both prime ministers agreed the idea was a good one, it was a long way from being realised.
"There are going to be defence and security scenarios in the future where it's going to make a lot of sense for both of us to quickly and jointly deploy," Mr Rudd said.
"And therefore, why not look through the possibilities to how we can institutionalise this between us?
"But we haven't reached a landing strip on this. We've agreed to explore it."
