Police were alerted to the discovery at a property in Guildford St, Burnside, about 9.40am today.
"An NZDF Explosive Ordnance Disposal team was advised and has now safely removed the object," a police spokeswoman said.
"A cordon was initially in place as a precaution but has now been removed."
A mortar fires explosive projectiles called mortar bombs at low speeds for short ranges and at high-arcing trajectories.
Mortars have been used for hundreds of years, originally in siege warfare.
By World War 2, mortars could fire as many as 30 bombs per minute and had a range of more than 2300 metres with different shell types.