Police were called to Williamson Ave in Ponsonby after 6.15pm on Wednesday.
Senior Sergeant Dave Plunkett said an SUV had been parked outside the supermarket while the driver was inside shopping. His young son was waiting in the car.
Plunkett said a man approached the car and yelled at the young boy to get out before he got in and drove away.
"Fortunately, the young boy was physically unharmed, and the alarm was raised with a road policing officer who was nearby."
Police responded and the Eagle helicopter crew found the car abandoned on Commercial Rd in Arch Hill shortly after.
Plunkett said police also received multiple calls about a man jumping fences in that area.
The man was later spotted on CCTV on the city's Karangahape Rd.
Plunkett said the man was arrested and a concealed knife and fake gun were found on him.
"I would like to acknowledge all our staff's exceptional response to this incident last night, which would have been incredibly startling for the young boy."
The 31-year-old man would appear in the Auckland District Court on Thursday.
He was charged with unlawfully carrying an imitation firearm, possessing an offensive weapon, unlawfully taking a motor vehicle and resisting police.
