Needing to beat world No18 South Korea by 12 points or more, the 35th-ranked New Zealanders held a 12-point lead with five minutes to go but faded in the frantic closing minutes.
The Tall Ferns had opened the tournament with an 111-54 win over the Philippines, but were outplayed 94-71 in Saturday's clash with world No8 China.
The top two nations from the Auckland tournament go through to a final play-off round for 16 teams in February, with the top 12 from there qualifying for the Tokyo Games.
But because South Korea had earlier upset China 81-80, the Tall Ferns needed a big win to progress, and just could not maintain the necessary margin.
The Ferns were missing injured forward Ashleigh Karaitiana, but started with intensity and rocketed out to an early 18-5 lead due to some outstanding defence and pinpoint shooting.
The New Zealanders maintained that margin to go into the first break with a 21-8 lead, South Korea reducing the gap to 37-28 at halftime then 55-50 at the three-quarter mark.
The Tall Ferns dug deep but could not sustain a mammoth final-quarter effort, Penina Davidson leading the way with 24 points and a 15-rebound haul.
