Otago won its second game of the season, and moved up to fourth in the New Zealand Football Championship, when Kelly latched on to a Tristan Prattley cross in the 89th minute.
Coach Malcolm Fleming was hardly gushing in his summation of the performance but was quietly satisfied at a win that sealed a useful start to the season for United.
"We didn't play that well, to be honest, but you always take a win away from home," Fleming said after the game.
"Both teams battled it out for most of the game. In the last 15 minutes, I think we started to get on top and we had six or seven chances."
Otago midfielder Andy Coburn controlled the centre of the park, goalkeeper James Waggett and the defensive line performed soundly, and Kelly and Nathan Knox warmed to their roles pushing forward.
It is too early to say Fleming has engineered a complete turnaround from two poor back-to-back seasons but at least the new coach can reflect on a reasonable start to the season.
"We're definitely happy. We're sitting fourth in the competition and we're looking forward to a big game against Canterbury now.
"If we can win that and push up to second or third before Christmas, we can't complain."
Otago created some reasonable early chances against a Waikato side missing veteran defender Che Bunce.
A Dan Ede header went wide of the post and a Kelly shot was blocked, but the best chance of the game was created 10 minutes before half-time, when Ede ran into the box and flicked the ball to Knox, whose shot was parried by Waikato keeper Sean Dowling.
The game meandered until the 80-minute mark, when Otago took the game by the scruff of the neck.
James Reichwein wasted a good opportunity to score and Kelly had another good shot.
Then, after some nice touches down the left-hand side, Prattley swept in his cross, Kelly poked his foot out and the ball went in.
Otago had to play without nippy forward Dave Dugdale, who picked up an injury at training on Friday night and did not make the trip.
It is hoped he will be back for the game against Canterbury United, at Carisbrook on Sunday.
With Auckland City on World Club Cup duty, Waitakere was able to shoot to the top of the NZFC table with a 2-1 win against Wellington.
Manawatu moved up to second with a 1-0 win against Hawkes Bay.
NZFC Result
Otago United 1
Stu Kelly
Waikato 0
Half-time: 0-0
