After last week's disappointment in their opening round loss to the Western Force the Waratahs banked their first competition points in the AAMI Park clash.
Waratahs centre Kurtley Beale put on a vintage display to seal the victory.
After temporarily leaving the field following a head-knock the former Rebel set up his team's first try with a deft chip kick over the defence which he collected and then off-loaded to Stephen Hoiles.
He then scored a 73rd minute try that broke a 28-28 deadlock.
NSW winger Taqele Naiyaravoro had a mixed night - scoring two second half tries, but was also yellow-carded for a deliberate foul when the Rebels were on the attack.
The Rebels were looking for their first back-to-back wins to start a season and threw everything at the visitors but came up agonisingly short.
It was another impressive display after their wooden-spoon finish last season.
The Waratahs clung to a 16-14 halftime lead, which should have been more but for a try after the buzzer by Rebel Lopeti Timani.
In a stand-out match the lock dived over the line after halfback Luke Burgess stole the ball from a Waratahs scrum.
Earlier Melbourne skipper Scott Higginbotham found a gaping hole off a nothing play to touch down for the first of the match in the 12th minute.
Both sides got plenty of try-scoring action in the second half with Naiyaravoro's double, while Burgess and Melbourne-born winger Sefanaia Naivalu also crossed.
But Beale broke the Rebels' hearts after a break by five-eighth Bernard Foley, who also added a late penalty for the 10-point margin.
Waratahs coach Michael Cheika said his team's performance was a huge improvement on last week's showing, particularly their physicality.
"We earnt our way back into the game or ahead just through some old-fashioned grunt," he said.
"I was disappointed in our physicality last week and needed to see more."
He said the way the team held its nerve when Naiyaravoro was sin-binned was key to the win.
"We were wobbling a bit and Dave got the team together and we held our nerve and we were able to edge ahead and that was really good character and that's the way you earn back your self-belief."
The Rebels were left to rue some missed opportunities with five-eighth Jack Debreczeni missing a long-range penalty attempt in the 71st penalty.
He was only taking the kick because their usual kicker, sharp-shooter Mike Harris, was forced off after a heavy hit by NSW skipper Dave Dennis.
"It was huge," said Melbourne coach Tony McGahan.
"We readily expected Jack to get that but if Mike's on the field there, it's 31-28 with a few minutes to go, it's a big difference."
