Tries from Nic White, Robbie Coleman, Jesse Mogg and Jarrad Butler gave the hosts a measure of revenge for their defeat in last year's final and earned them a place in next week's semi-finals, against the Waratahs.
The Chiefs were trailing 22-3 after half an hour but they did not win back-to-back titles by giving up easily and stormed back with tries from Bundee Aki, Tawera Kerr-Barlow and Tim Nanai-Williams to level the scores at 25-25 with 25 minutes to go.
Flanker Butler scored from a rolling maul five minutes later, though, and Chiefs flyhalf Aaron Cruden's failure to convert replacement back Gareth Anscombe's 77th minute try ultimately proved decisive.
"We knew it was going to be a full 80 performance, you don't knock a side like that over easily," Brumbies skipper Ben Mowen said in a televised pitchside interview.
Yellow cards at the start of each half for infringements at the breakdown had a significant impact on the contest.
Chiefs centre Tim Nanai-Williams was sent to the sin-bin in the fifth minute for cynically slowing down Brumbies ball and five minutes later the hosts were 14-3 up with two tries on the board.
Scrumhalf White scored the first, peeling off the back off a rolling maul when he spotted space down the blindside, and then plucked Dwayne Sweeney's chip out of the air to start a flowing move that ended with Coleman's second.
Mogg had showed his electric pace to set Coleman free for the line and the winger returned the favour 10 minutes later with a delicate pass that allowed his fullback to cross in the corner.
Christian Lealiifano added a penalty after 29 minutes but the rest of the half belonged to the Chiefs, who got their reward for sustained pressure when centre Aki wrestled his way over the line from close range after 35 minutes.
Cruden and Lealiifano traded penalties in the first three minutes of the second half but then Brumbies winger Henry Speight was shown his yellow card.
While Speight was off the field, Kerr-Barlow reacted quickly to dot the ball down after a ruck close to the line and when Nanai-Williams raced over in the corner after 55 minutes, the scores were all square.
The Chiefs came from behind to beat the Brumbies in last year's final and the hosts were clearly determined not to be overhauled again.
"It wasn't the result we were after, we started slowly and gave a quality side like the Brumbies a bit of a head start," Cruden said.
"It's pretty disappointing but we just couldn't quite get over the line."
