Mr Joyce secured a majority in the 21-member party room at a meeting in Canberra on Monday after long-time supporter Matt Canavan moved a spill motion.
Nationals whip Damian Drum confirmed the outcome of the leadership but did not reveal the vote tally.
"He has to go through a process now to be sworn in, to have all the conversations, to talk to the prime minister, and effectively get on with the job of representing our people," he told reporters at Parliament House.
Asked what the decision said about the junior coalition partner, Mr Drum said: "The most democratic party in Australia".
Mr McCormack was asked during a brief press conference after the meeting what he told Nationals colleagues.
"I said thank you for the great privilege of serving you," he told reporters.
Mr Joyce will address the media on Monday afternoon.
The change in Nationals leadership could have major implications for the Morrison government, with the junior coalition partner set for a ministerial reshuffle.
Agriculture Minister David Littleproud has retained his role as Nationals deputy leader.
Scott Morrison will have to navigate a new dynamic with Mr Joyce widely regarded as a more combative force within the coalition than Mr McCormack.
Before the spill, Mr Morrison - who is in quarantine at The Lodge - threw his support behind the incumbent.
"I've got a wonderful partnership with Michael. We've worked very closely together and provided great stable leadership for Australia," he told 2GB radio.
Mr Joyce failed to topple Mr McCormack at his last attempt in February last year but has prevailed at his second crack.
Mr Joyce lost his job in 2018 after a major scandal stemming from his extra-marital affair with a staffer who has now given birth to the couple's two children, as well as sexual harassment allegations he strongly denies.
Earlier, Mr McCormack vowed not to stand aside.
"If I survive then the people who actually run against me, they should think long and hard about their futures," he told reporters in Canberra.
"They should think long and hard about the role they need to play in government. They should stop being so destabilising."
Walking into the meeting, the deputy prime minister said he was feeling positive.
