Epsom MP David Seymour was the only Act candidate to get a seat in Parliament after the September 20 election.
The party did not get enough votes to also secure Mr Whyte a seat through the party list.
"Today I announce that I have tendered, and the board has accepted, my resignation as leader of Act New Zealand,'' Mr Whyte said in a statement.
"Clearly I make this announcement with regret, however the election result is clear, and I must now turn to my career and my family.
"I stood to lead Act because I believe in the party's ideas. I will continue to advance these ideas both inside and outside the party. I do not rule out returning to a substantial role with Act in the future.''
Act Party president John Thompson said Mr Seymour had been appointed as the party's new leader.
"Today the Act Party board announces with regret that it has accepted Jamie Whyte's resignation as leader," Mr Thompson said in a statement.
"Jamie has proven himself a principled and clear-headed advocate of Act's values of smaller government and greater personal responsibility. Under his leadership, the party has been rejuvenated and membership has swelled.
"We are proud to have had Jamie serve as leader, and hope the New Zealand public has not seen or heard the last of him."
Mr Seymour said he was "honoured" to lead the party.
"I look forward to the challenge and relish the opportunity," he said.
"It has been a privilege to work with Jamie Whyte. I want to acknowledge his tireless efforts through the past nine months and during the campaign.
"Through his efforts the party was rejuvenated and our membership increased. It is a substantial achievement and on behalf of all the members and supporters of the Act Party, we thank you Jamie."
Mr Seymour said he was excited by the opportunity to lead the Act Party, as well as being the MP for Epsom and Parliamentary Under-Secretary for Education and Regulatory Reform.
"I look forward to Act contributing to a stable and successful National-led government, and to expanding our presence in Parliament in 2017."
