Ms Ardern was driven to Auckland City Hospital by Mr Gayford, as was planned, before 6am yesterday.
Almost 11 hours later, the little girl - whose name has yet to be announced - was born at 4.45pm.
The PM's office also passed on information from the Auckland DHB that 16 babies had been born at Auckland City Hospital before the announcement of the First Baby's birth.
Former prime minister Helen Clark was one of many to wish Ms Ardern and Mr Gayford well with their new addition.
"First and foremost, today is a very happy day in the lives of Jacinda and Clarke as they welcome the arrival of their baby daughter.
"My full congratulations go to both Jacinda and Clarke, and I wish them and their new baby all the best."
New Zealanders took the news of Jacinda's pregnancy in their stride, she said.
"This is a sign of our maturity as a country and its acceptance that combining career and family is a choice which women are free to make.
"For New Zealand, these events and the way our country has greeted them will be seen as inspirational by all who advocate for gender equality and women's empowerment."
Attitudes had changed on such issues since she entered New Zealand politics and that was a "very good thing", she said.
Born on June 21 the baby girl shared a birthday with Prince William, Indonesia's President Joko Widodo, singer Lana Del Rey and NSA whistleblower Edward Snowden.
She also shares the date with former Pakistan prime minister and first female leader of a Muslim nation Benazir Bhutto.
Ms Bhutto was also, in 1990, the first head of government to give birth while in office.
Ms Ardern's close friend and colleague Finance Minister Grant Robertson said he was "thrilled and excited" and was tipping some family references would be included in the mix when the new baby was named.
"I'm so proud of her in so many ways. I see her as my friend, I have to say, having known her for a long time. This is just the most superb and wonderful news.
"And I'm really looking forward to meeting this very gorgeous little girl."
He had guessed it might be a girl but said Ms Ardern had not even let him in on the secret in advance.
"I know they've been thinking a lot about names and I'm sure there will be family connections as there always are for a lot of people in the names that come forward."
He hoped to meet the baby after the family had some time together.
Green Party co-leader James Shaw said it was an exciting time for the couple and a "historic moment for our country".
"I join with Marama and our Green Party caucus colleagues in wishing them both all the best."
A group of about 25 parents with their babies and toddlers took part in the weekly "wriggle and rhyme" class held at Mt Albert Library yesterday.
After half an hour of dancing, clapping and belting out well-known nursery rhymes, the Herald spoke to a number of parents about what advice they had for Jacinda and Clarke.
One mother said: "Take people's help when they offer it." Another said: "Take your time and mess can wait."
Another mother added, don't take "too much advice; listen to your own intuition".
"Enjoy it, you're never going to get this again."
Governor-General Dame Patsy Reddy posted a photo on Facebook, saying "A rainbow today at Government House, perhaps suggesting glad tidings are imminent".
In mid-January, Ms Ardern shocked the country when she announced, via social media, that she was expecting.
"We thought 2017 was a big year. This year, we'll join the many parents who wear two hats. I'll be PM & a mum while Clarke will be `first man of fishing' & stay-at-home dad," the couple announced.
Ms Ardern will take six weeks' maternity leave then return to work, while Mr Gayford will be the stay-at-home dad.
