Speaking at the New Zealand High Commission's residence afterwards, Mr Key said it was a sombre mood as dignitaries filed past the body of the former President of South Africa.
He said the memorial service the day before had been more of a celebration with music and dancing, but "today was different".
Mr Key said Mandela's body was covered to the chest with his head and face on view, covered by a Perspex box.
He said the world leaders had been quiet as they filed past.
"It was very sad. I took a moment," Mr Key said. He said he had never met Mandela. Asked if he felt it was unusual that this was the first time he had been in Mandela's presence, Mr Key said: "In a way."
"There is a massive connection between Mandela and New Zealand and South Africa and New Zealand. We have 50,000 South Africans living in New Zealand."
Mr Key said that Mandela would be remembered for two things - the fight against apartheid and finding the direction of the country.
"Mr Mandela was a remarkable human being. Coming here was the correct thing to do."
Asked about sections of the crowd at the memorial service booing South African President Jacob Zuma, Mr Key said he had been surprised to see and hear it.
"It speaks to the corruption allegations [against him]."
But he said it appeared the leadership of the ruling African National Congress was a one-horse race and that Mr Zuma was that leader.
Mr Key said it had been good chatting to dignitaries as they waited to view Mandela's remains.
"(Former Commonwealth Secretary-General) Don McKinnon is a maestro at working a room and he introduced me to lots of people."
Mr Key said there were opportunities for New Zealand to forge links with Africa through agriculture.
He said it had also been good to meet Irish rock star Bono from U2.
"Bono loves New Zealand and has great memories of his tours here."
The highlight of the time in South Africa had been United States President Barack Obama's speech at the memorial service and "being here" as it was a point in history, he said.
Accompanying Mr Key as part of the delegation was former Prime Minister Jim Bolger, who said it had been sad to see Mandela there but he looked at peace.
Mr Bolger said he had enjoyed the occasion "as much as one can enjoy farewelling a friend."
He said that as he was waiting to view Mandela's body at the Union Buildings, he had reflected on the last time he had been in the same building, which had been when Mandela had been inaugurated in 1994.
"In a way this was completing the journey," he said.
One fond memory he had of Mandela was a phone call he had received from him when he was Prime Minister wanting to discuss education.
"I remember it because it was the wrong hour of the day - it was early morning."
But the moment that stood out for above everything else was Mandela "memorably striding on to the field" wearing a No.6 Springbok jersey before the final of the 1995 Rugby World Cup which South Africa won.
"I remember thinking that if this helps South Africa to come together then that's fine."
- Andrew Austin of APN in Pretoria
