The Government is stepping in to take over management of the Auckland prison today.
Serco staff will remain on site but a Corrections Department prison director and team will oversee the prison.
Mr Key told TV3's Paul Henry programme today the Government did have the power to cancel Serco's contract and it would "absolutely" take that measure if an investigation found "they can't do their job".
The prison hit headlines this month after revelations of fight clubs, under-staffing, and inmates with drugs and homemade alcohol surfaced with footage filmed on contraband cellphones.
It has sparked a multi-agency investigation.
Responding to questioning that it could be difficult to cancel the contract with Serco, Mr Key said: "I don't think they're [Serco] enormously powerful, you've got to honour the contract and things are always relative.
"The broad legal advice I have without delving too much into it is, 'Yes, you can cancel the contract.' There's a range of options available ... if they can't do their job."
He denied that Serco hung Corrections Minister Peseta Sam Lotu-Iiga out to dry, but said "one day, maybe" when asked if he would give former Corrections Minister Judith Collins a call.
Mr Key said it would have been a "difficult time" for Mr Lotu-Iiga.
"My major concern would be are [Serco] keeping the Ministry of Corrections informed? I do [think Lotu-Iiga did a good job] but ... he can only work on the information he's got.
"He hasn't sat there and done nothing but the question always always is when these things get momentum of their own sometimes they're just a difficult time."
The Corrections investigations will be completed by the end of next month and phase two by the end of September.
Mr Lotu-Iiga told Radio New Zealand that he was meeting Corrections chief executive Ray Smith this morning.
"I will be getting an up-to-date report about how Serco is running today when we meet," he said.
Mr Smith told the broadcaster he would sign off about half a million dollars worth of penalties to Serco this morning - an amount which could increase in the coming days.
"We haven't worked our way through all of that yet, we need to review a lot of the footage and get their understanding of some of the incidents but I would expect more financial penalties to follow.
"We're invoking a very important part of the contract ... there has been a number of penalties over the years but going in and starting to take over leadership of the place while we try to sort things out is a big step."
