Mr Joyce said the Ministry of Education's Acting Secretary Peter Hughes had stopped all debt collection and was reviewing the policy.
Debt collection agency Baycorp has about 200 education-related debts, with seven of those relating to Novopay since they took over the contract last August, a spokesman for Mr Joyce's office said.
The other debts related to the previous pay provider for the Ministry of Education.
Teachers have slammed the the debt collection as heavy handed and hypocritical, with some still waiting for the wages they failed to receive as a result of the error-ridden Novopay system.
A west Auckland primary teacher described how she was being chased for $22.78.
"I was accidentally paid $1840 by Novopay in October, although I hadn't worked for 10 months at that time," said teacher Kelly Clarke, who is on maternity leave.
"I immediately realised I had been overpaid and tried to find out how I could pay it back. No one knew how to go about this.
"At the end of January, Novopay requested that I pay the money back. I repaid $1840 - the net pay that I had received. Since then I have received several requests that I repay a $22.78 [union] contribution which Novopay automatically made on my behalf when they paid me. I have made no effort to do so, since I never received that money ...
"Yesterday morning, a courier arrived and woke up my household. The delivery was a letter from Novopay informing me that debt collectors would be employed to take control of my $22.78 debt."
After she complained, Novopay representatives said they would put debt collection on hold while they looked into the case.
The Novopay payroll system, run by Australian company Talent2 and introduced six months ago, has been plagued by problems, and teachers and support staff have been underpaid, overpaid or not paid at all.
Ian Leckie, immediate past president of the NZ Educational Institute, said the collection action was "extremely heavy-handed".
He understood debt collectors were approaching teachers while they were working, and the visits were the first they knew about any overpayment.
He said the ministry owed money to his school, which had paid $45,000 in advance to teachers who had been victims of the Novopay system.
"But our school certainly isn't sending debt collectors around to the ministry for what we had to pay out."
