Labour MP Grant Robertson has said he will introduce a member's Bill to amend the Holidays Act and address the "unfair" situation.
Mr Robertson said there were 11 public holidays a year, but when Waitangi Day and Anzac Day fall on a weekend - as they do this year - they should be able to be taken on a weekday.
Waitangi Day, February 6, falls on a Sunday this year and Anzac Day, April 25, coincides with Easter Monday.
"I will be introducing a private member's Bill to make sure workers get the break they deserve," he said.
However, Mr Key told reporters that it was not something he was considering and while he did not rule it out altogether he could see problems with it.
"Certainly not this year. It's a bit self-serving for the Labour Party to be saying that it's a great idea. They had nine years in office, they faced exactly the same set of conditions we did and they did nothing about it."
He said the move could have substantial costs and there were other factors to consider.
"If you just take Anzac Day, that is the day we commemorate and pay respects to those who fought for freedom and democracy in this country. Are we really just saying that we are going to ignore that day and go to any other Monday?"
He said there were also issues around pay and entitlements if holidays were transferred.
Christmas and New Year holidays are celebrated on the actual day with holidays transferred to a week day when they fell on the weekend.
Mr Robertson said in Australia Anzac Day was commemorated on the actual day but there was always a public holiday on a Monday when the 25th fell during a weekend.
"Waitangi Day and Anzac Day are the only holidays that are not protected under the Act. Given that the situation only arises two out of every seven years we believe that it will be relatively easy for businesses to absorb the costs and in doing so ensure that workers get a fair go."
