The decision by Karzai could further complicate negotiations between the United States and Afghanistan over the presence of Americans troops in the country once most NATO forces leave by the end of 2014.
Speaking at a news conference in Kabul, Karzai's spokesman Aimal Faizi said villagers in Wardak province had lodged a series of complaints about operations conducted by US special forces and a group of Afghans working with them.
The decision was reached at a Sunday meeting of the Afghan National Security Council, chaired by Karzai, Faizi said.
"The Ministry of Defense was assigned to make sure all US special forces are out of the province within two weeks," he said.
"After a thorough discussion, it became clear that armed individuals named as US special forces stationed in Wardak province were engaging in harassing, annoying, torturing and even murdering innocent people," Faizi added.
Sunday's announcement came days after Karzai issued a decree banning all Afghan security forces from using NATO air strikes in residential areas, in a bid to curb civilian casualties.
That was in response to an operation in Kunar targeting four Taliban members which resulted in the deaths of ten civilians, including five children, during an air strike.
Karzai has long warned his Western backers that the killing of civilians could sap support for the foreign troops in the country and fuel the insurgency.
