Police stopped and arrested the attacker in the town of Wuerzburg by firing a bullet through his thigh. They said his injuries were not life-threatening.
"Three are dead and five very seriously injured," regional Interior Minister Joachim Herrmann said after arriving on the scene. "With the most seriously injured we are not sure they will survive."
The man had lived in Wuerzburg since 2015, the year Germany opened its borders to more than a million migrants and refugees fleeing war and poverty. His life was not in danger and he was being questioned by police in hospital, Herrmann said.
"His condition had been noticed in recent months, including violent tendencies, and a few days ago he was put into compulsory psychiatric treatment," Herrmann said.
Later, he told public television that, according to one witness, the suspect had called "Allahu akbar", an Arabic phrase meaning "God is great" and often associated with acts of Islamist terrorism before starting his spree.
"That suggests a possible Islamist motive, and that is also part of the investigation," he added.
Among the three dead was a young boy and one of his parents, Main Post newspaper said.
As is standard practice in Germany, police did not release the name of the suspect.
Police said there was no indication that there were any other attackers, and that the situation was now under control.
Videos posted on social media showed a seemingly barefoot man holding a long knife being warded off by other men with chairs until police arrived. Another video appeared to show blood on the ground.
The videos matched the reported location of the attacks on and around Wuerzburg's central Barbarossaplatz, though it was not immediately possible to confirm when they had been made.
Reuters footage from the scene showed dozens of police and emergency vehicles in attendance.
Wuerzburg, an ancient city of 130,000 people some 100km south-east of Frankfurt, was five years ago the scene of a knife attack on a train by a 17-year-old Pakistani asylum seeker in which five people were injured, two seriously.
