The Invercargill man lost his wife suddenly to an aneurism last August, leaving him to raise three children.
For his audition piece, which aired on Sunday, he chose to sing Tim McGraw's You Get Used to Somebody - a song he identifies with after losing Annette. His children, Jack, 19, Harry, 14, and Maggie, 11, stood in the wings.
"When you're hurting, sometimes you just want people to know how you feel and I guess that was my medium to get that across, subconsciously," he told the Herald.
"Lots of people, I have no idea who they are [have contacted me]. It's quite random to have people make the effort to make contact with you."
On Sunday night's show, the judges were impressed with his performance - the last of the night - with all of them showing tears.
Stan Walker asked Mr Townley why he chose that song. His voice cracked as he told the judges what had happened and what the song meant to him before pulling out a photo of his late wife and showing his "baby" to the audience. His children joined him on the stage and the judges gave him a unanimous "yes".
The 46-year-old hotel manager gathered with family and friends in one of his conference rooms to watch Sunday night's show.
Mr Townley said he was apprehensive about how his moment was going to be edited and how he would be portrayed but he was pleased.
"Right or wrong, that very much was a family moment and I'm very proud of that ... the kids are reasonably proud of their old dad, how cool is that?" he said.
"What people have been telling me has all been supportive, using words like 'inspirational', 'you're being strong' and 'great singing' and 'well done for getting up there and doing it with your family'. It's just really nice to think people who you don't know can take that time to send you a message."
Many of the people who have contacted Mr Townley have lost someone of their own.
He last saw his wife after dropping her at a morning tea after a funeral - he told her he'd see her in two hours. They had been together 27 years, since they were 19.
"It's just your worst nightmare."
Mr Townley said auditioning for The X Factor had been a positive experience."It's been so surreal, but also so humbling."
According to last night's ratings from Nielsen, the TV3 show pulled in an audience of about 454,000 - not enough to topple reality rival TV One's MasterChef, which drew more than 508,000 viewers.
