After 33 years in local body politics - and 12 years in the top job - choosing to withdraw from his bid for a fifth term as mayor was one of the hardest decisions he had made.
"I've had a wonderful experience, but I've been contemplating for about a month or two if I would stand [again]."
Mr Hayes (66) wants to spend more time with wife Diane and also concentrate on his forestry interests.
"I've had 48 years in sawmilling. With a new sawmill at Milburn, I want to preserve what assets I have up there ... and one word I have difficulty with is the word 'I'."
The Clutha District Council was an organisation that operated with unity of thought, where everyone was dedicated to achieving the best for the community, and it would be "unethical" to stand for the mayoralty again, he said.
"I haven't changed my thinking just because there has been a [mayoral] challenge."
There was a time to go, and he felt this was the right time.
Deputy mayor Hamish Anderson, who last week announced his intention to stand for mayor, said Mr Hayes' decision came as a surprise.
Mr Anderson (49), who has been a councillor for nine years and deputy mayor for the past six, said he had to "pay tribute" to Mr Hayes.
"He's done a lot of good for the district - hence my decision to stand was so difficult.
"He's obviously considered his reasons, and good on him.
"The sad thing at this point in time is we don't have a contest."
Mr Anderson and Mr Hayes both said the office of mayor should be determined by the public.
Mr Hayes, who was elected unopposed last term, said he hoped someone would stand against Mr Anderson.
"To be meaningful, it requires a mandate from the community," he said.
When asked if he endorsed Mr Anderson as mayor, Mr Hayes said "he has a lot of experience".
Mr Hayes said he did not have the words to describe what his time on the council meant to him.
"I will miss it like hell."