Mr Manning (80), a retired accountant and former chief executive of the Otago Council, lives at Alexandra, and was commenting during a visit to Dunedin.
''We can't carry on the way we are at the moment. We have to make some adjustments,'' he said.
The latest annual Statistics New Zealand business demography figures show Dunedin has lost a net 3450 full and part-time jobs in the seven years to February this year.
This was a drop of about 6% to 52,890 full and part-time jobs, and partly reflected a loss of manufacturing work, he said.
By contrast, inland districts in Otago had made ''notable'' gains over the period since the 2006 census. Central Otago was up 1370 jobs (16%), and Queenstown Lakes District up 2110 jobs (14%). Clutha was up 340 jobs (6%) and Waitaki up 420 (5%).
Much of New Zealand's overall job growth during the period had happened in Auckland, and important differences were also emerging within regions.
This was shown in the ''most spectacular'' way in inland Otago, where further recent wealthy lifestyle settlement had been part of the inland growth trend, along with international tourism and wine-growing.
Dunedin also faced other big challenges, including a plateauing in the University of Otago's domestic student roll. More should be done to boost the university's international student numbers, he said.
Dunedin should do more to reconnect itself with economically-thriving areas elsewhere in Otago, and he would like to see a regional development conference held in the city to help re-establish those links and boost planning.
But Dunedin had many strengths and he believed more could be done to attract a new wave of settlers to the city, also drawn by the lifestyle, but in a slightly different way from the inland Otago lifestyle settlers.
Such settlers included baby-boomers living elsewhere in New Zealand who were looking for a place to retire to and could be lured by the city's ''cultural energy'' and its ''rich array of cultural amenities, such as museums and art galleries''.
Such new settlers, including many nearing or in retirement, would bring ''fresh wealth and savings into the community''.
Otago Chamber of Commerce chief executive John Christie said Mr Manning was a highly experienced observer of trends in Dunedin and Otago, and he agreed the city faced big challenges.
There was no room for complacency, but, despite the recent loss of manufacturing jobs, Dunedin had many strengths, and the city's updated economic development strategy highlighted a constructive way forward, Mr Christie said.
The chamber was working collaboratively on related economic development initiatives, including Project Shanghai, to maximise benefits from Dunedin's sister-city links with that Chinese city.
There was scope to further promote the city's lifestyle appeal, and building closer ties with other parts of the region, as suggested in the economic strategy, would also be beneficial, he said.
Sophie Barker, a Dunedin City Council business development adviser, visitor industry, said the city's vibrant cultural life was a key part of the city's appeal, and was also generating jobs.
Dunedin's economic development strategy also highlighted several of the points Mr Manning was making, she said.
