DCC investigates computer outage

Mick Reece.
Mick Reece.
The Dunedin City Council says lessons will be learned after a major computer malfunction left hundreds of staff unable to access core electronic systems for most of a day.

The meltdown on Thursday morning took out one of three disk storage systems located in a secure room inside the council's Civic Centre building, which together held about 75% of council information.

Without it, staff were unable to access financial, document management or other systems, send or receive emails, or process receipts electronically.

It was the second significant malfunction involving the council's information technology systems this year, with old equipment blamed for the earlier incident, council network services manager Tim Oaks said yesterday.

The outage in the early hours of Thursday did not result in data being lost, but did cause disruption, he said.

That began as staff arrived for work and discovered some systems were inoperable, and continued until they began to be restored from mid-afternoon, he said.

The cause was "probably a bug" within the $60,000 storage system's firmware, which was diagnosed after the manufacturer - which he would not name - sent a representative from Christchurch to investigate.

Data logs had also been sent overseas for analysis in either the United States or India, with results expected in a few days, he said.

"We'll wait for the official diagnosis. It's like sending blood samples to the lab."

At one stage, up to 2000 emails were queued on the council's server, waiting to be sent once systems were restored, and other incoming emails would also have been delayed, he said.

"We had a big flood of emails coming in yesterday afternoon because of that."

Other systems, including the council's website, internal intranet, and access to the council's rating and geographic information continued unaffected.

Council financial controller Maree Clarke said the malfunction occurred on a "crucial" day in the council's monthly financial reporting cycle, and staff were "playing catch-up" yesterday as a result.

Accounts payable was also disrupted, with no invoices or payments able to be processed, and customer services agency staff had to manually process receipts yesterday, she said.

"We've managed quite well, really. It's just put everyone a little bit behind."

Luckily, the glitch had not occurred during payroll, meaning payments to staff - and their subsequent personal payments, such as mortgages - were not affected, she said.

"That would have been far more serious."

Council community and recreation services manager Mick Reece said the disruption was "difficult" for staff but he was impressed with the way it had been resolved.

"It was nice to see people talking to each other and actually using pens ... it just goes to prove how essential particularly email is."

Council finance and corporate support general manager Athol Stephens said there would be an impact on staff productivity, but it was difficult to calculate what the cost would be.

Mr Oaks said up to 25 staff had been "all hands to the pump" dealing with the malfunction.

Council staff had already begun a debrief, looking for possible improvements that could be identified, he said.

"We're already looking at that."

A firmware update could follow soon, once the analysis of the malfunction was complete, he said.

"We're going to need to know what the diagnosis of the root cause is before we do the permanent fix."

- chris.morris@odt.co.nz

 

Add a Comment