Battery failure hits city of Bend phones

Published 4:00 am Friday, February 26, 2010

An incorrectly installed battery system was to blame for a power failure that knocked out the servers that manage phone and Internet service at Bend City Hall for much of the day Wednesday.

City spokesman Justin Finestone said employees discovered the phones and Internet had gone down around 9:15 a.m., and that most service was restored by 2 p.m. Phone capacity was limited until the end of the day Wednesday, he said, with some calls apparently not getting through to city phone numbers, but all service was restored by Thursday morning.

Because the city’s phone system runs on a VOIP or Voice-Over Internet Protocol system, it’s powered by the same server system that handles city employees’ Internet use and stores city documents.

Steve Meyers, the city’s information technology director, said the origins of Wednesday’s problem track back to around three years ago, when the city upgraded its backup power supply.

In order to avoid problems during a power outage, City Hall servers are not powered on the same circuits powering the lights and electrical appliances, but on a circuit that draws current from the power grid to charge a bank of batteries. Power from the batteries is converted to an AC current that can be used by the servers for around 30 minutes.

Meyers said three years ago, the city installed additional batteries to double the capacity of the backup system, in order to accommodate the growing number of phones at City Hall and the increased power demands of the network.

However, the local contractor who performed the upgrade — Meyers said he could not recall which company did the work — installed the new equipment incorrectly. Instead of doubling the capacity of the system, the new batteries doubled the amount of time the batteries could power the servers — 60 minutes of power instead of 30 minutes.

The backup system functions like a breaker, limiting how much power the network can draw without overloading and shutting down.

“As we gradually added more equipment, we exceeded the maximum voltage it was capable of in the way it was configured — it won’t shut down until it’s 130 percent over maximum for 10 minutes,” he said. “We were going along and not aware of this, and eventually, enough things were added that it just shut down.”

Meyers said City Hall experienced a similar outage about three weeks ago, but employees were able to repair the problem without determining the cause. Wednesday, the IT staff contacted the manufacturers of the backup system, Meyers said, and believe they have reprogrammed the equipment to work as intended.

For now, they’ve bypassed the system and are operating the city’s network on current drawn directly from the power grid. A company recommended to the city by BendBroadband and Cascade Healthcare Community will be inspecting the equipment in the coming days to determine whether any changes need to be made before the backup system is reconnected.

Phone and Internet service at the city’s Public Works Department and Bend Police were not affected by the outage.

Meyers said one piece of equipment was damaged, but it is under warranty, and the city has borrowed a replacement from BendBroadband until the new equipment arrives. He said it’s doubtful the city will take any action against the contractor responsible for installing the additional batteries and equipment improperly, as the outage caused no clear monetary loss.

Finestone said most city employees were able to keep working during the outage, and apologized for any difficulty people may have had reaching city offices.

“It was an inconvenience, but it was only one day, and we’re all back to normal today,” he said.

Marketplace