News » Local News

Network nightmare

After soaking up millions, the city's IT systems still need help


Editor's note: This story has been updated to correct errors, made in the Independent's reporting, related to total city IT expenditures since 2011.

After more than $38 million in information technology expenditures in the last 3½ years, Colorado Springs Mayor Steve Bach says he needs another $2 million to make the system run — and that's just the beginning.

Though no dollar figures have been cited for future work, Bach's Chief of Staff Steve Cox says the city needs to rebuild its system "from scratch."

Last week, City Council begrudgingly voted 7-2 to allow Bach $1.1 million. But councilors expressed bewilderment over what's going on in IT, where a secret investigation in January was followed by the exodus of nine IT workers who left with $172,189 in severance pay, and the voluntary departure of its chief officer after barely a year on the job. The City Auditor's Office reported in October that the office lacked safeguards and procedures, which raised questions about data protection. A more recent audit was suspended amid the turmoil in IT. And Cox tells Council in a memo that the system has "multiple single points of failure" and has reached capacity.

All of which makes councilors question why Bach has turned not to community IT experts, but to Hogan Lovells, the international law firm that has collected $2.7 million from the city since Bach took office in June 2011.

Pre-dating Bach

City IT provides an array of services to support and maintain the city's networks, applications and uses that include email, phones, development plans, budgeting, billing, contracting and other crucial functions. It does this with 36 city employees, 12 contractors and a consulting team from Hogan Lovells, the city says.

That's 10 fewer workers than in January, when a reduction-in-force followed an investigation the city won't publicly discuss. Very little has been shared with Council, and nothing has been explained to the public.

Problems with city IT aren't new. Gene Bray, who worked in the department from 1999 to 2009 as a senior network engineer and principal network engineer, points to a Council-approved decision to replace the city's system in 2009 and 2010 that cost up to $17 million. The deal, he says, which involved one vendor, Cisco, included hardware and software that were designed to last only five years before support and replacement components would no longer be available. Other systems had been available with longer life spans, he says.

Bray also says the city's problems can be tied, at least in part, to a lack of qualified personnel — virtually all networking staff was laid off earlier this year. "The city has no one on staff who can manage the telecommunications system," he says, adding, "The city has not been staffed to take care of its IT network for years."

Deputy Chief of Staff Bret Waters disagrees, saying via email, "We have qualified staff currently that manage our telecommunications system and IT network."

But there's no denying the city has spent a lot of money on temporary IT workers: $2.3 million since 2011, and $686,014 in the first half of this year alone, records show.

But Waters admits temp workers were used in 2011 and '12 "to support the IT department" and acknowledges temps are now filling in for the nine staffers who departed earlier this year.

Bray notes that most findings of a recent City Auditor's Office review could have applied to the situation from 1999 to 2009 as well. Among them:

• City IT failed to document a city-wide data classification system to define criticality and sensitivity of certain information, leading to "risk that information that should be protected could be disclosed or modified."

• No written recovery plan for some databases existed, increasing the probability and impact of a major IT service interruption on key business functions.

• Database backups were kept at the city's data center for up to seven days before being stored off-site, increasing the risk of backup data losses if the data center were compromised.

Waters says the city has rectified all those issues. But another audit in June also found failings in internal controls. That audit report, which didn't disclose "identified vulnerabilities" in order to minimize security breaches, was suspended when the city called for major changes in its data center.

The stopgap plan

The city doesn't deny there are problems with the entire IT system. In a recent memo to Council, Cox said the $2 million sought by Bach is for "short-term remediation and support." Looking ahead, he said, the city must "begin ... redesigning and rebuilding its network" because the current system is "architecturally broken" and has experienced "a high frequency of failure."

He doesn't say in the memo how much that will cost. The system already has gobbled up nearly $8.8 million in capital spending since 2011 (excluding the 2009 upgrade), and nearly another $29.6 million for operations and smaller capital purchases.

The condition of the system recently was assessed with the help of Hogan Lovells, which has handled a variety of city legal matters since Bach took office in June 2011. As of July 7, the city had paid the firm $2,735,234 since 2011, including $751,298 so far this year, records show.

"I don't like how much has been spent on his [Bach's] favorite law firm," Councilor Andy Pico tells the Independent. "We have a lot of talent in town for IT, and I could put names together pretty quickly for who could be contracted."

Hiring lawyers to assess the city system came as a shock to Councilor Don Knight, a retired Air Force colonel who worked with IT networks during his military years. "Hogan Lovells is not a recognized computer expert," he said at the July 8 Council meeting where Council approved giving Bach $1.1 million to fix the IT system.

Cox defended the decision, saying, "They have a well-qualified group and act as eyes and ears for the mayor to be sure what we're proposing is valid." Bach wanted the firm involved, Waters says.

At the July 8 meeting, Councilor Jan Martin expressed concern about the "chaos" in IT, which she said erodes her confidence in its ability to handle projects. Councilor Joel Miller, too, said he is "not feeling a whole lot of confidence," and noted his repeated questions about IT to the administration spanning months were met with a "stone wall." He and Jill Gaebler were the councilors who voted against the $1.1 million in funding.

The department has had four CIOs since 2011: Curlie Matthews, who resigned after a run that began in 2007; Jesse James, who served as interim CIO until January 2013, when Joe Palmer was hired, and left a few months ago with a severance package; Palmer, who resigned without accepting severance; and current interim CIO Richard Valentine. Waters says the CIO job will be posted "in the future."

The Council's vote means postponing until 2015 a municipal court software program and other projects in order to fund IT. A South Academy Boulevard repaving project originally on the chopping block because of the IT funding request was not sacrificed, however.

Comments (3)

Showing 1-3 of 3

Add a comment

Add a comment

Clicky Quantcast