State paid $22K each for Internet routers -

[John]

... twenty four million dollars - and then they spent another million to modify them to work like the ones they could have bought for a few hundred in the first place.

It’s like buying a Ferrari and then paying extra to have it “customized” into a grocery cart.


From today's Charleston Gazette:

CHARLESTON, W.Va. -- Nobody told Hurricane librarian Rebecca Elliot that the $22,600 Internet router in the branch library's storage closet was powerful enough to serve an entire college campus.


Nobody told Elliot how much the router cost or who paid for it. Workers just showed up and installed the device. They left behind no instructions, no user manual.


The high-end router serves four public computer terminals at the small library in Putnam County.


"I don't know much about those kinds of things," Elliot said last week, before politely leaving to help an elderly patron select books. "I just work here."


The state of West Virginia is using $24 million in federal economic stimulus money to put high-powered Internet computer routers in small libraries, elementary schools and health clinics, even though the pricey equipment is designed to serve major research universities, medical centers and large corporations, a Gazette-Mail investigation has found.
The state purchased 1,064 routers two years ago, after receiving a $126 million federal stimulus grant to expand high-speed Internet across West Virginia.


The Cisco 3945 series routers, which cost $22,600 each, are built to serve "tens of thousands" of users or device connections, according to a Cisco sales agent. The routers are designed to serve a minimum of 500 users.


Yet state broadband project officials directed the installation of the stimulus-funded Cisco routers in West Virginia schools with fewer than a dozen computers and libraries that have only a single terminal for patrons.

Read the article at: http://www.wvgazette.com/News/201205050057?page=2&build=cache

[LFE]

Why do I always seem to miss the easy RFP's???

[cindylu]

Couldn't tell if the cost included labor/installation. I do know routers ain't cheap.

[Tony]

A well written article. But what's missing is: Who actually wrote the requirements, and what was their reasoning? Surely not this guy that apparently thinks they can take the unneeded T1 cards and use them for other things.

Gianato said the T1 cards have other uses -- video conferencing, wireless Internet and "voice over Internet protocol."
"I'm not an expert on the technical side," he said, "but these have all kinds of capabilities and applications."

[cindylu]

Good point Tony. Wonder if the person who wrote the article was tech savy to ask that question.

[chipgallo]

How can we get a copy of the original RFP and award documentation? I'd love to see the deployment schedule and maintenance planning.

The WV Office of Technology has some information online, but I couldn't find the exact stimulus spending documentation. It would be nice if they collected the IT procurement info here, but maybe WV has that all in one place elsewhere.

[androsyn]

Here's the thread where the reporter is asking for advice on the story... https://supportforums.cisco.com/thread/2146460

Also these devices can in addition to acting as simple router/gateways they can take the place of a few other devices that would typically need to be placed at a site. There are also some advantages to have identical hardware deployed to multiple sites if they are going to be managed remotely. Also the T1 cards..they are likely providing both internet access AND voice circuits for the buildings in question. Did these devices also replace old PBXes in schools and libraries as well? I'd sure like to see the requirements and what they are in fact actually doing with them.

[chipgallo]

Here's the Cisco link for this equipment. Note that WV bought five years of maintenance with these so that would be in the quoted price. I wonder if Verizon has offered them as managed devices? Otherwise, who will be updating the onboard operating system over the five years?

[androsyn]

Typically devices like these are centrally managed, not sure if Verizon would be doing that or if they are being managed by somebody with the state? Of course devices like these you wouldn't leave manuals with a librarian for, thus I suspect they're being managed remotely.

[androsyn]

http://www.broadband.wv.gov/Projects/Pages/BTOP.aspx <-- I believe they're related to this project. If the goal is to get 45mb/s in..they're starting to look a bit more appropriate and likely won't need to be replaced for 5-10 years.

[John]

The Gazette continued the story with an article today that includes this photo showing some of the 360 routers still boxed up and in storage - making management for the first two years of the five year contract pretty easy:

Router2_I120506162755.jpg

The second installment of the story is at

Internet routers have sat unused for nearly two years

[LazerFlash]

Any way you spin it, this is a sweet deal for Cisco and Verizon, and a waste of stimulus money in WV...

[derricksonb]

The purchase of maintenance contracts is a pretty standard deal for most government acquisition contracts as is installation and monitiring and generally tends to inflate the initial costs as well as raises the eyebrows of those unfamiliar with the acquisition process.

The Gazette continued the story with an article today that includes this photo showing some of the 360 routers still boxed up and in storage - making management for the first two years of the five year contract pretty easy:

Not sure who the integrator was for this project, but if that equipment has been sitting in a warehouse collecting dust for 2 years then this project was very poorly managed. All of our contracts stipulate that GFE is to be purchased and installed no earlier than 90 days prior to System Acceptance Testing. This ensures that we have the current revisions of software and also to make sure that the equipment has not gone end of life, end of sale or end of support by the manufacturer.

The 39xx series is a decent platform that will more than handle any capabilities that the State of West Virginia can throw at it, but I would hardly call it a "High Speed Router," and I wonder who made the determination to purchase 1,064 of them???? Do we really need 20(+/-) routers for each county to provide "High Speed" data to libraries, medical facilities and college campuses when most of them already have sufficient bandwidth? This reeks of Fraud, Waste and Abuse and I would not be surprised if the US Congress comes back and asks for an audit of the Stimulus Funds.

[Tony]

I guess we lose what we don't use if it's a grant.? So there's some encouragement to spend everything you can, quickly. But I agree they should take the money to Frontier (et al) and just hire them to run FIOS to the whole state. Buying routers for anyone is a different subject entirely.

[Kensey]

I guess we lose what we don't use if it's a grant.? So there's some encouragement to spend everything you can, quickly. But I agree they should take the money to Frontier (et al) and just hire them to run FIOS to the whole state. Buying routers for anyone is a different subject entirely.

Frontier is never going to run fiber in any way meaningful to most residents of Jefferson County. Frontier fiber is only going to new construction in the county, people like you or I will never see it on existing structures. They've said as much, in so many words.

[McMansion]

Frontier is never going to run fiber in any way meaningful to most residents of Jefferson County. Frontier fiber is only going to new construction in the county, people like you or I will never see it on existing structures. They've said as much, in so many words.

I work for Frontier in Martinsburg and can say that you aren't correct (and TRUST me, I'm no defender of the company). First, I just for everyone's benefit, we don't run fiber to homes anywhere (at least in wv). But, yes, if a new development or commercial area is going in, we will usually run fiber to the terminal in the area, and then copper from there. Also, fiber is usually going to older terminals as they are replaced and we want to bump up the capacity. So, even "old" homes are being served by copper.

But, we are certainly in a dying industry, that's for sure...

Sent from my 'effin Kindle Fire

[McMansion]

Looks pretty darn easy to maintain such a complex network to me!

Sent from my 'effin Kindle Fire

[Kensey]

I work for Frontier in Martinsburg and can say that you aren't correct (and TRUST me, I'm no defender of the company). First, I just for everyone's benefit, we don't run fiber to homes anywhere (at least in wv). But, yes, if a new development or commercial area is going in, we will usually run fiber to the terminal in the area, and then copper from there. Also, fiber is usually going to older terminals as they are replaced and we want to bump up the capacity. So, even "old" homes are being served by copper.

That's what I mean by "in any way meaningful to most". If all I have is copper for that last mile, it doesn't matter if it's backed by fiber or high-speed copper or a Magic Bit Bucket Brigade -- I'll never see more than a few Mbps at most, as opposed to the dozens or hundreds of Mbps possible with fiber-to-the-home. When people say they want "fiber" they mean something like FiOS right to the home, and Frontier is never, ever going to provide that unless there is a radical shift in strategy.

And actually I had thought from what I heard Frontier officials say previously, that there was fiber-to-the-premises in new construction. Not even that, eh?

I have read elsewhere that Frontier's tacit strategy is to run the bare minimum infrastructure to an area to qualify for government Universal Service Fee and/or broadband subsidies. No idea if it's true or not, but that would certainly fit the observed corporate behavior.

When you refer to "terminal", are we talking what would be called "fiber to the node" (>300 m from premises) or "fiber to the curb" (300m or less)?

[McMansion]

That's what I mean by "in any way meaningful to most". If all I have is copper for that last mile, it doesn't matter if it's backed by fiber or high-speed copper or a Magic Bit Bucket Brigade -- I'll never see more than a few Mbps at most, as opposed to the dozens or hundreds of Mbps possible with fiber-to-the-home. When people say they want "fiber" they mean something like FiOS right to the home, and Frontier is never, ever going to provide that unless there is a radical shift in strategy.

And actually I had thought from what I heard Frontier officials say previously, that there was fiber-to-the-premises in new construction. Not even that, eh?

I have read elsewhere that Frontier's tacit strategy is to run the bare minimum infrastructure to an area to qualify for government Universal Service Fee and/or broadband subsidies. No idea if it's true or not, but that would certainly fit the observed corporate behavior.

When you refer to "terminal", are we talking what would be called "fiber to the node" (>300 m from premises) or "fiber to the curb" (300m or less)?

Well, both FTTN and FTTC (mostly N). The vast majority of new remote terminals that go in are new DSLAMS to deliver traditional ADSL (1-3Meg). And we certainly do deliver fiber to the prem, usually for federal facilities or large buildings, cell sites, even. Frontier does have a few territories that have FTTH/FioS -- but that sounds a hell of a lot more fun than it actually is for the customers. It's complicated, but is just like everything else in the company: it cost's us more money to run than we get in revenue. And, as far as the USF stuff goes, we're hosed because the FCC is going to revamp that puppy a little more in the next FY and we'll be playing catch up to meet those requirements -but i'm no expert on that. After the Verizon sale, Frontier has a ton of different kinds of infrastructure all over the country and (hopefully) we can learn more about what works to best compete in the long term.

[LFE]

The fiber service to our neighborhood has been pretty terrible. Chopped & dropped calls, long outages etc.
The service guys seem rightfully frustrated as no good testing equipment seems to exist for trouble shooting.
- The service guys have been helpful and even provided person cell numbers allowing them to confirm back to dispatch that we do indeed have fiber here.