I like how the construction crew waited for my trip to NYC to cut the lines.

Accidental fiber cuts caused by construction workers took out telecommunications service for more than 750,000 customers in the New York City area yesterday.

There’s a technical term for this: fiber-seeking backhoe.

The fiber cuts hit the network of Level 3, an Internet backbone provider, and lasted for hours before being fixed. Problems hit several states: customer reports on DownDetector indicate that outages primarily affected Time Warner Cable (TWC) in New York and Cox Communications in large parts of Connecticut and Rhode Island and small parts of Massachusetts. Level 3’s network serves both TWC and Cox.

Think about this for a moment – one local event impacted people and businesses in 4 states. Remember, Level 3 is an Internet backbone provider. More than cable TV runs over their infrastructure.

The most specific outage numbers came from New York. The New York Department of Public Service (NYDPS) issued a statement saying that “more than 750,000 customers in the New York City area were unable to complete telephone calls.” Most or all of those customers are apparently Time Warner Cable users. Internet and TV service was also affected.

Level 3 confirmed the outage, telling CNN and other media outlets, “Our network is experiencing service disruptions affecting some of our customers with operations in the Northeastern United States due to a fiber cut caused by third-party construction. Our technicians are on site and working to restore service.” Time Warner Cable said the outage was caused by “multiple fiber cuts at one of our network providers.”

The NYDPS statement noted that Level 3 provides service to both TWC and Verizon in New York. But Verizon’s network did not suffer any problems related to the Level 3 trouble yesterday, a Verizon spokesperson told Ars.

Source: Big TWC outage: Fiber cuts take out service for 750,000 in NYC area | Ars Technica

Back in my Network Manager days my team and I spent a lot of effort making as certain as possible our major links – primarily our backbone and Internet connections – were truly redundant and diverse. Not only would we rarely rely upon a single provider (and Level 3 was one of those) but we would require geographic diversity as well.

For example, when I procured redundant backbone connectivity for a co-location center in Detroit, one circuit came from the East around Lake Erie and the other came from Chicago in the West.

In order to achieve this I worked hard on the contract language to place my employer in the best position possible while my engineers made sure the providers understood and deployed what we ordered. Even then, you never know when some rerouting might occur where once diverse paths now traverse a single MUX in an out-of-the-way unstaffed switching station.

Setting aside such edge cases, it is not only possible but the responsibility of an organization to make sure there’s as much Redundancy, Diversity, Reliability, Depth, and Simplicity (RDRDS) in the environment as practical without breaking the bank.

LinkedIn Tumblr