Utility outages can be a pain and we’re bound to see them over time, especially when severe weather conditions are involved, like this weeks devastating results of Superstorm Sandy.  Even worse, what happens when the problem is more than a down telephone line or cell tower, say when an entire central office is rendered useless.  That’s where the AT&T Network Disaster Recovery (NDR) Team steps in.  AT&T’s NDR’s goal is to reestablish communication access in affected areas and get the region running at its normal condition while repairing or rebuilding a facility.  We’re not just talking a momentary drop in service, the NDR’s goal is replace a physical building. In the case of New York, over 10 years ago, it was to replace a center that was damaged when the World Trade Center towers collapsed.

AT&T recently invited us to visit the Network Disaster Recovery Team where they were in town to run through training exercises with the dedicated national team as well as local AT&T members.  The setup was impressive.  The NDR Team set up dozens of trailers to provide everything from electricity, telephone, data, to U-Verse connectivity.

Sr. Technical Specialist, Kelly Morrison, gave us a thorough tour through AT&T’s NDR high tech camp.  We climbed in and out through giant containers, we were shown fiber-optic transport systems, switch trailers, generators, even a hardware store and machine shop.  As we entered each container, Morrison described in meticulous detail about the function of each trailer and the calculated number of trailers each disaster would require.  As it was explained to me, AT&T central offices deliver all AT&T services in a region, and if they were to be become damaged or destroyed it would take months to years to rebuild these facilities.  So it is important to be able to have the NDR team available to provide services even if the facilities are decimated.  The NDR is a long-term investment that AT&T has built up in order to maintain services.

AT&T has thus far invested $600 million in to the NDR in its 20-year history.  The NDR has over 320 pieces of equipment (including 90 trailers) stored throughout multiple sites that can be deployed anywhere in the United States and international sites (for enterprise services) in a matter of moments.  This particular exercise in Los Angeles, is AT&T’s 67th such exercise, is an opportunity to test out new equipment as well as work with local response teams to prepare and train for any possible emergency.  If you were wondering, AT&T’s NDR team practices these exercises about four times a year.  NDR continuously tinkers and tests new equipment, often times the equipment on these trailers contain the latest technology equipment, and thus needs to be tested before being implemented in real-life situations.

The NDR Team can set up infrastructure between 15 minutes to 24 hours, depending on what a situation requires.  This team can supply a region with support in the event a local team’s infrastructure is in such a state that they cannot deliver service.  The NDR is only called in after a disaster has passed, the local team monitors the situation and elevates requests for assistance through AT&T’s Global Network Operations Center (GNOC).  So, even if the equipment could provide services for large sporting or media events, its dedicated purpose is for emergency use only.

The NDR can bring in smaller trucks to help provide services within 15 minutes of arriving on scene.  These trucks can pipe into existing hardwire or microwave connections, but this provides additional support in a short matter of time.

The NDR can deploy Satellite (Cell on Light Trucks) COLTs to deliver phone and data services over dedicated satellite space to provide services when necessary.  AT&T has committed data feeds, though costly, ensures they have committed connectivity and do not have to scramble to acquire the capacity when it is most necessary.

In the event the work site is without power, the power distribution trailer provides electricity to the NDR team. A 500KW generator feeds electricity to the camp, mediating through a series of back up batteries, so even in the event multiple failures, the batteries would maintain power until the generators kick in.  In fact, the NDR packs a back-up generator along with its regular power distribution trailer, in the event those have a lapse in operation.  Talk about being prepared.

The NDR’s functions much like command and control structure found in military units.  This makes it easy and clear communication through the team and along side other teams.   The command center coordinates every aspect of the NDR’s movements and even travels with a finance team to coordinate the logistics of the NDR.

In most cases the NDR Team has to assume the site they are working from will lack reliable power and will not have all the necessary tools they will need to service the region.  So it is imperative that the NDR supply themselves with all necessary tools including generators, fuel, tools, food and even lodging.

The Hazardous Materials, search and rescue and other emergency services teams take part in these exercises as well.  The Hazardous Materials team trains technicians on how put on the proper safety attire to get access in to areas to service AT&T’s infrastructure.  The particular suit we saw in demonstration was a Level A, haz-mat suit, designed for the most severe situations.  These suits are worn in conjunction with oxygen tanks and it is not easy for technicians to work in these suits and in the short amount of time they have before their tanks are depleted.

I had the pleasure of meeting Rugby and Ben (and their humans Dennis and Eric).  AT&T works with local emergency services team and the Search Dog Foundation to provide communication tools in these emergency situations.

Final Thoughts

As much as I was enthralled by the scale and number of gadgets, I was even more impressed by the passion and dedication each member of the NDR had to restoring service.  As Morrison stated, their primary goal is to establish a temporary center while the inoperable permanent facility can be repaired or replaced, but ultimately the goal is to allow people to connect.  We met a variety of mangers, technicians, engineers and disaster relief specialists, but each one of these NDR members drove the point of helping people get in touch with their loved ones.

The NDR is an impressive collection of technical machinery and, even more importantly, dedicated team that restores phone, data, wireless capability at a moment’s notice.  It is comforting to know that these fine folks are behind the helm at AT&T’s NDR Team whether after a hurricane, forrest fire, tornado, earthquake or any disaster.