Three AT&T employees clad in heavy, bright orange hazmat suits cautiously descend a dimly lit stairwell. Flashlights shine around each corner and against the walls as the Network Disaster Recovery (NDR) team makes its way down two flights of stairs and through an empty hallway, leaving small plastic discs on the ground so they can find their way back through a potentially hazardous situation.
It’s a sunny October afternoon in New York, but for the sake of this exercise, which AT&T carries out three or four times per year, we may as well be in the middle of a hurricane.
Our destination is a quiet, well lit room usually off limits to the public. Humming AT&T servers and walls of back-up batteries serve as the stand-in for an actual disaster area. The team picks an area and scans for danger using thermal and gas sensors. After each area is cleared they quickly huddle and face each other to speak, though they can barely hear each through multiple layers of protective gear.
The purpose of this drill is to test how the latest wireless technology can help first responders. This is the team that would head into a contaminated cell tower or a server-filled AT&T facility to identify the problem. The hazmat suits and all the gear they carry is state of the art, and sometimes developed with input from AT&T. That includes a specialized smartphone, which streams the entire drill to a command center two floors up in the same building.
The Sonim XP7 runs Google’s Android operating system and connects to the Internet over AT&T’s regular LTE. If the network is down, which is one possible scenario NDR preps for, the phone would simply record the footage for later viewing.
It’s totally insulated, so there’s no chance a spark from the battery could set off a fire in a potential gas leak. It can also be scrubbed down if exposed to radioactive materials.
The command center can monitor each person’s heartbeat, and tell them when they need to slow down and catch their breath. It can also monitor the gas levels by connecting wireless to a team member’s smartphone.
NDR was officially founded in the early 2000s after the September 11 terrorist attack knocked out AT&T’s network. Since then, it’s developed into an official organization with the goal of going into dangerous areas to keep the carrier’s signal strong when it’s needed most. When Hurricane Sandy knocked out power across lower Manhattan, AT&T was able to keep its network up. Other deployments have included Hurricane Katrina in 2005, along with various wildfires, floods and tornadoes across the U.S.
In some situations, the team also uses a fleet of specially designed trailers that pack all the same equipment you’d find in AT&T’s main facilities. The carrier used these trailers to keep its signal up after Hurricane Sandy, and can send them anywhere in the country to set up a temporary office.
Once the dill is finished the team returns up the stairs slowly. At this point their air tanks are likely running low. Their faces are covered in sweat. Their hazmat suits are heavy. In a real disaster you get about 45 minutes of air. Less if there are a lot of stairs to climb
Two floors up in that same New York building, the team starts stripping their gear. Normally, a cold zone would be set up outside the disaster site, but for a controlled drill it’s easier to use a vacant floor and avoid causing a panic outside
First the big hazmat suits come off. That alone is a pretty complicated process, and requires an extra person or two helping out. After that the face masks and gas tanks. The test is over (it’s the last of multiple drills run throughout the week), and the team will head back to their regular jobs with AT&T across the country. Until the next test, or maybe sooner if disaster strikes first.