I was driving out to the suburbs of New Jersey a few weeks ago. It was a rainy day and I was on my way to visit AT&T's global network operations center (GNOC) for the second time since I started working as a journalist in 2007. I knew what to expect: an amazing James Bond-style control room that, typically, only journalists and AT&T's business partners ever get to see. And I knew I'd meet, for the second time, Chuck Kerschner, network operations director of the entire center. I didn't expect, however, to be blown away by its massive scale again.
When I entered the GNOC I was taken upstairs to a small movie theater where I saw a brief intro about what the facility is all about — maintaining AT&T's network stability. Then, and this didn't happen the first time I visited, the movie screens lifted up and I saw the gargantuan NASA-style center below me. What I saw before me was the room where, every day, AT&T employees work around the clock to make sure the company's global network is alive and well. Before long, I was down on the floor shaking hands with Kerschner yet again. This. Is. GNOC!
What is GNOC?
So what exactly is the GNOC? It's where AT&T monitors its entire global infrastructure. Its employees can see, at any time, how the network is performing in any part of the world. Those big screens I talked about earlier? They provide data on each network point and let AT&T's GNOC employees know whether or not network data needs to be rerouted in order to make sure the network's performance is up to snuff.
Kerschner showed me a number of instances where the network was flooded with data and how AT&T has to answer during a disaster: the tsnunami in Japan, for example, the flooding in New Orleans and even the attacks on September 11th all put massive strains on the network. In such a case, AT&T needs to reroute data so that those areas can stay in contact with the rest of the world.
A regular map on a quiet day might show a few spikes in activity at any given moment (possibly from a hacker), but, during a disaster, those maps suddenly overflows with data so intrusive that you can't even see the actual data map anymore. Since the GNOC is staffed 24 hours a day, the employees are able to immediately begin fixing the network issues. The eyes in GNOC are able to see issues before they occur in realtime, too. A yellow line might mean that traffic is beginning to increase and that rerouting should occur soon. AT&T uses these predictive charts to make sure that its customers never notice a hiccup.
Chuck Kerschner and the GNOC Employees
I imagine it's hard to fathom, from an outside perspective, what type of people work in such a demanding environment. The GNOC is open 24 hours a day and is fully staffed to handle anything that happens on AT&T's global Internet network. There are 141 huge screens that gobble up an entire wall, bigger than an IMAX movie theater — many of them with data that is so sensitive I wasn't allowed to photograph them up close.
So what kind of people work here? I used to think it would be filled with the stock broker type — people jumping over desks, yelling commands at one another to deal with a potential network outage on the other side of the world. That's not so. Chuck Kerschner towered over me and, just like when I met him nearly five years ago, he had a commanding handshake and looked me dead in the eye during our discussion. But he is a calm person and our conversation was akin to one might have on a relaxing fishing trip. That type of attitude expands across the whole floor, which is as quiet as the New York public library. Everyone is focused — Facebook and Reddit weren't up on any personal computer screens — and striving toward one thing: making sure AT&T's network is operating at its most optimal potential, every second of the day.
AT&T added a new screen that wasn't on display when I first made a visit to the GNOC in 2008: one purely dedicated to social networking. The display gives AT&T a glimpse at what's being discussed about its network on various social networks, including Twitter. This can help it take a head-start against issues that, likely, are also being reflected on other displays in the GNOC. So, not only is AT&T looking at data coming back from computers and its enterprise customers, but also from a human perspective from those who are also on the network.
AT&T has always had a focus on security — that much is obvious, otherwise the GNOC probably wouldn't need to exist in the first place. But it's also paying closer attention to mobile devices accessing its network, as well as security threats or malware that might be stemming from computers and phones, AT&T's executive director of security technology, Michael Singer, explained.
In a small room that sits inside the control center, Singer and other AT&T engineers are working daily to protect AT&T's network against these threats — often by creating custom algorithms that can predict and stop an attack as it happens. Like Kerschner, Singer was calm and seemingly easygoing. It must take that type of individual to work in a career that's constantly changing and adapting to new threats. In many cases, Singer said, AT&T is ahead of security firms and often works closely with them to make sure that both sides are prepared against attacks. Singer also showed me the following video, which gives a small overview of the security management he works on within the GNOC.
You have to see AT&T's GNOC to believe it. As a geek, I couldn't help but wonder what it would be like to play a Battlefield 3 session across all 141 monitors in the control room. But the global charts, news screens and the amount of data crossing those displays every second is enough to take anyone's breath away. It's incredible knowing that, at any given moment, despite whatever might be going on around the globe, Kerschner and his employees are walking the halls and manning the control room calmly, just like any other day at the office.