When you think about your cell phone do you ever really think about the network that powers it? More than likely you don’t… except when it doesn’t work. During a storm or other emergency your first thought is going to be how your carrier is letting you down. The faster the company can respond to your needs the better, and to do that there has to be a disaster recovery plan in place and the equipment to carry it out.
To that end, Verizon has gone a bit further than most in the case of its Emergency Preparedness Disaster Recovery (EPDR) team for the Kansas and Missouri by moving its equipment undergound – 60 feet to be exact – where it can always be ready to roll out at a moments notice and be kept in tip top shape.
After learning about the cave in a promotional video Verizon released, we contacted the press department to see if we could arrange a tour as I am located only a few hours away from Kansas City where the cave exists. We were met with an enthusiastic response to show off the endeavour and a meeting was quickly arranged.
What I found was not what you expect when you hear the word “cave” at all, and instead is a high tech facility perfectly suited to protecting a company’s valuable assets to make sure you suffer the least amount of down time no matter how bad a disaster may be.
It’s Not What You’re Expecting
I met up with Tony LaRose and Derald Price at a designated meeting location to be driven over to the Verizon cave, and already knew I was in for an odd journey when I overheard them discussing who had “the key to the cave.” Naturally you need to lock down this sort of equipment due to its value, but you just never think about locking a cave.
As we drove to the facility located in the greater Kansas City metropolitan area, LaRose participated in a conference call that was laying out the plans for where all of the equipment would be deployed in the coming weeks and months. “We don’t just send all of this out to emergencies,” LaRose explained after he had completed his call. “We will also go to events such as fairs, football games and race day at the Kansas Speedway. Anywhere we feel there is going to be extra strain on the network.”
When you enter the main cave facility, you drive past other companies that have taken up residence there from many different fields. You are also immediately struck by the immensity of the facility as you see a semi drive pass you in the opposite direction. This is not just some little cave in the river bank, this is a full-on cave complex where it isn’t too difficult to imagine yourself getting lost. If I drove back in there on my own today, I wouldn’t have the first clue how to find Verizon’s area again.
Arriving at Verizon’s spot in the cave system, there is just a simple office door in the wall with an unassuming sign. As we entered – they did have the key with them – and the lights were flicked on, you are standing in a barebones office area with a simple table, a few chairs and large, flat filing cabinets for plans. It is small and quaint, and looks like any office you would see in an industrial park.
But it was the next door that the interesting part resided behind.
So Much Technology
Passing through the next door revealed the “Barnyard” as they call it. Why Barnyard? Because it’s filled with devices named for various animals.
- COW – Cell On Wheels
- COLT – Cell On Light Truck
- GOATs – Generator On A Trailer
- HORSEs – HVAC On Road-Side Equipment
- CROWs – Cell Repeaters On Wheels
All of the vehicles and trailers are neatly contained, and those that have software in the need of updates are plugged into electricity in the walls with Internet lines so their software is constantly updated. “Before we had the cave, it wasn’t nearly as easy,” LaRose explained. “Everything was stored outside, so when we would arrive on site we would have to hook up to whatever fiber we could find and run numerous updates before it was ready to work. Now, everything is updated until the second we unplug it and drive it out the door so it’s ready to roll the second we arrive.”
The vehicles are all neatly lined up and stored, and everything is easily accessible. Beyond the vehicles and trailers themselves, Verizon is also able to store all of the cabling and parts the team will need on site.
What you are struck by, more than the equipment itself, as you walk around the cave is just how comfortable it is. It’s not hot, it’s not cold, it’s just a pleasant atmosphere that you could easily see yourself working in without hassle. This is not, however, normal for the cave. In the case of Verizon it is accomplished with a large dehumidifier hanging from the ceiling. LaRose took me through a door into a neighboring cave that is being worked on for a future occupant, and the moment he opened the door, the still, humid air hit you like a ton of bricks. It was oppresive and not conducive to breathing in the least.
Returning to the Verizon side of the door, and the much nicer atmosphere, I asked LaRose how quickly the team could deploy. The cave is not exactly in the center of town, and I could see where it might take some time to get the personal to their vehicles to roll out. “From the time we get the call to the vehicles rolling out the doors is about three hours,” LaRose said. “That includes unplugging them from the network and making sure we have everything, but it’s once we’re on site that we really save time due to all of the constant updates that are carried out.”
Throughout an average year, this particular EPDR team will usually have around 17 scheduled deployments, and five or six unscheduled emergency situations. Although its primary range is all of Kansas and Missouri, and part of southern Illinois, the assets can be deployed anywhere they are needed. Some of the equipment currently stored in the cave even responded to New York City following the terror attacks of 9/11, and were on site there for a month.
That led me to wonder about how long a deployment could be. Were we talking days? Weeks? Months? And it varied, they had one deployment where some assets were gone for a year, but in the case of 9/11 it was only about a month. So while the cave was fully loaded while I was there minus two COLTs and a COW, it seemed that this was a place that was probably fairly busy at any given time with equipment rolling in and out.
Why a Cave?
Despite the obvious benefits of the cave, it still feels a bit surreal as to why such a modern company would turn to such an old concept for storing its highest tech equipment. The answer is simple, yet catastrophic.
LaRose and Price were part of the team that were sent to Joplin, Missouri following the fatal tornado that touched down there on May 22, 2011. During their time there, and having witnessed the devastation, it crossed their minds about what would happen if such a powerful force ever passed through the then current outdoor barnyard. What would they do if all the equipment was destroyed? It would take a much longer time to restore service to customers as equipment from other areas would need to be brought in, and that would prevent those teams from covering their own areas.
The two men started tossing around the idea of going underground with the vehicles as an above ground warehouse wouldn’t offer the protection they were thinking of either. As soon as they returned to Kansas City from their assignment they started looking for spaces – Missouri has a tremendous amount of caves around the entire state – to see what they could find. Once they did find it, they had to seek approval from the company to move forward with this idea. LaRose said it only took two meetings for Verizon executives to see the benefits of the concept, and from start to completion the project took 12 months to open up the first 10,000 square feet of space.
I mention it as the first, however, as almost immediately upon moving in they discovered 10,000 square feet wasn’t enough space, and they needed a slightly higher ceiling if they were going to get the HORSE into the space.
The Verizon team spoke with the cave facility, and work began on adding another 10,000 square feet to the carrier’s space along with it being 3 feet lower to accommodate taller equipment. Having seen how much is in there now, it’s difficult to imagine how it was ever going to work at the original size, but now you can see how the trailers can easily be swung around and put into their parking spaces. And the HORSE looks quite comfortable in its parking space.
What Does the Future Hold?
As I left the cave with LaRose and Price, the former received a text message as soon as we were on the road that a COW was on its way back to the barnyard from its non-emergency assignment. “That may be the only drawback to the cave,” LaRose said as he put his phone down after showing me the text. “Every unit down there has a GPS tracker on it that is supposed to check in once a day so I always know where our assets are. So say a COW has been down there for a hundred days, as soon as it pulls out of the facility I get a text message for each day it was down there without the ability to connect.” LaRose said that they are thankfully working on a solution for that which involves running a wire up the top of the hill, but for now it was certainly a small price to pay for the peace of mind that the cave offers him and Price knowing that their equipment is protected from just about any natural disaster you can think of.
Since the cave opened, many Verizon executives have visited to see what exactly has been set up, and all have walked away liking what they see. Particularly interested are the managers of other Verizon Emergency Preparedness teams that have stated they are now looking for caves of their own. Having seen it for myself, and hearing not only the logic behind it but how it prevents even issues such as tire rot, and has cut equipment maintenance costs down significantly, it almost feels like every carrier should be doing this. Certainly there are areas where locating a cave wouldn’t be quite as easy as it was in the Kansas City area, but there are sure to be other solutions to be found.
It has not yet been determined if other caves will indeed be opening, but the research is being done and we’ll just have to see if anyone else actually takes the step to do so. For now, LaRose and Price just smile contently as they discuss the cave and all the benefits it offers them. “Plus, I can’t lie, the inner kid in me loves it when I get to say I work in a cave,” LaRose said with a broad smile.