We recently interviewed AT&T’s senior executive vice president of AT&T technology and network operations John Donovan, who sat down to answer a ton of our questions. We wanted to try to avoid the overly technical talk to get down to the questions we think you want the answers to, such as what goes into a carrier’s device and software approval process, why some carriers have great coverage in an area where other carriers barely have a single bar of service, and what AT&T is doing with its 4G LTE network in the future. Let’s get down to business, shall we?
Let’s talk about your 4G LTE network for a moment, since that’s one of your primary focus areas with AT&T. How’s that moving along? Why did you choose to roll out after Verizon?
Our network is largely based on architectural choices we made 3 years ago. The first choice was to delay our launch behind Verizon knowing that they had to go out and get LTE as fast as they could. We wanted HSPA+, 3G and LTE at the same time. That meant we had to do a number of things, including making our launch occur later than theirs. You can argue whether it was 6 months or 9 months, but our footprint is moving behind them at a rapid clip.
We’re at 384 cities* and we’re marching to a number we’ve announced: 270M people covered by the end of year. The decision we made had specific things involved. First —on a par for par basis, we have better battery life. They have two chipsets and we have one. They have one for voice and a second for data. That draws more battery and tends to make the footprint slightly bigger. Our solution has a single chip with simultaneous voice and data and can move between 3G/HSPA+ and LTE networks; if data doesn’t find LTE it goes to HSPA+. *[Editor’s Note: At the time of publication AT&T had expanded to 400 LTE markets]
Our timeframe was established by circuit-switch fallback technology. At the time, the chipsets didn’t work because there was big latency. Our big bet was to wait and make it work. The handovers between networks were complex, as were the setup times. It was very gut wrenching to go late. We ended up putting passive antennas on top of towers and taking some electronics on the ground and moving them up to make the antennas active. There’s a lot of loss that occurs between the baseband units in the ground and the antenna, and what we were able to do was eliminate that loss. We put more power at the tower and more power in the handsets.
Now, we have better battery, faster network up and down, and seamless network transition. Waiting proved to be be a prudent move. We wanted to make sure our core was integrated from 4G to 3G from a reliability standpoint. The second thing we did was deploy right out of the chute in over a dozen locations whereas competitors put up a network where the core was very different. You may have noticed 7 nationwide outages from Verizon, we had 0. That’s because of our core.
Ok, so we’ve covered LTE right now. What about in the next, say three years, where do you see it next? What’s up with HD Voice and LTE Advanced?
Remember, the penetration rates of smartphones are getting high, so networks are going to need to get denser, so if you look at the published numbers we have more sites than competitors. When you get to LTE technology, you can have performance degradation from lack of density. So we build additional towers; we’re adding 10,000 more, but also small cells. Small cells add capacity and make sure the footprint during heavy use periods doesn’t shrink into dead spots. Small cells can be in-building systems or on top of buildings, and we’ll deploy 40,000 more small cells and deploy Wi-Fi aggressively.
LTE Advanced, at its heart, needs to be augmented with that densification program. VoLTE, or high-def voice, is part of the evolution at the services level and will get enhanced with LTE. We’re on track and feel good about it. We’ll be running VoLTE trials this year and have targeted deployments this year. We have promising trials so far in the lab, but we’re not satisfied to deploy until we find the network performs equally well to what we have out there today.
We go into every deal with our eyes wide open. We don’t by default choose Qualcomm. We select them at a high percentage of the time because they have the roadmap out ahead for us. If you look at the next-gen chipsets we’re looking for, we’re looking for integration and very complex stuff. The FCC is talking about Band 17, Band 12 integration. We bought the D block from Qualcomm. If you take all of that complexity and you get it on paper and you get a chipset, Qualcomm tends to be to market first. That’s the bottom line. We look at who can give us the right chip footprint with the right cost, power and performance against those spectrum requirements and then get it to us the fastest. They really, really perform. Of all the companies that supply us in all of the things we do, they are one of the top performing companies.
What do you do when you know there’s a dead spot in your network? A bad speed test?
Today, because we’ve narrowed the problems down – we all have spots in our network we’re not proud of — we can do fix spots one by one. We can look at a specific location and figure out what’s wrong. Often you can perform antenna tilts on the tower. Sometimes it’s just that something got out of tuning. We can get immediate impact by messing with power and tilt of an antenna.
Next, we look at a permanent solution that’s fast. Small cells have rapid deployment cycles, versus a tower, and are simpler for zoning. If it requires an in-building small cell, you just negotiate with building owner. If it’s outside, we can put it a small cell on poles we already own and hang it below 18 feet you to meet with Homeowners association rules. Over 18 feet on the tower and you need industrial board approval or to talk with a city council sometimes.
If a small cell fails and we need a macro cell solution, then we need to go through zoning. We have to go through elaborate zoning processes that can take 7-9 months or up to 3 years. It’s painful to know that you have dead spots and you’re ready to make an investment and you can’t do it. So, to fix a dead spot we first try optimization, move to a small cell and then to macro cell. If it’s acute, we’ll get a temporary facility in like a CoW (cells on wheels) or something like that. We rent parking places, park a CoW and put in temporary solution. We can also place CoWs in buildings.
Why do some carriers have better coverage in areas where another might just have a single bar?
The industry is of the mindset of sharing sites. We sell tower capacity to Verizon, they sell it to us. If someone’s coverage is better than another, it’s a couple of things. It might be an old site that needs additional capacity. If I have a site that requires a major upgrade to lease to Verizon, I probably won’t do it for economic reasons, not because Verizon is a competitor. These get to be pretty prodigious towers, so the structure can’t accommodate upgrades and you need a new site.
Also, it comes down to the economics of the tower owner. The tower owner may have a 5 year lease with AT&T, reads the paper every day and says to himself, “Wow, this is a huge opportunity for me. I’m collecting $3,000 from AT&T but the next guy is going to pay $20,000.” So, property owners that lease the ground, or the building where the towers are, aren’t going to have the second or third carrier without making it very uneconomical. Still you try to never to let economics get in the way of a good customer experience.
Lastly, sometimes it’s the backhaul and not the tower itself. We want to run the backhaul as fiber wherever we can and microwave where we can’t. In all cases, we try for high capacity ethernet. In a lot of these areas, rural and remote areas, you have trouble getting that backhaul. It could be a small local provider, and the provider says “Look I don’t have enough money to put the backhaul in, so I’ll put it in next year’s capital budget.” We have to work around those local provider delays.
Can you talk about the approval process that new devices go through before they launch on AT&T? I feel like our readers are curious as to why sometimes software updates take longer, or devices roll out at different times on different carriers.
As a core foundation we are now the best in the world at looking at new devices, applications, operating systems and then evaluating what apps/devices/OSs will do to our wireless network. It’s challenging, and historically you worried about just the device. Now, we have to look at how apps, devices and an operating system effects our network geographically, nationally. We have to see what changes on all new and legacy devices. We can demand forecast ahead of a big launch to make sure we are getting sufficient capacity in the right place at the right time.
Normally, we categorize new devices into 3 types: our highest risk profiles, cycle times and rapid fire or iconic handsets. From our standpoint, the hardware is irrelevant. The radio performance could be hardware related because of antenna, but it’s mostly software. So we put a risk profile if it is a brand new chipset and firmware on a brand new device. Or if it’s a company who doesn’t have a track record of quality, we run it through a process that takes a 3-4 month time frame. We get it on a bench, tear it apart, learn what we can, then put it in Faraday cages and run state of the art simulations of body and performance characteristics. Then we take it out and do user testing on the network and gather performance stats. So, really, it’s chip dependent, software dependent and company dependent.
The second category we look at is a device with a faster cycle time with a known or existing chipset, a firmware stack that has a modest amount of change but is significant enough for testing. We don’t look at the new apps because that doesn’t matter a bit. If they (an OEM) change the protocol stack about how the dialer works for instance, that matters.
Then there’s our third category: rapid fire iconic devices. These are from big suppliers or are “hero” devices. We have a lot of confidence in the supplier and in the chipsets. Apple launches devices with chipsets in a second spin cycle, so from our standpoint that puts them in a much lower risk profile. Dating back to Steve Jobs, they didn’t take huge risks on the chipset side. We get an early start on iconic devices and move them quickly. There’s a really short cycle for those.
Also, we schedule an emergency (it’s actually always planned), EMR emergency maintenance software release. We plan it in so that if it launches on X date, well then say four weeks out we’ll plan a software upgrade and then you can catch your last bugs. That’s a big change in the industry. We worked with the the operating system companies and hardware guys to speed it up a lot; the last debug occurs in the field. We try not to miss big stuff.
That about wraps it up for our interview with Mr. Donovan. Hopefully you learned a bit about AT&T and how carriers in general operate from this interview.