Inside American’s operations nerve center in Fort Worth. Photographer: Matthew Johnson for Bloomberg Businessweek
At about 12:30 a.m. on July 19, Jessica Tyler was awakened by a phone call from work. That in itself is not unusual. Tyler is in charge of American Airlines Group Inc.’s Integrated Operations Center, located just outside Dallas Fort Worth International Airport. The IOC is American’s nerve center, managing the carrier’s more than 6,000 daily flights.
The range of problems the IOC has to deal with is dizzying: winter storms and summer hurricanes, mechanical failures and the occasional volcanic ash cloud. Every decision the people there make must take into account a lengthy list of factors, including the maintenance and replacement schedules for the myriad parts in every plane in the fleet and the collectively bargained time limits on the people who staff them. If Air Force One flies into one of American’s hubs, halting everything there for a few hours, that’s an IOC problem. If there’s a Taylor Swift concert in town and no hotel rooms available for overnighting American flight crews, that’s another one. As Tyler puts it, “it’s managed chaos.”
An employee monitors a volcano that could potentially disrupt flights. Photographer: Matthew Johnson for Bloomberg Businessweek
What was happening that night threatened another level of mayhem: The computer systems that the ops center relied on to manage flights from start to finish—and to relay information to pilots, track the weather, monitor delays and spot issues at airports—had simply stopped working. All of them, it would turn out, were running software from CrowdStrike Holdings Inc., the Texas-based cybersecurity company.
Tyler quickly drove the 20 minutes from her house to the American campus. The parking lot in front of the IOC, a low building clad in gleaming, tornado-proof glass, was beginning to fill with colleagues who’d gotten similar calls. They all made their way to the second floor, to a gray-carpeted bullpen full of clustered workstations.
Tyler. Photographer: Matthew Johnson for Bloomberg Businessweek
The spectacle that greeted them had a queasy, funhouse quality. The dimly lit space is full of screens—arrays at all of the desks and dozens of 82-inch monitors overhead. That night, rather than displaying route and airport maps, weather systems or real-time scheduling histograms, the sea of monitors were dumbly flashing on and off, over and over, blue then dark. “It was very eerie,” Tyler recalls.
It quickly became clear that American wasn’t alone. Reports came in through crew management that the systems were also down at the hotels where flight attendants and pilots were trying to check in for the night. Computers at some of American’s fuel vendors were also seeing the so-called blue screen of death, stuck in an endless reboot cycle.
The CrowdStrike outage, caused by a faulty software update from the company, would end up affecting 8.5 million computers worldwide, becoming front-page news and serving as a stark reminder of the fragility of many of the digital platforms on which modern life relies. For an airline, the problem had a special urgency—at the moment its systems went down, American had some 200 flights in the air. Later that morning thousands more were scheduled to take off, creating a branching web of turnarounds and scheduled connections that, if allowed to progress, would only grow more tangled as the day wore on.
Even on a good day, running an airline requires ingesting massive amounts of data. Decisions can go wrong in all sorts of ways. And like every other industry, airlines are trying to figure out how technological advances can benefit them. But in doing so, they have to navigate a distinctive set of countervailing pressures: thin profit margins, the demands of federal regulators and a workforce that is by contemporary standards heavily unionized. And then there’s the additional challenge of finding a way to overhaul a machine that can never stop.
In normal operation, American’s IOC is overseen by a director working a 12-hour shift from the bridge, a slightly raised set of workstations at the center of the bullpen. Also sitting in the bridge are representatives from flight dispatch, maintenance, crew scheduling, customer service, information technology, security, in-flight ops and pilots.
At one end of the IOC floor, visible through a large window but requiring special permission to enter, is a large room with rows of desks from which computer monitors rise at the touch of a button. Two giant Mercator world maps cover the front wall. This is the command center. Most of the time it’s empty or used for training courses. But during extreme events, representatives from every corner of the airline gather there to coordinate a response. The airline activated the command center when the US Department of Defense enlisted American and other airlines to fly thousands of US personnel and evacuees home in the chaotic 2021 withdrawal from Afghanistan.
Executives at American, speaking in detail for the first time about how they managed the CrowdStrike crisis, say they decided to stand up the command center almost immediately. Tyler went straight there when she arrived. The airline’s chief operating officer, David Seymour, joined her.
Seymour, chief operating officer of American Airlines. Photographer: Matthew Johnson for Bloomberg Businessweek
Within minutes, American’s information technology teams were piecing together what had happened from web posts on technical discussion boards and conversations with business partners. But the first priority was making sure the dispatchers at the IOC could remain in touch with the pilots of the planes in the air—late-night domestic flights heading west and overnights going to Asia or coming in from Europe. The outage, the flight teams realized, had compromised the communication links they used to maintain constant contact with planes in the air. As a workaround, they asked the Federal Aviation Administration’s Air Traffic Control Command Center, in Warrenton, Virginia, to relay messages to American’s pilots—a capability that wasn’t compromised by the software crash. That would suffice until everything came back online.
Still, that was only one of the functions that had gone down. Airport security systems that allowed crew members to badge in to the jet bridge weren’t working. Neither was the software that tracked checked bags. The caterers that supplied American’s flights were paralyzed. Getting the systems back or developing workarounds would take time. And the problems would only compound. Tyler evokes the famous I Love Lucy sketch where Lucy, taking a job in a chocolate factory, tries to keep up with a pitilessly accelerating conveyor belt of bonbons. “You have to stop the conveyor belt,” Tyler says.
At 12:45 a.m., at American’s request, the FAA issued a ground stop for all of the company’s flights, freezing in place every one of its planes that weren’t in the air. But it quickly became clear that would only delay the reckoning. The IOC was going to have to put in a cancellation package.
Airlines hate canceling flights. This is in part because travelers hate it. But they have additional reasons. Cancellations don’t just strand passengers, they strand the aircraft and crews that need to fly on from their arrival cities to some other place. Those disruptions ripple outward through the network. There are other knock-on effects: Airlines use their planes for deliveries to their maintenance shops, so canceling a flight can mean a replacement part needed by a specific plane at a specific airport doesn’t get there.
Some of the many screens at the IOC. Photographer: Matthew Johnson for Bloomberg Businessweek
Nonetheless, the IOC leaders on the bridge, working with a team of irregular operations managers, decided there was no other choice. (On top of everything else, the airline was dealing with thunderstorms at its Charlotte hub.) As always, the planners tried to minimize the damage, choosing as many out-and-back flights as they could and making sure there was gate capacity at the airports where the cancelled flights would temporarily be marooned. Between 4 a.m. and 5 a.m., American canceled 114 flights. The hour after that, it canceled 148 more.
As the cancellations were going into effect, teams of IT specialists were making their way across the floor of the IOC. Fixing the computers was straightforward: The offending file had to be removed, and then the computer rebooted. But it had to be done individually, machine by machine. By midday the airline was running more or less normally, albeit with residual delays. “By 3 p.m., the vast majority of our business partners had also recovered as well,” says Suzanne Williamson, who, as managing director of IOC operations, reports to Tyler. The next day, July 20, American’s completion factor—the percentage of flights that successfully complete their journey—was back up to 98.9%.
This sort of decision-making apparatus doesn’t always work perfectly. Delta Air Lines Inc., which has perhaps the best operational record in the US over the past few years, decided to hold off on canceling a significant number of flights in the early hours of July 19. Almost a week later, it was still dealing with stranded passengers and misrouted luggage, and it had been forced to cancel 7,000 flights. Delta declined to comment about its CrowdStrike response, pointing to what its chief executive officer, Ed Bastian, previously told the Wall Street Journal: “We did not want to cancel too quickly because it was a fluid environment.” The airline has sued CrowdStrike, putting the cost of the debacle at $500 million, and CrowdStrike countersued, accusing Delta of trying to shift blame.
The daily management of operations, both regular and irregular, is something American has very carefully been trying to modernize. In early November, American finally finished upgrading its flight planning system, a software tool for dispatchers. Its competitors have similar efforts in the works. Andrew Medland, the head of aviation at the management consulting firm Oliver Wyman, uses a medical analogy to describe where things stand with the industry: “The patient is on the table, the operation is ongoing, the organs are being transplanted, but there’s a lot of work to do.” And unlike most patients, this one needs to stay at work while it’s under the knife.
Airlines, and American in particular, were innovators in using computers to do everything from managing parts inventory and reservations to calculating optimum ticket prices. Some of those once-pioneering systems are still in place today, running, in some cases, on mainframe computers beneath layers of newer software tools. It’s partly the legacy of decades of mergers and the yearslong integration process that follows each. But it’s also a testament to the difficulty of changing a finely calibrated, human-based operation and assimilating the knowledge embedded in it. “We are slowly migrating these systems,” says Seymour, American’s COO, “because there are a lot of intricacies.”