Southwest Airlines suffered an outage in July, grounding its fleet nationwide and cancelling more than 2,300 flights. The culprit was determined to be a lone router at Southwest’s Love Field data center, causing an estimated $54mm outage cost to the company according to The Dallas Morning News.
The airline had a backup system in place for that router, but that system failed and created what its CEO alluded to as a “once-in-a-thousand-year flood.” Ultimately, Southwest determined the main router had a partial failure that didn’t allow it to function properly, but didn’t fail enough to signal the backup router to take over.
TBI often discusses hot-cold, hot-warm, and hot-hot backup environments with our cloud vendors. Whether managing data replication in a DRaaS solution or voice packets for a UCaaS solution, establishing a resilient and reliable backup environment is critical for successful application performance.
We believe a hot-hot environment where (for the sake of Southwest) both routers are always active and engaged in the network is best practice. Additionally, each of these routers should never exceed 40% of their individual capacities. Should one router fail, the other router does not need to wait for a signal to engage, and it can handle the additional workload without the performance issues that occur once a device surpasses 80% capacity.
Hot-hot environments are more expensive than hot-warm or hot-cold environments. However, by reducing the smallest chances for failure, a hot-hot environment can be the difference between making the concert at Red Rocks and sleeping on the floor at LaGuardia (thankfully this did not happen to me).
TBI works with world-class network and cloud providers that have built resilient and reliable systems to ensure seamless application delivery to your clients. Engage our channel managers and sales engineers to discuss where we can help expand your business by meeting your clients’ business outcomes.
About the Author
Seth Woodward is the Sales Enablement Specialist for TBI, the nation’s leading distributor of technology services – also referred to as a Master Agent. Woodward manages TBI’s portfolio of vendors and develops sales and training initiatives that empower the technology channel to expand services sold to its clients. Previous to TBI, Woodward worked at a technology firm bringing next-gen networking and video solutions to market for national retailers. He is an MBA-candidate at the Kellogg School of Management at Northwestern University. You can connect with Seth on LinkedIn.