July 23, 2024 An FCC investigation revealed that a botched network update by AT&T caused a massive nationwide outage on February 22, 2024. This outage blocked over 92 million calls, including 25,000 attempts to reach 911, affecting more than 125 million devices. The incident began three minutes after the update, disconnecting all devices due to a configuration error and took over 12 hours to fully restore services.
The FCC criticized AT&T for failing to follow best practices, such as adequate testing and peer review of network changes. The investigation highlighted various procedural flaws within AT&T, including insufficient oversight and controls, which contributed to the extensive scope and duration of the outage. An AT&T employee had placed a misconfigured network element into production during a routine maintenance window without the required peer review, exacerbating the problem.
The FCC report detailed that AT&T’s device registration systems were overwhelmed by re-registration requests, prolonging the outage. Additionally, the lack of adherence to internal procedures and inadequate testing before implementation were significant factors in the failure. AT&T acknowledged the flaws and pledged to make changes to prevent such incidents in the future.
AT&T faces potential penalties, as the FCC’s Public Safety and Homeland Security Bureau has referred the case to the Enforcement Bureau for possible violations of FCC rules. This comes after a similar incident involving Verizon Wireless, which resulted in a $1,050,000 fine and a mandated compliance plan.
