Around the same time, Cloudflare’s chief technology officer Dane Knecht explained that a latent bug was responsible in an apologetic X post.
“In short, a latent bug in a service underpinning our bot mitigation capability started to crash after a routine configuration change we made. That cascaded into a broad degradation to our network and other services. This was not an attack,” Knecht wrote, referring to a bug that went undetected in testing and has not caused a failure.
If you want a technical breakdown that isn’t “lol AI bad”:
https://blog.cloudflare.com/18-november-2025-outage/
Basically, a permission change cause an automated query to return more data than was planned for. The query resulted in a configuration file with a large amount of duplicate entries which was pushed to production. The size of the file went over the prealloctaed memory limit for a downstream system which died due to an unhandled error state resulting from the large configuration file. This caused a thread panic leading to the 5xx errors.
It seems that Crowdstrike isn’t alone this year in the ‘A bad config file nearly kills the Internet’ club.
There’s no such thing as bad data, only shitty code to create it or ingest it, and bad testing that failed to detect the shitty code. The overflow of the magic config-file size threw an exception, and there was no handler for that? Jeez Louise.
And as for unhandled exceptions, you’d think static analysis would have detected that.
Someone should make a programming language like Rust, but that doesn’t crash.
/s