Security vendor CrowdStrike released an update from their initial Post Incident Review (PIR) today. The company’s CEO has committed to being transparent regarding what the company learns about the causes of this disastrous incident.
There has been a great deal of speculation that the cause of the issue was found in a kernel driver but the company says this is not the case. The cause, according to their report, was a template file containing data that resulted in an out-of-bounds memory read. This triggered an exception that Windows couldn’t “handle elegantly”, causing the crash.
Whether it was a driver or a template file is a bit of a moot point, especially since the files were stored in a folder called \system32\drivers\CrowdStrike.
The PIR also indicates that this template did indeed pass through an extensive testing process. The company notes that two other templates were issued that day without any issues.
So why did their testing not catch this issue? Apparently, although the template passed through a number of stages of testing, none of these tested for bad data For that, there was a single point of failure in what they call the “Content Validator.” A bug in this validator allowed the template to pass through despite containing problematic data.
Once the template was in place, it has to be deleted manually, although some quick thinking from CrowdStrike staff put the template on their “known bad” list which in some cases, in multiple reboots, would block the file and allow normal processes to resume.
The company has outlined steps to prevent future incidents, including adding additional checks to the Content Validator and staggering deployment with a “canary” approach so that future issues could caught before the update has spread to a large number of machines.
Although this situation was identified and the offending template removed within a little over an hour of deployment, it still reached 8.5 million devices.
With the update on their site, the company has provided a large number of resources, including a step-by-step video guide to help users fix the problem which can be found on their site.
CrowdStrike assures customers that their systems’ protection remains unaffected. They emphasized the importance of relying on official information sources and provided a step-by-step video guide to help users fix the problem.
While we normally provide links to resources where appropriate, there are a number of what CrowdStrike CEO George Kurtz refers to as “adversaries and bad actors” who are trying to take advantage of this situation. Following a link in this case, even from a reliable source, is not a best practice. If you go to CrowdStrike.com you can easily find the additional information needed. Even in those cases, with a lot of “look-a-like” url’s being in circulation we urge careful typing. – Ed
