Security vendor CrowdStrike released an update on from their initial Post Incident Review today.
The first, and most surprising information was that CrowdStrike says that the incident was not caused by code or a kernel driver – the kernel driver was the assumed cause in most industry speculation.
This might be semantics to some extent especially the offending file is stored in a folder called system32\drivers\CrowdStrike
But clearly, whatever you call it, CrowdStrike acknowledges that this file contains data which “resulted in an out-of-bounds memory read triggering and exception” that Windows was not set up to handle “gracefully” and that resulted in the crash.
I’m not trying to attribute anything to CrowdStrike, I’m sure they take accountability, but the idea that if Windows had just been able to handle this “gracefully” might seem to some like passing the buck.
But if the driver/non-driver issue is a somewhat confusing, we did get an answer to the question that many of us were asking, and that is “why didn’t their testing catch this?”
The report lists a very thorough testing procedure before release which is, on our reading, pretty much consistent with industry best practices. So the template did go through a testing regime.
CrowdStrike notes that they issued two other templates at the same time and that both of these “behaved as expected.”
So what went wrong?
Apparently, it was a bug in what CrowdStrike refers to as the “Content Validator” which allowed this template to pass through despite having problematic data.
Reliance on that “Content Validator” was apparently the issue. Which still raises another question – whether you call this a template or a driver, shouldn’t the severe implication of a null value (or whatever the problematic value was) have been anticipated?
We got some additional answers as to why multiple reboots of a machine might be necessary to correct the error. Apparently, this “template” can’t simply be deleted remotely or simply over-written. It stays in the sensor’s directory.
Cleverly, their team, in a very short time opted to put the file on their “known-bad” list and after multiple reboot attempts it could be blocked and normal activity resumed.
A couple of assurances were offered to CrowdStrike customers. First, CrowdStrike is stating that there is no impact to the protection on working systems.
Second, they have listed a number of things that they will do to “prevent this in the future,” among these:
- Adding additional checks to the Content Validator
- Staggering deployment to gradually release to their base, starting with what they refer to as a “canary” deployment, likening back to the old idea of taking a canary into a coal mine because the canary would die more quickly than a human.
I’d normally say that there is a link to the complete text included in our show notes, but as reminder that this is a time when, as CrowdStrike CEO George Kurtz warned about “adversaries and bad actors” taking advantage of this situation – you should ensure that you take information only from the CrowdStrike site. So just go to CrowdStrike.com – the link to their remediation hub should be obvious.
And there are some good resources that you may want to refer to. The video that they issued on how and end user can fix the problem is well done, clear and provides a step by step that even a novice could follow – so if anyone still has people in the field affected or if you know small companies or individuals that somehow got hit, this is worth looking at.
