This is a special edition of Cyber Security Today and Hashtag Trending for Monday, July 22, 2024
I’m Jim Love, sitting in for Howard Solomon this week on the Cyber Security Today podcast although, Howard did come in on Friday to post a story on Tech Newsday, out online publication. You can find that at technewday.ca or .com – your pick.
Cyber Security Today for Monday July 22
This is a special edition of Cyber Security Today and Hashtag Trending for Monday, July 22, 2024
I’m Jim Love, sitting in for Howard Solomon this week on the Cyber Security Today podcast although, Howard did come in on Friday to post a story on Tech Newsday, out online publication. You can find that at technewday.ca or .com – your pick.
Our headline – one word. Crowdstrike.
We thought it was our job to summarize the situation to date. However, given that many of you are involved in this directly, some of this may not be news.
And for those of you working on front-line response, you have our support and sympathy. This is tough. And you are not alone. Forums and social media are rife with the frustrated comments of those who are probably still working on this.
Here’s just a smattering of the message we’ve seen, and as used as I am to some tough talk in tech forums, I don’t think I’ve ever seen so many F bombs.
And it’s no wonder. Cyber Security is frustrating enough. There have been a number of reports, you’ve no doubt heard them – anywhere from 28% of Chief Information Security Officers or CISOs and up to 51% of security professionals in general are likely to leave their jobs because of burnout in the next two years.
But here’s a sample from one Reddit forum on this incident….
ONE (expletive) MINUSCULE MOMENT OF NOT HAVING ANOTHER (expletive) THING TO BE ANXIOUS ABOUT????
(Expletive) we should just return to the caves. At least the bear attacks were quick.
And another describes what it was like for a lot of security professionals who realized right away that this was no ordinary outage. Here’s what one post said:
My wife’s machine BSODd live when this happened. I was like, babe, you are gonna read about this in the news tomorrow. I don’t think you’re gonna get in trouble with your boss
I felt like the cop in Dark Knight Rises telling the rookie ‘you are in for a show tonight’
When my pager started to go off tonight and my wife asked if it was bad, I said the same thing. “You’re going to read about this one in the news tomorrow”
So people are angry….
At who? Well right now, it’s a security firm called Crowdstrike.
Many of you will know them.
As as the head of IT in the former IT World Canada, I knew them as because they monitored and filtered out Distributed Denial of Service or DDoS attacks. Over the years, they’ve expanded their services and today they provide a wide range of cybersecurity products and services.
And it was one of those products, marketed under the name Falcon that is at the root of this disaster. Falcon, according to one description, provides endpoint detection and response products by monitor traffic passing through systems to protect against malicious files, viruses, and malware, relying heavily on cloud technology to protect internet-connected devices on corporate networks
In fact, they are second to only to Microsoft. Crowdstrike was reported by Gartner in 2023 to have up to 15% of the market share in the security segment. Still not Microsoft’s 40% but significant.
In fact, from the company’s own reporting
298 of the Fortune 500 companies use CrowdStrike
538 of the Fortune 1,000 companies are protected
– 43 out of 50 U.S. states use their services
– 8 out of the top 10 financial services firms
– 8 out of the top 10 food & beverage companies
– 6 out of the top 10 healthcare providers
– 7 out of the top 10 manufacturers
– 8 out of the top 10 auto companies
– 8 out of the top 10 technology firms
That’s just the top of their market. As I pointed out IT World wasn’t anywhere in that size and we – and a lot of other smaller companies used them as well. Hell, why wouldn’t you use the software that so many big companies used?
But there’s a flip side to that – and that is, if all of these companies use this software and if if they screw up, it hits a lot of firms in the US and around the world.
But it gets worse. What if that screw up affects the operating system that every company relies on? Microsoft Windows.
Then it’s a disaster.
And that’s what happened. An error in an update threw Windows machines around the world into a “blue screen of death.” Or BSOD. If you’ve been around in IT for any length of time you seen this. It actually used to happen regularly almost everyone Windows machines.
In the olden days, to make the system faster and more responsive, certain functions (somebody can correct me if I’m wrong, but if memory serves me – no pun intended – it was classically a graphics function) wrote directly to the kernel. It can address system memory and hardware directly. So when something fails at that level, it takes the machine down with it. And the operating system can’t restart unless it can use the kernel.
So – if the old days, and to this day apparently, any instruction was even a little wonky – the machine would crash and the blue screen of death would appear. But in most cases you’d power down and restart your machine and that would usually fix that.
Over the years, people have stayed away from writing directly to the kernel and put some software guard rails and error handling so that we’ve seen a whole lot less of these errors.
But, some software still directly interacts with the kernel – and one of those is – you guessed it. Crowdstrike’s Falcon Endpoint Detection and Response software.
I’m presuming, I’ve read and it makes sense, that the reason they do this is greater protection. If malware activity is detected, they can block it right directly. Whatever the reason, they do it.
So when Crowdstrike released an update to their software, an error caused it to cause an endless loop of Blue Screen of Death errors on many Windows machines.
It wasn’t a cyber attack or as many had thought a Windows update (and Microsoft was really clear about that) the source of the problem was the Crowdstrike update.
How big is the impact?
Microsoft says
“We currently estimate that CrowdStrike’s update affected 8.5 million Windows devices.”
The post by David Weston, vice-president at the firm, says this number is less than 1% of all Windows machines worldwide, but that “the broad economic and societal impacts reflect the use of CrowdStrike by enterprises that run many critical services”.
Microsoft keeps tabs on it’s software world-wide, so this is likely to be the best place to get an estimate of the damage.
The BBC reported that it was only Windows 10 machines that were affected, but we haven’t confirmed that, although if there is some feature that kept Windows 11 machines from failing you will see Microsoft pushing that story like mad as one more reason to move up to Windows 11. But somehow, I also think that Microsoft is going to keep a low profile on this one. They’ve taken enough hits on security failures in the past 12 months – and what do they say about people who live in glass houses? You don’t throw stones at a single pane of glass.
But even with the low key announcement, this is being called the worst cyber event in history. Some are calling it the Y2K we never had.
Microsoft was also quick to point out that this was not an issue with their Windows software. They also stated that “the incident highlights how important it is for companies such as CrowdStrike to use quality control checks on updates before sending them out.”
“It’s also a reminder of how important it is for all of us across the tech ecosystem to prioritize operating with safe deployment and disaster recovery using the mechanisms that exist,” Mr Weston said.
To put this in perspective, the WannaCry cyber-attack in 2017 was estimated to have hit 300,000 computers in 150 countries and that was regarded as a earth shaking crisis.
There was a similar attack called NotPetya a month later.
By comparison, a six-hour outage in 2021 at Meta, which runs Instagram, Facebook and WhatsApp or the DNS error that took out a lot of companies on the Eastern Seaboard last year seem pretty mild by comparison.
The damage was catastrophic. I’m sure we’ve all heard on the media all of the places that were affected. Airlines stopped flying. Many companies could no longer function.
Estimates of the economic impact are just guesses at this point, but it will be in the many millions of dollars or maybe even billions of dollars.
That’s from an error that was in place for a little over an hour.
One report shows the timestamps on the first erroneous update as 0409 UTC or coordinated universal time. That’s like Greenwich Mean Time that but it doesn’t change with Daylight Savings time and the fix at 0527 UTC. One hour and 18 minutes. But what an impact that had.
And in relatively short period, there was a fix identified and published. You can get the latest of that on the CrowdStrike site.
But there’s a catch. You have to install the fix manually and locally, machine by machine. And that is going to take a lot of time and resources, if indeed you can even get to all of the machines. The sheer volume alone will strain resources beyond the breaking point. Plus, we no longer have all of our computers in our offices – some are remote. And there may be remote devices running Windows that are really hard to reach.
So we understand the frustration of the people who have to handle this. And the companies who have been interrupted – and the customers whose lives were affected.
Everybody’s angry.
The stock price for CrowdStrike fell by as much as 15% and the stock closed down at a loss of almost 9%. Believe it or not, it’s not the worst showing for CrowdStrike stock. It fell by almost 15% in November of 2022, but it came back.
But as we all know, there has to be someone to blame. I may be cynical in my old age but I always said that if there was a disaster and you went to a meeting – if you couldn’t spot the scapegoat – it was you.
When something like this happens, someone usually loses their job.
Even those who are defending the programmer or tester who supposedly missed this error still have someone to blame.
someone is getting fired
(an eff) up of this magnitude is the result of culture and process, not 1 engineer. Blameless PIRs are critical for things like this, it’s how you understand the true cause that led you there, rather than just firing Gary because he was the one that pushed the button.
And I’m going to say, I don’t know what happened. But it is a valid question – why didn’t it get caught in testing?
I’ve been searching around to find an answer and before we rush to blame some programmer or tester, or even the corporate culture – I saw at least one good tech demonstration of how this could passed testing and still be issued with that error.
I’ll post a link in the show notes for today. Check it out yourself. And if, like me, your programming days are long behind you, it will still make a lot of sense.
And I want to be clear, that I’m not saying that’s what happened. All I’m saying is, if you are wondering how the hell this could happen, hopefully, if CrowdStrike is smart, we’ll get the facts of what happened exactly.
And for those who are saying that people should test updates – I think they should as well, but I’ve always told people that it’s important to keep your software up to date as the greatest protection against zero-day impacts, but we’ve always waited just a little bit of time to see if a Windows update would cause problems.
But I’ve always pushed to make sure that end point detection software is always the latest. New attacks come by the hour not the day or week. So, before we blame the folks who updated their end point software, we might want to take a deep breath.
And CrowdStrike head George Kurtz encouraged users to make sure they were speaking to official representatives from the company before downloading fixes.
“We know that adversaries and bad actors will try to exploit events like this,” he said in a blog post.
According to one source, there has already been a sharp rise in CrowdStrike themed domain registrations – which can only be to trick IT managers or others into downloading malicious software thinking they were going to “fix” the Crowdstrike problem.
Cyber security experts are urging everyone to only use CrowdStrike’s website as the source of information and help.
There will be lessons learned from this, I am certain. I’m thinking about having a special panel for our weekend show next week to have a deeper dive into this and with some perspective, help all of us understand what we can do to avoid this in the future.
That’s our special show from both Cyber Security Today and Hashtag Trending. We’ll be back we hope to the regular news in our next episodes.
Thanks for listening.
