When Trolling Goes Viral - Crowdstrike Edition
Day 6,089, 04:06
•
Published in Croatia 
•
by
•
by Janko Fran
READeR
When Trolling Goes Viral - Crowdstrike Edition
Last weekend was quite an expensive one for many companies. How expensive?
Very! The costs are estimated to $5.4 billion (USD) just for the top Fortune 500 companies.
However, last weekend, according to the Tech Business News, at least 538 companies have been verified to have experienced significant disruptions due to a widespread IT security outage, which affected more than 8.5 million Windows devices, according to Microsoft's report estimate.
While software updates may occasionally cause disturbances, significant incidents like the CrowdStrike event are infrequent. We currently estimate that CrowdStrike’s update affected 8.5 million Windows devices, or less than one percent of all Windows machines. While the percentage was small, the broad economic and societal impacts reflect the use of CrowdStrike by enterprises that run many critical services.
CrowdStrike’s cyber security services are responsible for 298 of the Fortune 500 companies, 538 of the Fortune 1000 companies, and 8 out of 10 of the top financial firms.
This incident, which affected various sectors including banks and airports, led to a search for information as services were brought to a halt.
However, alongside this genuine crisis, a trolling response by Vincent Flibustier on X emerged as a response. This response used generative AI to craft a plausible but unverified narrative, essentially as a joke, taking the "blame" and offering a false scapegoat for those seeking quick explanations without verifying the details.
A Harmless Trolling Incident - The Viral Impact
Click on the image to find out more about how to create a viral video and what are the consequences of bad journalism without factchecking...
The trolling incident quickly gained traction, amassing 45 million views in just three days. The story seemed very plausible, leading many to speculate whether something similar actually occurred at CrowdStrike. However, the details of the actual incident remain unclear. This highlights how quickly information, even if unverified, can spread when it taps into existing biases or fills a gap in the available details.
In this case, the trolling narrative, while misleading, was harmless as the author took all the "blame", but instead got the "fame". In this case, the trolling narrative exploited the lack of immediate, accurate information and the public's curiosity about the outage's impact. Still, it reminds us of the potential dangers of misinformation in today's media landscape, where sensational stories often overshadow the need for accurate reporting.
According to Vincent, "several things make it a good fake that worked :
1. No culprit named yet, I bring it on a platter, people like to have a culprit.
2- The culprit seems completely stupid, he is proud of his stupidity, he... takes his afternoon off on... the first day of work...
3- This falls right into a huge buzz in which people absolutely need to have new information, and a fake is by nature new, you won't read it anywhere else
4- In English = very easy to share internationally, with the vast majority of people who have no idea who I am.
5- Baby fingers are stupid, but they distract people from other things (like the fact that I have a horn on my head because of bad clipping)
6- Confirmation bias: People want to believe it, it's so funny. 'I like the information, so it is true.'
7- The information is pushed ironically by people who know that it is a joke, and then it arrives in the litteral zone which amplifies it even stronger.
However, we don't live in the age of good journalism anymore, but the age of monetized media, where every click counts, the entertainment rules and the truth is of secondary importance...
What Really Happened? The Truth Behind the IT Outage
The true nature of the incident involved a Blue Screen of Death (BSOD) at various locations, including Diego International Airport. Affected machines were stuck in a recovery blue screen at boot, a problem associated with a specific CrowdStrike update. The trolling response added confusion by providing a seemingly plausible but ultimately unverified explanation.
Unpacking the BSOD Bug - A Deeper Dive
Blue Screens of Death at Diego International Airport
For those looking to delve deeper into the technical aspects of the BSOD bug and how to resolve it, several resources are available. A Windows developer explains the CrowdStrike IT outage, and the company's CEO has assured that they are working on resolving the issue. Additional technical details are available in the Falcon Content Update for Windows Hosts provided by CrowdStrike and Microsoft's documentation on the related KB5042421 issue.
Affected machines are stuck in a recovery blue screen at boot. Image: Microsoft
And if you want to understand what really is the famous BSOD bug about and how to fix it, here is a video for you...
CrowdStrike IT Outage Explained by a Windows Developer
CrowdStrike CEO: ‘We know what the issue is’ and are resolving it
Here is more information about the issue directly from the source, from Crowdstrike:
Technical Details: Falcon Content Update for Windows Hosts
And similarly, from Microsoft:
KB5042421: CrowdStrike issue impacting Windows endpoints causing an 0x50 or 0x7E error message on a blue screen
How it Really Happened?
We don't know exactly, but...
Here is what a software engineer known by his online nickname topgun966 wrote on Readdit thread related to the issue:
"Here's the problem.
I have been a software engineer for almost 20 years.
I work in the cyber security field now.
My job focuses on container security KSPM and CSPM.
Crowdstrike at my company is handled by a different group. So this didn't affect me that much but I did help that team out where I could.
My biggest gripe is HOW they rolled it out.
First, updates on Friday.
That breaks so many unspoken rules (and some official policies at a lot of companies).
Next, not a phased rollout. Just a blanked push by region.
When you are deploying code to so many devices, regardless if it is an application code base update, or virus definitions, you do it in phases.
When you are modifying files in the Windows drivers folder, there are ALWAYS risks.
Windows has a lot of protections for its own files. If you modify a Windows file, it will recover itself.
But it was in the Crowdstrike folder in the drivers.
Windows doesn't monitor that obviously since it isn't its own files.
What they SHOULD have done and what is common practice is to slow roll it.
You do it in batches and have a pause in between.
The only exception to this is for critical updates like fixing an exploited zero day.
But routine updates can be tested till the end of time and will never flush out all the defects until it gets into customers systems.
This is why you do canary rollouts and 'scream' tests.
You roll it to maybe, 10k systems and wait. See if someone screams. If it is quiet, you bump it to 25k systems and wait again.
This is a solid reminder that proven software rollout policies need to be followed on an enterprise level.
That is where Crowdstrike failed the most, at a management level."
Kurtz from Crowdstrike Strikes Again
According to Business Insider, "tech industry analyst Anshel Sag pointed out that this isn't the first time Kurtz has played a major role in a historic IT blowout".
"On April 21, 2010, the antivirus company McAfee released an update to its software used by its corporate customers. The update deleted a key Windows file, causing millions of computers around the world to crash and repeatedly reboot. Much like the CrowdStrike mistake, the McAfee problem required a manual fix.
Kurtz was McAfee's chief technology officer at the time. Months later, Intel acquired McAfee. And several months after that Kurtz left the company. He founded CrowdStrike in 2012 and has been its CEO ever since."
More about it you can read here:
This is the 2nd time CrowdStrike CEO George Kurtz has been at the center of a global tech failure
LINKeR
Major Windows BSOD issue hits banks, airlines, and TV broadcasters
When Trolling Goes Viral - Crowdstrike Edition
Last weekend was quite an expensive one for many companies. How expensive?
Very! The costs are estimated to $5.4 billion (USD) just for the top Fortune 500 companies.
However, last weekend, according to the Tech Business News, at least 538 companies have been verified to have experienced significant disruptions due to a widespread IT security outage, which affected more than 8.5 million Windows devices, according to Microsoft's report estimate.
While software updates may occasionally cause disturbances, significant incidents like the CrowdStrike event are infrequent. We currently estimate that CrowdStrike’s update affected 8.5 million Windows devices, or less than one percent of all Windows machines. While the percentage was small, the broad economic and societal impacts reflect the use of CrowdStrike by enterprises that run many critical services.
CrowdStrike’s cyber security services are responsible for 298 of the Fortune 500 companies, 538 of the Fortune 1000 companies, and 8 out of 10 of the top financial firms.
This incident, which affected various sectors including banks and airports, led to a search for information as services were brought to a halt.
However, alongside this genuine crisis, a trolling response by Vincent Flibustier on X emerged as a response. This response used generative AI to craft a plausible but unverified narrative, essentially as a joke, taking the "blame" and offering a false scapegoat for those seeking quick explanations without verifying the details.
A Harmless Trolling Incident - The Viral Impact
Click on the image to find out more about how to create a viral video and what are the consequences of bad journalism without factchecking...
The trolling incident quickly gained traction, amassing 45 million views in just three days. The story seemed very plausible, leading many to speculate whether something similar actually occurred at CrowdStrike. However, the details of the actual incident remain unclear. This highlights how quickly information, even if unverified, can spread when it taps into existing biases or fills a gap in the available details.
In this case, the trolling narrative, while misleading, was harmless as the author took all the "blame", but instead got the "fame". In this case, the trolling narrative exploited the lack of immediate, accurate information and the public's curiosity about the outage's impact. Still, it reminds us of the potential dangers of misinformation in today's media landscape, where sensational stories often overshadow the need for accurate reporting.
According to Vincent, "several things make it a good fake that worked :
👇
1. No culprit named yet, I bring it on a platter, people like to have a culprit.
2- The culprit seems completely stupid, he is proud of his stupidity, he... takes his afternoon off on... the first day of work...
3- This falls right into a huge buzz in which people absolutely need to have new information, and a fake is by nature new, you won't read it anywhere else
4- In English = very easy to share internationally, with the vast majority of people who have no idea who I am.
5- Baby fingers are stupid, but they distract people from other things (like the fact that I have a horn on my head because of bad clipping)
6- Confirmation bias: People want to believe it, it's so funny. 'I like the information, so it is true.'
7- The information is pushed ironically by people who know that it is a joke, and then it arrives in the litteral zone which amplifies it even stronger.
However, we don't live in the age of good journalism anymore, but the age of monetized media, where every click counts, the entertainment rules and the truth is of secondary importance...
What Really Happened? The Truth Behind the IT Outage
The true nature of the incident involved a Blue Screen of Death (BSOD) at various locations, including Diego International Airport. Affected machines were stuck in a recovery blue screen at boot, a problem associated with a specific CrowdStrike update. The trolling response added confusion by providing a seemingly plausible but ultimately unverified explanation.
Unpacking the BSOD Bug - A Deeper Dive
Blue Screens of Death at Diego International Airport
For those looking to delve deeper into the technical aspects of the BSOD bug and how to resolve it, several resources are available. A Windows developer explains the CrowdStrike IT outage, and the company's CEO has assured that they are working on resolving the issue. Additional technical details are available in the Falcon Content Update for Windows Hosts provided by CrowdStrike and Microsoft's documentation on the related KB5042421 issue.
Affected machines are stuck in a recovery blue screen at boot. Image: Microsoft
And if you want to understand what really is the famous BSOD bug about and how to fix it, here is a video for you...
CrowdStrike IT Outage Explained by a Windows Developer
CrowdStrike CEO: ‘We know what the issue is’ and are resolving it
Here is more information about the issue directly from the source, from Crowdstrike:
Technical Details: Falcon Content Update for Windows Hosts
And similarly, from Microsoft:
KB5042421: CrowdStrike issue impacting Windows endpoints causing an 0x50 or 0x7E error message on a blue screen
How it Really Happened?
We don't know exactly, but...
Here is what a software engineer known by his online nickname topgun966 wrote on Readdit thread related to the issue:
"Here's the problem.
I have been a software engineer for almost 20 years.
I work in the cyber security field now.
My job focuses on container security KSPM and CSPM.
Crowdstrike at my company is handled by a different group. So this didn't affect me that much but I did help that team out where I could.
My biggest gripe is HOW they rolled it out.
First, updates on Friday.
That breaks so many unspoken rules (and some official policies at a lot of companies).
Next, not a phased rollout. Just a blanked push by region.
When you are deploying code to so many devices, regardless if it is an application code base update, or virus definitions, you do it in phases.
When you are modifying files in the Windows drivers folder, there are ALWAYS risks.
Windows has a lot of protections for its own files. If you modify a Windows file, it will recover itself.
But it was in the Crowdstrike folder in the drivers.
Windows doesn't monitor that obviously since it isn't its own files.
What they SHOULD have done and what is common practice is to slow roll it.
You do it in batches and have a pause in between.
The only exception to this is for critical updates like fixing an exploited zero day.
But routine updates can be tested till the end of time and will never flush out all the defects until it gets into customers systems.
This is why you do canary rollouts and 'scream' tests.
You roll it to maybe, 10k systems and wait. See if someone screams. If it is quiet, you bump it to 25k systems and wait again.
This is a solid reminder that proven software rollout policies need to be followed on an enterprise level.
That is where Crowdstrike failed the most, at a management level."
Kurtz from Crowdstrike Strikes Again
According to Business Insider, "tech industry analyst Anshel Sag pointed out that this isn't the first time Kurtz has played a major role in a historic IT blowout".
"On April 21, 2010, the antivirus company McAfee released an update to its software used by its corporate customers. The update deleted a key Windows file, causing millions of computers around the world to crash and repeatedly reboot. Much like the CrowdStrike mistake, the McAfee problem required a manual fix.
Kurtz was McAfee's chief technology officer at the time. Months later, Intel acquired McAfee. And several months after that Kurtz left the company. He founded CrowdStrike in 2012 and has been its CEO ever since."
More about it you can read here:
This is the 2nd time CrowdStrike CEO George Kurtz has been at the center of a global tech failure
LINKeR
Major Windows BSOD issue hits banks, airlines, and TV broadcasters
Facebook
Twitter
Discord
Comments
Umber bear was here
🙂
\\\\ /
o7
VOTE 🙂
Interesting info.
System crashes follow this guy around!
v + c+ e
Are you telling me that a first day idiot could destroy all this by doing a sht job and taking the afternoon off ?
lol. When this is true, we’ve other issues. Really.
Ah I didn’t read your full article. Nice background information. Yeah I guess this „first day rookie“ could be only bs. And being honest, who let first day dudes doing such things is either stupid or very very naiv. So on both ways the credits for the company is bad. Now even worst.