To the less IT literate Crowdstrike sounds like malware. It does to me anyway.
I am IT literate, and it sounds like that to me too 🙂
What actually is Cloudstrike?
Rather ironically, it's a system intended to stop hackers crashing PCs 😀
How true this is for the Cloud.
It's true for everything not just IT. A large part of my job is telling people that yes, this COULD go wrong and if it does it will cost you a lot of money, so mitigate it.
My work is unaffected but I have not been able to get onto STW all morning until now.
Its crazy there aren't enough fail safes built in to the system to prevent one component having a wobbly bricking things that integrate with it.
Reminds me of the air traffic control system says no moment a few years ago, but way worse....
There will definitely be a reckoning for how a trusted company like Crowdstrike has pushed out a dodgy patch like this, it just shouldn't be possible with correct procedures in place unless they've been compromised and what's gone out was never an authorised patch.
The resulting event is by far the biggest IT meltdown I can recall and as someone else has said the fix isn't easy if you have BitLocker running (which most IT literate companies will have on their EUDs) and don't have access to the recovery key (even worse if the issue has taken out your AD so you can't extract them centrally)
I'm just glad I work on an air-gapped secure network :p I think some colleagues are going to have busy weekends though 🙁
Reminds me I must re-read Second Sleep by Robert Harris.
it just shouldn’t be possible with correct procedures in place unless they’ve been compromised and what’s gone out was never an authorised patch.
That could be one possibility.
https://twitter.com/GossiTheDog/status/1814217357058842914
"I have obtained the Crowdstrike driver they pushed via auto update. I don't know how it happened, but the file isn't a validly formatted driver and causes Windows to crash every time."
https://twitter.com/GossiTheDog/status/1814217357058842914
How true this is for the Cloud. The man was totally ahead if his time.
There is no "cloud". Its just someone else's server. But because you've gone "cloud" instead of calling your IT guy to fix it you are now at the end of long queue of people waiting on the cloud provider to offer a fix*
*I know this is a massive over-simplification
Fix for a BitLocker enabled system if you don't have the recovery key BUT you do need to have local admin rights (might be a bit confusing without the accompanying screenshots), I haven't validated this myself but it's been sent out as a fix by our internal IT:
Start Computer
Press ESC (this is on the BitLocker passcode entry screen and takes you into BitLocker Recovery mode)
Press ESC again
Skip drive
Choose Troubleshoot
Choose Advanced options
Choose Command Prompt
Write command “bcdedit /set {default} safeboot minimal” and press enter. Afterwards write command “exit” and restart pc.
During boot enter Bitlocker and windows will run in to safe mode – there you will need enter Local Admin login.
Open browser and location C:\Windows\System32\drivers\CrowdStrike\
Delete all files with starting “C-00000291*
Once its deleted, open C:\Windows\System32\cmd.exe
Write command “bcdedit /deletevalue {default} safeboot
Restart computer and normally login – computer should work
In case it doesn’t work make sure in step 10 you removed proper file “291” have to be in first part not second or third.
I am sitting here with an update to our company's software that I've just finished writing. The news today has given me serious heebie-jeebies... think I'll do a little more testing, just in case haha
Hope no-one has shares in Crowdstrike
"Crowdstrike has lost a fifth of its value in pre-market trading in the US - down 21% in unofficial trading.
If confirmed when US stock markets open later today, that is a loss of $16 billion in its overnight valuation."
https://www.bbc.co.uk/news/live/cnk4jdwp49et
but I have not been able to get onto STW all morning until now.
That has been going on for a few days, I have posted about it a few times in the "report issues" sticky.
How can a company like Crowdstrike possibly be "worth" $80Bn? That's an insane valuation even without this. What kind of secret snake oil are/were they selling?
(Posted from my work Linux laptop).
Crowdstrke begins to learn rapidly and eventually becomes self-aware at 2:14 a.m., EDT, on July 19th, 2024.
I kind of hope it is malicious, otherwise I'm imagining some poor programmer in Crowdstrike's office hiding under his desk in a puddle of urine, gibbering to themselves while the company goes into meltdown around them.
It will be interesting to see what kind of "root cause analysis" gets released. IMO it is likely that all endpoint protection providers have similar processes, and trying to double guess who could have similar problems in the future from a one off incident probably isn't going to work. One theory would be that crowdstrile should now be much more careful for another few years at least, so would likely be more reliable for now than their competitors.
We run completely separate "chains" of computing in our operational controlling, maybe we should have different endpoint protection on each chain.
How can a company like Crowdstrike possibly be “worth” $80Bn?
Their customers are huge, their product is industry leading (up to now) and really, really expensive.
Very much NOT snake oil either. They offer a million dollars to anyone who gets hacked while using their software, which they’ve never had to pay out on.
IMO it is likely that all endpoint protection providers have similar processes
I wonder if Microsoft will make anything of it (as in "I told you so" as they're forced to open up this sort of low level access to vendors for competition's sake), maybe in Windows 12 MS Defender will be the only endpoint protection client that can work at this level...
My boss has been" working" from home since the pandemic . Does this mean he might actually have to come in and do some actual hands on?
Bloody hope not as he is clueless
I think crowdstrike is multi platform, which is 1 of the reasons companies use it, rather than having different security systems and processes for every operating system used.
£1m is absolutely **** all to big company so that is probably worth as much as Giant’s warranty. Any hack that takes a megacorp offline for a prolonged period of time will certainly cost more than that in lost revenue/compensation to customers etcThey offer a million dollars to anyone who gets hacked while using their software, which they’ve never had to pay out on.
how much do you think this **** up is going to cost Crowdstrike?
But the (previously) flakiest forum in the world just powers on without issue.
Have you visited the Wordle thread? It's chaos over there.
Greg's is working don't panic.
So far we have had a support supply chain group try and implement a fix they found on the web.
This has not gone down well apparently.
They offer a million dollars to anyone who gets hacked while using their software, which they’ve never had to pay out on.
they better hope this update wasn't a supply chain hack or their in serious debt 😀
Our local "Spotted" page on FB has gone full "cash is king, don't trust computers, or the government" which is quite unusual for us round here
To compound matters, there was actually an issue in US Central Azure region this morning too which meant storage became unlinked from VMs. Nice...
It's ok Sandwich, no issues it would seem in the shops here, tea levels are restored 🙂
Phew, both bins emptied.
This is why we have a no change Friday policy at work. If something needs pushing out we do it Mon-Thu so no poor sod is working over the weekend if it goes wrong. Although we do have planned downtime at weekends for mission critical stuff.
On Monday I’ll be walking into my workroom and making curtains as usual.
It affects a different version of Windows.
The resulting event is by far the biggest IT meltdown I can recall and as someone else has said the fix isn’t easy
It's going to take, optimistically, weeks to resolve.
How can a company like Crowdstrike possibly be “worth” $80Bn? That’s an insane valuation even without this. What kind of secret snake oil are/were they selling?
Crowdstrike is - well, was - very highly regarded. It's also very highly expensive.
Does anyone know if TicketMaster is affected? Trying to login and it says Email address not recognised despite it working yesterday..
Got a gig at weekend so need to access the tickets
Do you have an email copy of the tix maybe?
Unfortunately not.
Buy quill pens, parchment and make your own ink using soot and water.
Would love to like a lot of these comments but alas I’m unable to do so.
Love how this thread is a mix of IT helpdesk and comedy. Also love that I landed in the USA 12 hours or so before air travel went to shit. Phew. Also, is #humblesmug a thing?
I'm seeing a lot of predictable "Microsoft sucks" posts on places like Facebook.
For the record, this is nothing to do with Microsoft. An automatic update to a third-party application (CrowdStrike Falcon) pushed out malformed, unsigned code, and Windows - absolutely correctly - slammed on the brakes rather than allowing unverified and potentially malicious code to execute.
From the point of view of Windows this is intended, desired behaviour in response to something that shouldn't happen. Make no mistake, this is Bad, Falcon is in layman's terms a highly advanced antivirus product; it is supposed to be buried deep into the system and difficult to remove/bypass so that malware can't knobble it, which makes fixing it tricky. In many cases it's going to be a manual task on individual machines and I expect it is going to take weeks for some organisations to fully recover but a potential alternative could have been far worse.
CrowdStrike claims to have discovered a defect in their update system and rectified it, unsubstantiated rumours suggest that corruption may have happened "in flight" via their Content Delivery Network. Whether this is actually the case, I don't know.
In any case, I suspect there are going to be a lot of questions and introspection once the dust settles. Vendors like CrowdStrike operate with little to no regulation, "marking their own homework" if you will. I bet that's going to change.
Love how this thread is a mix of IT helpdesk and comedy. Also love that I landed in the USA 12 hours or so before air travel went to shit. Phew.
You should be ok until you need to make a card payment or get cash out of a machine! Good luck.
All is well. Amazon Prime Video is working so I can watch The Boys
Do any of the IT bods have a 'plain english' translation thing we can use to understand what you're wittering on about.
Also, using acronyms is only ever a way to present an 'aura of mystique' and exclude those who don't believe in communicating by using acronym soup. It's unnecessary bollocks.
Be clear, concise and, most importantly, intelligible - please.
someone made a boo boo
An IT security company (CrowdStrike) pushed out an update (not in the right format) to part of it's software, window's tried to use this updated file, didn't like it and then refused to switch on, meaning the computer is now effectively dead until someone comes along and manually removes the broken file.
The problem is that lots of big companies use windows computers and this CrowdStrike software, so lots of computers all stopped working at the same time.
A large part of my job is telling people that yes, this COULD go wrong and if it does it will cost you a lot of money, so mitigate it.
You work in the local off license...?
