IT End of World – STW going strong

What actually is Cloudstrike?

Hope no-one has shares in Crowdstrike

On Monday I’ll be walking into my workroom and making curtains as usual.

Hoorah for STW, however the ‘like’ button still doesn’t work.

pic.twitter.com/rf1mMGSP0s

— Universal Exporter (@FFF182) July 19, 2024

OK… Did anyone else (apart from me and the 20+ other blokes here in work) see Sky News boardcasting Pro Russian news on loop between 0630 and 0745…

(No! !…. None of us are wearing foil hats)

To the less IT literate Crowdstrike sounds like malware. It does to me anyway.

I am IT literate, and it sounds like that to me too 🙂

What actually is Cloudstrike?

Rather ironically, it’s a system intended to stop hackers crashing PCs 😀

How true this is for the Cloud.

It’s true for everything not just IT. A large part of my job is telling people that yes, this COULD go wrong and if it does it will cost you a lot of money, so mitigate it.

My work is unaffected but I have not been able to get onto STW all morning until now.

Its crazy there aren’t enough fail safes built in to the system to prevent one component having a wobbly bricking things that integrate with it.

Reminds me of the air traffic control system says no moment a few years ago, but way worse….

There will definitely be a reckoning for how a trusted company like Crowdstrike has pushed out a dodgy patch like this, it just shouldn’t be possible with correct procedures in place unless they’ve been compromised and what’s gone out was never an authorised patch.

The resulting event is by far the biggest IT meltdown I can recall and as someone else has said the fix isn’t easy if you have BitLocker running (which most IT literate companies will have on their EUDs) and don’t have access to the recovery key (even worse if the issue has taken out your AD so you can’t extract them centrally)

I’m just glad I work on an air-gapped secure network :p I think some colleagues are going to have busy weekends though 🙁

Reminds me I must re-read Second Sleep by Robert Harris.

it just shouldn’t be possible with correct procedures in place unless they’ve been compromised and what’s gone out was never an authorised patch.

That could be one possibility.

https://x.com/GossiTheDog/status/1814217357058842914

“I have obtained the Crowdstrike driver they pushed via auto update. I don’t know how it happened, but the file isn’t a validly formatted driver and causes Windows to crash every time.”

https://x.com/GossiTheDog/status/1814217357058842914https://x.com/GossiTheDog/status/1814217357058842914

https://x.com/GossiTheDog/status/1814217357058842914

How true this is for the Cloud. The man was totally ahead if his time.

There is no “cloud”.  Its just someone else’s server.  But because you’ve gone “cloud” instead of calling your IT guy to fix it you are now at the end of long queue of people waiting on the cloud provider to offer a fix*

*I know this is a massive over-simplification

Fix for a BitLocker enabled system if you don’t have the recovery key BUT you do need to have local admin rights (might be a bit confusing without the accompanying screenshots), I haven’t validated this myself but it’s been sent out as a fix by our internal IT:

Start Computer

Press ESC (this is on the BitLocker passcode entry screen and takes you into BitLocker Recovery mode)

Press ESC again

Skip drive

Choose Troubleshoot

Choose Advanced options

Choose Command Prompt

Write command “bcdedit /set {default} safeboot minimal” and press enter. Afterwards write command “exit” and restart pc.

During boot enter Bitlocker and windows will run in to safe mode – there you will need enter Local Admin login.

Open browser and location C:\Windows\System32\drivers\CrowdStrike\

Delete all files with starting “C-00000291*

Once its deleted, open C:\Windows\System32\cmd.exe

Write command “bcdedit /deletevalue {default} safeboot

Restart computer and normally login – computer should work

In case it doesn’t work make sure in step 10 you removed proper file “291” have to be in first part not second or third.

I am sitting here with an update to our company’s software that I’ve just finished writing. The news today has given me serious heebie-jeebies… think I’ll do a little more testing, just in case haha

Hope no-one has shares in Crowdstrike

“Crowdstrike has lost a fifth of its value in pre-market trading in the US – down 21% in unofficial trading.

If confirmed when US stock markets open later today, that is a loss of $16 billion in its overnight valuation.”

https://www.bbc.co.uk/news/live/cnk4jdwp49et

but I have not been able to get onto STW all morning until now.

That has been going on for a few days, I have posted about it a few times in the “report issues” sticky.

Global CrowdStrike outage leaves Mercedes fixing computers before practice

How can a company like Crowdstrike possibly be “worth” $80Bn? That’s an insane valuation even without this. What kind of secret snake oil are/were they selling?

(Posted from my work Linux laptop).

Crowdstrke begins to learn rapidly and eventually becomes self-aware at 2:14 a.m., EDT, on July 19th, 2024.

I kind of hope it is malicious, otherwise I’m imagining some poor programmer in Crowdstrike’s office hiding under his desk in a puddle of urine, gibbering to themselves while the company goes into meltdown around them.

It will be interesting to see what kind of “root cause analysis” gets released. IMO it is likely that all endpoint protection providers have similar processes, and trying to double guess who could have similar problems in the future from a one off incident probably isn’t going to work. One theory would be that crowdstrile should now be much more careful for another few years at least, so would likely be more reliable for now than their competitors.

We run completely separate “chains” of computing in our operational controlling, maybe we should have different endpoint protection on each chain.

How can a company like Crowdstrike possibly be “worth” $80Bn?

Their customers are huge, their product is industry leading (up to now) and really, really expensive.

Very much NOT snake oil either. They offer a million dollars to anyone who gets hacked while using their software, which they’ve never had to pay out on.

IMO it is likely that all endpoint protection providers have similar processes

I wonder if Microsoft will make anything of it (as in “I told you so” as they’re forced to open up this sort of low level access to vendors for competition’s sake), maybe in Windows 12 MS Defender will be the only endpoint protection client that can work at this level…

My boss has been” working” from home since the pandemic . Does this mean he might actually have to come in and do some actual hands on?

Bloody hope not as he is clueless

I think crowdstrike is multi platform, which is 1 of the reasons companies use it, rather than having different security systems and processes for every operating system used.

They offer a million dollars to anyone who gets hacked while using their software, which they’ve never had to pay out on.

£1m is absolutely **** all to big company so that is probably worth as much as Giant’s warranty. Any hack that takes a megacorp offline for a prolonged period of time will certainly cost more than that in lost revenue/compensation to customers etc

how much do you think this **** up is going to cost Crowdstrike?

But the (previously) flakiest forum in the world just powers on without issue.

Have you visited the Wordle thread? It’s chaos over there.

Greg’s is working don’t panic.

So far we have had a support supply chain group try and implement a fix they found on the web.

This has not gone down well apparently.

They offer a million dollars to anyone who gets hacked while using their software, which they’ve never had to pay out on.

they better hope this update wasn’t a supply chain hack or their in serious debt 😀

Our local “Spotted” page on FB has gone full “cash is king, don’t trust computers, or the government” which is quite unusual for us round here

To compound matters, there was actually an issue in US Central Azure region this morning too which meant storage became unlinked from VMs. Nice…

Pray for @longdog.

It’s ok Sandwich,  no issues it would seem in the shops here, tea levels are restored 🙂

Phew, both bins emptied.

This is why we have a no change Friday policy at work. If something needs pushing out we do it Mon-Thu so no poor sod is working over the weekend if it goes wrong. Although we do have planned downtime at weekends for mission critical stuff.

@jeffl do you work at NASA?

On Monday I’ll be walking into my workroom and making curtains as usual.

It affects a different version of Windows.

The resulting event is by far the biggest IT meltdown I can recall and as someone else has said the fix isn’t easy

It’s going to take, optimistically, weeks to resolve.

How can a company like Crowdstrike possibly be “worth” $80Bn? That’s an insane valuation even without this. What kind of secret snake oil are/were they selling?

Crowdstrike is – well, was – very highly regarded. It’s also very highly expensive.

Does anyone know if TicketMaster is affected? Trying to login and it says Email address not recognised despite it working yesterday..

Got a gig at weekend so need to access the tickets

Do you have an email copy of the tix maybe?

Unfortunately not.