IT End of World – STW going strong

Buy quill pens, parchment and make your own ink using soot and water.

Would love to like a lot of these comments but alas I’m unable to do so.

Love how this thread is a mix of IT helpdesk and comedy. Also love that I landed in the USA 12 hours or so before air travel went to shit. Phew. Also, is #humblesmug a thing?

I’m seeing a lot of predictable “Microsoft sucks” posts on places like Facebook.

For the record, this is nothing to do with Microsoft. An automatic update to a third-party application (CrowdStrike Falcon) pushed out malformed, unsigned code, and Windows – absolutely correctly – slammed on the brakes rather than allowing unverified and potentially malicious code to execute.

From the point of view of Windows this is intended, desired behaviour in response to something that shouldn’t happen. Make no mistake, this is Bad, Falcon is in layman’s terms a highly advanced antivirus product; it is supposed to be buried deep into the system and difficult to remove/bypass so that malware can’t knobble it, which makes fixing it tricky.  In many cases it’s going to be a manual task on individual machines and I expect it is going to take weeks for some organisations to fully recover but a potential alternative could have been far worse.

CrowdStrike claims to have discovered a defect in their update system and rectified it, unsubstantiated rumours suggest that corruption may have happened “in flight” via their Content Delivery Network.  Whether this is actually the case, I don’t know.

In any case, I suspect there are going to be a lot of questions and introspection once the dust settles.  Vendors like CrowdStrike operate with little to no regulation, “marking their own homework” if you will.  I bet that’s going to change.

Love how this thread is a mix of IT helpdesk and comedy. Also love that I landed in the USA 12 hours or so before air travel went to shit. Phew.

You should be ok until you need to make a card payment or get cash out of a machine! Good luck.

All is well. Amazon Prime Video is working so I can watch The Boys

Do any of the IT bods have a ‘plain english’ translation thing we can use to understand what you’re wittering on about.

Also, using acronyms is only ever a way to present an ‘aura of mystique’ and exclude those who don’t believe in communicating by using acronym soup. It’s unnecessary bollocks.

Be clear, concise and, most importantly, intelligible – please.

someone made a boo boo

An IT security company (CrowdStrike) pushed out an update (not in the right format) to part of it’s software, window’s tried to use this updated file, didn’t like it and then refused to switch on, meaning the computer is now effectively dead until someone comes along and manually removes the broken file.

The problem is that lots of big companies use windows computers and this CrowdStrike software, so lots of computers all stopped working at the same time.

A large part of my job is telling people that yes, this COULD go wrong and if it does it will cost you a lot of money, so mitigate it.

You work in the local off license…?

Do any of the IT bods have a ‘plain english’ translation thing we can use to understand what you’re wittering on about.

You remember the bloke with the submarine that was kind of winging it and then it imploded?

It’s the same as that, but with computers and trillions of dollars and broken transport and broken healthcare systems instead

Agree with cougar – more regulation is the likely outcome.

I work in IT. For an IT vendor. Risks are seen and tolerated chasing £. Corners get cut.

frankconway

Do any of the IT bods have a ‘plain english’ translation thing we can use to understand what you’re wittering on about.

To be fair I think @Cougar did that an hour ago?

“Well actually BSOD is Windows working exactly as intended.” Absolutely ****ing glorious! Chapeau.

I’m seeing a lot of predictable “Microsoft sucks” posts on places like Facebook.

I have seen a few posts flagging up similar stuff happened a couple of months back with a couple of Linux distros. Didnt play well with versions slightly behind and went pearshaped in a not dissimilar fashion.

Vendors like CrowdStrike operate with little to no regulation, “marking their own homework” if you will.

Its a tricky one because they need to be able to push stuff quickly to shut down zero days and who is going to regulate and mark their homework?

I would say getting sued might do the trick but Solarwind have managed to mostly defang an SEC lawsuit for their incompetent security practices.

The patch file is all zeros….

https://x.com/jeremyphoward/status/1814364640127922499

All I’m taking away from this is stop updating stuff. Windows updates are bad enough. After each one the menus, icons and general feel get closer to a Fisher-Price My First Computer vibe.

Does anyone know if TicketMaster is affected?

Your gig aside, that’s one company I’d love to see take a nosedive right down the shitter.

To be fair I think @Cougar did that an hour ago?

I attempted to.

“Well actually BSOD is Windows working exactly as intended.” Absolutely ****ing glorious! Chapeau.

Rewind far enough and you could BSOD a Windows box by unplugging the keyboard. Remember the days when you could have a power cut and you’d spend two days trying to recover it because it’d shat itself?  Times have changed.

Today Windows architecture is far more tightly secured, you cannot just slot in any old shit into it.  The issue in this particular case is that Falcon is essentially a rootkit, it operates at a very low level because it has to. A compromise at that level could be wildly catastrophic, so in the event of this kind of failure – ie fundamental code not being what it claims to be – Windows just stops rather than allowing who-knows-what to run amok with gay abandon. This is literally what a modern BSOD is, it’s damage limitation because code is not what it claims to be.  If your car caught fire would you rather it stopped to let you get out or just carry on burning in the middle lane of the M6?

As I said, the alternative is far worse.

Its a tricky one because they need to be able to push stuff quickly to shut down zero days and who is going to regulate and mark their homework?

That my friend is a very interesting question indeed.

All I’m taking away from this is stop updating stuff.

Don’t, that is akin to an anti-vax argument.  Once more with feeling, “the alternative is far worse.”

This incident is, in the panorama of incidents over decades, extraordinary.  Patching avoids many such incidents daily only a) they don’t hit the news any more than we see headlines going “still no Polio” and b) would you rather have an outage from a mistake or by deliberate malicious intent?

The semi-recent outbreak which took out half of the NHS, the vulnerability had been patched for months but it was never applied.  This scenario is FAR more common than what we’ve seen today.  In fact, I’d probably go so far as to say that today has been unique.

Patch your shit.

My brother is the Sky News newsreader that launches all their weekday live broadcasts at 6 am. Certainly was an interesting morning, when you become the news but don’t yet know the news as to why your systems are so broken. They drank a lot of coffee until they managed to fully get back on air at 9 am.

Interesting, I was absolutely oblivious to any of this happening until i saw this thread .

What’s been missing I haven’t missed

What got turned off I haven’t turned on

What couldn’t run well I always walk or cycle

I really do wish Whatever button got pressed stuck like a 13 year old reverb in a well used cotic soul and stayed off.

Even if that meant obviously losing here

The problem is that lots of big companies use windows computers and this CrowdStrike software, so lots of computers all stopped working at the same time.

Genius move to push an potentially bricking update to every single client machine in one go!

Have to admit, if an enemy government wanted the ability to screw with western critical infrastructure worldwide, all they have to do is start-up a cybersecurity firm in the US and wait a bit.

Didn’t the founder of Kapersky have some ‘interesting’ links with the KGB/FSB?

Interesting, I was absolutely oblivious to any of this happening

I realised how big an issue this is when I went to the doctor’s yesterday morning. Staff working from slips of paper with patients’ names and DOB on, no access to medical history, couldn’t prescribe or raise a referral. Basically, all they could do was have a chat and a physical examination with no follow-up.

The link to medical records is still down 24 hours later. Guessing they have a massive data validation job on their hands now, it’s not as simple as getting a few BSOD Windows boxes going again.

@Big-Bud most folk need this stuff at some point or another:

For our health

For family

For friends

For our jobs

Despite media reports, having a connected digital world parallel to the real one has improved health, communications, creativity and productivity. Going back would be a regression.

According to the CrowdStrike blog it wasn’t a code update signed or otherwise, just a config file update that caused ‘logic error’ making their Falcon engine bomb. These config files get released several times a day so they can quickly respond to new threats, but also promptly take down any machine online during a one hour window the other night. That explains the breadth and swiftness of the issue, and why booting into safe mode and deleting the config file fixes it.

Didn’t the founder of Kapersky have some ‘interesting’ links with the KGB/FSB?

There are/were rumours.  However you slice it though, they’re a Russian company.  Make what you will of having anything to do with Russia as your security provider.

it wasn’t a code update signed or otherwise, just a config file update

I find it funny that suppliers spew and customers accept this distinction as somehow more excusable

According to the CrowdStrike blog it wasn’t a code update signed or otherwise, just a config file update that caused ‘logic error’ making their Falcon engine bomb.

I’m no coder, but aren’t logic errors avoidable? Generally one checks, for example, that a value isn’t zero before trying to divide by it

Sounds a bit like the logic bomb from Portal 2.

Should have put a try…catch block around the bit of code that tries to read the config file to handle the exception cleanly lol!

Don’t, that is akin to an anti-vax argument.

Where’s the laughing face emoji when you need it. It was meant in jest.

Technology is great but we have become hugely over reliant on it. Worse, we’ve left all the people with no social skills in charge of it! Every IT department should have a protocol droid like C3PO to translate things from coder/IT bod in to language the rest of us can understand.  (Inset winking emoji here). Now go, roll your hundred sided dice and hope for a +7 to reversing ****!

it wasn’t a code update signed or otherwise, just a config file update

.

I find it funny that suppliers spew and customers accept this distinction as somehow more excusable

In CrowdStrike parlance it was a “channel file.”  But to report that would be meaningless to most people.  I was wrong earlier about it being a driver update – I’m piecing this together on holiday when I can – but the rest of the post stands, the component which uses the channel file is a system level driver.  Windows stopped procedures because it saw something it didn’t like and couldn’t terminate gracefully because of the level it operates..

Flights grounded,
Trains halted,
Stock exchange not trading,
Sky news off air.
Paxman and his underwear

You’ll be telling us you started a fire next…

Very well done, sir! Highly commended.

At least I’ve still got my source of emojis that works… *smug picachu face*

Does anyone know if TicketMaster is affected? Trying to login and it says Email address not recognised despite it working yesterday..

Got a gig at weekend so need to access the tickets

Ticketmaster are bloody awful, but I always try to save my tickets onto my phone, just in case. If you’ve got your ticket order reference, then, depending on the venue, if you get there early a member of the venue staff should be able to print a copy of the ticket – I had a problem with mine at a Roundhouse gig, and I had paper tickets printed at the desk beforehand.

Where’s the laughing face emoji when you need it. It was meant in jest.

Doh. 😀

I can’t find the article now, but I read earlier that one of the main individuals behind Crowdstrike sold a bunch of shares worth over a million dollars a day or so before this whole mess happened, which has both raised a few eyebrows, and instigated a call for an in-depth investigation. Unsurprisingly. Apparently, the sale was set up some months ago, with a deliberate delay to avoid charges of insider trading, the date it was set to go live unfortunately fell just before this all kicked off.
Still looks iffy, though.

Where’s the laughing face emoji when you need it. It was meant in jest.

Due to technical difficulties, we now have to do manual emojis ¯\_(ツ)_/¯

😉

:’(

Our Tesla thinks it’s a 30mph speed limit everywhere today until the cameras pick up an actual sign. It’s normally eerily accurate.

End of days I tell thee.

It affects a different version of Windows.

have you tried closing the curtains then opening them again?