[Closed] NHS in large scale IT shutdown

We moved to Win7 VMs 18/12 ago, which works fine except for the fact that I suspect some of our network is base 10 Ethernet.

However, there are a heck of a lot of old PCs about and the one on my desk has a 'Made for Windows XP' sticker on it (even if it's running a Win 7 VM)

The problem is that money is tight and it's really hard to justify replacing IT infrastructure that (sort of) works at the expense of clinical staff/equipment.

Right security appliance installed, AMP and Snort running, Windows now running again, just hope I don't get drowned in too many alerts now.

Anyway. I've just had a phone call, so now I'm going to have to drive to work. Not an infection but helping to keep it that way. Wish me luck.

[quote=MSP ]XP was superseded 11 years ago

Hang on - W7 wasn't released until 2009

[quote=richardkennerley ]A lot of the hardware we use is reliant on XP, upgrade the OS and the machines don't work. Not allowed to replace the machines because they're not broken.

Aargh - yes they are broken if they're running XP! Are they actually airgapped as Cougar suggests (which has to include a very robust policy regarding the connection of USB devices)?

Though I'm kind of surprised that W7 won't work - the real requirements are very little different to XP (I've run W7 on some very old hardware and extremely resource limited VMs - though it does help a lot if you tune it to get rid of a lot of rubbish).

aracer
MSP » XP was superseded 11 years ago
Hang on - W7 wasn't released until 2009

Yeah, but he said "superseded"

There's loads of xp apps that won't work on 7

Same for 2003 and 2008

Big corporate here. No un airgapped XP left for us though I suspect there are A few boxes hidden in cupboards just in case and it only takes one person..

We are currently rolling out a new patch on W7. Just to be sure to be sure.

[quote=nemesis ]There's loads of xp apps that won't work on 7

Not even using compatibility mode? I have a few which need that, but can't recall ever coming across one which didn't work with it.

I have a bit of experience of this, having done transition from XP to W7 in a school - the XP installation was supporting lots of really old software, including some from last century. Was fully expecting to tell the teachers they needed to move on to newer software, but in the event there wasn't anything we couldn't make work.

@aracer, opposite experience for me. Never found a single application where compatibility mode helped. We've still got a couple of xp machines kicking around our place running specific bits of software, but they aren't allowed a network connection.

@fifeandy & @aracer compatibly mode helps in 'some' simple cases when that fails I've never had any issues when I've made a proper compatibility shim using the app compatibility toolkit (or on occasion AppV to really fool an app into working on a later OS) yes its lots more work to do it that way but it works.

Though I'm kind of surprised that W7 won't work - the real requirements are very little different to XP

Also have kit with isa controller cards, def a few w2k boxes knocking around, think might be some earlier as well (none are networked)

Ah, we have the answer to one little mystery

http://www.bbc.co.uk/news/technology-39907049

Not even using compatibility mode? I have a few which need that, but can't recall ever coming across one which didn't work with it.

Anything that needs hardware drivers could be problematic.

Ah, we have the answer to one little mystery

I knew about that from Twitter - described by another security blogger as "stopping a speeding train." Saying it was an accident is a bit harsh, from what I could tell he knew exactly what he was doing.

So, I'm struggling to understand quite how its had such a big impact?

If it spreads via a SMB vulnerability, does this mean that each of these organisations have a WAN facing SMB port open on a machine that hasn't been patched? Then once inside it hunts out other SMB ports on the LAN and spreads itself?
If this is the case, in the NHS for example is it just infecting servers or also desktop machines? If so why do the desktop machines have the SMB port open anyway? Is it open by default?

Eg. at home I've got a little W7 PC that I use as a SMB server. It is behind a NAT and the port isn't forwarded to the WAN. Lets say its not patched (not sure if it is or not). So other than me clicking on a link in a spam email or visiting a malicious site, infecting one of the other computers on my LAN and it then spreading to my W7 SMB server, it can't get in right?

So how has it spread so far?!?

I'm not sure anyone can say at the moment how the original package got onto the NHS Network (N3). Could well have been a phishing email if you believe the media.

I think the problem may be compounded by the slightly misguided belief that N3 is nice and secure so you can trust anything on it. Not saying that applies to all NHS organisations but the impression I have personally is that some NHS organisations have their internet connectivity locked down far more than their N3 connectivity.

but once into the NHS network then, lets say via an email link, is it just infecting servers or do they have desktops with SMB services running? Can't understand why they would?

http://blog.talosintelligence.com/2017/05/wannacry.html

That's a good writeup of the issue.

At least this problem wasn't self-inflicted. Unlike the one a few months ago when an IT contractor emailed every nhs.net email address (about a million iirc), and then peoplpe started clicking Reply All to say "I don't think this is meant for me".

mrjmt - Member - Block User - Quote
So, I'm struggling to understand quite how its had such a big impact?

If it spreads via a SMB vulnerability, does this mean that each of these organisations have a WAN facing SMB port open on a machine that hasn't been patched?

No, its introduced via the usual- emails, downloaded attachments.

Then once inside it hunts out other SMB ports on the LAN and spreads itself?

AIUI, it uses any mapped SMB resources to explore, log, and then spread under that users credetials- easy with SSO setups.

If this is the case, in the NHS for example is it just infecting servers or also desktop machines?

Its agnostic re desktops or servers, it just wants to find as many files as it can and encrypt/infect.

If so why do the desktop machines have the SMB port open anyway? Is it open by default?

They need SMB ports open to access mapped resources, but internally only.

just think if a major bike forum and a few internet cycle retailers went down, there would be massive cycle anger in the uk, and a huge increase in production in most uk and world wide offices of cyclists

They need SMB ports open to access mapped resources, but internally only.

Ah, thats the bit I misunderstood, makes sense now I think about it, but I'd assumed that it was only computers that [i]hosted[/i] SMB shares that were vulnerable, but the ports sit open (and are vulnerable to the exploit) even if you only access shares on other machines and don't actually host any.

Aracer, it's pretty much been answered but in short, compatibility mode is sod all help ime. Big environment 1000s of users, multiple businesses and loads of bespoke or niche software that wouldn't run on w7

If it spreads via a SMB vulnerability, does this mean that each of these organisations have a WAN facing SMB port open on a machine that hasn't been patched? Then once inside it hunts out other SMB ports on the LAN and spreads itself?

No, its introduced via the usual- emails, downloaded attachments

There's nothing to indicate email was involved at all.

So yes, it could be that organisations have SMB exposed to the public Internet. It only takes one machine to be patient zero. It could also have brought in with a device; a user picks up an infection at home, brings their laptop into work where it automatically logs in to the Wi-Fi and boom, we're away.

Anyway, just got home.

We're a managed services provider for a lot of household names. Our internal machines are managed with WSUS, and our customer-facing cloud solution likewise falls under WSUS. Though of course, it's not that simple as the patch requires a reboot, so even though we've got multiple layers of redundancy you still have to be a little bit frosty about it.

We've also got various stand-alone servers, legacy systems, odds and sods dotted all over the place for various reasons. That's what I was asked to look at, so I've just spent seven hours hunting down and manually patching them. Fun.

All done? Will you sleep well now?

Now on a conference call...

It never ends.

Just got off the phone. That only took an hour. At least I get to (watch Doctor Who and) go to bed, the other guys are still at it. Reconvene at 9 tomorrow.

The last time this happened to me was the Millennium Bug, and we had ages to prepare for that.

Just had misfortune of hearing edwina currie on radio pontificating about this.
She clearly knows nothing other than the sensationalist headlines but that didn't stop her; one suggestion - 'go to imperial college, get a PhD student and tell them to sort it'.
Dopey cow.
Multiple references to 'hackers' but no mention of ransomeware.
No mention of widespread infections across multiple countries and business sectors.
No mention of NSA.
No mention of the 'kill switch'.
No discussion of possible perpetrators.
Much general criticism of Microsoft with no specifics or evidence.
She should not be used by broadcast media and then put down.

As for possible perpetrators - any takers for North Korea?

Cougar - Moderator

The last time this happened to me was the Millennium Bug, and we had ages to prepare for that.

And then ages after of people going "don't see what all the fuss was about, everything worked out fine, why do we listen to you bloody techies..."

I had the misfortune of hearing edwina currie on R5 a few weeks ago pontificating on cuts to disability benefit, I'd personally go on record as stating that I'd like to punch her in the face repeatedly until her face is reduced to a pulp, she is utterly repulsive and anyone who defends the decrepit Tory hag would be next in line to receive the same treatment.

@somafunk - so you're not her biggest fan?

We could start a thread - 'Who hates edwina currie - no reason required'.
Reasons would be good though.

She is vile, ignorant and condescending.

And then ages after of people going "don't see what all the fuss was about, everything worked out fine, why do we listen to you bloody techies..."

Yeah, that really, really makes me cross. "Nothing happened, what was all the fuss about?" Nothing happened because a lot of people put a lot of work into ensuring that nothing happened. At the turn of the Millennium when most people were having the party of their lives, I was sat in the office on my own with nothing further to do just in case "anything happened" (I spent most of the time shooting the breeze on the phone with a colleague in another company who'd got saddled with the same gig).

You have a shit IT dept and everyone notices, "what do we pay you for?"; you have a fantastic IT dept and no-one notices, "what do we pay you for?" It's a thankless bloody task.

(Point of order, I'm not IT any more though I've spent many years doing it. I got drafted in tonight as I was best placed to do this particular job and the primary teams were slammed with their own problems.)

so you're not her biggest fan?

That's certainly one polite way of expressing my disregard for her right to existence.

On the same program/discussion she mentioned that there was absolutely no poverty in this country and there was no need for societal use of food banks. Needless to say I utterly ****ing despise her and her ilk and would personally consider it a service to humanity to drag the decrepit whore by the hair round a few "choice" housing estates to explain her words to those who rely on such non-essential services. Of course for her to do this I'd have to remove or refrain from pounding her smug face with my fist which would leave me in a quandary, would I be satisfied with seeing her ripped apart by a baying crowd?, I suspect i would garner a certain thrill from seeing her fed to the wolves so to speak.

[quote=Cougar ]At the turn of the Millennium when most people were having the party of their lives, I was sat in the office on my own with nothing further to do just in case "anything happened"

Well to be fair, anybody with that job would surely appreciate that the start of the new millenium was actually a year later.

nice trolling NW - it was good to see Cougar so precisely following the script you'd given him the cue for!

NW wasn't trolling, he was empathising, is how I read it at least.

200!

So for seven years TM was Home Sec, head of a dept. taking decisions about matters like NHS security.

If soma had the choice, her or EC?

Oh, I remember 1999… a lot of work, a lot of good planning, a lot of problems caught live yet mitigated against before they went public. Well done all. Newspapers then acted like all that effort was a waste of time and money. Ever since I have NEVER relied on journalists working for newspapers, radio or TV to learn about anything remotely related technology of any kind. How many scientists or engineers work in the mainstream media? Next to none.

I'm still going. Started again at 9 this morning.

On the upside, I'll have netted about a week's pay by the time I'm done.

Sliver lining.

New bike?

Cougar - Moderator

NW wasn't trolling, he was empathising, is how I read it at least.

Kind of tying it to the current thing and some of the reasons why IT doesn't get the attention it needs

Still going. Got all the core servers done, now just mopping up all the shit like random Jump Servers. Just found my first XP SP2 machine... HULK SMASH!

I've MANUALLY patched and controlled-rebooted 60 servers so far. Yesterday all I had to eat until 10:30pm was a Mars bar. * malware writers, and *ing kill me now.

Just downed tools. 12 hours today.

No-one cares, do they? (-:

Nope 😉

Pah!....all you do is push buttons and bash keyboards, it's not as if you have a strenuous manual job that leaves you crippled by the end of the working day

Runs and hides under the stairs............. 😉

XP SP2? retro-tastic!!

No-one cares, do they? (-:

Depends who you are working for cougar - the NHS you get kudos for

Just downed tools. 12 hours today.

Anyway at last questions have been asked about those unsupported unpatched machines. I'm hoping they've been switched off and sod the consequences to the business.

Corporate, sadly. The NHS horse has bolted.

all you do is push buttons and bash keyboards, it's not as if you have a strenuous manual job that leaves you crippled by the end of the working day

I know you're joking, but in seriousness, I've done that. I used to have a job constructing kids' playgrounds, you know the soft play style things with ball pools and big plastic slides and stuff? We worked 12 hour shifts 7 days a week when doing installations (because the boss was a skinflint) and I've been more knackered after doing 8 hours thinking than I ever have lugging scaffolding poles about.

at last questions have been asked about those unsupported unpatched machines

You can still patch 2003 and XP (though I wouldn't advertise that fact).

corporate - you get no sympathy then - just charge the buggers a fortune for their lack of sense in not sorting out the vulnerabilities

You can still patch 2003 and XP

corporate - you get no sympathy then - just charge the buggers a fortune for their lack of sense in not sorting out the vulnerabilities

It's Not That Simple. Read back.

W2k??

Ye gods.

True story, when the Kaminsky DNS poisoning scare broke several years ago (2008?), we got a missive from a customer demanding what steps we were taking to mitigate the issues on their NT4 DNS servers.

Ye gods.

used to have a job constructing kids' playgrounds, you know the soft play style things with ball pools and big plastic slides and stuff?

Jeez!......this just gets better n' better....so you spent all day whizzing down slides and rolfing about in ball pits?, sounds like a perfect job 😀

Good man cougar.
How many/few organisations will block IT access until all (?) staff have been briefed on IT security?
Yes, I understand cost implications etc but how do they compare with damage caused by insecure processes and poor understanding/implentation of IT security.
Before the nhs mafia kicks off, I also understand the (broad) implications in terms of treatment and possible threat to life.

What about the use of third party media - personal memory sticks plugged into work network as an example. Sackable offence?

Banks have been (relatively) unaffected as far as we know; they should be prime targets which suggests that their collective efforts to secure their IT systems have been broadly successful.

Successive uk govs have commissioned so many unsuccessful large scale IT driven projects.
£10 billion worth of IT projects -
likely to be closer to £15 billion - written-off or scaled back since 2002.
No accountability but Crapita, Serco, PWC and others continue to flog dud solutions to gullible ministers & civil servants; jolly nice chablis, let's have another bottle and I'll sign the contract with this Mont Blanc pen you gave me at Christmas.

Gov criticises private sector; how about some self-flagellation based on incompetence and inaction in the public sector?

If I used random capitals and exclamation marks this could have been a rant.
It's not.
I'm one of the little guys who is thoroughly frustrated by self-serving and incompetent politicos; the ruling mentality can be summarised as....don't know, don't care but I'm important and you're not.

Did an audit on a prospective customer network last year, had uptimes on core switches and firewalls over 10 years and sounded shocked when we told them we wouldn't cover the targets they wanted us to meet for an outsource on their availability, change and security chapters buried within 1800 pages of contract docs.

You have a shit IT dept and everyone notices, "what do we pay you for?"; you have a fantastic IT dept and no-one notices, "what do we pay you for?" It's a thankless bloody task.

This x 1,000! I had the NYE1999 gig too. Got into the office at 14:00 on NYE and left site at 05:00 in the morning. Nothing happened. Apart from the 12 months previous that I'd spent arguing with the business that we really needed to harden up our AV protection.

Banks have been (relatively) unaffected as far as we know; they should be prime targets which suggests that their collective efforts to secure their IT systems have been broadly successful.

Or that their key IT systems are on completely different platforms.

Banks have been (relatively) unaffected as far as we know;

There's been two Russian banks affected that I'm aware of. And if you were an affected bank, would you disclose that?

@kelvin: agree but does not excuse culpability.

Jeez!......this just gets better n' better....so you spent all day whizzing down slides and rolfing about in ball pits?, sounds like a perfect job

Obviously, you have to test the UX.

What about the use of third party media - personal memory sticks plugged into work network as an example.

Pendrives have become less of a threat since autorun was disabled by default. But it's still a threat. (A fun trick for hackers / pen-testers is to 'accidentally' drop a loaded pendrive in the car park outside an organisation...)

And with round 2 starting at early breakfast time on Monday.......
'Cyber-attacks that have hit 150 countries since Friday should be treated by governments around the world as a "wake-up call", Microsoft says.
The computing giant said software vulnerabilities hoarded by governments have caused "widespread damage".

Let's work on the assumption that the 'kill switch' vulnerability has now been disabled so this could be an interesting week.

So is this attack purely ransomeware, or might there be other things happening, data gathering, perhaps?

And I guess it isn't a co-incidence that my Win10 had a big update yesterday.

Let's work on the assumption that the 'kill switch' vulnerability has now been disabled so this could be an interesting week.

Several reports on Twitter of version 2 with the kill switch removed already in circulation.

[quote=dirtydog ]Several reports on Twitter of version 2 with the kill switch removed already in circulation.

I would have been surprised if it wasn't.

Though whilst getting caught by the first round was excusable, anybody getting hit now when patches for the vulnerability are available on all platforms is completely incompetent. Any company with a clue should have been getting their Cougars in over the weekend.

Well now.

And with round 2 starting at early breakfast time on Monday

The thing here as that the worm (what causes it to spread) and the payload (what does the damage) are separate entities. What I've been doing this weekend is targeting the former*. Despite what the mainstream media would have you believe, the latter doesn't run on XP.

The elephant in the room is that it would be trivial to repackage this exploit. A different payload, a different (or more likely, no) killswitch. That's something I could talk about at length but is out of scope here. What makes this vulnerability so headline-y dangerous is its efficiency of propagation.

(* - the latter is broadly sorted but is a longer story.)

whilst getting caught by the first round was excusable, anybody getting hit now when patches for the vulnerability are available on all platforms is completely incompetent.

Maybe, maybe not, it's economies of scale. If you've got three PCs to worry about then sure. But we're a service provider, with the best will in the world this isn't a coffee break fix.

We've had our best people (obvs, hiya!) battering this at short notice over the weekend, and we've kicked the shit out of it. Seriously, with no intention to humblebrag we've kicked it out of the park, we had one non-critical accidental reboot due to a misunderstanding and beyond that there has been zero downtime whilst we've bounced every internal, customer-facing and other server we own. My company has its failings but we've been a well-oiled machine on this one an I'm truly proud of the work my colleagues and I have done here.

On top of all the mainstream managed machines that fall under WSUS, and the work my colleagues have done, I've personally patched ~60 servers manually in the last two days. These are the machines outside of domains, the ones that fall between the cracks. Legacy systems put in place decades ago.

Are we done? We've smashed the stuff we know about but practically we've still a way to go. We're now into the realms of plucking at test servers, at development machines, at VMs engineers have spun up on their own laptops under their own steam, this sort of thing. My day job is running a tech lab and I've not even had chance yet to look at my own estate, there's probably another couple of dozen uncontrolled servers in there. This is going to run and run, but we're ahead of the game. Any company who tells you they've sorted it right now either has exponentially fewer assets than we do, is naive, or is is lying.

We will nail it though, I have no doubt. And we'll nail it before many many others will, we've got an incredibly competent man at the rudder of this. It's been a while since I last found myself impressed by a role model.

Any company with a clue should have been getting their Cougars in over the weekend.

That made me smile, thank you.

And I guess it isn't a co-incidence that my Win10 had a big update yesterday.

Just downed tools. 12 hours today.

Any company with a clue should have been getting their Cougars in over the weekend.

It's really serious when they deploy the MILF's 😉

I suspect that there are some fortunate people who will have an easy budget session this year for their IT spending requirements.

I am not really too sympathetic about 12 hour days - my normal shift is 13 hours and I believe much tougher than computer wizardry no matter how difficult or important. What you describe would be an easy day for someone like Drac ( whose job is tougher than mine)

What you describe would be an easy day for someone like Drac ( whose job is tougher than mine)

Morning. Well yeah I do 12 hours day as the norm at one time they were 15-16 hour days but I'm lucky these days thanks to a job change I no longer get very many late finishes. That said I got a txt last night to say we have a meeint this afternoon, I've also got one on Wednesday monring. Both will be in my own time.

Howver that's my job it's normal for me. It's not normal for the others on here so I do have a little sympathy.

My wife is back in to work today first after her trust was attacked, she is not looking forward to it.