tjagain – Member
I am not really too sympathetic about 12 hour days – my normal shift is 13 hours and I believe much tougher than computer wizardry no matter how difficult or important

Given most of these people will have already worked a full week before giving up their weekends, cancelling plans and getting stuck in to fix a very serious problem I have a lot more respect and thanks for these people. Without their efforts many more systems would have been effected and more impacts would have been felt. Somebodies payroll server gets locked out, a delivery system or an order system. No drugs to hospitals, no money in your bank account etc.
It actually takes decent skills, effort and concentration to do some of this stuff, it may not be physically harder but it’s very mentally demanding.

Given the response from a few here whenever being asked to do extra or cancel plans comes up is foot down, call the union and it’s not worth my free time etc. a great number of people just got on with it.

You can play the I work in the NHS card a few times but don’t wear it out.

fair point mike

We patched an seriously large number of devices (end-user, servers but also Fiery printers etc) in a short space of time. Over 15000 laptops were not on the network as people had taken them home for the weekend or at least locked them in a cupboard. Hence a large part of that was managing not to get 15000 devices downloading a large-ish update of patches in a very short space of time on Monday morning when everyone arrived, but at the same time avoiding the kind of solution that meant 15000 people arrived at work and were told they couldn’t work… We also had fun because Sunday is a working day across much of the middle east, so had to work around business hours and live users there.

As some have said, not physically exhausting, but blimey a helluva lot going on to keep track of, and also brainstorm for solutions whilst preparing a list of all the random other systems that may need attention, and hatching a plan for them.

There’s at least one UK bank that’s been hit. Mate of mine did 36 hours straight, at work since lunchtime on saturday until midnight last night sorting it out. They work within IT security for one of the larger UK banks.

I’d guess (hope?) it’s the “non-banking” side of things.

They were apparently back in 7 this morning.

And no, i’m not surprised it’s not been publicised. I’d not tell anyone either.
Unless i lost all their money…….

Glad I’ve managed to dodge the bullet on this one so far, we do IT consultancy for some NHS Trusts but thankfully I’m not on those projects. That said the government agency I’m currently working on is a real eye opener when it comes to complexity that I think most, even experienced IT folk, don’t realise.

They finally moved to Win7 last year after an 18 month migration project, it took that long as they have well over a hundred bespoke apps (even those using COTS apps are heavily customised). Some of those are classed threat to life systems (as in downtime is an order of magnitude more serious than the boss can’t get to his Internet cat pics) and each one has to be extensively tested and issues fixed.

As for patching, it’s done quarterly as standard as it’s simply too risky to patch more frequently as patches are far from infallible (and again key systems need to be properly tested first). Fortunately the main environment isn’t Internet connected and end points are heavily locked down so the human error factor is largely mitigated but I can imagine IT in the NHS must be a nightmare to support and they have to be much more open and have a much less IT savvy general user base.

This does remind me rather of the Cory Doctorow story – ” when sysadmins ruled the earth”

https://craphound.com/overclocked/Cory_Doctorow_-_Overclocked_-_When_Sysadmins_Ruled_the_Earth.html

So cougar – which one are you?
““Yeah.” Van was a type-two sysadmin, over six feet tall, long pony-tail, bobbing Adam’s apple. Over his toast-rack chest, his tee said CHOOSE YOUR WEAPON and featured a row of polyhedral RPG dice.

Felix was a type-one admin, with an extra seventy or eighty pounds all around the middle, and a neat but full beard that he wore over his extra chins. His tee said HELLO CTHULHU and featured a cute, mouthless, Hello-Kitty-style Cthulhu. ”

Due to ‘precautions’ being taken at my work today email and access to networks and servers is being restricted until all the company computers are updated. Thankfully I checked the company news before going in this morning so I just stayed in bed instead.

Our latest technical guidance, recommendations come after the analysis:

https://blogs.technet.microsoft.com/mmpc/2017/05/12/wannacrypt-ransomware-worm-targets-out-of-date-systems/

More general customer guidance:
https://blogs.technet.microsoft.com/msrc/2017/05/12/customer-guidance-for-wannacrypt-attacks/

So cougar – which one are you?

He’s clearly a 2.

On topic.. I think that people have learned from the XP situation, and things are done quite differently now than they were 10 or 15 years ago.

We strongly dissuade people from customising our apps, even though they fully support it, because it makes things hard to upgrade so people don’t, and they end up in this situation.

so when does Hunt get fired?

https://www.thetimes.co.uk/edition/news/experts-told-minister-last-year-of-nhs-hacking-risk-qrpjbdh5d

june 8th with any luck.

Sure – wasn’t meaning to downplay what you’ve done. But if you’re not on top of it in the way you are, there are at least some other steps you can take to control the situation whilst you get there. Meanwhile I imagine some will have rocked up Monday morning and then started to think about what to do.

Not followed the whole thread but haven;t seen this;

civil service here, win 7 patches getting applied today. We still have many xp machines too! We had a ransomware attack just before xmas too.

Still waiting for the fix. Apparently it has to be applied to every pc separately one at a time yo check it’s worked.

so when does Hunt get fired?

I’d like to know how he’d get fired. From the end of a 155mm howitzer with a bit of luck. Useless hcunt

Hunt will not be fired – he is there to destroy the NHS and he is doing a good job of it.

Well the good news is – there goes all the shitty old W2k. Never to be turned on again. Result.

Given the response from a few here whenever being asked to do extra or cancel plans comes up is foot down, call the union and it’s not worth my free time etc. a great number of people just got on with it.

I’m a firm believer in “the door swings both ways,” and I’ve been afforded a -lot- of slack and freedom in the past to deal with personal issues. It’d have been churlish of me to say no, frankly. Plus, y’know, I get paid.

We patched an seriously large number of devices (end-user, servers but also Fiery printers etc) in a short space of time.

I’ve just had a conversation with a mate who was humblebragging about how he did 200 machines in 40 minutes. On our primary estates that’s precisely what happened, our internal servers & PCs and our cloud platform both have dedicated teams with robust patching policies and procedures in place.

However, I got to deal with all the off-domain cruft that was left over. We had to control individual reboots / failovers to redundant systems and so forth, with unique per-box login credentials, sometimes on systems that no-one we could find knew much about, on disconnected systems that weren’t necessarily accessible from a single management point. It just wasn’t practical (or safe) to to it in bulk.

And today, I actually got to make a start on my own kit. I manage what we call the Lab which is an area engineers can use to set up kit before it goes to site, build simulations for exams, and generally use it for their own nefarious purposes. I’ve got a VMware infrastructure with a homogeneous melting pot of OSes on there from Server 2003 to 2016, Windows 7 / 10, various flavours of Linux, virtual appliances and all sorts. Much of it predates my time there. So I’ve been playing “patch it or delete it” all day, if nothing else it’s done wonders for the disk space in the array.

Well the good news is – there goes all the shitty old W2k. Never to be turned on again. Result.

A couple of years back, I got asked to help an engineer with a wonky PC they’d uplifted from a customer. His question was “mate, WTF is this?” He’d never seen it before – it was Windows 3.11.

Ah yes Windows Mac looks likey. I was reminiscing yesterday over the pile of crap that was Windows 95.

Hunt will not be fired – he is there to destroy the NHS and he is doing a good job of it.

Well… probably not entirely accurate. His one job (which he failed to do) was to keep Health out of the papers. He’s only still in post as a Cameron loyalist because everyone else recognises Health is career suicide.

Hunt jealous of hackers who were able to cripple NHS in just 20 minutes

My NHS Day

No Internet, No Email, No Systems. Have access to MS Office though

My NHs day, 10,000 pcs in our estate, guys in all weekend keeping an eye on things. Not one of compromised so far.

Directors and senior managers nowhere to be seen, not one compliment incoming or even a comment that we must have been on top of our patching.

@Jonnyboi

Hmm sounds like you are providing more than the absolute minimum necessary service
Please consider yourself ready to be outsourced

My nhs day
Everything worked as normal all day.

My wife’s NHS day
Not allowed to switch computer on

No Internet, No Email, No Systems. Have access to MS Office though

No internet, internal email only, most systems working here.

[video]https://www.youtube.com/watch?v=88jkB1V6N9w[/video]

Decent summary in accessible terms.

Just as a postscript to this.

The guys who wrote this ransomware offer customer support.

Love the PS (I don’t, they’re leeches)

No Internet, No Email,

OT but yesterday external router down and email servers are off-site so no internet and no email.
did I send out an email to everyone telling them email was down then realised what I’d said just as I hit send?

The guys who wrote this ransomware offer customer support.

For clarity, “this ransomware” is the malware in the post, it’s unrelated to (but similar to) WannaCrypt.

In related news, I’m now working on servers that cannot be patched, disabling SMBv1. Yay.

Did we ever discuss that it appears the vast majority of computers affected were running W7 (I’ve seen suggestions of 99%)? So it wasn’t really a problem with stopping support for XP, but with not applying available patches to 7

I mentioned it a couple of times.

http://singletrackworld.com/forum/topic/nhs-in-large-scale-it-shutdown/page/3#post-8469449
http://singletrackworld.com/forum/topic/nhs-in-large-scale-it-shutdown/page/7#post-8472827

It’s almost certainly a primarily Windows 7 issue.

The best figures I have for the NHS is that XP accounts for about 5% of their workstation estate.

Interesting, missed that before

My win 7 machine refuses to install windows updates and just throws a hissy fit installing to 99% then uninstalling then installing again locking the machine up for a day so I’m relying on the fact that –
a) I don’t store anything on it that I’m not willing to lose
b) I don’t click on anything dodgy.

Have you added an internal DNS reg and a sinkhole server Cougar? Seems an easy way to stop the next variant firing, just by following twitter.. might even be scriptable / bott-able.

We did this first thing, SMB v1 was already off by policy on 99% of the clients, apart from GxP stuff (that’s pretty well hidden anyway) for performance issues. We’re near 100% W7 client wise.

My win 7 machine refuses to install windows updates

Google “windows update fixit”

Have you added an internal DNS reg and a sinkhole server Cougar?

Internal DNS is out of my hands these days. In any case, any new variant will almost certainly use a different killswitch (or not use one at all).