MegaSack DRAW - This year's winner is user - rgwb
We will be in touch
tjagain - Member
I am not really too sympathetic about 12 hour days - my normal shift is 13 hours and I believe much tougher than computer wizardry no matter how difficult or important
Given most of these people will have already worked a full week before giving up their weekends, cancelling plans and getting stuck in to fix a very serious problem I have a lot more respect and thanks for these people. Without their efforts many more systems would have been effected and more impacts would have been felt. Somebodies payroll server gets locked out, a delivery system or an order system. No drugs to hospitals, no money in your bank account etc.
It actually takes decent skills, effort and concentration to do some of this stuff, it may not be physically harder but it's very mentally demanding.
Given the response from a few here whenever being asked to do extra or cancel plans comes up is foot down, call the union and it's not worth my free time etc. a great number of people just got on with it.
You can play the I work in the NHS card a few times but don't wear it out.
fair point mike
We patched an seriously large number of devices (end-user, servers but also Fiery printers etc) in a short space of time. Over 15000 laptops were not on the network as people had taken them home for the weekend or at least locked them in a cupboard. Hence a large part of that was managing not to get 15000 devices downloading a large-ish update of patches in a very short space of time on Monday morning when everyone arrived, but at the same time avoiding the kind of solution that meant 15000 people arrived at work and were told they couldn't work... We also had fun because Sunday is a working day across much of the middle east, so had to work around business hours and live users there.
As some have said, not physically exhausting, but blimey a helluva lot going on to keep track of, and also brainstorm for solutions whilst preparing a list of all the random other systems that may need attention, and hatching a plan for them.
There's at least one UK bank that's been hit. Mate of mine did 36 hours straight, at work since lunchtime on saturday until midnight last night sorting it out. They work within IT security for one of the larger UK banks.
I'd guess (hope?) it's the "non-banking" side of things.
They were apparently back in 7 this morning.
And no, i'm not surprised it's not been publicised. I'd not tell anyone either.
Unless i lost all their money.......
Glad I've managed to dodge the bullet on this one so far, we do IT consultancy for some NHS Trusts but thankfully I'm not on those projects. That said the government agency I'm currently working on is a real eye opener when it comes to complexity that I think most, even experienced IT folk, don't realise.
They finally moved to Win7 last year after an 18 month migration project, it took that long as they have well over a hundred bespoke apps (even those using COTS apps are heavily customised). Some of those are classed threat to life systems (as in downtime is an order of magnitude more serious than the boss can't get to his Internet cat pics) and each one has to be extensively tested and issues fixed.
As for patching, it's done quarterly as standard as it's simply too risky to patch more frequently as patches are far from infallible (and again key systems need to be properly tested first). Fortunately the main environment isn't Internet connected and end points are heavily locked down so the human error factor is largely mitigated but I can imagine IT in the NHS must be a nightmare to support and they have to be much more open and have a much less IT savvy general user base.
This does remind me rather of the Cory Doctorow story - " when sysadmins ruled the earth"
https://craphound.com/overclocked/Cory_Doctorow_-_Overclocked_-_When_Sysadmins_Ruled_the_Earth.html
So cougar - which one are you?
"“Yeah.” Van was a type-two sysadmin, over six feet tall, long pony-tail, bobbing Adam’s apple. Over his toast-rack chest, his tee said CHOOSE YOUR WEAPON and featured a row of polyhedral RPG dice.
Felix was a type-one admin, with an extra seventy or eighty pounds all around the middle, and a neat but full beard that he wore over his extra chins. His tee said HELLO CTHULHU and featured a cute, mouthless, Hello-Kitty-style Cthulhu. "
Due to 'precautions' being taken at my work today email and access to networks and servers is being restricted until all the company computers are updated. Thankfully I checked the company news before going in this morning so I just stayed in bed instead.
Our latest technical guidance, recommendations come after the analysis:
[url= https://blogs.technet.microsoft.com/mmpc/2017/05/12/wannacrypt-ransomware-worm-targets-out-of-date-systems/ ]https://blogs.technet.microsoft.com/mmpc/2017/05/12/wannacrypt-ransomware-worm-targets-out-of-date-systems/[/url]
More general customer guidance:
[url= https://blogs.technet.microsoft.com/msrc/2017/05/12/customer-guidance-for-wannacrypt-attacks/ ]https://blogs.technet.microsoft.com/msrc/2017/05/12/customer-guidance-for-wannacrypt-attacks/[/url]
So cougar - which one are you?
He's clearly a 2.
On topic.. I think that people have learned from the XP situation, and things are done quite differently now than they were 10 or 15 years ago.
We strongly dissuade people from customising our apps, even though they fully support it, because it makes things hard to upgrade so people don't, and they end up in this situation.
so when does Hunt get fired?
[quote=kimbers ]so when does Hunt get fired?
> https://www.thetimes.co.uk/edition/news/experts-told-minister-last-year-of-nhs-hacking-risk-qrpjbdh5d
br />
june 8th with any luck.
[quote=Cougar ]Maybe, maybe not, it's economies of scale. If you've got three PCs to worry about then sure. But we're a service provider, with the best will in the world this isn't a coffee break fix.
Sure - wasn't meaning to downplay what you've done. But if you're not on top of it in the way you are, there are at least some other steps you can take to control the situation whilst you get there. Meanwhile I imagine some will have rocked up Monday morning and then started to think about what to do.
Not followed the whole thread but haven;t seen this;
[img] 
[/img]
civil service here, win 7 patches getting applied today. We still have many xp machines too! We had a ransomware attack just before xmas too.
Still waiting for the fix. Apparently it has to be applied to every pc separately one at a time yo check it's worked.
so when does Hunt get fired?
I'd like to know how he'd get fired. From the end of a 155mm howitzer with a bit of luck. Useless h[s]c[/s]unt
Hunt will not be fired - he is there to destroy the NHS and he is doing a good job of it.
Well the good news is - there goes all the shitty old W2k. Never to be turned on again. Result.
Given the response from a few here whenever being asked to do extra or cancel plans comes up is foot down, call the union and it's not worth my free time etc. a great number of people just got on with it.
I'm a firm believer in "the door swings both ways," and I've been afforded a -lot- of slack and freedom in the past to deal with personal issues. It'd have been churlish of me to say no, frankly. Plus, y'know, I get paid.
We patched an seriously large number of devices (end-user, servers but also Fiery printers etc) in a short space of time.
I've just had a conversation with a mate who was humblebragging about how he did 200 machines in 40 minutes. On our primary estates that's precisely what happened, our internal servers & PCs and our cloud platform both have dedicated teams with robust patching policies and procedures in place.
However, I got to deal with all the off-domain cruft that was left over. We had to control individual reboots / failovers to redundant systems and so forth, with unique per-box login credentials, sometimes on systems that no-one we could find knew much about, on disconnected systems that weren't necessarily accessible from a single management point. It just wasn't practical (or safe) to to it in bulk.
And today, I actually got to make a start on my own kit. I manage what we call the Lab which is an area engineers can use to set up kit before it goes to site, build simulations for exams, and generally use it for their own nefarious purposes. I've got a VMware infrastructure with a homogeneous melting pot of OSes on there from Server 2003 to 2016, Windows 7 / 10, various flavours of Linux, virtual appliances and all sorts. Much of it predates my time there. So I've been playing "patch it or delete it" all day, if nothing else it's done wonders for the disk space in the array.
Well the good news is - there goes all the shitty old W2k. Never to be turned on again. Result.
A couple of years back, I got asked to help an engineer with a wonky PC they'd uplifted from a customer. His question was "mate, WTF is this?" He'd never seen it before - it was Windows 3.11.
Ah yes Windows Mac looks likey. I was reminiscing yesterday over the pile of crap that was Windows 95.
Hunt will not be fired - he is there to destroy the NHS and he is doing a good job of it.
Well... probably not entirely accurate. His one job (which he failed to do) was to keep Health out of the papers. He's only still in post as a Cameron loyalist because everyone else recognises Health is career suicide.
My NHS Day
No Internet, No Email, No Systems. Have access to MS Office though
My NHs day, 10,000 pcs in our estate, guys in all weekend keeping an eye on things. Not one of compromised so far.
Directors and senior managers nowhere to be seen, not one compliment incoming or even a comment that we must have been on top of our patching.
Hmm sounds like you are providing more than the absolute minimum necessary service
Please consider yourself ready to be outsourced
My nhs day
Everything worked as normal all day.
My wife's NHS day
Not allowed to switch computer on
No Internet, No Email, No Systems. Have access to MS Office though
No internet, internal email only, [i]most[/i] systems working here.
Decent summary in accessible terms.
Just as a postscript to this.
The guys who wrote this ransomware offer customer support.
Love the PS (I don't, they're leeches)
[img] 
[/img]
No Internet, No Email,
OT but yesterday external router down and email servers are off-site so no internet and no email.
did I send out an email to everyone telling them email was down then realised what I'd said just as I hit send?
The guys who wrote this ransomware offer customer support.
For clarity, "this ransomware" is the malware in the post, it's unrelated to (but similar to) WannaCrypt.
In related news, I'm now working on servers that cannot be patched, disabling SMBv1. Yay.
Did we ever discuss that it appears the vast majority of computers affected were running W7 (I've seen suggestions of 99%)? So it wasn't really a problem with stopping support for XP, but with not applying available patches to 7
I mentioned it a couple of times.
http://singletrackworld.com/forum/topic/nhs-in-large-scale-it-shutdown/page/3#post-8469449
http://singletrackworld.com/forum/topic/nhs-in-large-scale-it-shutdown/page/7#post-8472827
It's almost certainly a primarily Windows 7 issue.
The best figures I have for the NHS is that XP accounts for about 5% of their workstation estate.
Interesting, missed that before
My win 7 machine refuses to install windows updates and just throws a hissy fit installing to 99% then uninstalling then installing again locking the machine up for a day so I'm relying on the fact that -
a) I don't store anything on it that I'm not willing to lose
b) I don't click on anything dodgy.
Have you added an internal DNS reg and a sinkhole server Cougar? Seems an easy way to stop the next variant firing, just by following twitter.. might even be scriptable / bott-able.
We did this first thing, SMB v1 was already off by policy on 99% of the clients, apart from GxP stuff (that's pretty well hidden anyway) for performance issues. We're near 100% W7 client wise.
My win 7 machine refuses to install windows updates
Google "windows update fixit"
Have you added an internal DNS reg and a sinkhole server Cougar?
Internal DNS is out of my hands these days. In any case, any new variant will almost certainly use a different killswitch (or not use one at all).
[quote=jambalaya ]Interesting, missed that before
Well to some extent it was just rumour at the time - there's now decent info on what actually was impacted. There has also been some dodgy use of statistics - apparently 90% of NHS trusts have some machines running XP and I've seen that figure totally misinterpreted by people who should know better.
I think if I had a W7 machine which wouldn't install updates I'd be doing a clean install - it's not just this vulnerability but other future ones (if it's on a network, it's not just the data on it you need to worry about).
^^ thanks. One of my French BILs has a number of XP machines connected to lab kit but none are on the internet. (Asked for help here when one died). Cost of software licence update is uneconomic vs kit which works perfectly well
[quote=jambalaya ]One of my French BILs has a number of XP machines connected to lab kit but none are on the internet.
There's not much risk doing that - provided of course you are also careful about other infection vectors such as USB sticks. There's nothing inherently wrong with XP and I've advised people to do similar (I also had a non connected XP machine here running after support ended, and we had XP running on VMs which were restored from snapshot every restart, though they've now been phased out).
Well to some extent it was just rumour at the time
I had it confirmed pretty quickly, an infosec took it to bits to find out. It was just reported by absolutely no-one, because who listens to experts when there's newspapers to sell.
There's not much risk doing that
"Not connected to the Internet" doesn't necessarily mean "not on a network." With unpatched machines it's perfectly possible for an Internet-connected PC to become infected and then for that to spread to other machines not on the Internet but still networked up.
"Quick! Update the servers and infrastructure so our data is protected."
tech:"But we've done no testing?"
"Just do it!"
*some time later*
tech:"We've done all the updates."
"but no one can login"
tech:"yeah, that's how secure it is!"
"and some of the data's gone"
tech:"errrm, yeah.Still the server's are all patched now which was what you said you wanted."
"why didn't you warn me?"
[i]The computer failure — that Queensland Health Minister Cameron Dick will tell Parliament of today — is most likely as a result of his department’s efforts in fending off “a very serious ransomware attack”.[/i]
[url= http://www.cairnspost.com.au/news/cairns-hospital-suffers-software-catastrophe-with-possible-loss-of-patient-data/news-story/c828de3f4a0f73132ec3d19284cbae88 ]http://www.cairnspost.com.au/news/cairns-hospital-suffers-software-catastrophe-with-possible-loss-of-patient-data/news-story/c828de3f4a0f73132ec3d19284cbae88[/url]
Aye, compatibility testing of hotfixes is a given, but that _should_ have been done in the two months between the patches being released and the malware's first hit.
People just do not like to patch if they know it is going to harm productivity.
The thing is most IT departments aren't staffed to cope with the bursty nature of proper patch testing. Where I work we deploy to a few test environment PCs but they only run some of the core apps (we can't afford to run every single prod app in the test environment, as I doubt many businesses can). Once the basic testing has been done it's pushed into live to a select set of PCs, if no issues are reported it's pushed to the remaining several thousand PCs, there's only time to do this quarterly and allow for a sufficient amount of testing (along with all the other work going on). Even then it's impossible to test everything (hundreds of apps and some functions are only run monthly or annually).
I'm not making excuses for the IT departments out there that still can't be bothered to patch routinely but the general perception from the public of this being the case with anyone that suffers an outbreak from malware where a patch already exists isn't correct.
.