“I accidentally ran, on all servers, a Bash script with a rm -rf {foo}/{bar} with those variables undefined due to a bug in the code above this line.”

Mr Marsala confirmed that the code had even deleted all of the backups that he had taken in case of catastrophe. Because the drives that were backing up the computers were mounted to it, the computer managed to wipe all of them.

http://www.independent.co.uk/life-style/gadgets-and-tech/news/man-accidentally-deletes-his-entire-company-with-one-line-of-bad-code-a6984256.html

Great headline but the guy was a dick. One line of code wasn’t the cause of his company’s destruction.

enfht is correct

rm -reputation?

makes me nervous every time I use that command, esp when scripted….

“Well, you should have been thinking about how to protect your customers’ data before nuking them,” wrote one person calling himself Massimo. “I won’t even begin enumerating how many errors are simultaneously required in order to be able to completely erase all your servers and all your backups in a single strike.

“This is not bad luck: it’s astonishingly bad design reinforced by complete carelessness.”

Sounds fair enough.

Mr Marsala confirmed that the code had even deleted all of the backups that he had taken in case of catastrophe. Because the drives that were backing up the computers were mounted to it, the computer managed to wipe all of those, too.
…
“All servers got deleted and the offsite backups too because the remote storage was mounted just before by the same script (that is a backup maintenance script).”

He didn’t have a reliable, resilient backup strategy is the problem here. Or indeed, any backups it seems, just replicated data. Remember the drum I beat here occasionally, “RAID is not backup”? Same principle. Jesus, college IT students learn about grandfather / father / son backup policies.

Dolt. No sympathy.

1500 clients and he just fires off that command?

Ballsy move 8)

You know, I wonder if this is clickbait. The whole story sounds weird.

Might be a stealth advert from a data recovery company.

Does seem odd. The original post to StackExchange doesn’t mention his name, so where have the press got it from? Doubt he would want to publicise his **** up.

It’s like a really boring episode of Mr Robot

I quite liked the ‘you don’t need tech support, you need a lawyer’ response

I’ve done this. 😳

Back in the day, long before Subversion and nightly jobs on build servers, I was using RCS for the company’s source control. I was in the habit of wiping (rm -rf) my local copy of the source so I could check out a fresh copy for doing a clean/verification build.

Except one late night I typed in the wrong terminal window and rm -rf’ed the entire RCS archive instead.

It was a good test of our disaster recovery strategy!

Worst one I’ve done was typing “shutdown -h now”* in a bash window on my laptop thinking it was a local window. Oh no, a remote cell site in China (I was in the same city at the time). Anyway, a taxi ride with my interpreter and lots of begging gained us access to a half built 30 storey office building at 3am and we finally located the cell in the lift room and brought it back online….

*Long before the days of GUIs for Linux on laptops….

The original post to StackExchange doesn’t mention his name, so where have the press got it from

A customer?

colleague had to explain to the CEO why the Windows equivalent was also performed one day, and that the “backup” was actually a sync, so had also perfectly synced all the deletes

my claim to fame was not that, but successfully deleting a single file… libc.so
at least there’s no chance of being able to delete anything else. or erm doing anything else.

another colleague managed to delete our complete RCS (actually think it was SCCS) and all checked out code. there’s good reason to run things like that on a remote server, and not locally c/o directory softlinks.

Surely data recovery could be able to restore one of the backups? I can believe it being an issue on the discs used by the server for hosting being unrecoverable but a hefty payment for a recovery expert should bring it back assuming he switched everything off before any other backup would have kicked off.

I smell something fishy.

Read that last night when I got home, journalists doing techy articles got me sofa raging at the hideous misuse of terminology.

Couldn’t work out if it was a spoof or just bad reporting of an idiot. I know bad practice goes on but testing a script in a live environment is just dumber than dumb.

Not a deletion but recently had a call where some software in a customers Citrix environment was corrupting, symptoms were similar to a rogue configuration, customer insisted all was standard…. except for the test server pushed into the live Citrix farm, to help with a busy period and had the rogue setting on it 🙁

Keep test and Live separate. Had it drilled into me as much as Cougar’s RAID comment above.

I know bad practice goes on but testing a script in a live environment is just dumber than dumb.

Along with mounting all your backup volumes on the same fs.