Waking at 1am with the sudden realisation that you’ve messed up at work….

Had the wake up in the middle of the night going “that thing – if x, then y. **** how did I miss that??” many times to various degrees of seriousness.

The difficult bit is remembering about it the next morning!

There’s also it’s more friendly cousin – waking up going “oh that’s how you solve that problem!” (Also happens quite a bit in the bath, for me)

Waking up in the bath sounds terrifying 😉

I set back a €2m project (and by extension a much larger €8bn project) by almost two months by using a newer, unvalidated version of a piece of software before I’d had chance to test it’s effectiveness. I was rushed, happened to use it by mistake and what I iterated in that version made the previous version incompatible, so I continued to use it. MASSIVE mistake. It was all kinds of wrong and the results it gave weren’t outside the realm of possibility, so I didn’t consciously notice (If I’m honest with myself, I knew something was off, but didn’t follow that sense of wrongness to conclusion) until it all came tumbling down in the days before a major aircraft design review. I had to own it in front of a room of peers and superiors, apologise and promise to correct it with all possible speed and diligence. I’ve never felt so small and stupid in my life, but they were all remarkably accommodating. I had it fixed in 6 weeks, but that was a LONG six weeks and I still kick myself for my stupidity.

Everybody makes mistakes. Its what you do once the mistake is found that counts.

^^^ This. You know the problem now and has taken the responsibility to remedy it, that’s a good start albeit at the last minute but you will solve it which is commendable.

I had a project that I spent several years (nearly a decade) working on (career dependent) but knew there was some issues but could not pinpoint the exact location of the problem. Handed in the project only to be spotted by the “clients” resulted in my ruined career. Practically destroyed everything I was supposed to be (career wise and life – lost everything) and ended up as zero hour worker. Spent another 20 years slowly picking up pieces. Now I am just at the start again. Can’t turn back the clock but just to move forward.

+1 for the reframing.
…I’ve found an issue, we can’t ship with it
…I think I know how to fix it, I will need x (time, resources)
…I expect it will lead to a delay of x

The culture of the company is also important here, how are staff treated when raising issues

The culture of the company is also important here, how are staff treated when raising issues

I’d also say that there’s only a certain amount of responsibility that a single person should own and if the company is that reliant on that person then they are actually not helping that person or being much good as a company,good companies make sure that processes and reviews are followed to capture oopsies early.

I’ve seen the odd manager go loopy(actually sectioned) in a bad company and off sick in others never to return quite the same.

Or maybe I’m the worst ever employee to manage 🙂

The cracker I owned up too was getting a million squid stuck in our system,was an automatic authorisation of life assurance payments to customers and I’d not set a flag correctly and the money was in no man’s land which was better than the bloke who debited the wrong payments from customer accounts that was a cracker but he survived another day although It took months to sort out.

One time (at band camp) I also a inadvertently triggered a purchase of 2O grands worth of shares but we made a profit on that so no pain.

OP,hope you are feeling better than you were at 1am. Some good replies in this thread,we are all human.👍

**** ups happen

It’s how you own them that’s important

^^ that needs to be on a T-shirt. 😀

Thanks all 🙂

Pretty low key compared to some people but the sinking feeling when I realised I was reformatting the hard drive containing loads of client tv shows instead of the internal hard drive containing something I didn’t need was something else. Instantly put my hand up though, and spent the next little while re-capturing a bunch of US reality shows from tape.

Lesson learned there, if reformatting a hard drive unplug all the other ones!

It wasn’t your design or quote, but you *did* find the problem in it. @wbo called it right.

That feeling when you wake up at 3am under a bush in someones garden in San Diego while on a work trip after getting roofied in a bar!

That worst feeling when you get back to hotel (thank you uber) and your boss starts messaging at 8am needing to desperately talk to you.

Then feeling of relief when you talk to the boss and find out your colleague got incredibly drunk, started a fight then spewed on the CFO of the company. Your slightly dishevelled and confused state forgotten about by all!

Excuse Number 1

@codybrennan have a read of this:

https://en.m.wikipedia.org/wiki/Just_culture

Anyone who works in aviation will immediately recognise it and why I am sharing.

Everyone makes mistakes – this will never, ever change, its human nature and is actually what makes us ‘better’ than AI or robots or software.

What we need to do, is learn from then, and we do that by being honest, opening up, and fixing the issue, and any decent human being will also understand that.

Oddly enough I’m on the other end as I’m the customer and one of our IT support suppliers has changed and configured a connection incorrectly. Not my job as we outsource but I found the issue. This affects all our reporting.
What is making mad is they are not telling me how it happened and being evasive. I just want it fixed but now I’m worried about their integrity, and may ditch them, not for the mistake but for not being honest about it.

Causally hold a claw hammer when informing the boss. It will soften his reply on the subject 😉

I once made a mistake that almost delayed the production start of a program by 12 weeks, a test wasn’t completed correctly and had to be repeated for certification. (The vehicle weight changed late in the program and i missed it, in my defense there had been a dozen weight changes in the preceding 6 months).

I admitted the mistake and managed the whole thing, pulled it back to about 6 weeks in the end. Cost about 20 million quid in lost production and a few percent off the stock market value of the company. Ended up with a promotion.

On the other hand a colleague missed something major, tried to cover it up, failed miserably and caused a fairly major issue. Had to examine and rework about 6000 cars. About 1/2 had to be scrapped. (None had actually been delivered at that point).

He’s no longer able to work in the industry as he’s been blacklisted.

One time when I worked for a bank, we lost a load of money. Not the usual way a bank loses money, ie fraud or betting on the wrong imaginary dog. No, this was a bag of physical cash, there was no way we should have lost it. It basically looked like a member of staff had nicked it, and it all was pretty fraught and depressing.

Til my colleague Helen was visited in a dream by her long-dead father, who said basically, “Lo, Helen, I have returned from heaven with this important message for you- you accidentally gave that bag of cash to a particular customer, in place of a different bag of cash”. Next day, she phones that guy up and he checks and yup, he had a bag of about five grand which should have been a bag of £500 in pound coins.

I nominated helen’s dead dad for employee of the month but apparently that was not ok

You’ll be respected for explaining the situation to your boss, giving you both the opportunity to work through it, figure out how to fix… We’ve all been there in our working life, monkey mind waking us up in the early hours to tell us about the error in calculations, why expensive bits of kit don’t fit into a process operating in 2022 because the drawings haven’t been as-builted since 1990….. Fortunately, the managers I’ve worked for have been suitably stressed by errors being identified, but equally appreciative that there’s the chance to resolve….that’s what the senior managers are paid for! I once worked on a project a few years after someone had been killed – the court case was in its final stages at that time, the ongoing stress for the project manager during that period and trauma the team suffered was horrendous – always think of the lads family, that he never got home from work that day. Not a great comparison, appreciate that, but when I do make a mistake at work, I balance it by saying everyone, including myself, got home safe to see their family at the end of the working day. Go easy on yourself, you inherited a mistake a spotted it. Good on you!

If it’s any help OP i woke up in the night once in a panic just like you…

I’d left a tray of (mildly) radioactive biological samples with a research value of about £150k on the counter of a bank on campus when I was a PhD student. Luckily they were clearly labeled as such.

I did that on the morning of September the 12th 2001.

Neither the police or the bank found it very funny, although the police did kindly return them.

I occasionally still wake up thinking about it but look back at the responsities that were given to a clueless 21 year old as a good lesson for what happens when an organisation wings it.

As per many others I have buggered up from time to time costing companies a reasonable amount of cash and I have also spent time tidying up after others buggering up. In all the latter cases I have been asked to provide review and feedback about what should be done in response. Only time I suggested firing someone was a good idea was due to them trying to cover it up which turned it from a few hours to fix to taking about 5 days including the weekend. Aside from that its generally we need to fix processes and get proper cross reviews in place.
From what you have wrote my immediate response would be its a failing in management not to have kept a few hours a week/month (depending on complexity/length of project) from the promoted expert to review your work whilst you settled into the role.

Any good manager takes Watson Snr’s advice.
“Recently, I was asked if I was going to fire an employee who made a mistake that cost the company $600,000. No, I replied, I just spent $600,000 training him. Why would I want somebody to hire his experience?”

Things to remember:
1. Everyone who is a builder, who does something, as opposed to sitting around watching others, makes mistakes. Everyone.
2. At the end of a project when it’s bedded in and running, no-one remembers the mistakes, they just think about the value it’s giving, not any bumps in the road to get there.

Don’t beat yourself up mate! And don’t think about it anymore until Monday.
You are exactly the kind of guy that employers love – you will own this and fix it :)}
Have a great Sunday!

I still wake up with a start/feeling of dread thinking I haven’t done my homework. I left school 31 years ago 😂

That project I inherited? I’ve checked it and found this problem. Don’t worry though, I’ve thought of these three possible fixes, these are the costs and benefits of each option, which should we do?
.
That kind of thing tends to be fairly well received🤷‍♂️

One time when I worked for a bank, we lost a load of money. Not the usual way a bank loses money, ie fraud or betting on the wrong imaginary dog. No, this was a bag of physical cash, there was no way we should have lost it. It basically looked like a member of staff had nicked it, and it all was pretty fraught and depressing.

Til my colleague Helen was visited in a dream by her long-dead father, who said basically, “Lo, Helen, I have returned from heaven with this important message for you- you accidentally gave that bag of cash to a particular customer, in place of a different bag of cash”. Next day, she phones that guy up and he checks and yup, he had a bag of about five grand which should have been a bag of £500 in pound coins.

I nominated helen’s dead dad for employee of the month but apparently that was not ok

This is a great story!

Try waking up and wondering if you put the correct solvent in the specimen processing machine and the 150 patients who the specimens belong to might have to be called back and re-biopsied 😳🤦

I love you guys/gals. So many points above that made me chuckle/wince/nod along to sagely. In terms of the impact of this thread- I slept well and deeply last night.

x

Been there before. As long as you own it and fix it, itll be forgotten about within a month at most even if it’s really bad.

My ones often aren’t revealed to me until I see them going wrong on TV to a global audience, never much fun when that happens. Layers and layers of checks but there’s always a chance something will slip all the way through. Fessing up with a clear “this happened, this is why it happened, this is what we’re doing to fix it and this is how we’ll ensure it doesn’t happen again” is the best I can do. I’ve still had clients who responded to my first proactive mention of it by CCing my boss in an arsey complaint back but I just think less of them for it and move on now. And hold a grudge forever, of course.

How did the morning meeting go Cody?

Yeah man update needed

Waking up in the bath sounds terrifying

It’s ok, he was with his friendly cousin…

Big national bank. Got mortgage rate wrong on an ad. ££.

Oh and not my story to tell, but I know someone who loaded a cash machine wrong, to the benefit of the public of Bristol.

As an aside doesn’t anyone review your work / cast a second pair of eyes over it? Seems like a poor process if not

This x100. Any high level designs for IT projects we do are reviewed by at least two other people (senior architects) and often sections are reviewed by technical architects for each technology involved (e.g. a network architect and a DBA). Before costs are provided to the client they go through a whole other review process. It’s mental that one person can submit a design with costs to a client without peer reviews for what sounds like a fairly big project!

I once worked for a company that provided software that made satellite television work, in this case for a popular provider named after the stuff that is above us, let’s call them “Cloud” or “Air” or something like that for now.

Anyway. I worked on hard disk backup code in this software. At one point, I very nearly delivered an update to the entire customer base that would have caused the hard disk emergency recovery process to be triggered (whether it was needed or not).

Emergency recovery would (give or take) reformat the disk in the customer’s unit.

“Oops.”

I set back a €2m project (and by extension a much larger €8bn project) by almost two months

Sounds like it was still quicker than most industry projects of that size.

Update: all is well. I followed the advice given, saw the boss first. He couldn’t have been any more helpful, but was keen to point out that I’d overstepped the role I have. He was right. He was keen to get the customer the solution, as was I, but pointed out that others had been failing to pick things up properly. We followed up in an email a bit later.

By the time I’d sent this, he’d managed to get a lot of assistance involved, which was great. And although I’ll still be advising, I’ve to step back from the coalface on this one and let others own their issues.

I couldn’t be more grateful for everyone’s advice. I’ve known for a while that I’m not great at pushing back where others are slacking off, and I need to work on that I think- for the good of physical and mental health, if nothing else.

Best regards to you all!

Result, good response from the boss.