Waking at 1am with the sudden realisation that you’ve messed up at work….

Bit of a first world problem really compared to everything else that’s going on, but I’ve just had one of the worst nights (for me) ever.

Story: a work thing. I design some IT solutions for customers, and generally this goes well enough. However, I inherited an in-flight thing some time ago from a colleague, the company’s expert in this product, who was promoted out of our role. The general design etc, and costs, had already been given to the customer prior to my involvement. There were missing details on this, it was on fire as usual, and the job was handed to me to get this firmed up.

Anyway, I’ve done so….but missed an important detail in all this that will make it very, very hard- perhaps impossible- to deliver, because of interoperability reasons on the customer and our kit. And I only twigged at 1am this morning, when some part of my brain realised and I woke up with a start, went through to my office, and saw the problematic elements on a diagram. Turns out its been under my nose all the time and I just hadn’t noticed. Shit. To be fair, the customer hasn’t realised this either. I haven’t slept since then. Its definitely not good.

So its on me of course, and I’ll have to tell the boss. Possibly this weekend, although I wouldn’t want to disturb him, and then of course the customer first thing on Monday. There will be badness all around, the customer will want their money back, everyone will question my competence and this may have longer-term career implications here. But there’s nothing else for it. Oh, and my ulcerative colitis has now suddenly flared up overnight for the first time in a decade as (probably) a result of the sudden stress.

Anyway- I’m only really posting this as I need to tell someone, and you, dear STW reader, are them. But chip in with work goof anecdotes or advice if you have them-

with love, your friend- cdb

It’s never as bad as it seems.

As an aside doesn’t anyone review your work / cast a second pair of eyes over it? Seems like a poor process if not

You’ve spotted it now so can be corrected before delivery and implementation. Not great news but better than delivering as complete and it failing.

Ah damn. Sorry to hear that. Presumably the first bloke didn’t spot it either so not really your fault. Try to announce it in a way that makes you look clever for identifying the problem maybe. Would have been worse to discover during final testing.

Honesty, humility and humbleness (and other words with “h”) will probably save the day.

It’s awkward, fess up, sort it and it’ll be different next time around. We’ve probably all been there at sometime but its not a good feeling to have….

What is done is done, take a deep breath and m<span style=”font-size: 0.8rem;”>ove on to the next stage. </span>

Hope you will be fine and have the discussion with your boss asap, don’t let it fester.

Then go for a bike ride.

Everybody makes mistakes. Its what you do once the mistake is found that counts.

Your work world is miles from mine but what i would be looking to do is go to the boss with something like ” discovered a balls up. The solution seems to be A ; B and C. Timescale for the fix should be X ; how do you want me to proceed”

Back in about 1995, a Mobile Phone Co had launched its first prepay offer. Top-ups were done via the Post Office, and there was a promotion where each top-up trigged a free additional one.

A file came in every day, and there was a semi-manual process to go through this file and add the free ones.

Christmas Eve, someone realised that there wouldn’t be any staff to process the top-up file over the whole Christmas and New Year period. A mate and I got the job of quickly scripting something to make the process fully automated. Essentially we wrote some code that said “if you see a top-up, add a free one”. We were the last people out of the office Christmas Eve but got the job done.

Christmas morning I realised that our code didn’t look at the top-up type, so a free top-up would be seen the next day and another free one applied. And the next day, and the next day…

Called out the on-call Christmas cover, talked them through the changes needed to the script over the phone (no laptops or remote access apart from the one on-call person).

Saved the company several million, or a PR disaster.

I may tell you about the time some of my code stopped all new mobile phone connections to Mobile Phone Co once you got to December.

Yes I sometimes wake with a start panicking about something I’ve done wrong at work. I’ve been retired 4 years!

Presumably the first bloke didn’t spot it either so not really your fault. Try to announce it in a way that makes you look clever for identifying the problem maybe.

This.

It is ALWAYS better to own up to the issue immediately it is spotted then delay or even try and hide it. The customer will ultimately be glad of the honesty rather than you winging it and hoping the issue never appears. The sooner people are aware of the issue the sooner it can be worked upon and fixed, regardless of the size of the problem. How many big projects have stalled or failed due to people hiding issues? Answer: thousands of them.

Based on a past career where there was some potential for it, my bar of CU is stuck at ‘no one died’. Anything else can be overcome.

But, yes, ruminating revelations of ‘nooooooo’ at awkward hours are a pain.

It’s why we put quality assurance processes in place?
And good design, review, and planning?
And we speak up when speed or cost may compromise quality beyond an acceptable limit?

**** ups happen

It’s how you own them that’s important

I mobilised a project at risk without all the terms having been agreed (Government contract) and once the legals has been reviewed, we’d never have done the work but had to proceed as we’d made employment offers (reputational risk was huge) eventually absorbing the millions it cost us. My director went off sick so I felt like I had to carry the can – it was rubbish.

Speak to your boss first and give him some possible solutions rather than letting the customer know – let those on the bigger pay grades deal with it.

Like others have said, be honest and try to fix it asap. That’s all you can be expected to do. Easy for me to say but try not to let it ruin your health.

Personally, I don’t judge companies by their mistakes, but how they deal with them.

Reiterate the above really, mistakes happen. It is not that you have made a mistake the most important thing is how you handle and deal with it. In my experience the fall out from a mistake has never been as bad as what I think it’s going to be….. and the longer I have to dwell on it the bigger and more catastrophic the mistake appears

Usually when I wake up in the night worrying about something, turns out next day that I’d been worrying about nothing.

Not a big help, I know 🙁

Two spring to mind for me.

One was a rounding error on an order forecasting spreadsheet, if we bought 1000 tonnes of a material we received a rebate on every kg purchased. 1 million kg x €0.2 = €200K  My rounding meant we were literally about 3kg short. Supplier stuck to their contract and refused the rebate. My boss was understandably pissed off but basically said as long as it’s not deliberate, we all make mistakes.

Second was an allocation decision I had to make which shorted a major toothpaste supplier of a key ingredient for a couple of weeks. Late Friday I got their purchasing director on the phone yelling and threatening all sorts, I had a nightmare weekend waiting to tell my boss on Monday. He was brilliant, said no-one talks to his staff like that and then called the PD there and then. Basically said if there were a few tubes fewer in Total on the shelves briefly (most likely in warehouses really) then he was sorry but just suck it up, no-one is going to die as a result.

It helps his previous job included making fuel allocations to the USAF in the first gulf war, where decisions were between a jet for fire support for troops in ambush vs a duster to casevac someone else somewhere, literally life and death, so against that he wasn’t inclined to give many **** about toothpaste or a few hundred thou in a $5Bn company.

when i got to the stage of waking up at night because i had done something wrong at work and because i did not want to go into work, i retired.
less money more time is a good ethos i think.

All I can say is your employer should be glad you have a conscience

As in understand it error free work is staggeringly expensive

I’ve woken up with the dreaded “I’ve screwed up” thoughts a fair few times over the years, my experience is that things are never as bad as they seem when your mind is running riot at 3am, in my case my mind starts going around in circles with a corresponding increase in stress reaction.
I reckon that going for a walk/ ride early on helps to calm and rationalise things and then go discuss some possible with your manager/ colleagues: it might be that they pick up on a detail you have missed.
Good luck with it

Generally speaking it is how you deal with issues that marks out a top performer. Issues happen. Those who can effectively manage issues are usually those who are marked out for higher things. It’s especially hard when it’s your own issue but that will usually be recognised by those above you so I’d suggest take the approach of “owning” the issue, go in on Monday with a clear explanation of what’s happened and what the options are to move on. Don’t email people about it over the weekend as (a) that gives you time to come up with a slick plan for Monday and (b) you want to be seen to be calm, collected, open and honest.

Don’t focus on the “why” it happened as that can come across as making excuses – just be factual.

You could easily come out of this better than you went in.

For a minute I was wondering if Kwasi Kwarteng was on STW 😉

The strongest relationships come when there have been headwinds. Don’t sweat it, do the right thing and it’ll be fine.

First. Take a breath and relax.

For a minute I was wondering if Kwasi Kwarteng was on STW 😉

No, no. The OP is asking how to acknowledge the mistake instead of walking into the office on Monday and going “we’re challenging the orthodoxy, please don’t tell me the numbers”.

Two weeks into my first proper IT job I thoroughly screwed up. Got a “hope you’re enjoying your new job” email from a mate. Unfortunately it was riddled with a malevolent payload which dispatched itself to everyone in our global address list (university Admin department). This then went onward to everyone in the entire university. I went and saw my boss immediately. There was no fuss, no drama we just worked out what had to be done and did it. Turns out we had no mail plug in on the workstation AV solution, no AV solution on exchange and none of the fundamentals closed down like, you know: backups that worked, open relay shut off. It was a disaster waiting to happen, I just triggered it. Spent 2 solid days fixing things. All was good in the end. Tonight me a fundamental lesson: so long as everyone knows the situation it can be dealt with. There’s always a solution.

I’ve often done the during sleep processing thing. This week I woke up having worked out that a script I wrote would permanently disable users who went on paid leave. Doubt I would have found that out until go live given the customer’s testing regime.

No need to lose sleep here. You inherited a project with a flaw, and have spotted the flaw. Could have been spotted earlier, couid have been spotted later. As everyone else said, mistakes happen , how you respond is what matters. If it’s going to help you mentally ,give your self an hour, spend that time to describe the problem, come up with a solution, tell your boss and park it till Monday. If your boss is pissed off, it should be at his project QC process.

when i got to the stage of waking up at night because i had done something wrong at work and because i did not want to go into work, i retired.
less money more time is a good ethos i think.

This – The stress was just too much.

And for the OP – Whatgoesup has the best plan – take some time, think through how to present the issue, what the solutions might be (including the difficult options such as cancelling the whole project), pros and cons, and present these to your boss on Monday. Might ant to make sure he/she is available Monday if that is possible though

**** ups happen

It’s how you own them that’s important

Absolutely this. Show me someone who says never had that horrible moment of realisation and I’ll show you a liar

We’ve all done it

Nobody died though, did they?

Mine was not updating the amended indesign files to the server at the end of the day. The next morning someone assumed they were right (which they should have been) and an unamended massive publication went to a big print run.

I was out of the office the next day and had that cold sweat ‘oh shit!’ moment when I realised

The lot then had to be pulped and reprinted at great expense and delivered late to an unhappy client

I fessed up straight away, said it was 100% my fault and took it on the chin. I was absolutely furious with myself for being so unprofessional

At the end of the day my boss came over put his arm around me and said ‘come on… I’m taking you for a pint’. He sat me down and gave me the ‘we’ve all been there…’ speech and I got some kudos for holding my hands up and then sorting it out

That was 20-odd years ago and I’ve never done anything like that again. Everything since is belt and braces!

The fact you’re waking up and you’re this concerned shows that you care. That won’t go I noticed if you take it on the chin, sort it out and make sure it never happens again

This stuff is complicated, mistakes will happen. My screwups have occasionally ended up on reddit, though so far, thankfully, nothing I’ve done has been bad enough to end up on STW.

Anyway, there’s probably a python module to solve your problem.

Anyone that hasn’t is either a liar or oblivious to their own mistakes.

Better you find and plan to make amends than someone else to find them and you get blind sided.

Based on a past career where there was some potential for it, my bar of CU is stuck at ‘no one died’. Anything else can be overcome

This.

The feeling as a suppression system discharges accidentally or a server room goes silent is a something else.
Especially compared to the fear followed by relief of opening an email about a fire on a ferry which was extinguished by a system I was the last person to touch.

I was Senior IT Manager/Head of/Director for many years. If one of my team made a cock up I just appreciated them speaking to me, accepting responsibility and to come armed with a resolution/mitigation.
Work on that then call him/her first thing Monday to tell them what the impact is and what you/them/team are going to do about it.
It may be just me but I would probably call over the weekend, it will ruin theirs but that goes with the territory and shows that you are committed by worrying about it yourself, that will help a little. Also look to introduce processes that mitigate against things like this in future (although dont necessarily mention that when you speak to them!!)

Call your boss up now, tell him you were working through the night looking everything over because something didnt sit right in what you’d been sent.

You’ve now found a big snag.

Don’t tell him you messed up and woke up stressing. Make out like you are an amazing employee helping the company.

Reframing it will make a lot of difference.

No need to lose sleep here. You inherited a project with a flaw, and have spotted the flaw.

This.

Also something is wrong with how the project is structured if it is relying on individuals to spot things such as this. Too much resting on one person’s shoulders. Needs to have multiple eyes on it to ensure problem are caught early, and to have some shared responsibility in situations such as this.

Personally I’d leave it unyil Monday.

What are you expecting will happen over the weekend that will be of any benefit?

I’ve had some brutally bad times at work, things where you just can’t imagine how you can dig yourself out of the hole you’re in. What you need to do is remind yourself that the sun will still come up tomorrow and the issue will get resolved one way or another. Constantly worrying about it will make it worse, so just go for a bike ride, drink a couple of beers or whatever, watch a movie, just do normal things when you’re not working, don’t let the worry take over your life.

What are you expecting will happen over the weekend that will be of any benefit?

Depends, presumably the OP will be looking at options regardless so their boss may as well not be blindsided and have the time set aside to deal with it.

@codybrennan this is a failure but its not yours. As pointed out the QC/QA seems to be severely lacking if its not been peer checked/independently verified up to this stage. You’ve caught it before it was deployed, think yourself lucky as I’ve seen an entire water tank for a safety critical function built before a designer realised it was actually above ground and needed a whole load of extra bracing that didn’t exist. Thankfully, before it was filled.

Cody, I have no advice to add but I do sympathise and hope it works out OK for you.

Oh, and my ulcerative colitis has now suddenly flared up overnight for the first time in a decade as (probably) a result of the sudden stress.

You need to reassess your priorities. As above; if no human is negatively impacted by your ‘mistake’, if the ultimate outcome is only that it might affect someone else’s profit margins, then it’s really not something to lose sleep over and make yourself unhealthy. Perhaps it’s time you walked away and looked at alternative work. Always put yourself first, in any job you’re doing.