The house across the road has just had a new roof fitted. The guys who fitted it are back today, up the scaffolding, taking it all off again. They do not look happy. Apparently somebody ordered the wrong tiles. Its also absolutely bucketing down
Oops!
What have you or one of your colleagues been guilty of....
When I was not long out of uni, I once sent the wrong files to print and had to get a print run of a few thousand catalogues pulped and reprinted. Since them, I've been extra careful about flight-checking jobs before I send anything to print.
OK....
This could go anywhere. Like the roof...
I suppose my biggest took place when I supported the systems used by one of the major energy suppliers to manage their smart meter installation jobs. This system held both the appointment details for that day and the backend installation steps for each meter. It was a pilot system that somehow ended up being 'piloted' to 1200 field engineers (yea, one of those sorts of pilots) so required a level of manual intervention at time to make it work.
On one such manual fix I failed to select a couple of lines in a where clause and the outcome was every job for every engineer for that day got nuked from their devices. Just over 5k customer appointments. It was a bad day.
This happened in the morning and we managed to get most engineers current job finished and they were able to do their final job of the day. It was quite an expensive mistake purely on the guarantee of service payouts!
20+ years ago I was reconfiguring a Cisco router for a customer to add a secondary IP address to a Token Ring interface (told you it was a long time ago) and forgot to add the "secondary" parameter to the command.
This replaced the IP address that I was connected to and I immediately lost connectivity. The device was heavily tied down and it was not possible to re-establish connectivity.
I was in Cheltenham and the device was in Istanbul. The work that I was doing was to allow the deployment of presentation equipment for a conference of senior managers from about 15 countries around Europe who were travelling to their most recently acquired site.
It was scheduled to start the following day. The guy from the customer that I was working with was desparate to fix it so he got one of his team to book flights and we drove directly to Heathrow, hopped on a plane, and then fixed the issue by working into the night.
I was lucky that it was a customer I knew well and was understanding and that I used to carry my passport in my work bag.
My girlfriend was not convinced that my reason for cancelling our date that evening was true until I returned the next day and showed her photos and my boarding passes.
Lack of WHERE clause on a SQL update query. Been there, done that. Soon learnt it was a good idea to wrap it in a no-commit transaction and check the data before committing.
Came in to see if anyone had said "Louise" yet.
Is disappointed.....
Someone I know started up a critical, and very expensive bit of safety equipment on an oil rig without first properly checking it was good to go. Equipment cooked itself and the cost to repair + loss of production while it was repaired was so high the majority financial backer decided to pull their investment rather than fund it. Result was the oil rig couldn't afford to continue producing, was decommissioned, and around 150 people lost their jobs.
Mine was probably after I'd collected a load of cast iron bollards from the powder coaters, they were stood up ready to be loaded onto a wagon going to Germany or something. I came past with a pallet truck loaded with something and yep. Touched one which then dominoed the rest.
Back to the powder coaters they went, luckily another was coming the next week for the same place ....
I used to work for a fella making beautiful one-off, bespoke furniture.
We made this desk once. Quite a big thing in Oak. Very nice and beautifully made (even if I do say so myself) to the drawings.
Went to the clients home to install it in their office. Didn't fit through the door. 😂
No amount of rotating or upending it would make it fit.
I mean, it wasn't my mistake, but it was a mistake.
In the end we had to take the door and door frame off which gave us just enough. Not sure if the bloke has ever moved since 🤔
Many moons ago someone I used to work with specced a piece of equipment for a petrol pumping system, but messed up on the spec and fitted one with seals only suitable for water. The petrol 'consumed' the seals and within a very short time was spraying at high pressure all over the roof.
Of a very large car plant.
Making very expensive cars.
Employing many thousands of people.
The whole place could very easily have gone up like a Roman candle.
It would probably have made the news...
Luckily it never came to that, but I always felt that the company was one small step away from disaster. I left soon after!
Deleted most of a live database once.
A) i shouldn't have had access, i was only a work placement student.
B) they suddenly discovered how important having *proper* backups was. (They were weekly, i did this at about 11 am Monday while very hung over).
C) off the back of the recovery process they got the budget signed off for *loads* of new IT kit they'd been trying to get sorted for 3 or 4 years.
D) we eventually only lost about 40 man hours of work. No one got fired.
In 1990-91 as part of my degree I did a year of industrial placement at a chemical plant outside Hull.
It was a massive site with a couple of dozen sandwich students and fresh trainee graduates working there. We tended to be given jobs where we would be involved, but any cock ups could be quickly rectified without any effect of production. In theory.
February 13th 1991. I was given the task of ensuring that the new handrails in a tank farm were up to standard. Intact, painted etc. It was miserable work in foul weather and nobody wanted to be there. So, the permit holder for the farm signed them off despite a couple of kick plates (the lip on the walkway under the handrail that prevents you going under the rail if you slip) being missing.
A tanker was due into the Humber to unload, and one of the valves that needed opening was in the tank farm. However, a member of the jetty crew had slipped the day before in a different area and broken his hip, so they were already unhappy. The crew get to the tank farm, see the missing kick plates stacked up, and refuse to unload the ship.
Ship has to go back out and wait somewhere off Spurn Head whilst the plates are put in, to return 24 hours later.
Whilst not directly my fault, because the permit holder had signed it off, I was still one of those invited to hastily arranged “meeting without biscuits” in the divisional chief engineer’s office. As we were about to go in one of the fitters wiped some mud across my face and jacket (which I thought was odd, but turned out to be genius). We got our bollocking, with the permit holder being removed from his position, but on the way out the head honcho looked at all the dirty faced fitters (and me) who had apparently battled manfully in the awful conditions to try and finish the railings and said something along the lines of “…and go and wash all of that crap off yourselves.”
I was working for a company making circuit boards in the early 90's in the North East. I designed the build for the latest Apple pc boards but when they came out of the press, they were bowed - not flat as they should be. I was known as the man who turned apples into bananas!
Many moons ago someone I used to work with specced a piece of equipment for a petrol pumping system, but messed up on the spec and fitted one with seals only suitable for water. The petrol 'consumed' the seals and within a very short time was spraying at high pressure all over the roof.Jaguar or Bentley/Rolls Royce? I know they've both had instances of cars getting covered in petrol from faulty gear! I think Jaguar moved to doing all their filling in a separate room (this was 25 years ago though).
On a similar note, we had a fitter who got the wrong spec material out of stores and couldn't be bothered to go and get the right one, i mean, they're both stainless. What could it matter.
Quite a lot actually. Once the piece was fitted and signed off, it took about 12 hours for the stainless to corrode and then dump several thousand gallons of the sort of chemical that dissolves 3/8ths of an inch of stainless in 12 hours...
New IT guy set up a fancy file syncing RAID system.
Then (whilst still in the 3 month probationary period) had to explain why file syncing / RAID is not a backup to the CEO.
Some by me but a mate topped that one summer: one that I remember was when I was a student a mate offered me a job in the warehouse he worked that sold and hired out laser projectors used by Pink Floyd, U2 etc. The fork lift driver went on a break and my mate decided he could drive a fork lift but lost control, drove right though the huge metal warehouse doors and scattered the projectors on the forks everywhere. and left a forklift shaped hole in the door.
He then just ran away as he knew that was it. I had to spend the rest of the summer saying I had not seen him. He went to Greece for a few months and never came back to the warehouse.
Misfuled an ambulance. Yup that’s about as exciting as my mistakes got.
When I was not long out of uni, I once sent the wrong files to print and had to get a print run of a few thousand catalogues pulped and reprinted. Since them, I've been extra careful about flight-checking jobs before I send anything to print.
Yeah, being in the design for print industry is a bloody nightmare. I am glad I only do web stuff now. But, back in the day, I was doing a press-check of some brochures for a client that they had given me authority to approve. It turns out that their absolving themselves of responsibility didn't go so far as actually agreeing with me that the special Pantone green, when printed on an uncoated stock (which they chose), wouldn't match quite exactly the glossy nature of the Cromalin proof they had signed off. Despite my arguing that it would never match exactly because of the nature of printing on uncoated stock, they wouldn't have it, and made us pay for the reprint (on a gloss stock). Absolute bastards – we had only been in business for about a year at the time and it almost sent us under.
In the end we had to take the door and door frame off
"Right" said Fred.
I was pretty mortified when I pressed the wrong button and shut down an entire rack instead of just the server I was working on. Pales into insignificance compared to some tales above.
"Congratulations! When's it due?"
Twice
To the same woman.
(Not me - I have a rule about age/weight/hair)
Probably declaring a manager to be a c--- in a classic 'he's behind me, isn't he?' moment.
Roof update: The guys have had to lug all the roof tiles they put up yesterday down the ladders and then lug back up the ladders the correct roof tiles. They now have to lay the roof
During the course of this, the rain has been coming down sideways, without a break
To say they don't look happy would be understating things massively
Someone used a Forklift to load a mould tool into the Autoclave when I worked at an F1 team, then proceeded to go on a break. Someone else came along and shut the door and ran the cure cycle with the Forklift still inside, so it was cooked along with the job...
Luckily it was a low temperature tooling cure rather than a hot one, but 55 degrees and 90psi wasn't in the design brief of the forklift I don't think!
Came in to see if anyone had said "Louise" yet.
That was a cock up at work, was it? Sounds like an issue for HR. 😁
Deleted most of a live database once.
It wasn't me (honest!) but I once dealt with an issue stemming from someone installing desktop client antivirus software onto an Exchange email server. It did a file-level scan, found a virus in the .EDB file (Exchange's database) and deleted it. The email database for the entire company.
Fortunately it was only quarantined so recovery was pretty straight-forward, but that was a bracing afternoon.
A few years ago, I got a call to investigate an aircraft coming in on emergency landing during it's delivery flight, as it had suddenly filled with toxic smoke and there were planes being grounded all over the place until the source could be identified.
The next few hours were rather exciting sitting in a plane (can't say manufacturer or location) on a apron, testing various scenarios with all of my analytical equipment and full SCBA wating for it to go smokey and hopefully not to firebally with the fire crews sitting outside wating for me to go boom.
The cost of grounding the aircraft and penalties for delivery delays was eye watering.
Turns out, there were a couple of issues, both as a result of someone buying cheap/incorrect seals used during manufacture in a couple of locations.
Rather than recall all of the new aircraft, they made changes to the Air Flight Manuals so the magic smoke scenario shouldn't happen.
suffice to say there are a number of aircraft flying about with known issues, and a manufacturer of posh planes that you couldn't pay me to get on
Mu biggest was working as a junior buyer for a packaging company. A works order came in requiring 120kg of paper. I got a bit confused and placed an order for 120 tonnes of paper. It all turned up, all 6 artics, from Sweden.
But in 2013 I was the health and Safety manager for an electrical recycling site. On a Sunday morning a fitter had been given some plastic push in fittings to improve the drain out of a waste oil tank. The fitter decided he could do a much better job by modifying the tank, so he gets out his welding kit and...
Firefighters tackle huge blaze at recycling centre in Milton Regis in Sittingbourne
I worked a Portsmouth Navy dockyard for a while. They installed a new gas turbine in a destroyer. They forgot to open the valves on the lubrication system and ‘cooked’ it first time they ran it. That cost a few million to fix.
Plenty of cases on projects I worked on where we proceeded ‘at risk’ because we hadn’t received the final contract from the customer. The customer added new terms that needed specialist external legal consultation - we sometimes ended up having to swallow a few million quid we hadn’t planned on. My director took the decision but as PM I got to share the good news with the Group MD
Undercalculated an insurance premium by $7,000,000. The total premium should've been $25,000,000 but I set it at $18,000,000. In my defence, it wasn't necessarily my mistake, it was an excel file with a formula that didn't cover the full range of rows for a quote this big, so it only added up the first $18 million worth of rows of insured items, rather than going right to the bottom of the list...
I've had better days.
No funny mistakes in my current job.
In a previous life, one of my mates at the ice cream van depot managed to fill a diesel van with petrol one day, and then put diesel in a petrol van the next day...
At the time, I thought that the boss, one of the owners, was a tyrant. I now realise that he was actually a paragon of patience, working with a bunch of 16-21 year old idiots.
In the end we had to take the door and door frame off
I used to work for an art transport firm and it was amazing how often clients hadn't taken into account the size of doors for jobs they'd tasked us to deliver - occasionally jobs we'd come to collect and the artist had made work that couldn't get out of the studio.
The most classic example of course is the Rothko Chapel - designed and build specifically to hold a set of Rothko's paintings - but without a door big enough to get them in.
My artwork mistake though is something I wasn't even aware of for years. Set up some artworks in the cellars under London Bridge station for a photoshoot - it was fountains that glowed in UV light and used a dye called floroscene - which is used for testing for leaks in plumbing. At the end of the day we sack-borrowed the dyed water out from the cellars in oil drums, poured it down the drain, got in the van and drove home to Brum
Several years later I did a pick up from an artist called Allison Wilding - my artschool hero and a big inspiration so very excited about it - and it turned out her studio was on the same street. Over a cup of tea I told her I'd worked there previously and she told me how someone had seen me pour green glowing liquid down the drain, dialled 999, and of the enormous emergency services response and lockdown that followed
I was pretty mortified when I pressed the wrong button and shut down an entire rack instead of just the server I was working on. Pales into insignificance compared to some tales above.
I've done that.
It was back in the days when power switches actually switched mains AC power click-on click-off rather than this 'soft reset' business. I had to power down a server, pressed in the power button click and then "****!" as I immediately realised that I was on the wrong box.
So I'm sat there in the comms room, finger holding down the power button to something mission-critical like the email server or the primary domain controller, sweat beading on my forehead. What to do?
I took a deep breath, released the power and immediately jammed it straight back in again. There was enough residual current slopping round in caps in the power supply to cope with a momentary supply blackout and it carried on as if nothing had happened.
Half an hour into a lesson a few weeks ago, I said, "right we should write this down, it's called Flemings Left Hand.........bugger".
I'd shown them it and got them to use it with the right hand!
I did forecasting for a large (nominally 1000 tonne / 1,000,000kg) contract that the purchasing team had negotiated. It was mildly complex, we had to order for next month's deliveries, but also forecast for M2 and M3 at the same time, and then for the full contract.
M1 was firm, it was orders, M2/3 could change but only within a variation set in the contract, IDR exactly but something like +/-10% for M2 and +/- 25% in M3. ie: if you'd forecasted 100Te at the outset, next month you could adjust it to anywhere from 72-125Te. If you didn't adjust, and left at 100Te say, then next month you turn your 100Te forecast into an order of between 90-110Te. Not overly complex, but through rounding on my calcs I managed to get us into a situation where we failed to meet the 1000Te threshold, because when I tried to up the last order to meet it they refused because it was 110.xx % of the forecast. Literally a few kg that we fell short by. As a result we didn't get the rebated price and had to pay an additional €200k
My boss tried to negotiate us out, but being a big German firm as supplier, the terms were the terms. I thought I was going to get fired, but he said it was an honest mistake and not to worry. I nver had any kind of rebuke for it.
In fairness, he'd been a supply operative managing fuel supplies in the first Kuwait Gulf war, making decisions which helicopter was getting fuelled next and which had to wait. The gunship to provide support to the ambushed patrol, or the medivac one? as he said, when that was his daily problem then this was insignificant in his book.
Not particularly big but rather funny. the charge nurse was ordering supplies to the ward I worked on a task usually delegated. they ordered 50 urine sample pots not noticing the unit of issue was "box of 100" so ended up with 5000
one postdoc colleague put fresh cow bones with remnants of flesh still on in a newly calibrated GC oven. the oven was specially configured to analyse organic components coming off mammoth bone fragments from La Brea tar pits. his rationale was that he was looking to compare modern organcs and how they differ. needless to say it clogged, required a full strip and service at great expense and took a critical research instrument offline for a few months. he was supervised from then on.
worst I have done is prior to realising my neurodivergent tendencies, walk away from things I'm doing and forget them completely. first time was filling a 25l container with phenolic resin from a 1000l tank. it was viscous, and slow, so I 'multitasked' i.e wandered off to do something else whilst it was filling. came back to the supervisors staring at the floor, now covered with resin. of course I owned up and spent the afternoon scraping it up and into a container. only about 300l..but quite expensive.
second time, filling up a large liquid nitrogen dewar from the bulk supply tanks. takes ages, went into the adjacent room to carry on running XRD samples, finished that and went on my lunch break. was interrupted by the company secretary asking why there was a pink floyd concert in the carpark.
from then on I learned about common traits of neurodivergence and problems with object permanence. :-/
Not me but...
Following on from the OP's roof one. A friend needed all his ridge tiles replaced. Arranged to have it done whilst he was on holiday. Came back and they didn't look any better, terrible job. Phoned the roofer who was adamant he'd done them very well. Much back and forth arguing followed before they both realised he'd done next door's roof🙈
.
Former colleague formatted a hard drive instead of a floppy and deleted a lot of stuff.
.
My predecessor at my current job sent £20k to some spam email which asked for it.
None of my major cockups have been at work but I'm an accountant and that's very dull and predictable.
I work on some very large Fire Suppression systems.
Not only do these systems dump large amounts of agent, in the form of gas, water or chemical. They also tend to shut down equipment they are protecting, set off the main building fire alarms etc.
If you test correctly there is always a chance of an accidental, evacuation shutdown, discharge etc.
So many stories, getting drenched, gassed, etc.
The one that stands out, an internet service provider. A very large server room I had to test the suppression system. Probably around 1000m3 volume of room.
I looked over the system, removed all the actuators and shutdowns I could.
The air conditioning shutdown was a normally closed contact, so I couldn’t remove it.
I warned the facilities guys that it’d kill the ac when I tested.
So I set it off. The air con shuts off and it starts to get hot in the room. Turns out they didn’t know how to restart the aircon…
I can’t say for sure as I was asked to leave, but it all went rather pear shaped afterwards.
Daft amounts of money in cost was claimed afterwards.
Luckily I’d got my sheet signed before I tested, to say I was ok to trigger the shutdown.
Not me, but the co-owner of a business I worked for. Family firm that did water utility work, fairly big £200m turnover. I was financial controller. Before I joined, one of the two Directors had been out to Qatar to price up installation of a water pipeline - about a 12km job (from memory).
He surveyed the job and gave them a per metre price. We shipped out a 'multi-million' pound 'Tesmec Trencher" - like below - massive machine.
TESMEC.COM Along went one of our operators.
The Director didn't survey the ground properly. Instead of sand they hit solid rock. This machine was digging down 5m or so. The job should have been a few weeks, it took nearly 2 years and it was costing more in 'metal picks' than it was in income, never mind fuel, maintenance and wages.
We lost quite a few million on the job, and the operator lost a load of weight as there was no drinking out there ! The Director said, had he not been the business owner, he would have been sacked.
I once closed the A66 Eastbound from Penrith 12 hours too early. I still have flashbacks about that after 12 years.....
Got an email from the Chief Exec. Then an email from my boss telling me how i should respond. I wrote a short diatribe back to my boss about how the Chief Exec didn't have a clue what she was doing and generally slagging her off. I hit send. The email from my boss was still there. A quick panic as i realised i'd responded directly to the Chief Exec instead.
Got an email from the Chief Exec.Currently dealing with one of these.
"Car doesn't do what i want it to."
"It's not meant to do that"
"I want it to"
"Your program team took it out of the program 3 years ago, look here's the decision documentation"
"I want it in"
"OK, i can do that"
"How does a next friday sound?"
"For the new decision? The meeting is usually on tuesdays."
"No, for a launch."
This is going to run and run and run.
At least I'm not the only one to miss the where clause of a sql stateme t against a production database. Mind you it had been checked by my boss.
