More bad gateways than a Stalingrad public footpath.

Why don’t you sack off the historical posts and just do a new forum

It’s not even like you can update the threads as they’re closed after a year or so…

Sure, some are informational, that goes without saying… but i think a new forum on a completely different platform has to be an option.

I’ll be the first to admit my forum is a lot smaller than here, we only have 300,000 posts (in 2 years) but using PHP i never get these issues… maybe twice a year when the host it’s sitting on has a problem… but i mean really really rarely.

I don’t know what to say really. Today has been unbearable at our end. One error after another that is directly related to the megasack comp. The software we use (and pay for) for the forms and data collection for some reason is causing havoc to our server. It’s the excat same software we used last year and the year before that. It’s exasperating and I can only apologise. We are ding everything we can to keep the site up and running along with the megasack competition. This is on top of ongoing, random server errors that we continue to try and track. Our server is plenty powerful.. We run Google cloud servers (2 of them) one with 16 cores and one with 4 (database). Honetsly we should be able to run the internet itself off the grunt that can kick out.

Honestly, I’m personally at my wits end here and I totally appreciate the frustrations for you guys. This is not the service I want to offer. But with extremely limited and rapidly shrinking resources each day we survive is a victory although it’s doing nothing for my health.

We will fix it. Or we’ll collapse trying.

I’ve explained a bit about how the stack here works in the past but it is as far from being just a forum as you can get really. By that I mean we run a forum. BBPress is the software. It’s common and supported and fits perfectly into WordPress. But our forum is linked across multiple database in order to do things that out of the box it cannot do and that’s where it gets really complex. Access to the forum and how it is used and who uses it is directly related to memberships, content access through paywalls and directly into multiple payment systems that manage the membership system. Add on to that API connections to discount providers and app access and it becomes a very complicated stack to manage. as I said, we have enough server grunt to run a small economy – we have software that is paid for and supported but we also have conflicts that are frustratingly hard to track down. Running just a forum is easy. There are many on here who have done it and still do it and don’t have our issues. What we have is scale (hundreds of thousands of users), complexity and **** all money to spend managing it. 

It must seem so simple to fix from the other side of the wire. It’s just a website with a forum. But we have 2 people to manage it when other similar media sites, like PB and Bike Radar have teams. It’s beyond frustrating and it’s ridiculously hard to fix with just 8 of us running the whole ship.

As I intimated earlier in the thread Mark there is a huge reserve of goodwill for you guys. We know things are on a shoestring and you are doing what you can. But in the last couple of weeks it’s been almost unusable at times as a subscriber.

It does help a lot when you come on here and explain the issues and how you’re trying to resolve them. The frustration mounts when people comment, we hear nothing and it feels like (probably incorrectly) nothing is happening to resolve the issues.

I really, really hope you get it sorted. I love it here, I just want to be able to check in without constant crashes. I get that you want that too. Good luck! 👍

Echo @blokeuptheroad. Must be hellish frustrating for you guys. Thanks for the update. 👍

Do it. 😉

Would it be possible to silo various elements, put them on different platforms so if one falls over it doesn’t drag the others with it? The megasack comp for instance. It would be no hardship at all as a user to have to log on to a separate site for that. Just that alone might solve some of the recent issues it seems.  Perhaps it could be extended to this forum, magazine/membership services, merchandise etc. if it made the individual elements more robust? If the issue is complexity and inter connectedness, would simplifying and separating be worth considering?  This is probably a naive question and I’m sure you will have looked at this and every other alternative, but thought I’d ask!

It’s not even like you can update the threads as they’re closed after a year or so…

On this,

If there is something you want to revisit, the moderators can reopen closed threads. You can pop them an email or even just Report one of the posts on it to ask for it to be reopened.

I think the auto-closure was an anti-spam measure.

Thanks for the update Mark, its a pain but knowing you’re knocking yourself silly trying to fix it is really good to know, reminds us there a real people behind this, not just us keyboard warriors.

I’m not convinced it’s anything to do with megasack though.

This. Megasack might cause a (small) surge in activity but surely not enough to overload the server to cause a 502. I’d be looking at 3rd party plugins and the underlying cloud (Google if I’m not mistaken?) networking setup*.

*Very much not a cloud/networking engineer, but I (used to) know a thing or two about websites/HTTP.

I don’t think it’s the traffic from the megasack, it’s the Forminator plug in that we’re using for entries that’s the suspect. Maybe we should just do answers on a postcard? All entries that include a fiver get doubled? Just don’t send pigeons. Imagine the poop as a couple of thousand pigeons turned up every day!

The updates/knowing what’s going on really help.

It *seems* to be a lot better just now – I’ve not seen a 502 since mid-afternoon now, although to be fair I’ve actually been off STW and doing some work…

Fair play to Mark and the team, the site has been running like a greased weasel today. Whatever you’ve done to the gateways (WD-40 on the hinges)? they’re all good now. Thank you 👍

Has anybody ever looked at what GDP rises by when the site goes down? 

I’ve no idea what a forum like this with a large income has these problems. It must be at the server end. Someone missing a connection wire 😆

While it may or may not be connected to the overall poor performance of the site generally I have a hard time believing that there is no correlation between that and the number of trackers/scripts/3rd party cookies etc that exist on this site.

I use ublock origin for much of the internet but singletrack is whitelisted to ensure I don’t contribute to the fall in revenue.  I temporarily reenabled it just now and refreshed this page..  it’s blocking 453 trackers etc.  a quick scan of other sites across the internet with the same settings and it’s blocking between 10 and 25.. 

As I said, might not be related to the current issue but it can’t be helping..  oh and rather than 502 errors I’ve been greeted with a WordPress login page about 40% of the time I visit the site for the last few weeks..  I usually have to refresh the page a half dowzen times or so before the home page pops up..  and even when it’s working the overall performance of the site is dire..  seems to be acceptable once you are here but from clicking the bookmark to the home page appearing (without wordpress logins taken into account) is getting on for a minute.. and that’s on a 1Gb connection, everything else online pops up instantly.

Our server is plenty powerful.

Unfortunately it needs to be to cope with such a sub-optimal website. If you are using Chrome just hover over the tab name and it will show how much memory the tab is using. For me this is currently 798MB whereas all my other tabs are in the 100-300MB range, in fact Lings Cars only uses 133MB! Ok, that’s resource usage at my end, but you will still be burning through a fair amount of compute power at your end to serve that all up.

I can’t even run the site on Chrome… it crashes my laptop. I need to open Edge as a separate entity otherwise i can’t do anything at all. It’s the only site i visit i have to do this with.

so this is what happens when the forum is powered by Chain Reaction.

I restarted my iMac, opened safari with one page showing singletrack and its immediatley using 352.2 MB with something attached called VTDecoderXPCService using 296.3 MB.

My Roon music server only uses 212 MB

VTDecoderXPCServiceis the video unit on the page.

Full members can deactivate the video player in profile settings

On checking in my preferences the “Disable video” is ticked

My wife’s chrome browser tab shows Mumsnet is allocating 1.2gb right now.

I just opened facebook and it allocated 600mb instantly. 1gb is high – not arguing that. But database driven sites cause browsers to store a lot of data. Simple sites that are essentially just html use very little of a browser. Ads also cause a lot of resources to be allocated by browsers. My logged in tab on this site right now are allocating 102mb. It varies depending on the thread I look at. It changes with each new page. That amount is not an amount being served and downloaded from our server. It’s an amount the browser is allocating on your device for use by the webpage. This is why Facebook instantly showed 600mb had been allocated in the memory of my machine before there was anywhere near that actually downloaded from their site. I’d love to have an internet connection that could instantly download 600mb. My point is not to say it’s not happening or it’s not a problem. What I am saying is that database driven sites like forums and social media sites with a lot of content and media (images. video etc) will cause your Chrome browser (In particular) to allocate a large chunk of your device’s ram to serving the page.

All performance issues with the site aside that is. No excuses for that. Today has been better and we feel like we have closed in a bit on the underlying issues. Tomorrow will hopefully be better and we’ll make more progress when we have both Andy and Anthony on the case. Andy works wed-fri.

My browser tab has assigned 90mb for this page. I’m logged in and have no ads, but I do have the video unit set to display. My Gmail tab is saying 584mb.

I’ll keep you updated as to how we get on tomorrow.

Yeah, I have 103mb for this page, 115mb for Facebook. There’s obviously quite a bit of variation but neither of those numbers seem unreasonable.

Why do some users have no problems (I’m a subscriber and I don’t have issues with the site on any device I use) and other people have no end of problems 

I’ve actually been having a few days of nice responsive forum after weeks of it being terrible,but sadly it’s reverted to type today with loads of 502s, very slow or ‘stuck’ pages or changes.

Freeloading scum on chrome android.

Again, not deflecting for our site problems but I have found this article helpful. I had a lot of extensions plugged in to my Chrome browser and I improved things for me after reading this.

https://economictimes.indiatimes.com/news/international/us/google-chromes-high-memory-usage-see-how-to-fix-it/articleshow/97066898.cms

This shows the issues we have had today. The spikes you see are server load. If the load stays below the red dotted line all is well and the site is fast and reliable. If you were on the site when those spikes happened you will have probably had the dreaded 502. If you didn’t but the site was awful slow at that time then you were lucky. When the server goes over the dotted line it starts to queue up requests. If you are in the queue then you will likely get the web page to load but it will be slow. Unfortunately the queue is finite and if there is no more room in the queue then you will get the 502 message which means the server can’t be reached. In other words you are not in the queue because there’s no more space. Once the load drops below that line then it doesn’t need to queue any requests and we all get the pages nice and quick. Our job is to pin down what is causing these spikes. It’s a lot easier to rule things out that find the culprit – today we ruled out quite a lot so we are getting closer. More tomorrow.

It must seem so simple to fix from the other side of the wire. It’s just a website with a forum. But we have 2 people to manage it when other similar media sites, like PB and Bike Radar have teams. It’s beyond frustrating and it’s ridiculously hard to fix with just 8 of us running the whole ship.

It’s really not simple. Anyone who’s actually worked in IT knows that. I know of whole teams for websites that have less daily users than you do with more impressive hardware than you do. It’s not easy, especially as you’re using WordPress. You have my sympathy. Good luck in sorting it all out. So cut yourself some slack.

I’m crossing my fingers here but we may have a breakthrough. How’s the site running for you guys?

Have you switched off the ads?!
😉

Depends when you had the breakthrough… I had a 502 earlier between 4:30 and 5 I think.

Edit: seems snappy ATM. 👍

OK here 👌

Like a dream currently

Mostly fine this afternoon. Had a series of 502s late lunchtime just as I was trying to submit my Megasack entry, but been fine since.

Zipping along still! 👍

Must have smuggled in some foreign hamsters to replace those work-shy British ones!? 🙂

Look how keen they are!…

All good again today 👍

Just had some 502s here…but no other issues so far 🤞🤞🤞

Site’s been running really fast all morning, whatever you and tech bods did, it hit the spot 👍

Phew. Thanks for bearing with us, folks. Tech has a little less hair now. Hopefully the ‘fun’ is over. Props are also due to Mark for holding it together enough to edit Megasack videos while also doing tech things. You don’t get props when you’re the boss.

I should imagine the chore of editing is softened a little by giggling at you loons in front of the camera!

Thanks Mark & crew. Feeling mildly guilty about starting this whinge thread. ☺️