Hello everyone. This is still a fluid situation but here's an update.
First some clarifications. I know this has caused some irrelevant confusion before. There are two Toms doing different stuff on this website. Most of my time should be spent on our other technical operations, while most of the stuff you see around you is Tom dB. Usually you can just say 'Tom' and at least we will know who you're referring to.
Second thing, the uptime has been a matter of some debate. Prior to 4 January this year we had 100.0% uptime on the 'new' forum software, I think. There was briefly some shonky table locking going on with the searches, but we innovated a solution to that quite a while ago and it never caused any serious problems. Since 4 January, when there was a sudden and unexpected crash, we've had some intermittent downtime. There have been a lot more continuous days of uptime than days with downtime since - I'm not saying it's currently good, but I suppose saying that I thought I had fixed it several times. Obviously there's an ongoing issue. The crashes happen mainly, but not exclusively, when the server's under load. If it happened regularly or predictably my job would be so much easier.
I posted the other day that we were waiting for our hosting provider to investigate a hardware issue. They have since investigated, and yesterday fixed, one particular problem. We are monitoring that situation further - I had hoped that would be the end of it.
However there is evidently another problem with the server. For those who know about such things, i/o is showing itself to have some sort of unbearable load. There's now a number of solutions I'm looking at implementing forthwith - one involves "archiving" some of the posts in the forum, another is to move to another server - not itself a trivial task. That won't be today. We are also in the process of offloading other load to yet another server.
So there you are. This is doing my nut in more than anyone else's. I've investigated a hundred possible bottlenecks this month, worked more hours than not, and I'm now narrowing in on a lasting solution. Sleep depending, I'll possibly arrange something for later today.
