Lissa Daniels

Site Update (part two)

Added 2024-10-20 14:22:34 +0000 UTC

A quick little update. As you can see, I’m making some progress (we’re not just getting the generic blank error page). The problem was threefold -

1.) We had an unexpected surge of traffic hammering our database (still investigating this)

2.) The new queries I built for all of the tasks were marginally more resource intensive

3.) Our server realizing we were overutilizing the CPU throttled the site. This aggravated the other two issues.

When I restored the site the first time, it was restored at a throttled level which I did not realize. This then almost instantly caused us to hit capacity again. The server, seeing we were at capacity of the new throttled level, throttled the site further. This caused a lot of broken queries, data stuck mid-stream and what not… and I’ve been trying to unravel.

It’s been a bit of a web to unweave trying to fix things one at a time and get us back open. I restored back to a backup taken a couple of hours before the first outage in order to get everything stable. A few players will have lost a post or two (myself included) that happened right before the first outage (or between outage one and outage two). Apologies for this.

I am currently working on expanding and debugging a more intensive cache system that should vastly improve the queries on the site and keep this from happening again. Unfortunately this is just taking me a little time and needs to be done in a closed state.

I expect to be able to open back up in a day or two (optimistically).

Hang tight all. We will be back shortly.