The heart of the beast

 THP has been a home to Touhou fanfiction has been around for 9 years now. But, its true rebirth as an independent community only came about a year or so after that. It was then that we moved on to the touhou-project domain and a server we controlled. In the first of these posts, I’ll be talking about how the site runs. I think it’s important information to have in order to understand why some things need to be changed going forward and how we’re going to go about them.  

But first, a little explanation as to what the point of this reward is about:  

I know that the tier description is a little ambiguous, but I plan to make these entries basically a blog about the technical side of things and let you know what I’m up to or what the site is doing. You can share this information on the site and elsewhere as you please as I don’t particularly care if the Patreon rewards are “private”. Everything on the internet is public, eventually. The only thing I’d ask is that you don't repost these entries for a while. Keep them to yourselves for some time, if only not to cheapen too much the incentive for donating at this level. From my perspective this is a little bonus to the kind souls that donate. Fanservice even! I’m going out of my way to write this since I don't normally write about these things on the site or elsewhere. Do consider that it takes me plenty of time and effort to make this sort of thing (hopefully) interesting and/or entertaining for you guys. 

Our hosting setup is fairly typical: we pay a hosting company who has allocated space at a datacenter to give us bandwidth, space and processing power in the form of a VPS. A VPS, or Virtual Private Server, is something of the middle ground between actually having a full server where all the hardware is being used for computing and a shared server where you can typically only do limited things like using a framework for a blog or static content only. In effect, what’s happening is that we’re allocated a set amount of resources from an actual full-fledged server but we’re allowed to do as we please within our space. This means we can install what we want and muck about for a much lower price than having a dedicated server. For a relatively low bandwidth site like THP without too many  external requests, it’s a pretty comfortable setup.

Of course, in the beginning, we had some serious teething problems. Any of you who were around when we first took over might remember that we had almost chronic issues with request timeouts as well as a spate of mysql errors. This was mainly because we had underestimated how many resources the site would take!  Running out of free ram was the main cause for these problems and after coaxing HY to upgrade the plan, we were mostly fine with the occasional hiccup. When I started paying for the hosting bills myself, my first order of business was to switch hosts and get a better, more competitively priced plan. For the most part, I chose well. There were some problems with the virtualization software they used for the VPS as well as with their datacenter. This resulted in some downtime every now and again. But, after another tiny migration after they upgraded their plans and software, we’ve encountered very few serious problems in years.  

Our operating system is Debian GNU/Linux. Why Linux? Partly because I’ve been  using it as my main OS since the mid 00s and partly because it’s perfect for the server environment. It’s free and libre, meaning that we can get it for no cost and modify it as much as we want. Plus, if I’m perfectly honest – Microsoft’s server software has always sucked and the various BSDs are less universally supported. My own knowledge of the environment has made swapping out software a breeze (more on that later), something which isn’t necessarily always the case. As a result, day to day administration is done through the command line remotely, just the way I like it!

Why Debian in particular as our distribution? Because it’s widely used, favors stability and was better than the other options in the initial host we used. Basically, we have a solid webserver setup because of these things. As of the time of writing this the site uptime is 163 days and 4 hours. The only reason it’s not longer (it was nearly 200 days before that) is that I had to reboot to apply security updates to the kernel. I was actually surprised that nothing broke with that update either, given just how many changes were being applied at once. Speaks volumes of Debian’s configuration and stability. The downside of all this meticulous packaging and stability is that the software itself is old. This can be mitigated by an extent by backports but, given how quickly the tech world moves, it can leave use without the latest and greatest until the next release cycle (can be ~2 years). And those are sometimes pretty big roadblocks to making significant changes ourselves.

In our Linux installation, we’re running a pretty standard combination of a web server, PHP and SQL. Originally that meant using Apache and MySQL. Apache is a great solution and still the world’s most popular web server software but can also be really complex to configure and, when we first started out, had a lot of performance overhead. The fact that Debian only had stable versions available meant that we wouldn’t get those new improvements that were making Apache more competitive for a long time. So, at some point, I chose to switch to lighttpd which is a simple, lightweight and no-frills piece of software. It has its shortcomings, too, but I chose it over Nginx because it had better documentation at the time. A similar thing happened to MySQL, we migrated to MariaDB several years ago because of technical superiority as well as its more active development cycle.  

The combination of PHP5, MariaDB and lighttpd allows use to run the next critical piece of the puzzle: the actual board software. The site runs off of a modified version of Kusaba X, which was a fork of the defunct Kusaba. Sadly, Kusaba X ceased development several years ago as well. There isn’t really any actively developed imageboard software these days save for proprietary stuff, which means that any and all support and improvements on the site falls squarely on me and any developers we may hire in the future. There’s a lot more to say about Kusaba X, its strengths and shortcomings but that’s something better left to a dedicated post somewhere down the line. Suffice it to say: this is one of the areas in which the site needs the most work and has been a cause for headache for a long time.   

There is also a few other pieces of software that run on the server that make life easier. Chief among those is iptables and other network tools that manage the firewall and can mitigate abuse. In conjunction with this type of software, I have made a few scripts to block people who were spamming the site or trying to DDoS us. This was needed a few years ago when a great host of Chinese, Russian and other IPs were hammering the site, saturating resources and making it unresponsive. We’ve been lucky not have been hit with anything quite so serious since then! Every system administrator dreads having to deal with this kind of stuff since it can be like playing an annoying game of whack-a-mole.  

Other than that there’s things like awstats (which analyses site traffic), mail server software (which for now only really emails me with stuff like reports for scripts that run on the site), a couple of bots for IRC and software that makes running the web server software easier. Like, you need ffmpeg (or Libav which is the fork Debian uses currently but will get rid of in favor of normal ffmpeg again soon) to create thumbnails for webms posted on the site. Lastly, the scripts that keep our SSL certificates up to date are worth mentioning as they have been of great help once I finished editing them and got them working just right.  

Sorry if this post hasn’t been that interesting! I know this might not seem at all relevant to Touhou fanfiction or the end user experience right now, but all of it matters since a lot of it has to be touched upon. I think that once we start delving into which areas need work and what we should be doing you’ll see how things are interconnected and why we’re proceeding in that manner. We’re likely not to talk about history much in the future since we seem to have just about most things covered.  

Do give me feedback in the comments, though. Since this is my first time doing this sort of thing I don’t know if I am being too verbose, not verbose enough or missing the mark entirely in content. I’ll change according to what you guys tell me. I have no idea when the next of these posts will be since I don’t want to be too spammy either. Whatever feels right, I suppose.  

Have fun in the meanwhile and make sure to read, write and discuss things on the site too!