Step by Step (1)

Hey all, hope you’ve been doing well! I’ve been busy at work doing a little bit here and a little bit there and, as of a few hours before this post, have deemed it time to push through many of the changes that were ready.

I’ve tried to keep things brief and categorized but there’s a lot to talk about the things accomplished and some of the challenges I had to face.

Images

Since THP is an image board, naturally the actual image posting aspect of it should be robust, right? Well, a bug was brought to my attention a while back, relating to transparency not being kept when thumbnails were generated. This was likely a bug that’s been in the code base for a little under two decades now and more or less went unnoticed.

I tackled the image generation process head-on. Transparency was being ignored outright and had to add a way to handle that. As you might imagine, reading up on how images are processed by code is not exactly intuitive and requires holding a lot of abstract ideas while designing code; the various states of a passed image needs to kept track of, things like color spaces, blending and transformations applied. There’s dozens of in-built functions and tools available and, often enough, very many parameters that can be passed along each step of the way. From initial validation to resizing to final validation and saving, it’s 100+ lines of code and several functions of testing and tuning to get things right.

Now, to be clear, I’ve dealt with the image generation process for THP in the past. Things like adding WebP or WebM support required my messing about and I have excised a lot of dead or useless code over the years. Still, it’s not all gone. There’s references to other mechanisms like the mature and robust ImageMagick library that I’ve kept as a fallback option. All the same, it’s a little bit surprising when in the course of debugging things I saw a section of code that was attributed to someone from 2004. I plan to eventually get rid of most of that old image-generation code as the new features I have in mind for THP in the longer run will need more flexibility. That said, just reading over and analyzing what is being used and what’s useless gave me a pretty good idea of the scale of the job. I’ll see about doing more of that over the coming weeks.

But, well, before I get carried away, the transparency issue was fixed. For the most part. Turns out that GIFs with their limited 8-bit per pixel and limited color range are tricky to get right. I spent altogether too much time trying to get it working correctly and, in the end, figured that it wasn’t worth dedicated so much time for a somewhat niche problem. It’s not forgotten by any stretch but, rather, I would rather prioritize other things first. It’ll need a custom solution to fix fully as even the various discussions and examples of code I found on the internet with people with similar problems all have their own drawbacks or aren’t applicable. I’ll have to start from the basics and build a proper solution.

Scripts

A lot of THP’s functionality is driven by user-facing scripts. As new features are added and systems interact, there’s greater complexity. So I spent a fair amount of time dealing with several outstanding bugs as well as new ones that I discovered as I went along.

The most prominent issue was with the preview function. It failed to retain at least one specific character (+) and also had issues with outputting correctly-formatted links and images. The latter was fairly easy to debug and fix in the generation process that’s done on the backend.

When hovering on links or embedding posts inside a preview post, things should now behave correctly, as should edge cases related to spoilered images. Another, more important, enhancement was related to how the previewer works when the page width has been limited (either by option or different screen sizes) and it should now look the correct size and orientation no matter how a user previews a post. Add to that the removal of some cruft that’s not needed in a temporary post and the feature is ever-so-slightly optimized. Oh and a few changes to other parts of the code base mean that so users are now warned they can’t preview a reply in locked threads.

What did my brains in was trying to figure out why some things were garbled. I figuratively tore down the walls trying to follow step by step where it was failing. And when that didn’t illuminate things, I basically rewrote a more limited version in my test environment to eliminate the possibility of other bits of code interfering. No joy either. Thankfully, after spending most of an afternoon trying to fix things (though, admitted with breaks and I did other work when I got too frustrated), I finally saw the one tiny thing that subtly broke things. To make a convoluted story short, basically a lot of data is sent asynchronously to the server which then parses it, assembles a post and then returns it so it’s inserted on the page. One specific bit of the data that was being sent wasn’t quite encoded in the proper format but every other part of the data was correctly being parsed. The object as a whole was valid, which made the thing a real head-banger.

Embedding post or loading up a post on hovering on a link also got a look beyond how they related to post previewing. A lot of it was bug fixes for certain cases (with all the combination of options, it can lead to unforeseen and strange situations) like how highlighting certain posts caused the page to shift and trigger other knock-on events that could be confusing. It’s my ultimate goal to get both features working with archived posts some day so cleaning up the code base and making sure the fundamentals of the design are solid are in my best interest as a maintainer.

Other than that, there were a couple of quality-of-life improvements to the existing reminder buttons that tell users when they can edit their posts or if they should add it to the story list. That’s the kind of thing that just requires a little tweaking usually but also uncovered a few issues with other interrelated systems. Like, in one case, a certain combination of options and states of reminders blocked some of the pop up menus used for reporting or submitting certain data by users. I sometimes think to myself that it’s odd that no one else encounters these sorts of things (or, well reports them) but I suppose most users don’t really use many if any of the advanced site features.

Moderation and other things

While it’s not really a big deal, some ways of moderating the site had been broken for some time since I stripped out old code and design principles from the original developers earlier this year. Think things like assuming that moderators are board-specific first and using cookies to expose features. As a result, posting as a moderator from a board page did not allow for setting special flags like locking a thread, making a sticky, displaying staff status etc. Most of those things could be done from the moderation page interface anyhow, so it wasn’t a high priority to fix.

It took some sleuthing through several files to see how every loose thread of code unraveled others but, while I was at it, I cleaned up a lot of things I should have done earlier. Moderation tools are slightly less silly now and, as an added bonus, it led me to simplifying how a lot of the RSS feeds are initiated.

I don’t want to harp on too long about the questionable design decisions of the original developers but in this case it’s worth a brief explanation: when a post is made it goes through one process; when a post is deleted is goes through a similar process to a post but with some caveats; when a post is deleted by a moderator through the mod page it goes through another process; multiple posts or due to ban, another; when it is deleted by a moderator from the board pageit goes through yet another!

So if information is updated at one of those points (like, say, regenerate a board-specific RSS feed) it might not if you do it a different way. Add to that that those are very different environments, with different data and variables and creating a universal solution can be tricky. I had to wrangle a few different classes and write a special new function that acts as an intermediary to get behavior working for most cases. For some reason when all posts by a user are deleted it fails to remove some entries here and there in the backend. Given that that’s a rare action, took my hours of improvements as a win and will revisit things whenever I have time.

Before I move on, I did also want to mention that there were some changes to the few cookies that the site does use, basically making sure that most of the security and privacy-focused settings are enforced everywhere they’re possible. This was already mostly the case but I saw some of the code when I was rooting around in the mod page and figured why not?

Great Cleanup

In the course of doing everything else, I realized that there were legacy portions of code that either had no function or got a little in the way, mostly by being variables to pass on when rewriting or expanding upon other parts of the code. While I’ve already mentioned some of the pertinent parts in regards to previews and moderation parts, a lot of this is more structural.

I’ll give you a brief refreshed on how data is stored in THP’s database: there’s various tables that keep posts, board info, staff details, etc etc and they each have several columns that hold data. In the case of posts these will be who posted it when and if there’s an image attached and so on. There’s some functionality that’s used rarely or not at all or that was implemented in a highly inefficient fashion.

I’ve mentioned it before but it’s one of my eventual goals to normalize data and optimize how it’s stored and thus separate, shuffle around, or its organization and relations. But that ultimately goes hand-in-hand with any changes to the site generally as we move away from the classic image board format into something purpose built for us. In other words, there’s no need to change it all now nor all at once. For the time being, whenever I can, I whittle down parts of it and simplify what’s stored and how it’s retrieved. Previous posts have talked about things like spoiler functionality, story list and other things having to do with this.

So that I don’t finish you off by being too boring, most of my attention in this area in the past few weeks has been aimed more narrowly at the data for the boards. I removed several columns of data: the one that keeps track if the forced anon feature is enabled, forced redirect when posting, the max age for a post before it’s culled, at which page a thread is marked for deletion, and the custom option to start posts on a board from a specific post number.

All in all, useless features for our community. In each case it required changing or removing code that made reference to those things in about a dozen or so different files. It simplifies things a lot and makes the table easier to manage and I decided to also add a new column: one that tracks if a board is not safe for work overall, which has implications in how thumbnails are generated for post previews and on the front page. That way /at/ does not have to be specifically exempted from the various things, and a simple toggle keeps it (and any other  designated boards and their content) out of the way.

As you might imagine, testing that I didn’t break anything else was the biggest time sink. I prefer to be slow and steady with this sort of thing rather than to move fast and break things. It comes with the added bonus that it gives me time to reflect and think about how to tackle future cases and I’ve got several notes jotted down for when it’s time to reform the posts table (and reintegrate archived posts).

It’s nice to also be able to change some minor things that kinda bothered me while I was at it. The options interface for boards on the private side of the site was also slimmed down and made a little less overwhelming. That does need a lot more work … especially when it comes to how boards are “regenerated” after settings are changed but it’s not something too urgent and I can do it piecemeal besides. Same as some of the behind-the-scenes tweaks I did to the front page and will continue to do for the time being.

Wait, there’s more!

There actually is a fair amount more but I feel like I’ve written out too much for a single post. I feel like I might overwhelm my dear readers with too many different subjects, with not that much depth dedicated to each one.

So, I think I’ll post the next sometime soon. Seems like this is what happens when I’m too lazy to write these posts regularly—there’s a lot to cover when I do get around to it!

Look forward to another post soon and, as always, let me know if there’s something that needs clarification, if there’s anything that you’d like to hear about, or just feel free to give me feedback about whatever else.

Until next time, take it easy!