Massive upgrade to the scale names

One of my pet peeves in this whole music theory project is how often I get snide remarks about the collection like "that's not a scale, you made that up".

In response to this, and for the benefit of researchers, published yesterday: a huge upgrade to how I organized and publish the lists of scale names. If you go to the website now, you'll see that scale names are still organized by musical traditions, but now you also get citations showing where the name came from.

This upgrade follows about a month of work tracking down published sources, parsing scanned books and online databases, building a new schema for storing it all, and designing a new rendering for the UI.

The collection was previously organized poorly and I did not store the provenance of names that I put into the database. As a result, I could not verify names nor identify how they got into the system.

Now, every name is meticulously catalogued along with what scale it is meant to describe, and a complete citation. My database currently consists of 13.201 rows (with tons of duplication) with the names provided by 26 different books and websites. For example, the scale "Lydian" appears as 24 different rows, once for each of the sources that mentioned it.

Those names are then massaged into proper case with consistent spaces and hyphens, "normalized" for spelling variations (e.g. "flat 5" and "♭5" are deemed equivalent) or name equivalency (for example, "Locrian Major" and "Major Locrian" are considered equivalent).

Then the names are filtered by an "errata" database that I also maintain. Some of the books I own have names that are just utterly incorrect. Some of the scale books out there are riddled with typos, bad spellings, and pitch inaccuracies. Rather than correct them in the source data, I keep the incorrect rows and mark them as incorrect so they do not show up on the website or in any other materials.

Finally, the aggregate collection is deduplicated and displayed with optional extra meta-information, and warnings when an ambiguous name is used to describe more than one scale.

There are still names in the collection that have no citation... they will remain there for now, as I diligently hunt down sources for them. But eventually I will want to cull the collection removing any names that don't originate from any published source or legitimate musical tradition.

The new system is elegantly architected so that I can easily add more sources, when I acquire a new book or discover some other resource. It will also be easier for me to edit the names as needed and republish scale pages so everything stays up to date with those changes.

This upgrade will be really useful when I start working on version 3 of the youtube videos... that's a project for next year.

The book has made a little progress, but not a lot since most of my time has been dedicated to getting this upgrade done.

Names are messy. They won't all be perfect or correct. If you see a name that seems incorrect, let me know.

Cheers
Ian