rabid.audio

JTM45 Build Part II

2026-04-08T00:00:00-04:00

A few weeks ago I wrote about the Marshall JTM45 amp I put together. In it, I teased a bigger mod I wanted to do.

Comparing the schematics between the JTM45 (actually the Fender Bassman but the circuits are basically identical) and another classic British amp, the Vox AC30/6, I found the topologies to be incredibly similar. The Marshall sounds great for sure, but I’d rather sound (in the words of a friend) “more Brian May, less Jimmy Page.” So I converted my amp to a hybrid of the two.

Below are both schematics with the common sections highlighted. We can step through them one by one.

Comparison

Power (red)

The power systems are very comparable between the two, as expected for an AC/DC converter. The most notable difference is the lower output voltage of the AC30, a 30 watt clean amp, compared to the 45 watt high-gain JTM45. The JTM45’s power transformer lines are 325VAC on each side, compared to 180VAC. The schematic shown here for the AC30 has a solid-state rectifier, but earlier versions used a tube rectifier also.

Both use a choke on the B+ lines. The power capacitors have different values to account for the different voltages and noise profiles. The JTM45 has a solid-state half-rectified B- line that the AC30 doesn’t. The JTM45 progressively steps down the B+ lines from the later stages to the earlier stages, while the AC30 uses a common B+ line. But at the end of the day, DC volts are DC volts, the specific topology of the power supply matter little beyond the output voltages (as long as they are properly filtered).

Trem (green)

The AC30 has a tremolo circuit, highlighted in green. This adds 3 additional tubes which I don’t have the space for, and honestly the circuit makes my head spin a little bit¹. I’ve got a decent tremolo pedal so we can ignore this (treat the output as ground).

Input buffer (teal)

These two sections are almost exactly identical. Bright and normal channels, each with a low and high input with a 68k divider. Both are 12AX7s (at least on the JTM45; the Bassman used a 12AY7) buffering each channel. The only difference is the tube biases: the B+ line is lower on the AC30 (probably about 238V compared to 432V) and the cathode and plate components are different. This will be a recurring theme throughout this analysis.

Stage 2 preamp (blue)

Here there’s a big difference in design but a small difference in practice. The Bassman/JTM45 mixes down the two channels here (in brown), causing both channels to go through the tone stack. The AC30 sends only the bright channel through this stage, sending the normal channel directly into the phase inverter stage (note the summing resistors in brown are in the orange section instead). Frankly I prefer the flexibility of getting tone controls on both channels, so we’ll stick with the JTM45’s approach here.

Notice the bright cap is the same in both circuits² (100p). Now the value of the volume pot is different by a factor of two (500k vs 1M), which moves the cutoff by a factor of two. But still this is much more similar than I would have expected between circuits known for sounding so different. Again the B+ and cathode resistors are different changing the biases slightly. The AC30 has a higher cathode resistor but that’s likely to compensate for its lower B+, the gains are probably similar. The cathode resistor is actually smaller on the current buffer, probably to compensate for the higher loss of the AC30’s tone stack.

Tone stack (yellow)

This is another major difference between the two circuits, and probably has the largest contribution to the tone differences between the amps. The JTM45 uses the Marshall tone stack, with Bass/Mid/Treble controls, while the AC30 has just Bass and Treble. You can use the ubiquitous Duncan Tone Stack (for which there’s an online version now) to see the differences between the two curves (Marshall is blue, Vox is red).

You can interactively compare the two here. The Vox has a lower overall output and much more of a mid scoop.

Phase inverter (orange)

The phase inverter uses the same differential amplifier approach in both circuits. Apart from the summing differences discussed earlier, there are a few other differences. The Bassman/JTM45 has a presence filter on the opposite side of the amp, optionally bleeding off the highest frequencies. It also has a negative feedback line from the output (salmon). It has a 47pF capacitor across the outputs for stability, and an asymmetrical plate resistor design, with the input side at 82k and the reference side at 100k. The AC30 has symmetrical 100k plate resistors, higher input coupling capacitors (47nF vs 22nF), and different biasing. The output coupling caps are comparable (100nF vs 150nF).

Power amp (magenta)

This is another section that’s pretty different in theory, but not that different in practice. The AC30 uses two stages of lower-gain EL84 tubes, while the JTM45 uses one stage of EL34s. In addition to the gain differences, these tubes have different pinouts, so switching them is not trivial. At the end of the day, most of the tonal character has come from the phase inverter and the pre-amps; the power amp serves to make that signal loud, so using the louder JTM45 design makes sense. The AC30 sends the 220k grid leak resistors to ground, while the JTM45 connects them to its -50V B- line. This is just a more complicated example of the differences in tube biases.

The AC30 also has an additional “tone cut” filter: a 250k ohm pot and 4.7n cap acting as a high-pass filter shorting the high frequencies across the differential amplifier. This filter sweeps from above 30kHz down to 130Hz. The AC30 has a 1.5k gate resistor, while the JTM45 has a 5.1k gate resistor (not in the schematic).

Changes

A complete conversion to an AC30 would be an enormous undertaking, but a hybrid of the two is quite doable. I did this in stages, with the most impactful changes first. In order to speed up the iteration cycle and not wear out the tube holders by constantly removing the tubes between every change, I rigged up a little mount with some leftover plywood to hold the amp while I work.

Tone stack

Even though the two tone stacks are pretty distinct, the changes required are minimal. Both use 22nF caps for C2 and C3. Both are input through a capacitor into the top of the treble knob and output on the center of it. The bass pot is 1M in both cases, though the treble pot changes from 250K to 1M. I had a 1M left over from the master volume mod for this. I changed the 250pF cap to a 50pF, the 56k resistor to a 100k, swapped the treble pot, removed the mid pot and moved the bass pot into it’s spot, added a 10k resistor across the bass pot, and sent the bottom to ground, and sent the 22nF cap that had been going to the mid pot to the center of the bass pot.

Tone cut

Using the 250k pot that used to be the treble pot, I added the tone cut filter across the differential amp. Because I’d already done the master volume mod, I tied these into the master volume pot, but if you haven’t done this you can place these where the grid leak resistors jump to the the power tube gates.

I didn’t like the way the 4.7nF capacitor sounded, so I ended up using a 47nF capacitor instead, making the extreme end of the low-pass filter cutoff at around 13Hz instead of 130Hz.

The difference between the 100pF and 150pF coupling capacitors would normally be somewhat negligible, but because adding this pot effectively halves the output impedance, increasing the capacitance compensates here by keeping the cutoff in a similar spot, maintaining the low-frequency response. I added 47nF capacitors in parallel to bring these up to 147nF.

The JTM45 presence knob provides a similar control as the tone cut with a different approach, but since there’s already a hole cut for the pot and it’s already working, it doesn’t hurt to keep both.

Re-biasing

So far these changes have been pretty easy to accomplish and had large effects on tone. Adding a tremolo and switching EL34s to EL84s are extremely difficult tasks and would require more tubes which we don’t have space for. Outside of that, the only difference is the tube biasing.

Originally I’d planned to re-bias all the tubes to match the AC30 versions. I bought all the components to change the cathode and plate resistors to match. However, the significant difference in B+ voltages adds a significant complexity to this operation. The transformer outputs a much higher baseline DC voltage with the JTM45 and I definitely don’t want to replace it. Simply adding a voltage divider to drop the voltage by a factor of almost two would be quite inefficient and makes me nervous.

I’d rather keep the B+ lines the same and adjust the biases so that the gain of each stage is the same as the AC30, even if the components are different. However, it’s difficult to calculate the gain at each stage. I couldn’t find an AC30 schematic that had test point voltages indicated. If I had one, I could measure them but I don’t. I tried simulating them in a circuit simulator but the load, including the coupling cap values (which seems unintuitive to me), seems to play a significant role in the resulting gain, making it hard to compare.

For example, I simulated the first stage pre-amp gain of the JTM45 and found it to be about 50. But the AC30 I found anywhere from 35 to 60, depending on the B+ voltage and the size of the coupling cap and volume knob.

I tried measuring them on the oscilloscope, but the tube rectifier would flicker and scream at me if I connected the scope across the differential amplifier. And if I don’t have an AC30 to compare it to it doesn’t do much good regardless.

I tried replacing the B- with ground, hypothetically reducing the gain of the power amp stage. However this also made the tube rectifier angry, suggesting I probably need to adjust some other biases to compensate. I didn’t try removing the negative feedback as that would make the already over-gained amp even louder and dirtier.

In the end, I decided to keep the tube biases where they are. I’ve got a louder amp than a true AC30, and at low gain it sounds quite pleasing, on the edge of breakdown depending on how hard I play. The master volume means I can tune the gain and therefore the cleanliness independent of volume, so there’s no harm in keeping the gain low. And there’s quite a lot of bite there if I want it.

If I were to re-bias anything, the phase inverter would be the most important, followed by the preamps, since these are were the most clipping is likely to occur. In hindsight, I might want to try biasing the phase inverter evenly, since the 82k/100k imbalance could be the source for asymmetric clipping.

Solid-state rectifier

I also attempted the solid state rectifier mod I discussed previously. But adding a diode bridge in parallel with the tube rectifier made it very unhappy (despite the tube rectifier being disconnected from the output). I think I have to disconnect the transformer center tap when using the diode bridge, but exactly which lines need to be disconnected and which can stay connected is unclear, and I don’t want to have a 5-pole switch as that would be incredibly expensive. I don’t have a big enough switch for these voltages right now anyway, so I decided to come back to this if/when the tube rectifier blows.

Conclusion

At some point I want this to be an amp rather than a project, so I decided to take it home in this state for now. I did a side-by-side comparison with my clean amp. It sounds lovely in this state; with the gain low it has a comparable brightness but a deeper low and mid range, and a bit more saturation when playing hard. The low strings almost sound compressed. Cranking the gain and the treble gives it this great energy that reminds me of Marc Bolan. I think I’m going to use both on the Punter album I’m working on, though I’m not sure yet if I want to use them in parallel or for different guitar parts.

I have kinda caught the amp building bug though. Might do a Princeton Reverb build for a friend one day; she’s wanted one ever since we played one at Big House Guitars and I think doing a build with a spring reverb would be soooo cool.

I happened to look at the schematic for a Princeton Reverb last night and that tremolo is much more straightforward: one triode as an oscillator (feeding back on itself) oscillates the power stage grid bias between a slightly negative B- and a highly positive B+.
There are a few things I can identify in the AC30 schematic though. On the output of the sub-circuit there’s three 1M ohm resistors and four 4.7nF capacitors. These are acting as an extremely sharp (4th order) high-pass filter with a cutoff around 30Hz. This likely is to filter out the DC offset while keeping the audio signal and as much of the low-frequency content of the tremolo (probably around 5-20Hz) as possible. Also interesting is that rather than summing the signal with the normal and bright channels, they put it on the opposite side of the differential amplifier. I suppose the thought is you aren’t going to be using both channels at the same time, so you can treat the opposite side as ground. I am curious what it would sound like to run two guitars through this amp at the same time… ↩
Its interesting to me that back in the day amp designers would add extra jacks and buffers to have multiple channels when the only difference between them is a single bright cap. Seems much simpler, cheaper, and user-friendly to have 1 channel with a switch that optionally connects the bright cap. That’s 2 jacks, half a tube, and a pot saved. Heck, you can switch the input resistor as well and have just one jack instead of 4 for the price of two switches. But IDK, maybe switches were more expensive, or they really expected multiple channels to be used at the same time? If someone knows the history I’m curious. ↩

Introducing chicy.online

2026-03-27T00:00:00-04:00

Background

I’m a big fan of static sites. Why so much of the web uses Wordpress with its bulky, error-prone, and insecure PHP backend to store what’s effectively just some HTML and CSS never made sense to me.

Tooling for this sort of site is incredibly flexible. This blog is created using Bridgetown, and an earlier iteration of it used Jekyll. Site generators allow a really pleasant and flexible abstraction for creating content that can then be packaged into static HTML/CSS/JS/etc files for deployment. My personal links site okaysure.cool is literally just a single html file.

The great benefit is without a backend server, hosting can be accomplished entirely through a CDN, making it incredibly cheap to run. There’s no reason a low-traffic site like this should need a $20/month server running constantly in the cloud waiting to handout files which are public.

Unfortunately, actually hosting this somewhere is kind of a pain. GitHub has Pages, a free and popular option. But deploying to it is trickier than it should be. For a long time, this blog was compiling code and committing the results to another repo, where it would then trigger another CI job to update the site.

Meanwhile, for some simpler sites I was using Kinsta, a for-profit company offering a free static hosting service as a lead-gen for their other products, which is going through some changes and is dependant on the graciousness of a for-profit company.

At FIXD, we had a Terraform module that would set up an S3 bucket configured as a web host with CloudFront in front of it. Since there were no backend servers and S3 storage is so cheap, this was an essentially free way to run a static site. Since then I’ve wanted an easy way to do that for myself and my friends.

`chicy init`

Introducing chicy.online, a hosting service for static sites and a CLI for managing them.

Getting started is easy. Simply install the CLI from NPM:

npm install -g chicy.online

Next, create a new site:

chicy init --name "My Cool Site"

Authentication

The first time you use it, it will send you to GitHub to authenticate. I needed some way to keep track of users, and GitHub OAuth seemed like the best approach. The app only stores your login and email, it does not request or require any access to your account or any of your repositories.

Deploy Key

Once authenticated, chicy will create a site for you, giving you a unique subdomain on sites.chicy.online, e.g. teeny-angle-dwas5. It will generate a deploy key which you can use with CI/CD to publish your files. If you aren’t using CI/CD you can ignore this, as you can publish from your own machine using the GitHub authentication you’ve already performed. This deploy key is only shown once, but you can regenerate a new one later with chicy regenerate.

Config file

It will also generate you a .chicy.json file that you can use to configure the site. We’ll come back to it later.

`chicy deploy`

Next, compile your assets (if needed), and upload them:

chicy deploy path/to/dist --wait

This will create a tarball of all the assets and upload them to S3, attaching them to a new deployment, with an ID like d_00001122334455aabbccddee.

Behind the scenes, chicy creates an S3-backed git repository and commits your files to it. The --wait flag (optional) waits for this process to complete. Depending on the size of your site, this can take a little while.

You can verify the files that are going to be uploaded with --dry-run. This will print the paths for all files to be uploaded.

`chicy promote`

By default, new deployments are staged and not immediately published. To publish your site, a deployment needs to be promoted:

chicy promote d_00001122334455aabbccddee --wait

Under the hood, this will check out the S3 git repository at the given deployment and load the assets into place to be served by CloudFront. This will also trigger an invalidation of all assets so that CloudFront will begin serving your updated site. This process can also take a while.

You can promote and deploy as one command. For example on CI you might do:

npm run build
CHICY_DEPLOY_KEY=dk_ac3d... npx chicy deploy dist/ --promote --wait

Custom domains

At this point your site should be live on your unique subdomain, e.g. teeny-angle-dwas5.sites.chicy.online. But no static site service would be complete if it didn’t let you bring your own domain. chicy will automatically provision and validate an SSL certificate and serv e your site via HTTPS.

DNS Records

You’ll need to set a CNAME record pointing your custom domain to the site subdomain. Check with your DNS provider for directions on how to do this.

www.example.com CNAME teeny-angle-dwas5.sites.chicy.online

If you’re using a subdomain like www.example.com, that is the only record you need.

Apex records

Unfortunately, if you want to run your site at the root/apex of your domain instead (e.g. example.com), CNAME records are not supported at the root. You can still do it, it just requires more steps.

First, you’ll need to find the CloudFront distribution domain. As of writing this is dwuuvq1nesgb7.cloudfront.net but should I need to re-deploy the server it could change in the future. You can check it on the API health check endpoint: api.chicy.online

Then you’ll need to create 3 records. First, you’ll want A and AAAA records for the IP addresses of the Cloudfront distribution, which you will need to look up:

dig dwuuvq1nesgb7.cloudfront.net A   
dig dwuuvq1nesgb7.cloudfront.net AAAA

Create records for all the IPv4 and IPv6 addresses.

example.com	A	18.64.236.128
example.com	A	18.64.236.93
example.com	A	18.64.236.57
example.com	A	18.64.236.79


example.com AAAA	2600:9000:2335:4000:16:ea16:e4c0:93a1
example.com AAAA	2600:9000:2335:6200:16:ea16:e4c0:93a1
example.com AAAA	2600:9000:2335:7a00:16:ea16:e4c0:93a1
example.com AAAA	2600:9000:2335:fe00:16:ea16:e4c0:93a1
example.com AAAA	2600:9000:2335:c000:16:ea16:e4c0:93a1
example.com AAAA	2600:9000:2335:6a00:16:ea16:e4c0:93a1
example.com AAAA	2600:9000:2335:7800:16:ea16:e4c0:93a1
example.com AAAA	2600:9000:2335:3c00:16:ea16:e4c0:93a1

If you’re using AWS as your DNS provider, you can do this through alias records instead.

Finally we’ll need to validate ownership. Normally CloudFront can verify domain ownership through the CNAME record but if using A records you’ll need to verify using a TXT record pointing _cf-challenge to the Cloudfront distribution.

_cf-challenge.example.com TXT dwuuvq1nesgb7.cloudfront.net

Attaching domain

Once the necessary record(s) are created, simply associate your domain with the site:

chicy configure --domain www.example.com

You can do this before or after your first deployment. If you’ve already got a deployment published, your site should be available on your custom domain!

If you want to remove it later, you can do chicy configure --no-domain.

Rollbacks

To roll back to an earlier version of the site, simply promote an earlier deployment.

chicy deployments # list the deployments for the site
chicy rollback d_00001122334455ffffffffff --wait # `rollback` is simply an alias for `promote`

Config file

When you run chicy init, a config file is generated for your site. The default template looks like this:

{
  "siteId": "teeny-angle-dwas5",
  "exclude": [],
  "retain": [],
  "rewriteRules": [
    {
      "match": "^$",
      "replace": "/index.html",
      "last": true,
      "flags": ""
    },
    {
      "match": "^(.*?)/$",
      "replace": "${1}/index.html",
      "last": true,
      "flags": ""
    },
    {
      "match": "^(.*?)/([^.]+)$",
      "replace": "${1}/${2}/index.html",
      "last": true,
      "flags": ""
    }
  ]
}

siteId

Most commands require a --site flag to know which site we are working with. This can be inferred if using a Deploy Key, or if this config file is in the current working directory. If not specified, the CLI will prompt you to select a site.

exclude / retain

Both of these options are a list of globs relative to the upload path.

exclude (in addition to the --exclude CLI flag) will skip files from your asset directory from being uploaded to the site.

Use retain to specify file paths that should be carried over between deployments, even if they aren’t included in subsequent deployments. For example, if you use cache-busting asset fingerprinting like script-{hash-of-content}.js, you can keep the earlier versions of the asset around for any clients still using them.

rewriteRules

Rewrite rules allow you to alter URLs to point to different files in your deployment. It’s loosely based on Apache’s mod_rewrite. Rules are evaluated in defined order. If the passed in URI path (only the path; hostname and query parameters are not included) matches the match regex, the rule will be evaluated. The path will change to the value of replace. Any group replacements (e.g. ${1}) will be replaced with the corresponding capture (...) index from the match regex. Rules will continue to be evaluated with the new path, unless last is set to true, which stops rule evaluation with the matched rule. The regular expressions use Javascript syntax.

The default set of rules will redirect any bare directory roots to an index.html file within that directory which is often the desired behavior for websites. For example, the top-level example.com will return /index.html and /posts will return /posts/index.html.

To add a prefix to all your paths, add a rule early in execution which appends the prefix to the beginning of the path, for example /page.html to /contents/page.html:

{ "match": "^/(.*)$", "replace": "/contents/${1}" }

That’s pretty much all there is to it!

Under the hood

In order to avoid having a backend server running, all the processing is done with Lambda functions and AWS resources. State is stored in DynamoDB, content is stored in S3, and served with CloudFront’s (relatively new) multi-tenant feature, which allows multiple domains with just one distribution. The app uses the Serverless framework to manage deployments, and CloudFormation to define all the AWS assets (I much prefer Terraform but using the native functionality meant less dependencies and complication). The URL rewrite feature is implemented using CloudFront’s Functions and Key-Value Stores, which allow a small amount of state and Javascript to run in a limited environment before every request. There’s also Lambda@Edge which I might switch to later but it doesn’t play very nicely with Serverless.

Lambda functions can execute for up to 15 minutes, but API Gateway calls must complete within 6 seconds. For this reason, longer tasks are triggered by S3 events. When a deployment is created, temporary AWS credentials are generated which have only permission to write to a particular deployment file in S3. The CLI then uses these credentials to place a tarball of the site content at that location. When the object is placed, a Lambda function is invoked which downloads the tarball and clones the site’s internal git repository. It deletes any existing files, unzips the tarball in place, makes a new commit for the deployment, and pushes the repository back to S3.

On promotion, a dummy file is placed into S3 to trigger the promotion (this is dirty but simpler than other methods of invocation). It clones the repository, checks out the desired deployment, and copies the files into the directory on S3 that acts as an origin for CloudFront. The site configuration at the time of the deployment is pulled out of DynamoDB and put into CloudFront’s Key-Value Store by domain name. When a request is made, the Function looks up this configuration by requested hostname and applies any rewrite rules before handing the request to CloudFront.

An earlier design iteration kept all the past deployment tarballs and unzipped them into place to switch deployments. To make storage more efficient, the app now uses git-remote-s3, a git plugin that uses S3 as a remote git server, and commits new files on top of the old ones. This allows leveraging git to store differential versions. It also uses LFS for large binary files, although to what extent this is configured correctly I am unsure. In the future, I may change the CLI to push the new files directly to the S3 origin instead of the intermediate tarball step, which will offload some work from Lambda and reduce execution times.

Lambda functions make sense for small and/or bursty traffic because it’s billed per-call. For low traffic volumes the price-per-call is less than having a server sitting idle waiting for requests. And when traffic is unpredictable, serverless applications are able to scale instantly. But if your volume is constant and predictable, having a dedicated server running becomes cheaper. Fortunately because the Lambda handler is just a wrapper for a Node.js server, it would be trivial to run it with a server later.

Costs

S3 storage, at the least-optimized on-demand tier, is as of writing only $0.023/GB/mo. Amazon is really pushing their new flat-rate pricing for CloudFront (probably to compete with CloudFlare) but I’m pretty sure at scale it’s a bad deal. Outside of the generous free tier, serving content is $0.085/GB + $0.020/GB transfer + $0.0100 per 10K requests. There’s also a charge of $0.10/tenant/mo for the multi-tenant feature. Invalidations are $0.005/path. CF Functions are $0.10 per 1 million invocations and KVS is $0.03 per 1 million reads. DynamoDB on-demand pricing is complicated, but $0.625 per million write request units $0.125 per million read request units plus $0.25/GB/mo of storage. Lambda is $0.0000166667 for every GB-second and $0.20 per 1M requests.

If we estimate liberally that a site uses 5GB of storage, deploying 1GB of assets once a week, each requiring 100 API calls and 10m of Lambda execution time, and receives a 100K page views a month at an average of 200kb for a total of 20GB transferred:

0.023*5 + 0.085*1 + 0.020*20 + (100,000/10,000)*0.0100 + 0.10 + 4*0.005 + (0.10 + 0.03)*(100,000/1,000,000) + 4*(20*0.625 + 200*0.125)/1,000,000 + 0.25*(1/1024/1024) + 4*10*60*(256/1024)*0.0000166667 + 4*(100/1000000)*0.20 = 0.8432

…comes just shy of $1/site/month. And this is excluding the volume discounts and large free-tier benefits. The largest of this is by far the cost per-10K Cloudfront requests.

I’d like to confirm this at scale (cloud pricing is incredibly hard to estimate by design), but ideally I’d love to charge around $1 per month per site to users (probably billed annually due to transaction costs). Currently I haven’t built any user limits or billing infrastructure so if you want to test it out you can do it for free (just please don’t rack up my AWS bill thanks).

Self-hosting

It is possible to not use my deployment of this infrastructure and instead deploy it yourself, giving you your own private API and Cloudfront distribution to host your sites with. I haven’t documented this yet as I haven’t built out all the features for it, but if you’re interested you can probably read the code and figure things out. You’ll need to set up an AWS Hosted Zone for the domain you want to run on and pass in a couple of arguments via environment variables to serverless when you deploy:

HOSTED_ZONE_ID=Z093471597392O2DOAPL
BASE_DOMAIN=yourstatichost.com
# Create a Github App for authentication
GITHUB_CLIENT_ID=...
GITHUB_CLIENT_SECRET=...

# for the CLI you'll want to override the API endpoint to point to your deployment
API_HOST=https://api.yourstatichost.com

Eventually I want to make multiple authentication backends, one of which is a single root user which you could use for private deployments.

Future

The app is usable today as-is. This blog you’re reading now as well as my personal links site are both now running on chicy. But there’s still more work I have planned.

The biggest is to make a web interface as an alternative to the CLI, hosted as a single-page app on sites.chicy.online itself. If it starts to scale I’d also want to build out subscriptions and user limits so that I can manage the costs. Some additional minor features include support for custom error pages (e.g. /404.html) and the ability to squash old deployments for storage optimization.

I had previously started a similar project linky which was a Linktree-like single page site generator which would then host sites on a similar multi-tenant distribution. Momentum stalled when there wasn’t yet a Terraform interface for creating multi-tenant distributions. But I may revive that project as a simple layer on top of this.

One last thing, what’s with the name?

IDK. I was originally going for something like “static-chick” but I typo’d the first domain (static-chic.online) and didn’t want to fix it. When it came time to make a CLI I wasn’t sure what to name it so chicy came out of that. It’s short and there wasn’t already a package named that on NPM. How to pronounce it is an exercise for the reader. It could be “sheek-e” (like the French word “chic”) or “chickey” or “cheeky”. I’ll accept any of these.

JTM45 Amplifier Build

2026-02-25T00:00:00-05:00

Over the weekend I joined an amp-building class run by Daniel Klein of Port City Amps. Over 3 days we put together a JTM45 kit. This is an early Marshall amp which was based heavily on a Fender Bassman. We were given some high-quality parts over-spec’d to get a really premium, low noise version of the amp.

Daniel gave us a crash course in tube amplifier design and general analog electronics. He got some of the EE theory wrong here and there but the practical intro to amplifier design was seriously invaluable. I’d never worked with tubes before and frankly was afraid of them but I discovered that the theory is overwhelmingly similar to transistor amplifiers.

I won’t go too much into detail about the design of the amplifier since it’s his secret sauce (you should take his class) and I’m going to link a different schematic that doesn’t have his annotations (which I also think happens to be easier to read).

Here’s the high-level. There’s a 12AX7 (a dual tube) pre-amp buffer stage, with one side for the bright inputs and one for the normal inputs. The only difference between the two channels is the bright capacitor on the bright channel. They are mixed down into another 12AX7, the first stage being a voltage amplifier and the second being a common cathode amp, which just like a common-collector BJT amplifier has a voltage gain of 1 but a high current gain. This high current output allows it to drive a passive tone stage. From here it goes into one side of a differential amplifier implemented using a 12AX7 as a long-tail pair, making this a Class B push-pull design. Finally there’s the power amplifier stage, with two matched-pair EL34 tubes cranking out a high-voltage output. This is stepped down to a low-voltage, high-current output to drive low-impedance speakers using an output transformer.

I was surprised to learn how much of the heavy lifting is done by the power transformer. While I understand the theory of AC-DC converters, I’ve never actually built one. I’m generally scared of mains power, and since much of what I do is +/- 12V DC or less, I usually just buy a wall wort or use my bench power supply and avoid AC power all-together. But with tubes that’s not really an option. The tubes need plate voltages in the range of 200-400V DC, plus a low-voltage AC line for the heaters.

I asked about why in modern days we use an expensive, inefficient, and fragile tube rectifier (in this case a GZ34) instead of a $1 diode rectifier. The answer seems to be that people think it adds warmth. I’m a little skeptical how much the power supply line actually effects the guitar tone; this may be a case of gear-heads insisting that tubes are always superior. Mesa/Boogie makes a dual-rectifier amp with both types and a switch between the two. I’ve never tried it but supposedly there’s a subtle difference.

The amp sounds great. I’m more of a clean amp girlie; my gigging amp is a Rivera Suprema 55 which I keep on the clean channel, with both the bright boost and mid notch enabled. So I was interested in getting high-gain amp to kinda explore that world, and see if I could get some good heavier sounds for when I need a wall of sound with Serious Times.

Unfortunately the JTM45 is so loud. I’ve been keeping the volume around 1.2 out of 10, which has a lovely warm clean tone, but to get the real tone this amp is known for I gotta crank it to where I’m bothering the neighbors and have to put my earplugs in, making it harder to decern the tone.

Master Volume

A very popular mod for this amp is to add a master volume. I would have naively done this either by putting a volume knob before or after the power stage, or by adding wire-wound resistors in series with the speaker. But the common (and much better approach) is to convert the 220Kohm grid leak resistors between the phase inverter and the power amps into pots. This keeps the gain and tone from the pre-amp stages, and at full volume behaves transparently.

In order to do this, I swapped the separate 1M audio taper volume pots into a stacked 1M audio taper, and used the other slot in the faceplate for the new master volume (a stacked 250K audio taper pot). This way the knobs become essentially “gain” and “volume” as found on more modern amps.

The output to the grids of the power transformers are on the center pin, and the signals in from the phase inverter are on the left if the knob is facing you. The -50V supply rail is on the right, and there are some 1Mohm resistors from the supply to the grids, likely for stability or grid leak when the pot goes to zero.

Load

It’s really bad for the output transformer to run the amp without a load, but I’ve already once forgotten to plug the speaker in before powering it on. So I added a default load if the speaker is unplugged. I used the plug-detect on the output jacks to send the output through a big wire-wound resistor in case I forget. This is also useful for hacking on the amplifier, as I can power up the amplifier on the bench.

I screwed a 100W 8ohm wire-wound resistor to the side of the case, through some holes which were already drilled, probably for a can-cap.

Faceplate

The gold plexi faceplate is an iconic aesthetic of a Marshall amplifier. The text reads correctly when put into an amp head enclosure, but in a combo amp the text ends up reading towards the back of the amp instead of the front. Unfortunately, despite the large number of kits for this amp available online, and some discussion in forums of some folks making custom one-offs, no one seems to sell a flipped version of the panel.

This, combined with the fact that my mods are going to change the behavior of some of the pots, makes me think this is as good a time as any to get the laser cutter in the makerspace up and running and make my own panel. I ordered some acrylic. I’ve laser etched text in acrylic before but I wasn’t sure how printed text is done. I watched this video and it turns out it’s incredibly straightforward. Basically you etch with the protective sticker film still on, apply a layer of lacquer, and then use a paint pen to fill in the etch. Hopefully I can try this out soon.

Other Mods

I’ve got a list of more mods I want to do. One is significant enough to merit it’s own post, but I’ve already ordered the parts for it, so expect an update soon.

Here are some of the smaller mods I’ve been considering:

A switch for a solid-state rectifier. This is useful in case the rectifier tube blows; I could switch to a solid-state one and still play a show. It would also allow me to test how important the tube rectifier is to the tone as discussed above.
Effects send/return. This one isn’t super high priority since I’ve never bothered to actually use one. Rather than add more tubes for buffers I’d probably just add a TLO72. I might be able to drive them from an AC-DC converter off the heater line. This would probably be below the minimum power supply requirements for the opamp from the datasheet but I’ve run the TL072 off a 9V battery before without issue so it would probably work. Definitely a lower-priority mod; I lot of work for little return.
A switch to invert the phase, in case you’re using two amps and they are out of phase

2025 Retrospective

2026-01-01T00:00:00-05:00

It’s been a while since I wrote a blog post, and a lot has happened this past year. I figured I’d dump it all in one post.

Kelly Romo

In April I joined Kelly Romo as her full-time bass player. I played six shows with her this year, culminating in a headlining show at The Masquerade in December. With Kelly I played my first “real” gig at a ticketed, paid venue. I was so nervous before the show but as soon as I got on stage it all went away and I had such a great time.

She’s working on a new album, and a couple singles from it have dropped already. My favorite of the songs so far, Back Again, has a music video.

Punter

Unfortunately, the Stolen Velour project kind of fizzled out. We played two house shows and put out two songs, but I wanted to take my original songs in a different direction and ultimately it got difficult to get everyone in the same room.

I took some of my unreleased songs that SV had been performing and a few new ones, and have been performing under the name Punter. I played my first show in July. It was a little rough; my vocals were inconsistent and the friends I asked to join me only got 3 rehearsals to prepare. But I think the vision comes through.

In November I booked a show of transgender folk artists at Waller’s. It gave myself and my friends, all of whom had performed only a handful of times prior, a safe and welcoming place to perform. It was the first time I felt content with my performance of my solo material. And the turnout was better than I expected for a bunch of still-establishing acts.

Watershed

My (now ex-) boyfriend has this brick shed in his backyard in Reynoldstown. When he first moved in, he said it would be a cool place to do shows. Looking at the dark, dusty room I was skeptical. But after pressure-washing the walls, sealing the leaks in the roof, re-running power, and adding proper lighting I had a change of heart.

In May for my 34th birthday I co-booked the Shedwarming show. I opened with both Chris Kraemer and Stolen Velour, before a stacked lineup of really talented artists.

It was an absolutely magical night, my best birthday party ever. That show ended up launching what would be called Watershed (a triple-entendre; starting as literally a shed full of water, a play on the name of Atlanta’s city water department, and a turning point for house venues).

The folks who lived at the house (all musicians) continued to organize shows there, the second being later that month.

Janie Danger organized a July 4th show there, featuring performances from herself and several transgender bands and a cookout of hotdogs of her own recipe. Girls brought bathing suits and hung out in the kiddie pools, or alternated between the mosh pit and the slip-in-slide. Over 100 people showed up and we raised over $1000 for Palestine relief. Lucca of Split Silk told me it was her favorite show she ever played. It was truly an unforgettable night and I am so proud to have been a part of it.

View this post on Instagram

A post on Instagram

From there word about the venue spread fast and soon out-of-town bands were booking shows. We got a reputation for being quite well-organized for a house show. All ticket sales went to bands, as all the organization was volunteer. Local house show veteran Austin of Dead Echo volunteered to run sound at many of the shows, hauling gear multiple times a week. Bee of Just Rats organized a week of events to celebrate her going-away.

Some incredible bands played that stage throughout summer and fall, including Janie, Tongues of Fire, T.E.N.T., Hillview #73, and Split Silk. Several bands played their first shows, including Baby Bites, Code Unknown, Sweet Arcana, and myself.

Unfortunately there were several noise complaints culminating in a cease-and-desist. Combined with the cold weather, burnout, and the busy lives of the organizers, the venue has gone dormant these past few months. But I’m hoping come spring we kick things off again.

Rock Camp compilation album

I organized and mastered a compilation album to raise money for Y’all Rock Camp ATL. A number of past and present volunteers contributed some incredible songs. There’s even one of mine on there. We dropped the album on Bandcamp Friday and raised nearly $250 dollars for camp.

Causerie: A Compilation Benefiting YRC ATL by Various Artists

I learned quite a lot about mastering by doing a comp album where all the songs are from different styles and artists and recorded under different circumstances. It really put into perspective what mastering is and why you do it. I also think I did an okay job with the album art considering I’m not much of a visual artist.

The Coven makerspace

I co-founded a small, private makerspace called The Coven. We provide space for transgender makers in Atlanta, specializing in equipment and expertise in the areas of analog audio/video and cosplay. We have 3D and resin printers, a laser cutter, drill press, sewing and embroidery machines, a analog/digital media workstation, and two robust electronics workstations.

Eventually we will be formally opening the space for members but with our busy lives it’s taken time to get everything set up. But getting all my electronics equipment out of my house has allowed me to convert my office entirely to music and get my living room back. I’ve been spending quite a lot of time working in the space, and have worked on some smaller projects which I might cover in future posts.

Serious Times

My girlfriend Madison and I started writing together, and have a couple of demos that I’m pretty proud of. We’re going to try and put together an album or EP this year, and potentially try playing these songs live. We got a mess of looper pedals with which we’re hoping to combine with some backing tracks to get pretty thorough recreations of these songs with only the two of us.

Other stuff

I built a lot of infrastructure to manage accounting for Fourth Strike and am in the process of taking over administrative responsibilities for the organization. I took part in the Fourth Strike Song Jam, which I’ll post about eventually. I took vocal lessons with a classical vocalist, culminating in a performance in a chapel which without a doubt had the best acoustics of anywhere I will likely ever perform. I volunteered again with Rock Camp, becoming the de-facto stage manager for the showcase in Heaven at the Masquerade. And I became an aunt!

Looking back

I feel like this is the year I became fit to call myself a musician. In the past I played instruments, recorded music, and even had performed some open mics and talent shows. But this was the year that it went from a hobby to a full time focus.

The year started with a house show that I booked and played two sets in, the first house show I ever played, and culminated in headlining Purgatory at the Masquerade. I have seen over two dozen shows at the new Purgatory over the last 10+ years. It’s probably my favorite room in town and every time I looked up at that stage I imagined myself on it. To finally get to be up there was magical.

I’ve grown as a musician both qualitatively and quantitatively. My vocals show a stark improvement just from the summer to now, putting aside how much better they were this summer than only a few years ago. My stage presence has also improved noticeably and I’m more confident talking between songs.

I track all the shows I’ve played and the numbers have exploded this year. From 6 shows, all open mic or talent show-type gigs, over the previous 15 years to 20 sets across 15 shows in 2025. 7 of those were my own original material. I booked 5 shows and volunteered or organized 7 more. I worked on 36 songs, wither that was performing, recording, producing, mixing, or mastering. I wrote 7 songs, two of which have released publicly, with the rest hopefully coming soon.

Looking forward

Speaking of which, my goals for 2026 are to continue growing as a musician. Specifically:

put out an album of original music
play a tour or at least a show outside of Atlanta

The first one will likely be Punter and/or Serious Times. Serious Times is further along, with those songs just needing some final polish and mastering.

The second one is more likely with Kelly, who has expressed similar goals for the year. Maybe some bigger band will take us on a regional tour.

Ultimately I hope I can look back on 2026 with a similar list of accomplishments.

Introducing Stolen Velour

2025-03-13T00:00:00-04:00

I’ve started a new band! We’re called Stolen Velour (a play on “stolen valor”, “velour” is velvet fabric). My friend Emmaline and I are “co-fronting” the band. We each bring our own songs and sing lead on them, with the other playing lead guitar. This dynamic might not work forever but for now it’s suiting us pretty well. My songs tend to be sad girl slowcore/emo songs while hers tend to be a blend of indie rock and shoegaze. We’re calling it “grungy lovewave.”

Our first demo is one of mine, it’s called “floorboards” and you can hear it below. Our next one is one of Em’s and it’s very catchy. We plan to drop an EP this spring.

floorboards by Stolen Velour

I’ve also created a record label to put out music, Killian Street Records. I had planned on doing this for a while, assigning Chris Kraemer’s first EP which I produced as KST-001 and my solo folk EP as KST-002. The label will let me use one account for DistroKid and manage multiple Bandcamp and Spotify pages.

I’ve been working on some new material for Serious Times also, which should be out later this year.

All in all, lots of music happening. I’m so happy I get to do this~

External Guitar Power Supply

2024-11-26T00:00:00-05:00

I have this 1998 made in Japan Squier Vista “Jagmaster” (a cross between a Jaguar and a Jazzmaster).

I use it as my project guitar, but because it’s kind of rare I’m careful that any changes I make to it could be reverted to the condition I got it in. So I got a second pickguard and left the original electronics attached to the original pickguard. I nicknamed “The Woman In the Red Dress”.

I had won some JH EMGs at a charity auction, and I wasn’t completely happy with the P-Rails I had in there previously, so I decided to put them in the Jagmaster. I did have to carefully file the pickup cutouts larger to fit these beefy boys.

When I first but put them in I thought they could be active or passive, but the output was way too low; the pickups really do need to be powered. What to do then? I definitely didn’t want to cut a battery box into this guitar. So I came up with an idea to have a battery box external to the guitar.

It’s just a box with two stereo jacks and a 9V inside. The positive 9V line runs along the ring of the cable into the guitar where it is attached to the pickups. The negative of the battery is connected to the ring of the output jack, which means the battery is connected through the sleeve to ground when a cable is connected (this is a common technique inside active guitars, we just moved it to the box instead).

Then I run a short stereo cable from the guitar to the box and tie it down to my guitar strap. I can just leave it attached all the time.

Albums of the Year - 2023

2023-12-18T00:00:00-05:00

2023 has been a great year in my personal life, and an appropriately strong year for music as well. This year I converted my spreadsheet of new releases into a Notion app, along with a script to import releases from Bandcamp and Spotify. Even still I’ve been hit or miss with keeping up with new releases. Here are my favorites of the year. This is just the top 10, I decided not to include records that I liked but didn’t listen to enough to stand strongly by.

Also check out my list from last year.

Fourth Strike Song Jam 2023

2023-12-10T00:00:00-05:00

Every year, Fourth Strike (the record label/not-for-profit music collective behind the garages) does a little community song jam. Participants are encouraged to make one song a week for for weeks. Each week has a theme and some optional additional challenges. It’s a fun, low-pressure environment to practice making music.

🎤Announcing the FOURTH STRIKE SONG JAM 2023! 🎵

The Song Jam is a non-competitive space to complete challenges in songwriting & producing

FOUR CHALLENGES, FOUR SONGS, FOUR WEEKS (+a break week)

Running Nov 6 - Dec 10

Join on our discord server: https://t.co/bve5GExAzN pic.twitter.com/OSmnFuzFzM
— Fourth Strike (@StrikeFourth) October 19, 2023

This year I did my best to participate consistently. I’ve put the results on Bandcamp, with the more post-rock songs under Serious Times and the more twee emo songs under a new name, Slow Church.

Update 2024-03-01: The song jam album dropped today. You should listen, there are some real gems on here. Particular standouts are PINK HEART, polaris, Prelude to Avunculicide, and Pure Propaganda.

Fourth Strike Song Jam 2023 by Fourth Strike Records

Week 1: PHONE IT IN

CHALLENGE: The title of your song must be the name of one of your “most used” emojis

BONUS CHALLENGE: Oops, your phone is running out of storage. Better make the song shorter than 1m 59s!

For week one, I made a little spoken word piece which was what I’d wished I’d said during some mildly transphobic incident (basically a shower argument), and placed it over some sloppy twangly midwest emo guitar. Trying to to channel american poetry club or something like that. It’s cringingly twee.

Bikeway Narrows by Slow Church

Week 2: ART HEIST

CHALLENGE: For this week’s inspiration, we’re breaking into a modern art gallery. Your inspiration are the two pieces of artwork below - feel free to take inspiration from one or both of them however you like!

Yellow, Cherry, Orange by Mark Rothko

Surrealism on TV by Robert Heinecken

BONUS CHALLENGE: Good artists copy, great artists steal. Incorporate one or both of the “Bonus Sample” audio files in the folder below - you may edit, sample, or rerecord them however you like

The Rothko gave me vibes like the sun, oppressive brightness and heat in a desert or something like that. Meanwhile the second sample is like a de-tuned drone in the Bb/C range that felt similarly oppressive. I decided to detune the instruments to match that sample pitch, and write a chord progression that utilized that pitch somewhere in each chord, keeping it droning through the song.

I added some processing to the sample, and layered in a heavily processed sound of bowing a de-tuned low B on a 5 string electric bass. Added some high-reverb guitar feedback, some Ah’s (thanks to my partner Kathryn for singing), synth strings, and some sax (both a pretty doubling of the string part and some frantic squealy playing and mouthpiece noises, thanks to my friend John). Another friend really outdid themselves with this insane drum part, which really caries an otherwise kinda boring track.

If Fashion is Suffering Then I'm a Narcissist by Serious Times

Week 3: I LIKE YOUR STYLE

CHALLENGE: Hey new friend! Collaborate with someone you haven’t collaborated before.

BONUS CHALLENGE: Make your song “in the style of” another artist - a fellow jammer, a famous musical act, your mate jim, anyone!

I had plans to do a song in the style of Modest Mouse with a friend. We had some good ideas but were both too busy and nothing ever materialized. But I did finally finish a song I had written back in June, unreleated to the song jam.

Week 4: GOLDEN RECORD

CHALLENGE: There is an alien, somewhere. Or an astronaut, waiting. You’ll never reach them, but you could broadcast. We’re making a new “Golden Record” What are you going to send?

BONUS CHALLENGE: The space boffins really don’t want to waste any space on this disc - make it as long as you like, but make it count! Your song must be through-composed (no repeated sections).

BONUS BONUS CHALLENGE: Humanity is at its best when we work together: have each track or instrument be performed/recorded/made by a separate person.

I ended up not doing a song of my own for this week either, but I did contribute a string bass part to someone else’s song:

Fourth Strike Song Jam 2023 by Fourth Strike Records

Goodbye, FIXD

2023-08-04T00:00:00-04:00

As I enter the last week at FIXD Automotive, a company I co-founded and built over 8 years, I’m reflecting on the journey; the successes, the learnings (and there were many!), and the friendships.

But I think of all the things I accomplished there, the thing I’m most proud of is the engineering team I built and the culture around it. In the early days when we had no cash, we had to be scrappy. We couldn’t recruit people with a high amount of existing experience because we simply couldn’t afford them. Instead, we recruited high-potential people early in their careers and grew them internally. And around that team we had a culture I would describe as always willing to help and step up, hungry to learn new things, and thoughtful about balancing scrappiness and reliability in an optimal way. Our team is small and very fast, despite covering a pretty insane spread of products and technologies.

Below I’ve shared what I think is a summation of that culture. It’s our shared Philosophy from our Engineering Handbook (a concept borrowed from GitLab). It covers our (mostly my) opinions on how to build software, much of which was learned the hard way along the way. I’m sharing it here because these are the default set values I plan to bring to future software teams in the future. I realize some of this is specific to FIXD’s size, tech stack, market, etc. and YMMV depending on context, but I think this is a healthy starting point for any software team of any size. Note that in some places it links out to other parts of the handbook, which are internal, so I’ve formatted those as links but removed the hyperlink itself. It’s also been gently edited to remove some references that were too company-specific.

It’s a bittersweet feeling to be moving on, but I’m excited to focus my attention on new problems. I’m leaving to pursue music technology full-time. Exactly what that will look like I’m unsure of right now, but I can’t wait to dive in.

Engineering Philosophy

This is a collection of values and opinions that you’ll find on the engineering team. They certainly aren’t rules; every decision depends on context and even if you generally agree with these you can find a situation where the opposite would be the better choice. But these can act as guidelines to aid in technical decision-making.

People and Processes

Move fast and break things (in B2C)

Many technical decisions come down to weighing velocity against quality. Moving fast increases the probability of a quality issue that degrades or breaks the customer experience. In B2B, losing a big client can make-or-break a business. But in B2C, we get 100s of new customers every day. Some of them are likely to have a bad experience, and often its easier to give them a refund and move on; there will be more to replace them. If we embrace this, as long as changes can be isolated to a limited number of customers, we can prioritize velocity over quality in many situations and iterate faster.

One-way vs two-way doors

Many decisions at a company are trivially reversible. The risk of trying new things is low because if it doesn’t work out as expected, the decision can be reversed and the changed rolled back to the previous state. A/B tests of new features rely on two-way doors like this. Some decisions are one-way doors; once a decision is made, rolling back is difficult or impossible. Be quick to pass through two-way doors, and slow to pass through one-way doors.

50-20-20-10

When broken down into categories, a healthy Engineering should be spending:

50% of their time on net new features and capabilities
20% on keeping the lights on (scaling systems, incident response, updating dependencies, etc)
20% on quality improvements
10% on internal tooling

Strive to be T-shaped

Skills and knowledge can grow in two dimensions: depth and breadth. T-shaped people have basic skills in many areas and very deep knowledge in one area. Never stop learning and stretching out in both directions, but don’t try to be square-shaped (deep in everything), you’ll stretch yourself thin.

Run a data team like a product team

Code reviews

Code review is a valuable step in the software development process as both a quality check and an opportunity for domain knowledge transfer. There’s some basic etiquette to follow on both sides of a code review.

Code reviews are not enforced at FIXD, only recommended. Ultimately you alone are responsible for the correctness of your code. You should ask for a code review because you desire an outside perspective, not because you feel obligated to do so. You don’t need to ask for a code review for a small or inconsequential change.

Accept contributions from anyone at the company

Anyone should be able to submit merge requests, even if they typically work on another project, or perhaps don’t even work in engineering at all. There shouldn’t be an expectation of this, but the option should be available. Sometimes it’s easier to just change an upstream dependency yourself rather than wait for another team to have the bandwidth to make the change for you. In these situations, be sure to get a code review from someone who is more familiar with the particular codebase in case you missed something.

Tools

Enforce styles, not tools

Engineers have strong opinions on tools. Give them the freedom to use whatever environment they like as long as the code they contribute matches that of their peers. If the code works and matches the style, it doesn’t matter if that code was written in a fancy JetBrains IDE on Windows or vim on a Raspberry Pi. See Developer Machines.

Follow the conventions of the language/framework’s community and culture. Use aggressive automated formatting and linting, combined with code review, to enforce code style. Style conventions that are consistent throughout makes the code approachable for other team members. Avoid bike-shedding about things like spaces vs tabs; instead pick a popular, opinionated set of rules from the community (e.g. standard) and don’t waste time over-configuring it.

The right tool

Not everything is a nail, so don’t always use a hammer. Use the right tool for the job, even if it means introducing a new technology to the stack. The “right” tool depends on context; we should pick boring tools when they aren’t core to the business and avoid chasing the shiny new technology if it’s not the right tool. But we shouldn’t force a familiar technology onto a problem it’s not good at solving.

Example

Data engineers know and love Python, it’s where all the tools are. If you make them use Ruby, you’ll have a harder time recruiting, spend more time training, and have to re-invent the wheel often.

Example

The industry is moving towards Flutter and away from native for mobile development, and if we don’t keep up it will become very hard to recruit for and maintain.

Aside

Some companies, notably Mailchimp, have the opposite culture value: they build everything in their chosen framework (PHP) even when a better tool exists in order to keep the stack consistent. This allows developers mobility within the organization, but creates a worse developer experience. Here at FIXD we prefer to use the right tool for the job. We can train people in a new stack if needed.

Embrace open-source

The entire technology space has flourished on the back of a community that is willing to share advancements and build on each other’s work. The right tool still applies, but given the choice between otherwise equal tools, one closed source and one open, prefer the open one. Open-source is inherently cheaper, contains fewer security issues, and can be adapted if a key feature is missing. But be wary of small projects with minimal support; there’s a higher risk of the project being abandoned, and FIXD having to maintain a version until an alternative can be migrated to.

Using open-source software isn’t completely free. It comes with a responsibility to contribute back to the community. This could be in the form of pull requests for bug fixes, new features, or improved documentation, or by publishing tools FIXD has created for others to use. Open-sourcing code has benefits beyond just supporting the community. It can be used to build visibility and trust for recruiting, for example. Any code that has been sufficiently abstracted to be free of FIXD-specific content and generally useful to others is open for consideration to be open sourced. Note that by open-sourcing something we are implicitly committing to maintaining the project and engaging with the community, even if we stop using the code ourselves, so the decision should not be taken too lightly. See open source licensing for directions.

Debugging > logging

A behavior that clearly separates junior developers from mid/senior developers is the way they debug code that isn’t working. Junior developers have a tendency to rely on logging, spreading print statements throughout the code and reviewing the logs to try and find where the behavior deviates from expectations. More experienced engineers use the debugger to step through the code in question, checking state against assumptions at each line. This drastically speeds up the debugging process. If you can do this in a unit test rather than in the running application, the cycle time is even further improved.

Example

debugger/byebug in Ruby, import pdb; pdb.set_trace() in Python, breakpoints in Android/iOS, etc.

Logging is a useful tool for debugging in production, but excessive logging can be expensive and too much logging can create so much noise that it’s hard to narrow down the specific issue. Keep application logging to a minimum, logging only key low-level interactions (e.g. incoming request/responses and background job execution on backends; new screens, high level user behavior, and network requests on front-ends) and errors. Avoid stacktraces for expected errors, only print stacktraces when an error violates defined behavior.

Debugging in production

Ideally all bugs can be recreated and resolved in development environments. But not infrequently, a bug will arise that seems to only apply to production. As much as we try to keep staging environments consistent in setup with production, issues specific to an environment will always arise. In those situations, being able to debug in production is an incredibly powerful tool. An example of this is being able to open a Rails console in a production container and run arbitrary commands. This allows you to manually call specific parts of the code (even private methods thanks to Ruby’s flexible meta-programming) to find what the issue is.

Danger

With this power comes great responsibility, as a mistake in this environment can have catastrophic effects (nothing is protecting you from wiping the production database, notifying actual customers, or charging real credit cards!). Thus this power should be used judiciously and only granted to engineers who are comfortable and familiar enough with the environment to operate safely. But the ability to respond to a bug in minutes rather than days is worth the risk.

Note

This power is a strong case for using an imperative language (such as Ruby) for backends rather than a compiled language.

Code

Code for developers first, machines second

Code is a unique form of rhetoric in that it has dual audiences, machines and developers. Machines have strict grammar rules and will interpret writing literally. For machines, code can be objectively correct or incorrect. Developers are more forgiving about grammar but speak machine as a second language, and therefore can struggle to comprehend complex ideas. Correctness for developers is subjective rather than objective. Good code clearly communicates the same idea to both audiences succinctly.

Readability over performance

Engineering hours are 1+ orders of magnitude more expensive than infrastructure. That means code that is highly performant but hard to understand will net out more expensive most of the time, because the cost of maintaining it is more expensive. For example, Ruby is not a particularly performant language, 100s or 1000s of times slower than C. But it’s much easier to write, maintain, and recruit for. Throwing a bit more compute to compensate for performance is much cheaper than writing code in C and allows us to move much faster.

Similarly, code that deals with every possible edge case in a graceful way is more code to maintain. While we should avoid too much undefined behavior, falling back to simple, defined, but sub-optimal behavior for obscure edge cases is easier to maintain and reason about.

Declarative over imperative

For the purposes of this paragraph, “imperative” code is code that does things (loops, branches, IO), while “declarative” code describes the intended behavior. Put another way, declarative code is that “what” and imperative is the “how”. Separating these two things generally leads to code that is easier to read and to change. Make the business rules declarative, and encapsulate the imperative part away from the business logic.

Examples

Config files define data or behavior, and application code reads those config files during execution

Domain-specific languages, an idea popular in Ruby and Kotlin, such as serializers

Bundling a collection of infrastructure into a Terraform module, and calling it with configuration arguments for the specific environment

If logic needs to change on-the-fly without a code change and deployment, that logic can be stored in a database. For example, instead of hard-coding every checkout funnel, we can create a Funnel model that describes the behavior, and these models can be created at runtime. But doing so looses the benefits of version control (consistent environments, automated testing of behavior, searchability and readability, code review, etc). Consider if in place of a database model and a UI that allows editing behavior, if instead updating a config file and redeploying code is really that much slower to change.

Tip

If the declarative code is simple enough, you may even be able to train non-engineers how to make the change without involving engineering. Editing of config files and opening MRs can be done entirely within the GitLab UI.

No code is sacred

The codebase is a description of the company, and the company is always changing. Things that were once true are not true any longer. While we should always strive to make code extensible, sometimes it’s easier to simply rewrite a feature than to learn the existing code well enough to understand how to change it. If the new code better describes the behavior in a way that’s at least as maintainable, then it doesn’t matter who wrote the original code or how old or new it is, the new code is better.

Tip

When doing this, test against the old implementation’s unit tests as much as possible so that you know the new implementation is consistent.

Documentation is code, code is documentation

Documentation is a form of code that does not have machine checks like tests or type checking, making it harder to maintain than code handled by a compiler or interpreter. Ideally, code should be straightforward enough to follow without paragraphs of explanation. But because we have to write for two audiences, sometimes code alone is not enough to communicate an idea to developers. In this case, it’s best to put documentation as close to the code as possible (i.e. comments, README, etc). Documentation that is separate from the code (e.g. a wiki), while more searchable, won’t be maintained and will soon become incorrect. Incorrect documentation is worse than no documentation at all.

De-normalize on insert

Note

This applies to backend apps that wrap a relational database, particularly those written in Rails. YMMV for other contexts.

While backend apps are typically just wrappers around a database, they have some complex logic associated with them, often transforming data read in into a different form on the way out. There are generally two approaches to putting the business logic: before writing to the database or after reading from the database.

Follow a fully normalized database schema design, inserting data into logically organized schemas with no duplication, and then write complex queries/read logic to re-organize that data in a form ready for consumption by the UI.
Pros: guarantees about data consistency
Cons: query logic can be slow, and often hard to maintain
Write complex insert logic that stores data both in its raw form and in a format optimized for querying, and then perform simple queries to return data to the UI.
Pros: Easy to read later, easier to maintain write logic
Cons: possibility for data inconsistency

Note

Our backend code challenge for engineering candidates is designed around this idea, although we don’t necessarily look for people that solve the problem the same way we do, but rather if they can think thoughtfully about these trade-offs.

We generally prefer the latter solution, which we often refer to as “de-normalization”. For queries beyond selects with a few filters and maybe a few joins, an ORM will break down. And hand-crafting optimized SQL is a maintenance problem; it sucks to write in the moment, it sucks to change later, and it doesn’t leverage any language features like type checking or encapsulation. If you structure your data around how it will be consumed at write time, your logic can be less efficient because it only operates on one unit of data at a time rather than possibly the whole set, and results only need to be computed once at creation time rather than repeated on every read request, and you can still leverage the ORM.

The trade-off there is we give up data consistency. There is now more than one way to answer the same question. If the insert logic is flawed, different ways could produce different answers. This terrifies DBAs, but developers are usually okay with this. To avoid data consistency problems, we typically store the raw source data as well as it’s structured representation, so that in the case when a mistake is made or a feature needs to change, the structured version can be re-generated. We also rely on aggressive automated testing and database constraints where applicable to avoid these mistakes.

If you didn’t measure it, it’s not a performance improvement

It can be tempting to start optimizing a bit of code for performance when you see an opportunity or suspect a performance problem. Before you dive immediately into optimization mode, measure the performance first, and then compare it to the performance of your more optimized solution. You’ll likely encounter cases where what you thought was an issue wasn’t actually a big deal. Maybe you gained a small improvement but it wasn’t worth the added complexity in maintaining the more optimized solution. Maybe your new solution is actually slower (this happens more often than you think).

Also by measuring you can then brag about it in your merge request.

One-way data flow UIs

In the mid 2010s, React revolutionized the frontend JS world by designing a framework around the concept of one-way data flows. In this paradigm, UIs are declarative, and pass data down to their children. When data changes, the UIs are simply re-rendered with the new state. Events (typically user interaction) bubble back up or trigger background operations which eventually change the state at a higher level, causing new data to flow downwards and re-render the UI.

React is not the only JS framework that uses this paradigm, and the design pattern can be implemented in any language/framework. Following this pattern makes it easy to reason about and change.

Example

In frontend JS, we often use Svelte for this. On mobile, SwiftUI, Jetpack Compose, and Flutter all work this way. This pattern can also be implemented manually in any language.

Meaning of `TODO`

The term “TODO” implies a task that should be done later. With this understanding, you can make a good case for not leaving TODO comments in the code; stuff that is to-do should be put on a task backlog instead.

However, historically in FIXD codebases, TODO does not mean tasks to-do. Instead, it acts as signpost to other developers about the existence of technical debt. It might indicate a known shortcut or workaround, an extension point for a possible subsequent iteration, or

It doesn’t mean that a change needs to be made now or in the future, just that a change might be necessary in the future depending on how the company or application evolves.

“Why use TODO if it doesn’t mean to-do?”

Good question. TODO is an accepted and well-known comment pattern, and many IDEs and tools support special highlighting or other features for them. A count of TODO comments as a percentage of LOC could be used as an approximation for the level of technical debt in a codebase.

Versioning with mobile apps

In web environments, typically your backend and frontend deploy together, so there’s no need to maintain backwards compatibility of endpoints used by the frontend. Public APIs used by third parties need versioning, but deprecation cycles can pressure users to migrate to the latest version.

Mobile apps are unfortunately a different breed. End users are terrible about keeping apps up-to-date. Even if the app store pressures users to have auto-update on, users can still turn it off manually, and there’s a transition period during rollout where multiple versions can be simultaneously active, as updates often happen during times of low usage, such as night charging. A long tail of users will stay on incredibly old versions indefinitely, and even if you drop support for these users, you need a solution for phased rollouts.

FIXD deals with these problems using a couple of conventions:

The backend reserves the right to add additional properties to models and additional optional arguments at any time. Clients should not break if a new property is added. This allows backward-compatible changes without versioning when additional functionality is added (the most common case).
In situations where backwards-incompatible changes need to be made, we use a migration pattern to support older implementations while encapsulating and hiding these older implementations from the main codebase. These change files operate as middleware, transforming old input data into new input format and new output data into old output format. Really old versions can travel through multiple layers of transforms. This adds a slight performance overhead to clients on an old version but makes it trivial to support old versions indefinitely if necessary.

Tip

At FIXD we use per-endpoint versioning using dates in headers. Path-based versioning is difficult to maintain. Per-endpoint versioning means that not every endpoint has to update for one change. It also means that clients can selectively opt-in to some changes but not others, which provides some flexibility. And using dates rather than integers adds useful contextual information about the change to developers.

Make the change easy, then make the easy change

for each desired change, make the change easy (warning: this may be hard), then make the easy change
— Kent Beck 🌻 (@KentBeck) September 25, 2012

Often a behavior change requires rethinking the underlying architecture which didn’t account for this new use-case. In those cases, refactor the underlying architecture first, then implement the simple change under the new architecture. This can be harder but will scale the architecture along with the application, and will create less bugs.

Testing and Quality

Automated, not manual, QA

Manual QA is much more expensive than automated QA. While manual QA is more flexible to system changes, it doesn’t necessarily catch more issues. Running quality checks earlier in the product lifecycle (“shift left”) allows teams to move faster, but often manual QA has to be farther to the right, slowing down development. Most importantly, a manual QA team that is separate from the development team creates a toxic culture, where developers are tempted to throw bad code over the wall and let the QA team deal with it, and QA feels that the developers are incompetent. Automated checks are harder to write at first and can be hard to maintain, but create better outcomes and find issues faster. The more testable your application is, the less it tends to need tests.

Follow the testing pyramid

Small, isolated tests are easy to write and maintain, but they don’t tell you that the whole system works together correctly. As tests grow cover more surface area or realistic cases, they become increasingly more brittle, eventually reaching a point where the cost of maintaining them outweighs their value.

The testing pyramid is a way to conceptualize striking a balance between these. It indicates that you should have many small, focused, isolated tests (such as unit tests), some medium-complexity tests (e.g. integration), and a few complex tests (e.g. UI).

On the backend for example, you might add a dozen unit tests that cover all the functionality for your core service class, 2-3 controller/request tests of the happy path and any expected edge paths, and a single Cypress test that renders the content in the browser and tests the happy path end-to-end.

UI tests can be really difficult to maintain, but the confidence that good UI tests give you to push code to production quickly is worth the headache. If a UI test breaks, fix it early; it’s the strongest defense you have against a catastrophic mistake.

Recreate the bug in a test before you fix it

Find a bug? Before you fix it, take the time to write a test to recreate it. It shouldn’t take you very long, proves you understand the bug, and most importantly keeps it from popping up again later.

You can also write the test after the fix, then temporarily revert the fix to make sure the test fails without it.

Not all bugs can be easily reproduced in a unit test, but I highly recommend doing this for the ones you can.

Tests are documentation

Most (if not all) unit test frameworks in FIXD projects are of the “spec” variety; tests are written using describe/context/it. This pattern allows your tests to function as documentation of the behavior under test, saving you from having to explain behavior in comments and ensuring that the “documentation” is always in sync with the code (since the tests will fail otherwise).

When writing unit tests, think about all the cases worth covering, and then write tests as if you’re describing their behavior to your peers. Behaviors that are not supported by a class can be just as important as behavior that is, so you might write a test for that also.

Infrastructure

Monolith over microservices (at our scale)

For a small team, a monolith is much easier to deploy and reason about. Microservices make systems complicated and create a network of fragile dependencies. The benefit of microservices is that it clearly delineates code boundaries, so that teams can remain small and focused. This is necessary and worthwhile only at the scale of massive teams with 100s of developers. Resist microservices for as long as is reasonable. Use modular monolith practices until the team reaches the size that microservices is inevitable.

Infrastructure as code

Tools like Terraform and Cloudformation enable infrastructure configuration to be declarative. This makes infrastructure easy to reason about, change, and recreate, enables version control and code reviews, and also serves as documentation.

Automate security scans

Even the most senior engineers aren’t necessarily trained to approach or evaluate solutions with security in mind, and don’t have the bandwidth to review every change for security issues. At a small company, we don’t necessarily have the resources for a dedicated security person, and even if we did, they can’t possibly keep up with everything.

We can supplement manual review with automated security checks. Plenty of free SAST tools can evaluate code for common security issues like SQL injection. Other tools can check dependencies against known CVE databases looking for packages that may introduce vulnerabilities. Other more powerful automations exist but these two are relatively easy to implement and maintain and will go a long way to catching issues.

Make vendors interchangeable, but don’t change them

It should be possible to change critical pieces of infrastructure, e.g. switch from Postgres to MySQL, move to GCP from AWS, etc. The fact that this change could be hypothetically made is a sign of a good, declarative architecture. But you should never do it in practice unless a problem is existential to the company.

CV-MIDI clock converter

2023-05-29T00:00:00-04:00

In addition to my other NiftyKeyz mod from this weekend, I also modded the clock behavior.

My new clock module works great. It’s designed to be the source clock for the whole rack. However, the NiftyKeyz has it’s own internal clock, which it uses for the arpeggiator and for it’s clock output CV. There’s no external clock input jack, but there is the ability for it to take in an external clock over MIDI.

So I used a Teensy 2.0 to take in clock signals from a CV input and output MIDI clock messages.

MIDI clocks

Unfortunately it seems the MIDI org has put a lot of the spec behind a login page, and the clock function doesn’t seem to be all that well documented. Fortunately it’s pretty simple and I was able to figure it out with a little Googling and some trial-and-error.

Unlike other messages such as note-on/note-off, these are global messages that aren’t sent to a specific channel but instead broadcast to the whole chain. The clock source sends a start message 0xFA, followed by a series of clock messages 0xF8, 24 per beat. It can be stopped with a stop message 0xFC.

The choice of 24 clocks per beat is curious. This is likely to make it easier to support 3/4 and 6/8 time signatures, but it’s pretty annoying from a microprocessor perspective to have a prime factor of 3 in the math.

Implementation

While I could have used just about any microprocessor, I used the Teensy because it’s got native USB including MIDI-over-USB, which was useful for testing, and the ATmega32U4 includes 2x 16-bit timers, which ended up being central to the solution.

I wanted to be able to send in the base beat clock and have the module sub-divide it into 24 clocks. First, Timer1 is configured to run in normal mode, where it just counts up to 65535 and triggers an interrupt when it overflows. If I set the timer counter to zero on the first clock, I can very accurately measure the time between internal beats by checking the counter value on the next pulse.

I wanted to support as close a range to my clock module as possible. After some math, a prescaler of 1024 on a clock speed of 4MHz allows the input clock as low as ~3.5 BPM before the timer overflows. At a 1600 BPM, the counter value would be 145, which is still a reasonable resolution.

So with Timer1 we are able to measure the input clock speed. Then, we can set Timer3 in CTC mode to trigger at that rate / 24. When it’s interrupt triggers, we output a MIDI clock message. A prescaler value of 64 gives counter values from 96 to 41665 over the same range (1600 to 3.75 input BPM, or MIDI clock messages at 640Hz to 1.5Hz).

In order to convert the counter values of Timer1 to the counter values of Timer3, we have to take these prescaler values into account.

T3 = T1 * (1024 / 64) / 24
T3 = T1 / 1.5

Again unfortunately since there’s a factor of 3 in here, we do have to do floating point math in order to calculate this. Fortunately I measured the time of this computation on 16-bit unsigned integers to be around 16us on the ATmega32U4 at 4MHz.

Logic

The code works mostly from within interrupts:

Clock input rising edge interrupt:
- First time:
  - Reset and start Timer1
- Subsequent time:
  - Check Timer1 counter value
  - Set Timer3 to Timer1 / 24
  - Reset both timers
Timer3 interrupt:
- Trigger MIDI clock message
Timer1 overflow:
- Turn off timers, as it seems the clock inputs have stopped

This debugging chart shows the typical behavior. Here, the red signal is Timer3, the blue is Timer1, and the purple represents the number of MIDI clock messages left. Timer3 is often aliased due to undersampling, but at slower rates you can see that it does indeed trigger 24 times for each blue line.

All of this works if the input clock is stable. However, we also need to account for input clocks that get faster or slower. If getting slower, all we have to do is send only the first 24 clocks of the beat. When the next input clock eventually comes in, the new clocks will start at the updated tempo.

If the input clock is getting faster, the best solution I could find is to quickly beat any of the remaining 24 beats and then start at the new rate.

You can see the well-commented source code here. Eventually I’d like to make a helper library for generating these timer registers. In the meantime I simply commented the configuration of the registers.

Finally, I added a button to start and stop the signal. I left room for a MIDI-in jack to pass through any signals along to the keyboard, but I didn’t solder it yet.

I may eventually run a board of this and hack on the keyboard to put it in the case itself. In the meantime though I just made a protoboard version and taped it to the back.

As usual, source code and schematics on GitHub!