Telling a Computer to Do Things

For the first few years of my career, I didn’t know how to tell my computer to do things. I could kick off a few commands from the terminal – run those tests, install that dependency, start that container, ssh to that machine – but I was limited to running simple commands one at a time. My terminal was the world’s worst GUI, and I thought that the shell was the way to start programs that didn’t have application wrappers.

At a fundamental level, I wasn’t able to tell my computer to accomplish anything that involved logic or stitching together multiple programs like:

• Do one thing and then do another
• If a command fails, log out an error message
• Kick off a program when the computer starts
• Run two commands at the same time
• Loop through all of the files in a directory and take an action on each one
• Use the output of one command as the input for another

In theory, I could have used NodeJS to write those sorts of programs. In practice, I never did. This was partially mindset: I wasn’t used to thinking about the programs I used on the command line as things that I could control. And the rest of it was a lack of skill: I didn’t know the programs that I was using well enough to integrate them into a script that I’d written.

I kept trying to learn the shell though, and I slowly got to the point where I could muddle my way through scripts like this one that had basic logic:

set +e
npm install
status_code=$?
if [[ "$status_code" != "0" ]]; then
    echo "Something went wrong with your npm install. Check your ~/.npmrc to make sure it's authed to our registry."
    exit 1
fi
set -e

Depending on your familiarity with shell scripting, You might be gibbering right now. Sorry. (If you’re not bleeding from the eyes yet, here’s why you should be: 1)

Even though the commands and scripts I wrote had problems, it was transformational for me; I had the sudden ability to tell my computer to stitch together existing programs to accomplish my goals. It felt a little bit like the change that came from learning to program in the first place.

Over the course of years, I slowly started to learn more shell tools and figure out shell syntax. I’d often learn a new tool or pattern and then have a moment of pain when I realized how much easier a past problem would have been to solve if I hadn’t used a hammer to solve a problem that needed a drill.

Over that time, I’ve worked with a ton of (incredibly strong!) engineers who didn’t spend as much time learning the shell, and instead relied on GUIs to do things like run tests, manage git, talk to databases, and do day-to-day tasks. Relying on a GUI works well until you want to accomplish something that the GUI wasn’t set up to handle, and I think it’s easy to slip into a mindset where you’re limited to what the GUI is capable of. I’ve seen skilled engineers spend a ton of effort because they didn’t know how to do things like use while to keep running a command or write a for loop to do the same operation on every file in a directory.

I think that same GUI-focus can be a problem when it comes to editing and maintaining shell scripts. I’d wager most companies have a decent amount of essential logic to build, deploy, validate code, and test in languages like Bash or Zsh. If you’re not comfortable with the language your tooling is built in, then you won’t be able to easily read or improve it. You might be able to tell the remote servers that your code runs on how to behave but not be able to tell the computer that you work on how to do things like run linting and tests in parallel – that’s a bummer!

Let’s pause to take a quick detour: Why the heck are so many of these scripts end up written in languages that aren’t the main ones the team uses? I think it’s often more ergonomic to write a script that stitches together commands in a language that’s been designed to be easy to stitch together commands. Let’s take a super simple example of running a test until it fails: while pnpm exec mocha ./pathToFile.test.ts; do true; done. There are obvious things to critique about this syntax, but let’s take a look at what it looks like in NodeJS:

const { execSync } = require("child_process");
while (true) {
    try {
        execSync(`pnpm exec mocha ./pathToFile.test.ts`,  { stdio: "inherit" });
    } catch (err) {
        console.error("failed", err);
        break;
    }
}

There are a lot of rough edges and gotchas here, and I personally think the shell is easier! I don’t need to worry about creating a file, requiring child_process, or setting { stdio: "inherit" } to see output. And this is a pretty simple example that doesn’t even stitch together multiple programs with a pipe, capture any output, or use a temporary file!

💡

This doesn’t mean that you need to resign yourself to writing in Bash or Zsh or any similar language! For teams that know JavaScript well, one tool I’ve enjoyed is zx. I think it can make these scripts pretty ergonomic to write and maintain. Ruby and Python are both easier than NodeJS is, but I think there are plenty of languages out there that require even more ceremony to write a quick little script like this.

I’m certainly not arguing that shell scripts will always be easier for build scripts! When problems are simple enough that you’re just stitching two programs together, a tool like Bash or Zsh feels pretty ergonomic. As soon as you want more sophisticated logic and data types, you’ll want to choose a language that makes it easy to represent (and test!) more sophisticated logic and data types.

I suspect that many engineers who gripe about build scripts being written in a shell language haven’t actually tried converting them to a different language. Aside from the syntax (potentially) being more complicated, a huge part of "learning to write shell scripts" isn’t actually syntactical. If you convert a script that stitches together commands but don’t actually know how the commands you’re stitching together behave, the resulting script is likely to be similarly impenetrable.

Knowing the shell – being able to tell a computer to do things – depends a ton on knowledge of the programs that do the things that you want to accomplish! I’d argue that knowing the shell is 20% syntax and 80% having a good toolbox:

• If you know fzf, you can build quick utilities with interactive fuzzy-searching. (Example: git checkout $(git branch --sort=-committerdate | fzf) will let you fuzzy-choose a branch.)
• If you know tldr or eg, you can pull up usage examples for any other command in this list
• If you know rsync, you can copy changed files on to a faster remote server to run something heavy and slow
• If you know xargs, you can build up commands incrementally and parallelize work
• If you know sed -i or ast-grep, you can quickly rewrite complicated patterns across a bunch of files
• If you know direnv, you can make sure the right environment variables are set for everyone who runs commands in a codebase.
• If you know duckdb, you can write SQL to query CSV and JSON files locally as part of a larger script.
• If you know gh, you can build scripts to check on your PRs and open up new PRs from the cli.
• If you know ngrok, you can quickly serve a local port to test something out on a different machine.

Each additional program you learn expands your capabilities more because each new tool can be used with every other tool you already know.

I can’t stress enough that I’m the furthest thing in the world from a shell scripting expert, and I’m terrible compared to people who know it deeply,2 but I’ve still gotten a lot of value out of knowing enough shell syntax to stitch together programs and knowing enough programs that I actually want to stick together.

Telling your computer to do things is great!

---

1. This script was written with good intentions, but it was overly complicated because I didn’t know that if's main mode is to take commands. [[ (or test) is just a special command. This means that our if check can just be ! npm install.

   • With this approach, there’s no need to set +e to allow commands to fail. (set +e and then setting it back with set -e is to avoid the shell-script as a whole failing because a command failed – Unofficial Bash Strict Mode)

   • It also means we don’t need to capture the exit code with $?

     if ! npm install; then echo "Something went wrong with your npm install. Check your ~/.npmrc to make sure it's authed to our registry." exit 1 fi

    ↩︎

2. From an HN comment on a previous blog post: "This post […] means I no longer wonder why 99% of shell scripts I come across look inept." ↩︎