Documenting what you're willing to support (and not)

Sometimes, you just need to write down what you're willing to do and what you're not. I have a short tale about doing that at a job, and then bringing that same line of thinking forward to my current concerns.

I used to be on a team that was responsible for the care and feeding of a great many Linux boxes which together constituted the "web tier" for a giant social network. You know, the one with all of the cat pictures... and later the whole genocide thing and enabling fascism. Yeah, them.

Anyway, given that we had a six-digit number of machines that was steadily climbing and people were always experimenting with stuff on them, with them, and under them, it was necessary to apply some balance to keep things from breaking too often. There was a fine line between "everything's broken" and "it's impossible to roll anything out so the business dies".

At some point, I realized that if I wrote a wiki page and documented the things that we were willing to support, I could wait about six months and then it would be like it had always been there. Enough people went through the revolving doors of that place such that six months' worth of employee turnover was sufficient to make it look like a whole other company. All I had to do was write it, wait a bit, then start citing it when needed.

One thing that used to happen is that our "hostprefix" - that is, the first few letters of the hostname - was a dumping ground. It was kind of the default place for testing stuff, trying things, or putting machines when you were "done" with them, whatever that meant. We had picked up all kinds of broken hardware that wasn't really ready to serve production traffic. Sometimes this was developmental hardware that was missing certain key aspects that we depended on, like having several hundred gigs of disk space to have a few days of local logging on board.

My page became a list of things that wouldn't be particularly surprising to anyone who had been paying attention. It must be a box with at least this much memory, this much disk space, this much network bandwidth, this version of CentOS, with the company production Chef environment installed and running properly... and it went on and on like this. It was fairly clear that merely having a thing installed wasn't enough. It had to be running to completion. That means successful runs!

I wish I had saved a copy of it, since it would be interesting to look back on it after over a decade to see what all I had noted back then. Oh well.

Anyway, after it had aged a bit, I was able to point people at it and go "this is what we will do and this is what we will reject". While it wasn't a hard-and-fast ruleset, it was pretty clear about our expectations. Or, well, let's face it - *my* expectations. I had some strong opinions about what's worth supporting and what's just plain broken and a waste of time.

One section of the page had to do with "non-compliant host handling". I forget the specifics (again, operating on memory here...), but it probably included things like "we disable it and it stops receiving production traffic", "it gets reinstalled to remove out-of-spec customizations", and "it is removed from the hostprefix entirely". That last one was mostly for hardware mismatches, since there was no amount of "reinstall to remove your bullshit" that would fix a lack of disk space (or whatever).

One near-quote from that page did escape into the outside world. It has to do with the "non-compliant host" actions:

"Note: any of these many happen *without prior notification* to experiment owners in the interest of keeping the site healthy. Drain first, investigate second."

"Drain" in this case actually referred to a command that we could run to disable a host in the load balancers so they stopped receiving traffic. When a host is gobbling up traffic and making a mess for users, disable it, THEN figure out what to do about it. Don't make people suffer while you debate what's going to happen with the wayward web server.

Given all this, it shouldn't be particularly surprising that I've finally come up with a list of feed reader behaviors. I wrote it like a bunch of items you might see in one of these big tech company performance reviews. You know the ones that are like "$name consistently delivers foo and bar on time"? Imagine that, but for feed readers.

The idea is that I'll be able to point at it and go "that, right there, see, I'm not being capricious or picking on you in particular... this represents a common problem which has existed since well before you showed up". The items are short and sweet and have unique identifiers so it's possible to point at one and say "do it like that".

I've been sharing this with a few other people who also work in this space and have to deal with lots of traffic from feed reader software. If you're one of those people and want to see it, send me a note.

At some point, I'll open it up to the world and then we'll see what happens with that.