QConSF 2016: A Dark and Stormy Night - Operational Antipatterns
I gave this talk at QConSF 2016.
Abstract
“It was a dark and stormy night; the rain fell in torrents — except at occasional intervals, when it was checked by a violent gust of wind which swept up the streets (for it is in London that our scene lies), rattling along the housetops, and fiercely agitating the scanty flame of the lamps that struggled against the darkness.”
This sentence exhibits so many writing antipatterns that it’s inspired an entire literary competition for terrible opening sentences. It’s complicated, repetitive, and barely gets its point across. In the same vein, common distributed system antipatterns have inspired many informal competitions—ops folk telling ghost stories around a bottle of whiskey. Building a large software system without an eye to operability can lead to disaster. If your system does what you want it to do today, but upgrading software packages takes months of engineering time, it’s not doing what you want it to do a year from now.
This talk will introduce some common operational antipatterns, and a few tactics to help avoid shooting your future self in the foot. From a system with completely opaque fractal queues, a multitenant system with no circuit breakers, or a piece of software that requires hours of manual operations, expect a rapid-fire succession of stories and lessons that will terrify and delight!
Bio
Kiran Bhattaram is an engineer on the Observability team at Stripe. She has previously written code for the New York Times, LinkedIn, and MIT CSAIL. In her spare time, she enjoys making things, whether tinkering with circuits, sewing dresses, or woodworking.