Dimitris John Raptis

625 days ago

Tech and culture debt are a real and present threat to all startups: shortcuts taken for growth too often become pitfalls when growth arrives and you try to stabilise your platform.

"The infamous November 2014 Azure outage happened for just that reason. At around the same time, a dev at one of Azure’s competitors overrode the rule that you shouldn’t push a config that fails tests because they knew that the config couldn’t possibly be bad. When that caused the canary deploy to start failing, they overrode the rule that you can’t deploy from canary into staging with a failure because they knew their config couldn’t possibly be bad and so the failure must be from something else. That postmortem revealed that the config was technically correct, but exposed a bug in the underlying software; it was pure luck that the latent bug the config revealed wasn’t as severe as the Azure bug."

Normalization of deviance in software: how broken practices become standard

danluu.com

Have you ever mentioned something that seems totally normal to you only to be greeted by surprise? Happens to me all the time when I describe something everyone at work thinks is normal. For some reason, my conversation partner’s face morphs from pleasant smile to rictus of horror.