It’s 5AM. Your pager rings to bring you the news: people on the east-coast started their day, and your system has crawled to a stop. You are on-call and it is your responsibility to fix the issue. You start looking at metrics, logs, and obviously something fishy was part of yesterday night’s deployment. You rollback, check metrics again, everything back to normal, back to bed. It’s 6:30AM and you hate your life.

Later that day the incident is discussed in a retrospective. The problem was coming from somebody else’s commit that hadn’t been load tested. This is the third time this year. Your manager sternly announces that we cannot carry on like that, load-test your things, people! Everyone in the team agrees: yes, we will load test our stuff. They write that in a summarizing email in passive voice: “All code that has impact on performance needs to be tested”, then close the incident.

A month later, something else breaks for the same reason. The root cause is not poor application performance and bad code. The root cause is wishful thinking. The problem is clearly identified, as well as the behavior you want to see to fix it. Wishing behaviors to appear and saying “we should X, really we should!” (or worse “X should really be”) does not magically make these appear.

You can transform wishful thinking into actions through mechanisms. A mechanism is a step inserted into a process to enforce a behavior, in order to achieve a desired outcome.

The four characteristics of a mechanism are:

  1. A desired outcome - what you want to achieve.
  2. A behavior change to achieve the desired outcome - how you want to modify actors’ way of doing things to achieve the outcome. This is usually where wishful thinking stops
  3. A process definition - how you are systematizing the behavior
  4. An insertion point - how the process will be inserted in the existing workflow, enforced and put in the way so that actors have no choice but follow it.

A common mechanism to increase the quality of a software is to add code coverage measurement to the build pipeline, and block build if coverage drops a certain threshold:

  • The desired outcome is increased quality.
  • The behavior is that developers add automated test coverage for everything they build.
  • The process is checking code coverage metric against a threshold
  • The insertion point is the automated build

This example is automated, but mechanisms can also be manual. The most common instance is a checklist added at a particular step of the process (pull-request template, manual test step, deployment approval, …).

  • The desired outcome is that performance is satisfactory under high load
  • The behavior is that developers load-test their changes
  • The process is a check-list item: “I ran load tests and they pass”
  • The insertion point is a checkbox in the pull-request template

Automated mechanisms are usually cheaper to execute and more reliable, manual mechanisms are usually faster to implement, and can be a good temporary way of fixing the problem until automated.

Each of the four characteristics are crucial. A mechanism without a desired outcome or behavior is pure administrative burden1. A mechanism without a process definition is wishful thinking and does nothing. A mechanism that doesn’t stand in the way is not enforced and will not be followed.

Mechanisms are useful at a personal level. This is the central theme of frameworks such as Atomic Habits or the happiness advantage principle #6 which aim at getting people to change by creating habits: don’t trust cheer will and instead create a system that will force you to act the way you want. Put the TV remote away, and your guitar in the middle of the room. Track your calories and your exercise. Build processes.

The last key to successful mechanisms is the revision process. As practices evolve, mechanisms can become redundant, irrelevant, and useless. Manual testing and formal deployment approval can become useless if everything is covered through automated tests. In such cases, maintaining the mechanisms only add to the burden. Mechanisms can also fail to achieve the desired behavior or outcome, or at being executed systematically. A meta-mechanism introduced to regularly review mechanisms through frequent continuous improvement meetings (e.g. retrospective every two weeks) and challenging the status quo is necessary to keep things lean.

Do not accept wishful thinking. Good intentions are pure procrastination. Either things are acceptable the way they are (sure, things broke, but it’s not that big of a deal and we have bigger fishes to fry at the moment), or they aren’t. In which case: don’t let go until you have a solid and precise definition of how you will change how you work, and how you will enforce it, in order to fix the problem.

notes

  1. Interestingly, mechanisms where the outcome or behavior is not clearly communicated can be seen as administrative - so explaining the why also is important for people to understand and follow.