14 things you need to be a successful software developer - Number 3 - No-one is best positioned to avoid a problem than those who just made it
In 2017, I was working at AWS, and AWS went down... and went down hard. It was a tough night to be an engineer, and when the dust settled hours later, it turned out that it was caused by a typo. The amount of money lost by AWS—and, more importantly, their customers—that day was huge. One person was to blame. What would you do if you were Andy Jassy (CEO of AWS at the time) with that employee? Fire them? Demote them? No... nothing like that! That engineer was first supported, and then led the work in their team to ensure it never happened again. Their work then rolled out to the rest of the organization over the next few months. Ultimately, the entire organization improved from that. This absolutely speaks to building a safe organization for making mistakes, which leads to success.
There is more coming on how to build safety in teams in this series, so I won’t cover those details now. Rather, what I want to share today is about how to build knowledge—and, unfortunately, it’s a lot of work. I worked with a tech lead who told me, “Failing is the best way to learn”... and he was entirely wrong. Failure is not learning; failing is failing. So how do we build knowledge from failure—or any other situation?
Obviously, the safety of the environment matters, but it’s also about taking time to step back, gathering diverse views, and discussing it to find the learnings.
What practical tools do we have to help with this?
- Do your sprint retros!
- Norm Kerth’s wonderful retrospective prime directive should be stated every time you review a sprint—or every time you review an incident.
- When you have an incident, take time to do a blameless post-incident review (or post-mortem). The best I’ve ever seen is the AWS model. That article is so detailed and covers an amazing tool—which you can use not just in a CoE, but in many situations—and that’s the 5 Whys.
In short, failing in a safe environment—where learning is an action, not just a statement on a company’s values—can lead to powerful improvements. You can help that by building blameless cultures and ensuring that learning actually happens!