Robert MacLean

You never debug standing up

Published on Estimated reading time: 3 minutes

humility_road_sign

Being a professional developer means following standards—standards set by smarter people (smarter than me at least)—and when you start to break those standards, you’ll go wrong. I know this! However, last Friday, I broke many rules and failed because I thought I knew better this one time. So this post is a retrospective of what went wrong, so that I never forget again.

The project is a fairly complex backend system for a special mobile device, and when I say complex, I mean in size and scope: VB.NET, C#, Java, JavaScript, and a ton of inline SQL. The work itself is done by three different companies, each with its own agreements, policies, and methodologies. That sounds like a recipe for disaster, but it isn’t—we’ve exceeded every goal. That is, until Friday.

On Friday, we were told the customer was coming at 9 a.m. on Monday for a demo. We had eight hours to prepare. The software was in a state between alpha and beta, with known issues that needed resolving by demo time. We worked hard and made great progress until about 3 p.m., when we hit an issue that everyone in the room thought was on the mobile device. The first problem? The mobile dev company wasn’t onsite, so we were really guessing.

confidence

We started experimenting with fixes, pushing seven builds to QA in an hour—lucky if we got five in a day normally. At the same time, we were also adding bug fixes and tweaks for other issues. Then, the answer came in from the mobile device developer: they were breaking because their expectations of the spec didn’t match ours. More fixes and adjustments—and then the server just stopped.

At this point, I checked my watch. I had a two-year-old son at school, and it was almost time to pick him up. Tick. Tock. As a single parent, I had to leave—and soon. The mobile guys had gone offline completely. Another dev from the third company was rushing to propose (she said yes 😊)! We were all rushing: skipping testing, skipping local builds before checkins… I even checked in code without comments. No matter what I tried, it kept breaking. Tick. Tock. Same error every time. I couldn’t find it. The PM was looking worried, and I felt the weight of needing to solve it. Out of time.

I apologized and left, feeling terrible. I had failed. I had let a client down.

Saturday, I sat down to dig into the code. Without the time pressure, I finally read the stack trace properly—not assuming what it was saying, as I had been in my rushed state—and quickly found the issue. Resolved it with one code check-in (built locally first, with a check-in comment, assigned to work items). It worked. The test client worked too!

In retrospect, what I should have done was simple: we knew there was a demo, even if it was last-minute. We should have chosen what to show before the cut-off time after lunch—and then not included broken things. Better to show 80% working flawlessly than rush for 100% and end up breaking everything.

I should’ve also stepped back earlier to clear my mind—gotten coffee or something. Stepping away often helps; how many times have you asked for help only to realize you’d solved it yourself in the process?

Finally, skipping the standards that make us professionals leads to disaster. I should’ve ensured:

Monday came and went. The customer loved it, and the demo was great. We succeeded. It required a sacrifice from the team—some of their weekends—and one that, if we’d acted more professionally and kept our heads, could’ve been avoided. This doesn’t happen often to me; I’m usually surrounded by professionals who pull each other back when needed. This post is a reminder: that’s why we do that for each other, and what others can learn.

Management is doing things right; leadership is doing the right things —Peter Drucker.