I recently created an event website and needed to create links for people to add the events to their calendars - the documentation for how to do this for Google cloud is a mess so this is what I eventually worked out.

Start with https://www.google.com/calendar/render?action=TEMPLATE and now each piece we will add is an additional parameter.

1. Title is specified as text, so append that to the URL if you want that. For example if I wanted the title to be Welcome, then I would add &text=Welcome, e.g. https://www.google.com/calendar/render?action=TEMPLATE&text=Welcome
2. Date/Time is next, starting with the keyword dates. Here we specify the date as YYYYMMDD, e.g. 1 July 2022 is 20220701, and times are formatted as HHMMSS, e.g. 8:30 am is 083000. The date and time are seperated with the letter T and the starting and end pieces are seperated with /. For example if we start the Welcome at 8:30am on 1 July 2022 and it ends at 10am - the value would be 20220701T083000/20220701T100000.
3. Timezone is optional, and without it you get GMT. If you want to specify a timezone, use ctz and the value is a tz database entry. For example, if you want South Africa it will be Africa/Johannesburg.
4. Location is also optional and it is the key location with free form text in it.

If we put the above together as an example you get https://www.google.com/calendar/render?action=TEMPLATE&text=Welcome&dates=20220701T083000/20220701T100000&ctz=Africa/Johannesburg&Location=Boardroom

Notes:

1. You must URL encode the values you use. For example, if you had a title of Welcome Drinks, it needs to be Welcome%20Drinks
2. There are other parameters for description etc… but I never used them so I do not have them documented anymore.

I was asked the question a while ago, How do I handle tech debt? and I am not sure I ever put it down in a form that makes sense; so this is an attempt at trying to convey several tools I use.

Pay back early

The initial thinking to handling tech debt is not my idea, but stolen from Steve McConnells’ wonderful Code Complete. Steve shows in his book that is if you can tackle tech debt earlier in the lifecycle of a project, the cost of tech debt is a lot less. Basically the quicker you get back to repaying that debt, the cheaper it will be.

One aspect to keep in mind is that while the early part of a project may refer to greenfields/new projects in your case, it is not limited to that idea. The early part could also mean epic-level-sized pieces of work that are started on an existing project, thus the “catch it early and it is cheaper” applies to existing teams just as much as it does to teams starting a new project.

Organisation

When I do think of what to do specifically to handle tech debt, the one that comes to mind first is also the only one that won’t fit into an existing team easily and that is team organisation.

I’ve seen this at Equal Experts, and previously when I worked at both AWS and Microsoft. The simple answer is that teams above 10 people fail. Why is 10 the magic number though?

1. First is a bit of how we are wired, 15 is the number of close relationships we can have and a closer team performs better; but the eagled eyed reader you are will note that 10 and 15 aren’t the same and this is because you need to allow team members develop close relationships across an organisation, not just in their team.
2. The second reason why 10 is the magic number is that as we develop increasingly complex systems, the ability for people to hold all the information in their heads gets increasingly difficult. If we maintain teams at about 10, it forces limits on the volume of work they can build. This natural limit on size limits the complexity too meaning you end up with many small teams.

When I say team, I am not referring just to engineering resources but the entire team; POs, QAs, BAs, and any other two-letter acronym role you may come up with. The entire team is 10 or less - so you may find you only have 4 engineers in a team.

Cross-skilled teams

There is also something just right about having teams of 8 to 10 people who have the majority of the skills they need to deliver the team end-to-end deliverables. The idea of a dedicated front-end team or dedicated back-end team, where they own part of a feature should be more unique in an organisation. The bulk of teams in a healthy organisation should own features end-to-end. This will force teams to work together and when you have teams holding each other responsible for deadlines and deliverables that helps trim fat in many aspects of delivery.

Having a team responsible to other teams, equally empowers the team to push back on new work because making sure their debt doesn’t overwhelm them and prevent them from actually meeting the demands of the teams which they are responsible for.

The focus in an area also helps people become more a master of the tech they use, and less a generalist and that mastery means the understanding of required trade-offs that lead to tech debt are better understood, compensated for and implemented right and that will, over time, lower the tech debt.

DevOps

I always encourage clients to adopt the DevOps mindset. The above cross-skill, end-to-end ownership hints at that, but one of the best pillars in DevOps to get in early is “You build it, you run it”. This term is simply the idea a team can write code, deploy it, monitor it and support it.

This mindset might feel like it goes against the above idea of a team having the skills they need and being empowered for a whole feature because how can a team own everything from first principles? But we will get to that solution later on.

Where I want to focus on how DevOps helps in regards to lowering tech debt, is that while a lot of serious outages are not caused by tech debt; how long it takes to recover from an outage is often directly related to the tech debt of the team. Recovery from an outage is not just “the website is back”, but includes all the post-incident reviews and working out how many clients were impacted etc… Nothing I have found motivates a product owner and empowered team to cut down on tech debt than the risk they will be woken up for incidents at 2 am and not spending a lot of time after trying to root cause it.

Happy I can even share more specific information from my latest project, as I have a YouTube video the client made with us on this I very aspect.

Reign in tech

A powerful tool in large organisations is to limit technology choices across an organisation, as tech which is kept up to date and used is a lot less of an issue for tech debt, compared to an old system written in a language or tooling that few understand.

Bleeding edge is called that cause it hurts

A small solution to tech debt is to pick stable tech for your organisation. Nothing builds up tech debt faster and is more painful to deal with than bleeding-edge tech.

Horizontal scaling teams

That cost of getting started and frustration just leads teams to not invest which is yet another major worry. This is also the solution to how a small team deals with the first principles that I mentioned earlier.

To solve this we often build horizontally focused teams that have a single feature or set of features that other teams build on top of. IDPs are a great example of this. Another example is having a team that handles all the web routing, bot detection, caching etc… and other teams plug into their offerings. In this case, the team building the web tech might say “We only support caching with React ESIs” so teams in the business can choose to use React and get the benefit of speed and support. They are still empowered to choose something else, but they now need to justify the trade-offs of lost speed compared to using the “blessed tech” from the horizontal teams.

A great example of this is covered in the Equal Experts playbook on Digital Platforms.

Trickle down updates

An interesting side effect of the horizontally scaled teams is that they also become their mini places which force other teams to keep their tech up to date. This happens naturally as the horizontal team updates and forces new updates to consumers of their solutions.

I was looking at that recently where the team responsible for the deployment pipeline runners we use, issued new runners and that meant we needed to take on operational work to migrate from the old runner system to the new runner system. This forced work meant cleaning and fixing how we worked with the runners; it wasn’t a great time for the team but the system coming out at the end is in better shape.

Bar raisers

An aspect that was unique to AWS which I loved, and also is easier to adopt than organisational changes, is the concept of bar raisers.

The idea is the bar raisers are a group of people who give guidance to others to improve them, but they are not responsible for the adoption of that guidance.

For example, at AWS if you wanted to do a deployment which was higher than normal risk, you would be required to complete a form explaining how it will happen, how you will test and how you will recover if it does wrong. You would then take that to the bar raisers who would review the document and give you feedback. This is great because they are not gatekeepers, they are not there to prevent you from doing a deployment (again teams need to own what they build), but they bring guidance and wisdom to the teams.

We had set times for the bar raisers and set days when each of us would do it, which helped the senior people not be overwhelmed with requests. The concept of bar raisers was used in all aspects, including security and design. This sharing helped teams find out about each other’s capabilities, and shared knowledge and helped teams from falling into holes others had found while not bringing in the dreaded micromanagement.

Tech debt is normal work

The last two concepts are two of the easiest to adopt in any organisation. The first is just to capture all tech debt as normal work in your backlog. This helps teams prioritize and understand the lifecycle of their projects better.

We have done some experiments recently to measure avg. ticket time, and when coupled with operation tickets (as we call them) they drag your avg. down if they not getting attended to. This helps the product owner to prioritize correctly and understand the impact.

Even if a team doesn’t pick up the work immediately that is ok because an important aspect of teams that do adopt “you build it, you run it” is they will have natural ebbs and flows in their work. For example, the festive season might be very quiet since you’ll have someone on call in case something goes wrong, but a lot of the team is not there. This quiet time becomes a great opportunity to get tech debt resolved.

Lastly on capturing it; you can’t fix what you can’t see - so shining a light on it, and just going “Well that is worse than we expected” is a great first step.

Tech debt sprints

The last one is the idea we had from my days at Microsoft: tech debt sprints. I spoke of this at Agile Africa, in case you want to watch a video. The idea is to add an extra sprint into every feature at the end and just allow the team to tackle tech debt. At Microsoft, this let us go fast, ship MVPs to customers, get feedback and make trade-offs all knowing we were piling up the tech debt, but also gave the team confidence that it would be fixed sooner, rather than later or never.

If you work with JavaScript or TypeScript today, you have a package.json with all your dependencies in it and the same is true for JVM with build.gradle… in fact, every framework has this package management system and you can easily use this to keep your dependencies up to date.

In my role, every time I add a new feature or fix a bug, I update those dependencies to keep the system alive. This pattern originates from my belief that part of being a good programmer means following the boy scout rule.

I was recently asked if I believe that these dependency upgrades are risky and should we rather batch them up and do them later since it will make code reviews smaller and our code won’t break from a dependency change.

I disagreed but saying “the boy scout rule” is not enough of reason to disagree… that is a way of working, the reasons I disagreed are…

Versions & Volume

All dependency version upgrades have the chance to fail. By fail I mean they break our code in unexpected ways.

There is a standard that minor version changes should be safe to upgrade, which is why I often will do them all at once with minimal checks while major version changes I approach with more care and understanding. Major changes normally happen by themselves. This is because the major version change is the way the dependency developer tell me and you there are things to be aware of.

Major vs. minor will never be rules to rely on perfectly, rather they are guidance of how to approach the situation. Much like when you drive a car, a change in speed of the road is a sign that you need more or less caution in the next area.

As an example that the type version changes and also the volume of changes are not factors let me tell you about last week. Last week I did two minor version updates on a backend system as part of normal feature addition. It broke the testing tools because one of the dependencies had a breaking change. A minor version, with a breaking change.

It was human error on the part of the developer of the dependency to do a minor and not a major change; which impacted how I approached updating, and that will always increase the chance of issues.

Software is built by humans. Humans, not versions will always be the cause of errors.

Risk & Reward

I do like the word “risk” when discussing should you update, because risk never lives alone; it lives with reward.

How often have you heard people saying updating is too risky, focusing on the chance of something breaking… but not mentioning the reward if they did update?

Stability is not a reward; Stability is what customers expect as the minimum

When we do update we gain code that performs better, is cheaper and easier to maintain and is more secure. The discussion is not what will it break, it is why do we not want faster & safer code for cheaper?

I have an inherited piece of code from a team that did not update the versions, it has a lot of out of date dependencies. It is a high chance of breaking when we start to upgrade those dependencies because it was left uncared for.

However, if I look at the projects my team has built where we all update versions every time we do a change, we only ever going to be doing one or two small updates each time. It is easy to see when issues appear which makes fixing the issues easy too.

Death, taxes and having to update your code.

As a developer there is only one way of escaping updating your code: You will hand the code to someone else to deal with and changes teams, or eventually, you will need to upgrade - doing it often and in small batches is cheaper and easier for you.

Using the backend system example again from above. I only had two small changes to dependencies, so my issue was one of them. I could quickly check both of them and I ended up in the release notes for one of them within 15min where the docs clearly showed the change of logic. That let me fix the code to work with it and thus we could stay on the new version. If I had 100 changes… I would’ve rolled it all back and gone to lunch and future me would hate past me for that.

Architects & Gardeners

Lastly, our job is not to build some stable monument and leave it to stand the test of time. I deeply believe in DevOps and thus believe the truth that software is evolutionary in nature and needs to be cared for.

We are gardeners of living software, not architects of software towers.

In our world, when things stop… they are dead. Maintenance and fixing things that break is core to our belief that it is the best way to deliver value to customers with living software.

1. Build for sustainability
   We embrace proven technology and architectures. This will ensure that the system can be operated by a wide range of people and experience can be shared.
2. Code is a liability
   We use 3rd party libraries to lower the code we directly need to create. This helps us go fast and focus on the aspects which deliver value to the business
3. Numbers are not valuable by themselves; We focus on meaningful goals and use numbers to help our understanding
   We do not believe in 100% code coverage as a valuable measure
4. We value fast development locally and a stable pipeline
   We should be able to run everything locally, with stubs/mocks, if needed. We use extensive git push hooks to prevent pipeline issues.
5. We value documentation, not just the “what” but also the “why”
6. We avoid bike shedding by using tools built by experts, to ensure common understanding.

1. Software is not magic
2. Software is never “done”
3. Software is a team effort; nobody can do it all
4. Design isn’t how something looks; it is how it works
5. Security is everyone’s responsibility
6. Feature size doesn’t predict developer time
7. Greatness comes from thousands of small improvements
8. Technical debt is bad but unavoidable
9. Software doesn’t run itself
10. Complex systems need DevOps to run well

From Tom Limoncelli; his post goes into great detail

Today I was honoured to be able to share about Git as part of the Front End Development South Africa’s work with Keep The Receipt.

If you would like to see it, it will be on YouTube soon (will link to it here) and the slides are immediately available.

Recently been talking a lot about the OWASP Top 10 and have created some slides and a 90 min talk on it!

So if want to raise up your security, this is a great place to start.

I spend a lot of time in the kitchen, I love to cook and so I am often in there with my phone listening to a podcast or, if it is a Saturday morning, watching Show of the week. I am not alone in this behaviour, everyone in my home does this and often at dinner we share youtube videos by propping a phone up on the toaster and huddling around it. This was clearly time to improve the experience with a kitchen screen - a smart TV would be perfect but their lack of support means it is a show stopper for me… so I put together a smarter screen.

Build List

Powering this is a Raspberry Pi 4. I grabbed the 4Gb model, just cause… I don’t have a smart reason for that decision. If you are in South Africa, I grabbed mine from PiShop and grabbed the essentials kit. Putting together the “case” was maybe the most head-scratching aspect since it is just screws, plastic and a fan… no instructions.

Also ordered from PiShop was the remote control since I want this to be like a TV, I do not want a keyboard or mouse. I opted for the OSMC Remote Control which has a small USB dongle and uses radio signals rather than infrared, which means it does not need line of sight. Since the Pi will be behind the screen, the line of sight will be an issue. The remote “just worked” which was so awesome.

For the screen, I ordered a LG 24MK400H which was the perfect side for my needs, wall-mountable and on special 😄 The mounting solution I grabbed a Brateck LDA18-220 Aluminum Articulating Wall Mount Caravan Bracket, which is really awesome and easy for mounting. This came with instructions but they were poor and going with experimenting first helped me find a happier setup.

With all of that, I had everything I needed to get running.

OS

The Pi kit came with a MicroSD card with NOOBS preinstalled on it and all I needed to do is when booting, hold shift and select the LibreElec OS to install. LibreElec is a really basic OS which is “just enough” to run the Kodi media centre software. Going through the setup on that got it up and running within about 30min.

Add-ons

I don’t have a “library” of media, rather I just stream the content I want so installing the add-ons I needed was key to set up, and I went with:

• YouTube
• TubeCast, this lets me cast from my phone to the Kodi
• Twitch
• Amazon Prime Video (VOD), this is for streaming Amazon and not for the buying of movies
• Netflix, this has a really great guide to getting started with 3rd party add-ons which are worth your time

Notes

The only strange part of the setup was that each time the Kodi booted, I got a prompt saying there is an update for LibreElec… but the settings for LibreElec had nothing in it for the update and no way to do the update. Thanks to Reddit I was able to switch it to manual and update the setup and then switch it back.

Go to Settings > LibreELEC > System. Change automatic updates to ‘manual’ (I’m not even sure if auto update works at all, I’ve had it set to that before and it never auto updates). Change update channel to LibreELEC-8.0. Select available versions and select the newest one (8.2.3 at time of writing this).

If this was how you were trying to update, then I’m not sure. I would say backup your LibreELEC install and then start fresh with a new version.

Today I was honoured to be part of the second DevFest in SA with a talk sharing about Kotlin, Micronaut, DataStore and other fun tech... but more on how we ended up where we are with our current project. It is a tech lead doing a retrospective with tech sprinkles to get everyone involved.

If you want the code it is on GitHub and slides are below:

Here is the story of all the work I did this week, and it is so odd I feel it needs to be shared… but lets talk about the world the problem can be found in.

The world

It is a µservice (that is the ohh, look how smart I am to use a symbol for micro) written with DropWizard and deployed in a Docker container, with Traefik in front of that. To hit it you go through a ingress controller and a load balancer.

 +---------------+
 |               |
 |   Internet    |
 |               |
 +---------------+
         |
         |
         |
 +---------------+
 |               |
 | Load Balancer |
 |               |
 +---------------+
         |
         |
         |
 +---------------+
 |               |
 |    Ingress    |
 |               |
 +---------------+
         |
         |
         |
 +---------------+
 |               |
 |    Traefik    |
 |               |
 +---------------+
         |
         |
         |
 +---------------+
 |               |
 | Microservice  |
 |               |
 +---------------+

The microservice acts a BFF (backend for front end), so it does some auth fun and makes calls to an internal API and manipulates them (e.g. changes the data structure). We have a number of different REST style calls across GET, POST, PUT and DELETE.

In terms of environments, we obviously have a production environment and we have an integration environment which are setup the same way. We have a stubs environment where we fake out the internal API. Lastly we can run the microservice on our laptops, but there is just the microservice… no traefik etc…

The Problem

When we run our load test from outside everything works except 1 call which fails 98% of the time with a HTTP 500. Other calls (even the same method) all work. Load tests run against the stubs environment and the same call works perfectly on our laptops, in integration and production.

We can even run the fake internal API on the laptops, with the load tests and it works fine there. Basically, one call fails most of the time in one environment… 🤔

Grabbing logs

When we pull the logs for the microservice in stubs things get weirder… it is returning an HTTP 200 😐 This is the same experience we get everywhere else, it works… except in stubs to the load tests it gets a 500

+---------------+   500 Here
|               |
|   Internet    |
|               |
+-------+-------+
        |
        |
        /
        |
        |
+-------+-------+   200 Here
|               |
| Microservice  |
|               |
+---------------+

Further pulling of logs show 500s in the load balancer and the ingress so somewhere between the microservice and traefik the HTTP 200 becomes a HTTP 500… but we don’t have logs on traefik we can pull which makes this a bit harder to determine…

Logging onto Traefik

Next we logged onto Traefik and decided to curl the microservice directly and lo and behold… we get a 500 🤯 and to make it more interesting that the microservice logs still show it is returning a 200 - like what the actual?! Could there be an network issue or magic?

Interesting the 500 came with an error saying “insufficient content written”

Insufficient content written

This led me to looking at the content and looking at what we are sending and I see we are sending Content-Length and the body and guess what… the length of the body does not equal the Content-Length… oh 💩

This is a client side HTTP error… where the server sends the incorrect amount of body, so the client goes, well the server is wrong and raises a 500. I’ve always thought 500 errors were server error and thus could only be raised on the server.

The fix

The fix is simple, in our server we were using the Response.fromResponse to map internal API to the public API and so it was copying the Content-Length from the internal API and we were sending that along.

This meant the fix was to delete the Content-Length header before we call fromResponse to ensure it would rebuild the header and be correct.

The reason why it didn’t fail else where, the version of the mock API we use added Content-Length but newer versions and the real APIs used chunked encoding which never set the header so there is no issue there.

This was a long road to understand the issue, and one line to fix it, and totally a new experience in learning that server errors can occur client side too.