From a two month to a two week release cycle

In GoHealth, we help people to find and enroll in the most fitting health insurance plans. A part of the process is filling in an enrollment application using our web interface, which is then submitted to the insurance company that provides the plan. That’s where my team comes in — we own the enrollment user flow and integrate with major US health insurance carriers such as Humana or Anthem.

When I joined the enrollment team, we were doing ad-hoc releases. We did not have a regular release schedule which meant we performed releases when we felt the time was right — typically we felt a strong urge to release ahead of critical fixes or business imposed deadlines. This often led to a gap of four to six weeks between releases. Big releases mean big problems — each release contains too many changes, it is hard to reason about what actually goes into the release and the testing is a nightmare. Less frequent releases also mean that people need to remind themselves how to perform the release both process-wise (e.g. who to notify) and execution-wise (e.g. which buttons to push to deploy the application). As a result, there was a lot of anxiety and apprehension around every release.

All these issues would snowball into increasingly less frequent and more painful releases. After one such particularly painful release with 6 weeks’ worth of changes in it, it was clear to us we had to do something significant.

We have set on a path to improve our release cycle and while there is always room for improvement, we got to the point where we stick to the regular two-week release cycle, occasionally getting down to a release or three a week. Here is what we have done and learned in the process.

Stick to schedule

Our release days last month. It was a busy one.

Create checklists

One of the first improvements was to document all the steps in a simple one-page document to make the release process repeatable and faster. This page included everything you needed to perform a release:

  • Documentation on the release process
  • List of deployment jobs
  • Monitoring dashboards and logging

Once we had a decent grasp on the process, we created a checklist to outline all the steps in a release:

  • Which systems to release
  • People to notify
  • Human checks needed (e.g. backwards compatibility)
  • Testing areas

Creating documentation and checklists tremendously helped us to get more people involved in the process of releasing.

All checked (checklist fragment circa May 2021)

Automate deployment

Just click the right button

Automate the release preparation

Automate testing

With the whole team cooperating, we are able to run regression suites automatically and even verify deployed versions without human intervention — that’s the peace of mind you need for frequent releases.

Tests are turning out exquisite this season

Make each change releasable

Feature flags don’t have to be anything fancy — simple property settings work well for us.

Unfortunately, it is not always effective to keep the new code and old code side by side. In such moments we deploy from hotfix branches but it is an exception that raises more than one eyebrow — releasing from the hotfix branch means we are doing something wrong and need to go back to main quickly.

We still need to work out kinks on the flux capacitor

Tag all the versions

This is complicated by the fact that applications we deploy consist of multiple dependent libraries which contain business logic for different integrations. Again we have the tooling to help us see changes down the multiple layers of the dependency graph.

The critical enabler is publishing libraries and containers to Nexus. We tag each git commit from which we publish a library. It took us a little work to become consistent but is a lifesaver when you go investigating what could have gone wrong.

Four steps to an epiphany — the simpler the better

Have a rollback plan

It sometimes happens that the release is hard to roll back due to environmental changes or other external dependencies. In such cases, we look for mitigations on the code level (e.g. extra code to allow for backward compatibility) and proceed extra carefully.

Conclusion

At the end of the day, keeping good release practices is like going to the gym — everyone knows it’s useful but it takes determination to stick to it.

Are you interested to find out more? Come work with us! Get in touch at slovakiateam@gohealth.com and join our team.

Author: Michal Kostic

Follow us on our web and social media!

GoHealth.sk | Facebook | Instagram | LinkedIn | YouTube

Leading U.S. healthtech provider now located also in Slovakia. Modern technology, functionality and exciting projects are what motivate us.