Signed, Sealed, Delivered — It’s Shipped

For the past year, 9 a.m. and 1 p.m. Pacific Time were when we were most likely to have an outage. Why? Because that’s when we shipped new code. While engineers were encouraged to shepherd their code to production, test in lower environments, and review dashboards and metrics to make sure it worked as expected, incidents would continue to escape. Engineers sometimes spent several hours a day just to get their code out the door.

To reduce the amount of time spent releasing code and to increase the stability of our releases, we decided it was time to prioritize migration to a continuous deployment process.

Four months ago, our deployment process constricted engineering resources and velocity, was clunky, made it difficult to respond to incidents, and was time-consuming. While there is still a lot more we can do to improve our new CD processes, we are in a much better place now with faster rollbacks, a higher deployment velocity, and better incident and release management. It took some time to make the transition but was well worth it in the end.

Where we started

Manual release cuts

Our release process before switching to CD required manual intervention. We knew manual deployments were not a tenable solution, but we were OK with that at the time. To deploy code, we instituted a merge freeze twice a day so an engineer could go cut and tag a release for each service that had changed since the last release window. Once tagged, the release would be tested manually by a release engineer and then our CI pipelines would deploy the new code. Assuming the new deployment was successful, engineers could resume work on the repository.

This was a rough solution for several reasons:

• All work was stalled on our services twice a day to allow a release to be cut.
• Our on-call release manager, which changed every deployment, had to stop working on other projects just to cut a release.
• Engineers sometimes didn’t know they were on-call to be a release manager, and a release would be delayed until someone else stepped in to cut it.
• A release engineer had to stop other work to run our manual smoke-test suite.

Updating our branching strategy

While long term we wanted to get to a “true CD” process, we realized getting there would require many small steps—the first being the removal of our daily code freezes. To do that, we created a release branch that existed parallel to our main branch and automatically merged changes into our release branch when deployments were scheduled.

last_tag=$( git tag — sort committerdate | tail -n 1 )
last_tag_hash=$( git log — pretty=%H -1 ${last_tag} )
release_head=$( git log — pretty=%H -1 origin/release )
main_head=$( git log — pretty=%H -1 origin/main )

# the merge leaves a merge commit which can throw the hash
# comparison off, let’s take a diff too to avoid false
# notifications in Slack
merge_needed=false

if [[ “${main_head}” != “${release_head}” ]]; then
  git checkout main
  git pull
  git checkout release
  git pull

  if [ `git diff main | wc -l` -gt 0 ]; then
    merge_needed=true
  fi
fi

if [[ “${last_tag_hash}” == “${release_head}” ]]; then
  lagging_release=false
else
  lagging_release=true
fi

Every commit to the main branch would trigger a deployment to our development and load-testing environments. Then, at 9 a.m. and 1 p.m. Pacific time, a scheduled CI job would run, checking the main branch for any changes that weren’t already on the release branch. If changes were found, they would be merged into the release branch, and an automated Slack message would be sent notifying the team that a release is needed.

That Slack notification cued our release engineers to manually verify different test cases which ensured our golden paths and other critical features weren’t broken. We’d keep track of the test cases and their results for each release in a Google Sheet. Once complete, they would notify the release manager of the results. If the tests failed, the release engineer would coordinate with the application engineers to determine what broke in the release and how to fix it. If they passed, the release manager would tag the latest commit on the release branch and a GitHub Action workflow would trigger a Helm deployment to put the code in production.

- name: Helm deploy
  id: helm-deploy
  run: |
    echo “**Image tag:** ${{ env.IMAGE_VERSION }}” >> $GITHUB_STEP_SUMMARY
    echo “**Deploying ${{ env.SERVICE }} to:** ${{ env.DEPLOY_ENV }}” >> $GITHUB_STEP_SUMMARY
    helm upgrade \
    --install \
    --timeout 15m0s \
    --wait \
    --atomic \
    -n ${{ env.SERVICE }} \
    -f ./kubernetes/${{ env.SERVICE }}/values/values-${{ env.DEPLOY_ENV }}.yml \
    --set repository_version=${{ env.IMAGE_VERSION }} \
    ${{ env.SERVICE }} \
    ./kubernetes/${{ env.SERVICE }}

Our smoke-testing Google Sheet

While creating a new branching strategy stopped us from implementing a twice-daily code freeze, there was another massive issue we needed to resolve before we could implement CD: our manual, Google Sheet-based smoke-testing suite. We had a suite of about 26 tests that our release engineers would manually run through before each release. If they had the tests memorized, they could complete them in about 10–15 minutes, but if not, it would take 30–45 minutes. Then, if a bug was found during testing, it would often take a couple of hours to fix the release and retest before deployment.

At Whatnot, we believe that effective CD lives and dies by our testing suite, so we knew that we needed to find a better solution. We were already using Datadog for our observability suite, so we chose to take advantage of their synthetic testing tool to replace our manual smoke-tests. We audited the tests to determine what was still needed and what could be removed. Then we recreated the suite as Datadog synthetic tests. These synthetic tests could be triggered from a web browser or via an API call, making it easier to automate. Additionally, the results of these tests could be seen directly from the Datadog UI, allowing us to get rid of our scrappy Google Sheet-based smoke-testing log. Now, with all aspects of the deployment process automated, we were ready to wire them up with a CD tool.

Where we are now

Connecting the dots with Argo CD

To orchestrate our deployments, we chose Argo CD and Argo Rollouts. While there are other alternatives, we picked Argo because of its interoperability with our Kuberenetes clusters, its ability to integrate with our SSO tool, its user interface, and our engineers’ previous experiences with it. A dedicated cluster was created for our Argo CD instance to increase its security and to give it dedicated computing resources, and we created connections from it to each of our application Kuberenetes clusters. We opted for a single Argo CD instance instead of one instance per cluster so engineers could have a single pane of glass for all application deployments. With Argo CD deployed, we could use it to deploy Argo Rollouts and other cluster add-ons required by each application cluster. Finally, rather than serving the Argo CD UI through a public-facing load balancer, we opted to use Zero Trust Network Access (via Twingate) to give our engineers a secure way to connect to it.

Once Argo CD was running and Argo Rollouts was deployed in each application cluster, we started migrating each application from our manual Helm deployment process to automated deployments managed by Argo CD. To do this, we moved our Helm charts and Kubernetes manifests for each application to a separate code repository, changed Kubernetes Deployments to Argo Rollouts (taking advantage of Argo’s CRDs), and created new GitHub Actions workflows in the deployment repositories. Those new workflows would trigger an Argo CD sync via the ArgoCD CLI, in turn deploying the application.

- name: Sync app
  id: sync-app
  run: |
    echo "start-time=$(date +%s)" >> $GITHUB_OUTPUT

    argocd app get ${{ steps.get-app-info.outputs.app }} --hard-refresh --grpc-web

    argocd app sync ${{ steps.get-app-info.outputs.app }} --grpc-web
    sleep 10

    argocd app wait ${{ steps.get-app-info.outputs.app }} --health --suspended --degraded --timeout 5400 --grpc-web

    status=$(argocd app get ${{ steps.get-app-info.outputs.app }} -o json | jq -r .status.health.status)
    echo "end-time=$(date +%s)" >> $GITHUB_OUTPUT

    echo "Status: ${status}"
    [ ${status} == "Healthy" ] || [ ${status} == "Suspended" ]

Testing and monitoring releases

Once we got the deployment processes wired up with Argo CD, we needed to add visibility into it and stability to the deployments themselves. We did this by using Slack notifications from GitHub Actions workflows and the Datadog synthetic tests we created earlier.

For visibility into what was getting deployed in each environment, we added steps to our GHA deployment workflows that send Slack messages when deployments begin and fail or succeed for each environment. Originally, these notifications used the framework that was provided by Argo Rollouts, but we switched to GitHub Actions because it provided increased flexibility in message formatting and allowed better threading of Slack messages. Additional notifications were added to releases that included a list of commits being deployed, Slack mentions of the release’s committers to make them aware their code is going out, and a reminder with a dashboard link to monitor the application’s health during deployment.

In addition to Slack notifications, we used tags and dashboards in Datadog to know what metrics, log messages, and traces are coming from what version of our application. We could reference the version tag in Datadog to see what version of an application is running, create dashboards to show us how rollouts are progressing, and unlock Watchdog (Datadog’s AI monitoring tool) to create automated alerts based on version deployments.

To ensure the stability of our releases, we added analysis templates to our deployments which called the Datadog API to trigger the synthetic tests. We also checked the custom metrics of the service in Datadog to see if they were healthy. If the tests failed or the metrics were abnormal, the deployment would be automatically rolled back. Sometimes faulty features still get released, but when they do, it’s easy for us to add a new Datadog synthetic test to make our test suite more strict in the future. Our planned future iterations of release stability include taking full advantage of Blue/Green deployments by running synthetic tests against the “Green” deployment before migrating traffic and increasing the number of custom metrics we monitor automatically during deployments.

Iterate and simplify

With the observability and stability pieces of our CD process in place, we were prepared to cut all our services over to it. Afterward, we solicited feedback from engineers on what they liked and what they wanted to be improved.

While implementing their feedback and iterating on our previous work, we noticed a pattern across all our GitHub Actions workflows. To reduce code duplication, we pulled out those patterns into shared workflows and referenced them from our application and deployment repositories. Doing so made it easier for us to maintain and improve our CI/CD pipelines.

name: cd

on:
  workflow_dispatch: {}
  schedule:
    - cron: '*/10 * * * *'

concurrency:
  group: cd
  cancel-in-progress: false

jobs:
  cd:
    uses: Whatnot-Inc/gha-workflows/.github/workflows/cd.yml@main
    with:
      service: web
      deploy-repo: whatnot_web_deploy
      image-wait-time-mins: 10
    secrets: inherit

Switching to a CD process also simplified rollback and code freeze processes since we only needed to work off of one branch. Now, if there is an issue with deployment, we can quickly log into the Argo CD UI and click a button to roll back to the previous healthy version, and we can pause our CD GitHub Actions workflows until the fix is implemented or bad code is removed to ensure the bug isn’t deployed again. If we need to implement a code freeze for a company holiday or during peak business hours, we can use Merge Freeze to easily manage branch and PR freezing from Slack.

Our transition to continuous deployment has decreased the amount of time needed to roll back issues from about 30 minutes to five minutes, increased our deployment frequency by 5x-10x, and unlocked easier release management. The biggest win for us is that we no longer have releases containing 20–30+ changes. Having only a handful of changes per release has made our applications more stable and has made it easier to track down changes that caused issues, in turn helping us provide a better experience for our users.

These improvements in our deployment process have been a huge boon to our velocity, but there is more work to do! If you’d like to be a part of this effort, let us know at https://whatnot.com/careers.