Open Letter on Scalability & Reliability

In July, we were informed that Post Malone—the singer/songwriter who has won 10 Billboard Music Awards and has an Instagram following of 22 million—would be live streaming a giveaway on our platform. The prize? An opportunity to play him face-to-face in Magic: The Gathering and win $100,000. We had hosted large live streams before and knew that this event would generate a lot of traffic for our app. We took our usual steps to prepare for the event: informing Agora, our streaming video provider, about the event, vertically scaling the compute resources of our main back-end and live event service, and setting up a bridge call for incident response. However, when the stream started, we quickly learned that those usual steps would not be enough.

The Giveaway

On Aug. 4 at 5:50 p.m. Pacific time, the stream began, and users following Post Malone on Whatnot or had this stream bookmarked received a push notification to join it. A few minutes into the stream, we began receiving video choppiness reports. Having seen this issue before, we knew it was often due to a poor host network connection, so we had our host stream from a different room with a more robust connection in an attempt to fix that problem. At the time, it looked like the network connection was the only issue, but our systems worsened as we attempted to fix it. The marketing team had sent out push notifications to 700,000 users inviting them to join the stream, and though we were aware this notification would be sent, we were unprepared for the traffic patterns that would emerge.

The steps we had taken to pre-warm our systems worked for previous streams bigger than this, but because our application changes and grows so rapidly, those steps were not enough to support this new load. New traffic patterns emerged since our last big event preventing services from scaling effectively. Because of these shortcomings, users could not connect to Post Malone’s live stream, viewers already in the stream could not interact with it as they should have been, and our hosts couldn’t run their giveaways.

The failures caused by the load in Post Malone’s stream cascaded into additional failures for other buyers and sellers. Landing pages wouldn’t render when new users entered the app, feeds of live streams and categories wouldn’t load, and experiences throughout the app were failing—we had a complete outage.

The Outage

Ten minutes after the stream began, we called a SEV-0. We started our emergency response by diagnosing the issues, identifying what went wrong, where it went wrong, and determining if there were immediate actions we could take to salvage the stream. Engineers from across the globe came online to help triage these problems and identify if there was anything immediate that we could do.

While digging around our logs and metrics, we noticed several inefficiencies in our code that compounded on each other and caused this failure: our systems were making unnecessary DynamoDB requests and getting throttled, we failed to take advantage of caching as well as we could have, and we weren’t handling errors gracefully enough.

These technical pitfalls combined caused our critical systems to fail, making it difficult to diagnose and resolve the problem quickly. Because we only had Post Malone for a short time, the decision was made to “Post-Pone” Post Malone, but we didn’t know when the event would be rescheduled. We restarted our critical services and allowed DynamoDB to cool down. Forty-eight minutes after the outage began, our app was back to a stable state; however, the question remained: “How are we going to pull this event off?”

The Response

While there were alternative ways to run the giveaway, we knew that none of them would be as effective or exciting as doing it with Post Malone live on our app. Regardless of if we executed a giveaway contingency plan, we would still need to be able to host a live-streamed Magic game in a week about 15,000 viewers had already bookmarked. This stunt with Post Malone was one of our most prominent marketing events yet, and we knew that failure was not an option.

Once the system was healthy, we analyzed the logs and metrics during the outage and used them to create a list of actionable items and next steps. We decided that the best path forward was to determine the root cause of the outage as fast as possible and address those issues immediately—even at the cost of using an unsustainable long-term solution or temporarily diminishing some users’ experiences. The call would be made at 10 a.m. the following day if we would be attempting a second giveaway with Post Malone that evening or if we needed to use one of the contingency plans.

After our analysis, our west coast team began working to solve the problems identified until our Poland-based team woke up and began working. Around 11:30 p.m. Pacific time, the two teams synced, and work continued in Poland throughout the night. There were ten and a half hours until the “go/no go” decision had to be made; we were forced to move uncomfortably fast to try and remediate any issues we could or create workarounds for problems we knew could not be fixed in time. As soon as our engineers throughout the US woke up, they joined the effort to prepare for the second attempt of the giveaway stream; it was an all-hands-on-deck situation.

Because of the engineering team’s around-the-clock effort, issues were identified and fixes were implemented to mitigate potential incidents similar to the previous night. Caching was added to reduce calls to DynamoDB, kill switches and feature flags were created for non-essential, resource-heavy features that could be turned off temporarily during the stream, shortcomings in infrastructure were identified, and configurable rate limiting was put in place to manage network load. By the 10 a.m. deadline, we could confidently say the app would be ready to support the load of the giveaway that evening. However, we continued implementing fixes, testing different configurations and code changes, and load testing the application throughout the day to be more sure the app could support the anticipated load.

The Giveaway (take two)

By 4:30 p.m., all code changes required for that evening’s event had been deployed, additional infrastructure was scaled up and provisioned, and we were standing by for the event. An hour before the stream was scheduled, we began flipping the kill switches deployed earlier in the day to turn off features that we suspected were aggravating the problems caused by increased traffic. After confirming that we were ready to go, global push notifications began to be sent to all users inviting them to join the stream; this time, notifications were sent in groups of about 100,000 users every seven to ten minutes with the hope that the number of users would ramp up gradually instead of spike up.

Around 5:45 p.m., the stream began again. There was choppy video again due to poor internet connection on the host’s side of the stream, but the stream was still functional. DynamoDB began throttling requests again, but because we had already bumped up the number of reserved capacity units for the tables and indexes, we could increase the number of reserved units before our application began spiraling to an unstable and unusable state. The stream continued without problems, and the winner was successfully drawn. Part One of the event was a success, but issues remained, and unsustainable solutions needed a more stable fix before Part Two of the event the following week.

The Mitigation

After resting over the weekend, the teams returned on Monday to begin preparations for streaming the Magic game on Thursday. The live stream already had 14,000 bookmarks, and we wanted to be sure we didn’t have a repeat of issues from the original stream. Because of the production value, complexity, and logistics our marketing team was planning for this event, we knew there would be no second chances and that we only had one shot at success. The consequences of the app failing during this event would be detrimental to the marketing stunt and the company.

To prepare for the event, we scheduled daily standups that included team leads and other engineers to ensure we were on track to remediate any outstanding issues. We created a list of findings and key takeaways from Thursday’s and Friday’s events and used it to guide and prioritize our work.

Throughout the week, we created a run-book that listed the engineers responsible for specific features of our app and assigned individuals to specific incident management roles. The run-book included a list of actions to take, triggers for when to take action, links to where we could monitor the action’s triggers, an assignment of an individual to actions, and a place to record when we executed action. This run-book provided a “single pane of glass” experience for the event’s incident response plan. We ensured the run-book was overly specific to help us execute it confidently without asking additional questions when we needed to take action.

The Game

About one hour before the event, we began executing the scheduled tasks in our run-book. We took preventative actions similar to those from the previous event on Friday but modified them to allow us to be more targeted in the features we adjusted and the effects they would have on users.

At 6 p.m., the stream started. The engineering and marketing teams huddled in a video call as we watched the event proceed, on-call for any possible incident. We had engineers monitor video quality metrics to adjust the stream quality in real-time. We also had the support team from Agora on-call with us in case anything went wrong on their end. Thankfully, because of the engineering team’s preparation, no contingency plans in the run-book needed to be executed. The stream, which held 6,000 viewers consistently and peaked at around 8,400 viewers, was completed successfully, our giveaway winner walked away with $100,000, and we were able to make a lot of good “POSTponed” and “POST Malone” puns.

The Takeaways

While the best-case scenario, streaming both events without issues, didn’t happen, we were still able to take away key learnings from these events. We have already started putting plans in place to enhance our incident response processes and the reliability, scalability, and flexibility of our systems. The experiences we underwent to gain this knowledge were difficult, uncomfortable, and, at times, embarrassing. But sometimes, that happens when we move uncomfortably fast and set audacious goals. When we fail, we take complete ownership of our mistakes, quickly get back up, and keep moving forward while betting on the future.

If this sounds like the kind of team you’d want to be on, let us know! We’re hiring and would love to hear from you.