Refactoring Authentication at Whatnot

Authentication is critical to converting visitors into daily active users. It’s troublesome to constantly force re-auth, yet keeping user data safe is significant. Classic friction vs. security tradeoff. We must ensure users can successfully and securely access the service through reliable login. At the same time, we must be able to reduce churn and work with a secure architecture that fosters product growth.

We found there was a lack of visibility and auditing with the previous flow, and had received reports of users being logged out from time to time. As a solution, we introduced oAuth 2.0. This enabled us to automatically extend sessions via a more secure approach, schedule dedicated dashboard reminders, and have logout instrumentation and a greater understanding into the authentication architecture and challenges we face.

Design

We previously used JSON Web Tokens (JWT) in a simple way—a token was generated for 30 days and when it expired, a user was forced to log in again. We had some options to revoke a token, but this had a serious implication—every service that wanted to verify if a token had not been revoked had to contact the source-of-truth to check if the token can be accepted.

And since this goes against JWT philosophy of being stateless, we needed a design that would be future-proof, especially if the number of services grows.

We decided to go with a scheme similar to the one utilized in the OAuth 2.0 standard, where this problem is solved in the following way:

• A short-lived (for a few minutes) access token can be used across the whole system to authenticate users;
• A long lived (for a few weeks or even months) refresh token can be used to generate a new access token.

All the centralized checks (like user being banned, or token revoked) need only to be made when an access token is being refreshed, instead for every single operation.

This of course comes with a trade-off—a user can still perform certain actions until the access token expires. However, we can keep the token refresh process completely in the background and transparent to the user, while keeping that window very short.

Redesigning the authentication system was also a good opportunity to increase security of the token storage on the client side. We decided to follow OWASP best practices to make it much more difficult for a malicious user to steal credentials. Consequently, this reduced the probability of gaining access as an unaware user.

Additional protection of a token in this scheme is ensured by a fingerprint paired with the token, which is stored as a cookie. Only a valid pair of a token and a fingerprint can be used to successfully authenticate a request, just like two keys are required to open a deposit box.

Since the cookie is hardened, it’s impossible to impersonate a user with data sidejacked using a cross site scripting (XSS) attack. That’s because the token can be retrieved with an XSS vulnerability, but the fingerprint cannot. Since a custom authentication header needs to be present in every request, this works as a protection against Cross Site Request Forgery (CSRF) attacks too.

This solution is very handy as we can use one method of storing and passing the tokens for clients across all platforms. We also don’t need to add any CSRF tokens to the requests sent from the web browser, on top of the token and the fingerprint pair.

Bug Shooting Methodology

Authentication issues can be tricky to debug—for example, a user might report being accidentally logged out, but it’s working perfectly fine on your end. The standard client side development process to manually test different flows may not result in success repros, with all different client differences, network condition variance, and user activity that exits the app or switches between apps, etc. In response, we developed a couple of methods in the overhaul.

Loggings, loggings, loggings

We quickly realized when there are no loggings, it’s almost IMPOSSIBLE to debug random logouts. On the other hand, having extensive loggings are a great way to root cause and find out edge cases quickly.

The first thing we did was add loggings into every if/else condition in critical authentication logic on both client and server side. user_id is always handy to stitch together telemetry. There are also many times we need to add additional parameters into DataDog loggings to figure out what went wrong.

Customizable Access Token TTL Endpoint for Internal Developers

The other challenge we faced was that developers needed to wait for the access token to expire (usually about 5 minutes or 10 minutes) to test the next refresh. On the other hand, for developers who need to test resource API, 5 or 10 minutes is too short, resulting in developers constantly needing to acquire new tokens. We created an endpoint to make the expiration customizable for developers, so it could be set to 30 seconds, or 10 minutes, depending on need. This significantly shortened the development and testing cycle for authentication.

Stress Test

Stress test is another great way to expose issues. (For example, on the web we once ran into multi-tab racing conditions.) While it is hard to reproduce with a 5-minute refresh schedule with two tabs open, it is much easier to spot issues if keeping 10 tabs open, and setting refresh schedule to be 1 second. Then you will see a lot of racing conditions leading to failed refresh requests at once.

Edge cases

We ran into a series of edge cases and thought some of them (listed below) might be common for other developers working on this too, so we’ll spend some time sharing how we ended up mitigating those issues.

iOS

Keychain Issues

The iOS Keychain is a set of Apple-provided APIs and services that allow app developers to store credentials and other sensitive data on the device in a safe, encrypted manner. As such, it was a natural choice when it came to selecting the best storage for user authentication data.

The keychain API supports safe storage of arbitrary key-value pairs. Our authentication data always consisted of a couple of fields, thus, when new ones were added for the second iteration of the solution, it was only natural to introduce an additional set of keys. This deceivingly obvious choice was soon to come back to haunt us when we started noticing in our logs that some users were being logged out of the app for no obvious reason.

A closer investigation revealed there was a mismatch between the different pieces of authentication data provided by the client to the backend. This initially made no sense at all since that data was always being refreshed in a transaction as a whole.

After a careful and somewhat time consuming root cause analysis we concluded that the way we interacted with the keychain API was a bit naive. Deep down in the keychain logic we had uncovered a false assumption that probably every developer has made at some point in their career, which is that I/O operations are guaranteed to always succeed. The keychain code was assuming that the save operation always succeeds and was not handling any exceptions at all.

To verify our theory we had added some additional log statements and had improved keychain error handling. It wasn’t a big surprise when we started seeing app sessions where the majority of authentication data was being saved successfully to the keychain, with only a single field failing to persist. It was usually the very last field that would throw an exception. Since this issue occurred extremely rarely, we came to a conclusion that the OS was most likely cutting off access to the keychain when the device was locked while the app was still persisting authentication data in the background.

To work around this problem we simply stored the data atomically as a single blob of information. If the write was successful, the app would continue to operate as usual. If it failed though, the entire set of authentication data would be discarded. Due to the specifics of our implementation, it didn’t pose an issue since the existing credentials were likely to still allow the user to access their account.

More Keychain Issues

We were in the middle of patting ourselves on the back for being clever and figuring out the keychain issue, when we received reports of an even more mysterious case of users being randomly logged out. They would open the app and be surprised to land on the sign-in screen for no apparent reason.

This time around it was even trickier to find any leads since our logs were showing no authentication errors whatsoever on the client or the backend side. The client logs indicated that the app was successfully refreshing and saving tokens the whole time, only to find itself unable to access this data a while later.

As we continued analyzing logs of more app sessions affected by this issue, we started seeing an interesting pattern. The logic responsible for displaying the sign-in screen was always firing off at app launch. This made us think—if writing to the keychain is bound to fail from time to time, maybe it’s the same story for reading from the keychain?

We had added some additional log statements around the code that was accessing authentication data at app launch, and were soon enough able to confirm our theory—the OS was politely yet unexpectedly telling us that regretfully it could not allow us to access the keychain content.

Another assumption that many iOS developers (including our past selves) make is that the app is always in the foreground when the launch-related app lifecycle delegate calls are being executed. Since we were using the default keychain access level where the encrypted data is only accessible while the device is unlocked, it was probably reasonable of us to assume that we should be able to access our keychain data at app launch to verify that the user is authenticated.

This was a fair assumption until iOS 15 came around and Apple introduced app prewarming. The clever yet unexpectedly problematic idea was to automatically launch the most frequently used apps in the background, ahead of the user actually tapping on the app icon, so that when they actually decided to use a specific app, it would launch pretty much instantly.

As a result of this subtle change it was now possible for an app to be launched when the device was locked, making the keychain inaccessible and rendering any attempts to read it as failed. Fortunately, the solution was quite simple. Since there’s no significant logic in our app that needs to execute prior to signing in, we could simply wait for the protectedDataDidBecomeAvailable notification, and pause the app launch/authentication logic until the required data became available.

Web

Token Syncing across multiple tabs on web

One of the unique challenges on the web is syncing auth tokens and states across tabs.

To solve this, we used localSync module.

When one tab receives back responses from v2/refresh, it will publish signals to other tabs. Other tabs, upon receiving the message, will feed LOGIN_USER with new tokens and update redux states of its own tab.

Racing condition caused fingerprint mismatch

One of the biggest challenges on the web is when two tabs send out refresh requests at the same time. This racing condition would invalidate one of the requests, resulting in the user logged out.

The situation can be very bad especially when a user has lots of tabs open, and under poor internet conditions.

To address this, we implemented a local storage based lock to sequence refresh requests across multiple tabs. When one request goes out, a lock will be set. We also store the id of the tab sending out the request If a second request comes in when this happens, we abort the request. The lock will be lifted when the network request returns.

Results

Since implementing Authentication 2.0, we’ve received less logout reports on iOS. And, with the new and improved dashboards, we’re able to quickly know when things go wrong, and identify issues, leading to greater developer efficiency.

If you’re interested in working on solutions like this, join us!

Huge kudos to Rafal Studnicki whom we cannot drive through without. Shout out to Mikolaj who has heavy-lifted Android Auth V2 extremely fast. And special thanks to Doug Richardson, Raphael Furlan, Jeff Avery and Vincent Tian for their awesome thoughts and support.