Evolving Feed Ranking at Whatnot

This is part of our ML blog series and pairs well with 6x Faster ML Inference: Why Online≫Batch, which goes deeper into how we built the online inference ML system for scale

A journey from batch to online predictions

Over 2 years ago in March 2023, we hosted a Whatnot Machine Learning day in our San Francisco office. As part of that day, we went into the mechanics of our batch prediction-driven ranking framework. A lot has changed since 2 years ago.

First, Whatnot has grown a lot - we have scaled faster than any marketplace in recent history, surpassing $3 billion in GMV in 2024, over 175,000 hours of livestreams are hosted each week, and we have expanded to over 140 categories in nine countries. With this growth, we faced data size scaling issues with our batch prediction approach, which required us to calculate predictions for n buyers x m sellers. This O(nm) scaling was not sustainable, and we found ourselves adding more and more hacks to manage, filtering out predictions offline for inactive sellers and buyers.

Second, Whatnot is a dynamic marketplace. The ranking challenge at Whatnot is unique- the core entities we are ranking (shows) are constantly changing. As a result, we build for the real-time nature of the marketplace- Real-Time Retrieval and Ranking, Feeds with Real-time signals, Scaling our Data Stack with Kafka and Real-Time Stream Processing, Powering real-time magic moments through notifications, and more. A buyer may be interested in a show selling Air Jordan’s in their size right now, but much less interested when the seller starts selling Crocs 5 minutes later. The batch prediction approach, which required us to calculate our predictions ahead of time and rank based on this limited information at serving time, meant we were not able to capture the dynamic nature of the ranking problem.

Our personalized ranking journey

The For You Feed is the main feed which surfaces shows that are highly relevant to each buyer. It is the landing point when a buyer opens the app. Whatnot is a Discovery-based shopping experience, and the For You Feed is the main surface where buyers discover shows and products they want to buy- 65% of overall GMV comes from feeds; 50% of GMV comes from the For You Feed alone.

To connect buyers with the sellers, products, shows, and communities they love, we formulate feed ranking as a prediction problem: what is the likelihood that a buyer will watch or purchase in a particular show? Based on these predictions, we rank shows according to the estimated likelihood of buyer-show interaction, which is a proxy for relevance.

Batch predictions

Our first ML-based ranking solution consisted of offline predictions calculated in batches daily. Every day we generated watch and purchase likelihood predictions between all possible pairs of buyers and sellers in our system. These predictions were then fetched and used to rank shows when generating the feed for each buyer. This approach did not scale well- the O(nm) growth since the beginning of 2024 is below. Whatnot surpassed $3B in livestream sales in 2024 alone, over 175,000 hours of livestreams are hosted each week, and viewers on Whatnot spend over 80 minutes per day watching livestreams.

Our earlier feed ranking process can be split into 2 pieces: Offline components and Online components

1. Offline: All of our event logging (see here for more details on our event logging) lands in our data warehouse. This includes information about the feed request and generated feed, which we log as backend events. Every day, we created a training dataset and a prediction dataset. The training dataset contained historical features and labels; the prediction dataset included the most recent feature values for all n x m buyer/seller pairs. Within our internal Machine Learning Platform, we trained a new model using the training data, and ran a large batch prediction job, which used the trained model to make n x m predictions. These predictions were then uploaded into our prediction retrieval store.
2. Online: When a new feed request came in, our feed ranking orchestration layer (see: Whatamix) retrieved the set of currently playing shows and looked up the relevant prediction scores using the buyer ID and the set of seller IDs associated with each show. We then used these scores to rank the shows for the buyer.

Batch predictions were a great choice for our first ML-based ranking solution- there was minimal complexity and it was relatively easy to handle prediction issues. However, batch predictions quickly ran into several shortcomings:

• Scaling limitations. Running predictions on all possible pairs of buyers and sellers means that most of those predictions will be unused. It also means that, as our user base grows, the compute and storage needed for such predictions explodes, becoming effectively unmanageable.
• Lack of show-level features. Because we needed to compute our predictions ahead of time, we were not able to use show-level features, which are only known at request time. As a simple example, we could not use the show category as a feature- this feature is only known reliably at request time. Many sellers consistently sell in multiple categories, or change the category of their show before going live to reflect the products they plan on selling in the show.
• Lack of real time signals and staleness. Batch predictions can become inaccurate or irrelevant. A lot may happen in between daily batch prediction runs which may be very indicative of buyer-seller affinity (a buyer might make significant purchases from or start following a particular seller, for example). A model that runs every 24 hours will not be able to capture these in a timely manner.
• Cold-start problems. Batch predictions not only miss real time signals about existing buyers and sellers or shows, but they also are unable to include new buyers and new sellers- we were unable to make predictions for a buyer or a seller until at least 24 hours after they signed up for Whatnot and started using the app.

Online predictions

To address all of these challenges, we worked closely with our ML Platform team to migrate our ranking model from batch inference to online inference.

With online inference, much of the ranking process is moved from Offline to Online

1. Offline: Like before, our event logging data lands in our data warehouse. Every day, we create a training dataset, which includes historical features, labels, and request-level features (which are logged directly as a backend event). We use Chalk, our feature store, to combine our historical features, request-level features, and labels into a training dataset. Within our Machine Learning Platform, we train a new model using this training data with a couple types of negative sampling: random negative samples (easy) and livestreams that were generated for a user that the user did not interact with (hard). The model is then made available for predictions at an endpoint.
2. Online: When a feed request comes in, our feed ranking service (see: Whatamix) retrieves the set of currently playing shows. Given the buyer ID and list of show IDs, we retrieve the relevant feature values from our feature store, then use those features to make a set of predictions, which we return to our feed ranking service for ranking. We directly pass the request features (like the category of the show) as part of the feed request. To ensure we have offline/online consistency, we also log these feature values directly, as noted above.

These changes had a hugely positive impact on the business. We saw large improvements in the For You Feed’s effectiveness at connecting buyers with shows they love:

• FYF GMV increased 7.5%
• FYF orders increased 5.2%
• Watches over 1 minute increased 3.0%
• Watches over 10 minutes increased 3.7%

This is the first in a series of blog posts where we will share more details of the technical architecture of our recommendation systems. Moving to an Online Inference framework, while unlocking a lot of business value, also introduced new challenges, including:

• Latency challenges- predictions need to be made at serving, which takes time
• Stability and scaling- our Machine Learning Platform needs to scale up and down with the usage patterns of the app, including our big marketing events
• Feature point in time correctness- we need to ensure that the feature values we use for training align with the ones we use for inference
• On call/monitoring- with an online system, we needed to build complete monitoring with real-time alerting so that we can identify and fix problems immediately

What’s Next?

Moving to online inference gives the team a pattern that unblocks a wide range of further improvements

• We shipped a new user ranking model (to be covered in a future blog post)
• We unblocked a whole new set of feature types, including:
• Additional request level features– we can continue to utilize features known at request time, like the onboarding selections a new user made.
• Hourly features– instead of calculating our features once per day, we can calculate features every hour to make more up-to-date, dynamic predictions.
• Session-level features– personalize at the session level. For example, if a buyer who typically buys Pokémon cards searched for vinyl records in this app session, we can recommend Vinyl Record shows to them.

Enjoyed learning about our feed ranking evolution and the move to online predictions? If tackling exciting challenges at a rapidly growing startup sounds like your next move, explore career opportunities at Whatnot today!