Whatamix: Blendable feed construction

How we leveraged DAGs to construct the Whatnot feeds

Feed functionality plays a pivotal role in the user experience of e-commerce apps, allowing users to discover and connect to the products, sellers, and other content types they are interested in. Additionally, feeds must not only provide ways to improve discoverability for users, such as supporting different content types like videos to watch and products to buy, but also balance business needs, such as ensuring visibility for special events, certain groups of sellers, etc. Across these different discovery applications, there are a set of common steps such as candidate retrieval, feature hydration, model scoring, filtering, blending, and so on.

Within the complex feed construction process, independent development across different content types, discovery methods, and potentially teams can lead to code duplication, inconsistent patterns for interacting with external services, disjointed logging and observability, and additional maintenance burden. At Whatnot, our approach to solving this problem early on was to develop Whatamix, a platform for describing feed recommender systems as directed acyclic graphs (DAGs) with interchangeable nodes that can be used across different use cases.

Here, we’ll share how it works, how we built it, and what features we’ve found particularly useful—ultimately enabling us to significantly enhance the feed experience on Whatnot.

Problem Statement

From the moment a user first opens the app and lands on the ‘For You’ feed (FYF), feeds are a core part of the Whatnot experience. They are a primary driver of user engagement and livestream discoverability. However, our variety of feeds, such as FYF, category-specific feeds, feeds for followed content, and more, can serve very different content and be organized in different ways: some feeds may need more personalization, some may be more exploratory, some require insertion of different display modules such as shelves and banners that themselves may lead to other feeds or parts of the product. For example, see the event banner and popular show shelf in the images below.

This is necessary both to ensure a full user experience, letting users cater each app session to their current intent, and to meet business goals, letting us provide visibility for certain events, helping new sellers and categories gain traction, etc.

Our primary goal at Whatnot was to allow for a high degree of flexibility: developers should be able to craft feeds with different content, composed together in different ways, using consistent building blocks that can be reused across all feeds. This consistency provides meaningful value, specifically:

1. Reduces code duplication, increasing dev velocity and mitigating the risk of errors
2. Observability: all feeds can have the same caching, logging, and alerting mechanisms, making it easier to detect and resolve issues, and to have clean data across all our feeds
3. Provides common modules and levers that can be used on any feed once it’s been built for one: e.g., shelf insertions, retrieval and blending for different candidate sources, reranking by various attributes, etc.

Our secondary goal was performance. Before Whatamix, our feed construction was synchronous, where each component was generated sequentially, even if they had no dependencies on one another. Post-Whatamix, any pieces of feed logic that aren’t dependent on each other (e.g., retrieval from different candidate sources), are automatically run asynchronously.

Another secondary benefit that we only realized after the fact was that Whatamix rendered our SaaS vendors partially fungible. Whatnot is a relatively heavy user of SaaS, and in two separate instances, our vendors were acquired and we had to migrate off them in a short time frame. While Whatamix didn’t spare us the work of finding alternatives in these instances, the common DAG and node implementations used to interface with these SaaS vendors did make it significantly easier to switch to new alternatives across all our feeds.

Whatamix

Nodes

Whatamix is, at its core, a DAG orchestration layer where the different functional components of a recommendation system are represented as DAG nodes. Each node is a wrapper around one modular piece of logic: e.g., retrieving candidates from a data source, feature enrichment, model scoring, combining different candidate sets, etc. And a DAG can be composed of however many or however few nodes.

All nodes come with some generic benefits:

• Parallel execution: Any non-dependent nodes are run asynchronously
• Consistent error handling: developers can set a timeout on any node, as well as a flag for whether a failure of the node should be treated as a hard failure (fail the entire DAG run) vs a soft failure (catch the exception, log it, and move on).
• Consistent observability: DAG nodes have instrumentation for node latencies, success/error rates, and traces. We also have special node wrappers, which include sampled event logging of inputs and outputs.

As DAGs have rapidly increased in size and complexity, it has been important to facilitate and enforce DAG- and node-level timeouts and error handling. Most nodes within a DAG are not critical to the overall DAG succeeding, so developers are able to not only limit the maximum time spent executing a particular node, but also can supply node-level defaults in case of failure, limiting exposure to the rest of the DAG.

To illustrate what a DAG might look like in practice, a simplified DAG to construct a feed could be represented by the following five sequential nodes:

1. Retrieve candidates from ElasticSearch (ES).
2. Hydrate the candidates with a certain subset of features by doing lookup from an external key-value store.
3. Perform real-time model inference upon the candidates using a predictive model.
4. Sort the candidates by their predicted model scores.
5. Adjust the order of the candidates to achieve a desired category diversity.

In order to add a popular shows shelf to this feed, we would simply add a second “branch” to our DAG, which is a single node that retrieves candidates from ES ordered by popular score, followed by an insertion node that inserts the populated shelf into the ranked list of candidates from the other “branch” of the DAG. The only dependency between the two branches is at the insertion node which combines them, so the two can run asynchronously/”in parallel”.

By writing our feeds with these mostly generic nodes, we can avoid adding much custom code, without compromising the flexibility we have to build feeds with different layouts, requirements, and product goals.

In general, Whatamix DAG nodes fall into roughly 4 categories:

1. Retrieval
   a. Retrieving candidates from a number of different sources
2. Ranking
   a. Feature and score hydration
   b. Real-time model inference
   c. Value modeling
3. Business logic/twiddling
   a. Category diversity adjustments
   b. Static position insertion
   c. Splitting a list of candidates by feature value
   d. Combining two candidate lists together
4. Budgeting
   a. Bandits
   b. New Seller Exposure
   c. Ads

Since these nodes are often used across multiple feeds, implementing/improving a node can benefit many feeds at once. Most nodes also operate on what we call a “feed entity,” which is an abstract representation of any type of content we want to put in a feed, including livestreams, products, users, categories, and shelves/banners/other containers of the individual object types just listed. This ensures that we can easily create feeds of different content types or mix and match content types in a singular feed with only a single node implementation for any piece of functionality.

DAG Construction and DAG Execution

There are two important phases at feed request time for a Whatamix DAG:

1. DAG construction
2. DAG execution

During DAG construction, all of the nodes to service a feed request are fully specified, outlining all dependencies for each step necessary to build and return the user’s feed. Constructing our DAGs at the time of a request enables us to build different DAG graphs depending on the properties of the user, feed context, or any experiment group assignments.

The DAG is only executed after it’s fully constructed, meaning that prior to execution Whatamix automatically identifies any non-dependent node paths and then runs them asynchronously/”in parallel” during execution. As a result, developers modifying feed DAGs in Whatamix do not need to think deeply about how to maximize code “parallelism,” instead they only need to specify which outputs of upstream nodes are inputs to downstream nodes, and allow the Whatamix engine to decide when to schedule different nodes.

This pattern of construct before execution gives us the benefit of increased parallelism, but has the limitation that you cannot decide to remove or add nodes from a DAG during execution. While overall this has been a worthwhile tradeoff, it does force developers to write any logic dependent on upstream node outputs into node execution instead of DAG construction. For example, if node A returns apples and oranges, node B only operates on oranges, and we have a DAG of A → B → final result, node B will always be part of the DAG regardless of if node A’s results include any oranges. The logic to skip doing any operations if there are no oranges must be written into node B itself.

Params and A/B Testing

When constructing our DAGs, we may want to conditionally add or remove nodes for certain users or types of requests, whether that’s to improve the product experience or to meet specific business requirements. Some common examples are gradually rolling out a new predictive model, or adding some reranking business logic.

In a majority of cases, we make these changes via A/B testing (via Statsig, our A/B testing + dynamic config vendor), so that we can verify and measure the quantitative impacts of our changes. Since A/B testing is used pervasively throughout our feeds, and multiple A/B tests could be running at any time for a certain feed, we wanted to ensure that:

1. All experimental parameters are available in a consistent way throughout feed generation, during both DAG construction and execution
2. Experimental parameters are not looked up from Statsig multiple times for the same experiment since lookups can be expensive
3. The experimental parameters applied in a certain request can be easily logged for debugging purposes

To meet these requirements, we introduced the concept of “feed params” within Whatamix. At the start of the feed request, before DAG construction, we do a one-time lookup of all the applicable feed-related Statsig experiments and dynamic configs for the request. We combine these experimental values with hardcoded defaults and store the resulting parameter values in our “feed params,” a container of parameters that’s available throughout the rest of the feed request. These params can then be used during DAG construction to add or remove certain nodes, or during DAG execution to decide what logic should be run within a node. The param names, values, and source experiments are also automatically logged to allow for easy and consistent debugging across all our running experiments.

This allows developers to avoid adding their own lookup logic for each new experiment, and for node logic to be agnostic to which experiment, dynamic config, or hardcoded value a parameter value is sourced from.

For example, we may have a feed_params.ranking.model_id, where the actual model ID comes from experiment A for some users, experiment B for others, and doesn’t exist for others. Then during DAG construction, we will only add model scoring nodes to the DAG if feed_params.ranking.model_id exists, and will pass in the feed_params.ranking.model_id as an input to the scoring nodes. Then later on, if model ID A shows promising results and we want to launch it to production, we can simply hardcode feed_params.ranking.model_id to model ID A without making any further changes to the DAG itself.

Cross-team Collaboration

Cross-team collaboration can be trickier, since each team may have different goals, requirements, and deadlines. To make it easier to avoid experiment and code collisions, we encourage collaborating teams to create, if possible, self-contained sub-components of the DAG (sub-DAG).

To demonstrate, if we look at the problem of constructing a feed with ranked organic content and ads, our Discovery team is responsible for constructing the organic non-ads component of the feed, and the Ads team is responsible for running ads auctions, picking which ads should be inserted into a feed, and deciding where they should be inserted. The overall DAG is partitioned into 3 sub-DAGs, which together produce an ordered list of both organic livestreams and ads:

1. One which constructs the ranked organic feed
2. One which constructs a ranked list of ads
3. One which combines the organic feed and the ads into a single ordered feed

Developers can also partition out a subset of feed params for their specific sub-DAG, e.g. feed_params.ads vs feed_params.ranking. This partitioning means the different sub-DAG owners can run A/B tests on their sub-DAG without needing to know what A/B tests are running on other sub-DAGs.

Observability

We built multiple levels of logging and metrics into Whatamix.

1. We emit standardized metrics for latency, throughput and success/failure at the DAG- and node-level, which underpins our service-level monitoring.
2. We have standardized logging that captures detailed request inputs and constructed feed outputs for every Whatamix request.
3. For a sample of users, we emit detailed node-level inputs and outputs. Both the DAG-level and node-level inputs and outputs are ingested into our data warehouse for further analysis.

When new feeds, DAGs, or nodes are developed using Whatamix, developers get all of this instrumentation out of the box automatically, and usually don’t have to add custom metrics or logging of their own.

The crucial idea of our Whatamix logging is that every request has a unique ranking request identifier generated by Whatamix. This ranking request ID is sent back to the mobile and web clients along with the constructed feed, so that it is attached not only to the logs and events generated by Whatamix on the backend but also to the user engagement events on the frontend. This means that we can attribute downstream user actions such as livestream watches and purchases back to the originating feed and the ranking decisions that constructed it. More details will be provided and this will be explained further in a future blog post on our session id-based logging!

Contributing to Whatamix

Designing nodes for reuse is no simple task, and is often more art than science. While abstractions for some functionality are relatively obvious, for others, there is a balance to be struck between being general enough for reuse while specific enough that developers don’t need to make adjustments for each particular configuration.

To help facilitate these decisions, we established a core group of maintainers to make and review changes to the overall DAG construction and execution platform. These core maintainers also help design and review contributions from other developers for potentially reusable nodes.

Developers can also add “experimental” nodes meant for rapid execution when the reusable path is not obvious or immediately necessary. Everyone is free to use both reusable and experimental nodes, with the understanding that anything marked as “experimental” may not be supported by the core Whatamix maintainers, is likely customized for a specific use case, and may be subject to change. If eventually an experimental node proves to be a common functionality, Whatamix maintainers will work with the owner to clean up the API (if necessary) and promote it into the set of core reusable nodes.

Next Steps

We have a few exciting ongoing or upcoming initiatives:

• Search on Whatamix: Just as the recommendations/feed ranking is like performing search with a user context as the query, Search is like constructing a feed with the user context and an actual query.
• Whatamix offline replay at scale: We have offline Whatamix execution functionality for local development and e2e correctness testing, though the scale of the feed requests used for such uses is relatively limited. On the other hand, generating personalized daily email digests or deciding which notifications to send a user are nearly the same problem, though at a significantly larger offline scale.
• Whatamix debugger/gdb: Unlock the ability to “step through” a DAG offline using a specific set of feed requests and observe the inputs and outputs for each node. Similarly, allows for visual display and debugging of a DAG.

Whatamix has allowed many contributors to rapidly iterate on multiple feed and discovery use cases across Whatnot, and deliver significant improvements to the buyer and seller experience. It was built deliberately to solve practical challenges we face when scaling machine learning systems. Recently, we presented Whatamix for a roundtable discussion at the Workshop on Applied Machine Learning Management at KDD in Barcelona, sharing our experiences with fellow ML experts across the industry. If you are interested in practical applications of machine learning in real-world products, join us!