6x Faster ML Inference: Why Online >> Batch

This is part of our ML blog series and pairs well with Evolving Feed Ranking at Whatnot, which explains our For You Feed recommendation system and its crucial role in connecting users to relevant livestream auctions.

TLDR; Our batch ML inference system was collapsing under explosive growth, losing 5% coverage weekly as we tried to score 10 billion+ user-seller pairs daily. We transformed it into a real-time system that serves millions of predictions in <200ms p99 latency with >99.9% reliability. Key optimizations included: migrating from DynamoDB to Redis (3x latency improvement), compiling Gradient Boosting Decision Tree(GBDT) models to native C binaries (20x faster inference), implementing gRPC for data transmission (6.7x improvement), and systematically tuning thread pools. The result: 5.8x overall latency reduction while capturing millions in GMV that would have been lost to coverage gaps.

The Breaking Point

Our machine learning pipeline was suffocating.

Every day, our batch inference system scored 10 billion+ user-seller pairs to power personalized recommendations across our marketplace. But with our explosive user growth, coverage was slipping by 5% weekly. Each percentage point of lost coverage translated to approximately six figures in unrealized GMV per month. We were fighting a losing battle against scale.

To support our company’s crazy growth, we needed a system that could serve predictions for arbitrary buyer-seller pairs in under 200ms p99 and handle 300+ queries per second(QPS) at peak traffic, running inference against feature matrices of size 1500x150, all while maintaining at least three 9’s Service Level Objective(SLO) success rate with potential for growth to four. An online system was the natural choice, as we could know at inference time which buyer-seller pairs were relevant to infer against instead of running all possible pairs.

The Engineering Puzzle

Our batch system followed a daily cycle: dbt models in Snowflake calculated features, batch jobs exported them to S3, SageMaker batch transform jobs generated predictions, and finally, scripts loaded these predictions into Rockset, where they could be queried as needed by our backend. The entire process took up to 18 hours, uncomfortably close to 24 hours which is obviously not ideal for a daily job.

When we embarked on this journey in late 2023, our ML platform essentially did not exist. Our Discovery team had built bespoke training and batch inference pipelines for their models, but otherwise we had very little. We framed the gap between us and online inference in three components: a feature store for point in time correctness and low-latency feature retrieval, scalable model serving infrastructure, and a client to integrate into the app backend.

We considered several approaches:

1. The DIY route: Deploy an open-source feature store and model serving layer to Kubernetes. This option offered maximum control, and colocation of online inference components with our backend, but carried the highest operational burden.
2. The AWS-native approach: Leverage AWS SageMaker Feature Store for feature management and SageMaker Endpoints for model serving. The implementation of this option was simpler, but more costly. We also had concerns about whether AWS’s out-of-the-box tooling would match our future requirements.
3. The hybrid solution: Combine a third-party feature platform (for feature definitions, processing, and serving) with AWS SageMaker Endpoints (for model hosting), connected by a lightweight client library.

Given infinite resources and time, we would have chosen the DIY route, as it would have all but guaranteed our ability to support arbitrary use cases at relatively low cost. However, due to the company’s rapid growth and our team of just three, we had to move uncomfortably fast. To strike a balance between speed and scalability, we opted for a middle ground with a hybrid architecture.

Solution Architecture

Find below a diagram of the online portion of the system we designed. We’ll go into more detail about offline, supporting systems later. Our initial architecture, stripped of bells and whistles like observability, a/b testing, feature sharing, feature distribution monitoring, load testing, and interfaces, looked like this:

Whatamix, a Whatnot system that blends signals into coherent recommendations, gathered buyer and seller IDs for ranking and sent them to a SageMaker endpoint which hosted a model using TorchServe. The size of this payload varied over time, but at the time of design it comprised 1 buyer ID and roughly 1500 seller IDs at peak traffic.

SageMaker received IDs, marshalled them into a feature store query (powered by Chalk), and dispatched it to a query server. The query server generated a query plan, split the IDs into batches, and for each batch, fetched approximately 150 columns of features from DynamoDB. From there, the query server would return features to SageMaker, which would run inference against them and return predictions along with ids back to Whatamix, where they could be used as signals in feed ranking. Under the hood, features were materialized daily from our data warehouse to our offline and online store.

This architecture was a journey to achieve, and presented significant technical challenges, each of which was its own story–more on that below.

Implementation Journey

Initial System Performance

Our first working version included several key components:

• An asynchronous HTTP client within our Whatamix service
• A standardized deployment framework for SageMaker endpoints
• Traffic-responsive autoscaling machinery
• Feature infrastructure (query server, HTTP client, online store, feature materialization machinery) from Chalk
• Centralized repositories for feature definitions, deployments, and data materialization triggers

While functional, this system fell short of our targets. Success rates barely hovered above 98.9%, and latency peaked at 700ms p99 which is far from our target SLOs. Our investigation revealed four critical bottlenecks:

1. Our Whatamix client exhibited periodic latency spikes every 3600 seconds
2. Model inference showed excessive latency at the tails
3. DynamoDB showed unexpectedly high tail latency (~250ms p99) for parallel batch operations, while transmitting large feature matrices (1500×150) created substantial network overhead.
4. Endpoints weren’t scaling properly during traffic spikes, causing request queues to fill and generating 503 errors

So we got cracking!

Optimization 1: Whatamix Client

With some sleuthing, we uncovered that our client’s cross account credentials(provisioned by AWS STS) were expiring every 3600 seconds. When this happened, the client made a high-latency synchronous call to refresh these credentials, causing predictable hourly spikes.

The fix was straightforward: we implemented an in-memory credential cache and a background refresh timer that proactively renewed credentials before expiration. This eliminated the spikes and protected us from potential AssumeRole limits at the account level, which could have caused throttling for all clients if exceeded.

Optimization 2: Model Inference

Our model endpoints were experiencing severe performance issues, 700ms p99 latency with maxed-out CPU utilization despite running on high-core machines. This created a classic performance puzzle that requires systematic investigation.

Discovery Phase: Our analysis revealed three critical bottlenecks:

1. Threadpool oversubscription: CPU utilization remained maxed out regardless of worker count, indicating resource contention
2. Inefficient model implementation: Our Gradient Boosting Decision Tree(GBDT) model wasn’t optimized for the large feature matrices we were processing
3. Synchronous blocking: Workers were stuck waiting for slow feature fetches (500ms p99), creating cascading delays

Systematic Approach: Rather than guessing at solutions, we built a comprehensive load testing framework that deployed multiple endpoint configurations and systematically tested different parameter combinations. This allowed us to isolate the sources of contention and find optimal resource allocation through grid search methodology.

The Breakthrough Solution: Our systematic approach yielded dramatic improvements:

1. Resource optimization: Through systematic parameter tuning, we found the optimal balance between worker threads and system resources, reducing CPU contention
2. Model compilation: We implemented a compilation step in our deployment pipeline that converted our models to Treelite format and compiled them to native C binaries using tl2cgen, a technique that dramatically improved inference performance for large feature matrices

Results: These optimizations reduced p99 latency from 700ms to 400ms (over 40% improvement) while significantly improving resource utilization. With model inference no longer the bottleneck, our attention could turn to the remaining performance constraint: feature fetching.

Optimization 3: Feature Fetching

We determined two primary bottlenecks affecting latency in our feature fetches:

• DynamoDB’s Problematic Tail Latency: While DynamoDB performed reasonably well for single batch operations (with p99 latency of single-digit milliseconds), we discovered a critical insight about its behavior in our multi-batch context. When our system needed to fetch 100 batches in parallel with all needing to succeed before query completion, DynamoDB’s long tail latency above p99 became our Achilles’ heel. In this scenario, what should have been a p99 latency guarantee effectively degraded to p90, resulting in observed query server latencies around 250ms p99.
• Inefficient Data Transmission Protocol: The HTTP/1.1 protocol between our model endpoint and query server was creating unnecessary overhead when transmitting large feature matrices.

Our Two-Pronged Solution:

Data Store Migration: We replaced DynamoDB with Redis for our online feature store. Redis provided more consistent performance at scale, with dramatically tighter tail latency for batch fetches. This change alone substantially reduced our feature fetch latency.

Protocol Optimization: Working with our feature store vendor, we implemented a gRPC client (built on HTTP/2) for feature store queries. This protocol change dramatically improved data transmission efficiency for our large payloads.

The combination of these optimizations, along with strategic caching mechanisms in the query server, brought our fetch latency down to a very respectable 80ms p99, a 3x improvement from our starting point.

As a future direction, we’re exploring compression strategies specifically designed for recommendation system feature matrices, which tend to be extremely sparse.

Optimization 4: Endpoint Scaling

Initially, we approached scaling our model endpoints using a standard QPS-based autoscaling strategy. However, this led to frequent 503 errors (indicating full TorchServe request queues) that followed a distinct daily pattern.

Our investigation uncovered a counterintuitive insight: endpoint performance wasn’t primarily determined by query volume. Instead, with feature fetching consuming most of worker time, the critical factor was feature matrix size which varied throughout the day based on how many sellers were live on our platform.

Rather than immediately optimizing our scaling strategy, we took a strategic approach:

1. First, we addressed the feature fetch latency issues described above
2. Once feature fetch tails were tightened, endpoint performance became more predictable and QPS-dependent
3. We then conducted systematic load testing to identify appropriate QPS targets
4. Finally, we reconfigured our autoscaling policies based on these findings

This approach eliminated our 503 errors without requiring complex scaling logic, an elegant solution to what initially seemed like a difficult problem.

While embedding feature fetches inside our endpoint initially simplified our architecture by containing the query server interface within our inference code, it created this scaling complexity. Our future roadmap includes plans to disentangle this interface, which should further improve scaling performance and flexibility.

From Suffocating to Soaring

Our journey from batch to online inference transformed not just our technical capabilities, but our entire approach to ML operations at Whatnot. Through methodical optimization and strategic architectural decisions, we achieved remarkable results:

• Reliability: >99.9% success rate, meeting our three 9’s SLO target
• Performance: 120ms p99 latency, well under our 200ms requirement—down from ~700ms, representing a significant speedup.
• Scalability: From struggling with coverage loss to confidently serving millions of predictions daily.

These metrics represent more than technical achievements, they translate directly to business value. By maintaining complete coverage of user-seller pairs while supporting our explosive growth, we’ve captured millions in GMV that would otherwise have been lost to scaling limitations.

Enjoyed reading about our machine learning platform evolution? If diving into exciting challenges at a rapidly growing startup sounds like your next move, explore career opportunities at Whatnot today!