Recap: Whatnot at Data Council 2023

Whatnot’s modern data stack processes tens of millions of events across hundreds of different event types daily. At the Data Council conference, Whatnot data engineers Alice Leach and Zack Klein discussed managing and scaling the dynamic and growing system powering our social commerce platform.

Extreme Self-Service: Turning Data Consumers into Data Constructors, Alice Leach

Watch “Extreme Self-Service: Turning Data Consumers into Data Constructors | Whatnot” on YouTube

Small data teams invariably have a supply and demand problem. From requests for new data products to the classic ‘why does my data look funny?’, triaging and prioritizing data work can become a full time job. However, in many cases with a small amount of training, data consumers can create their own data products. How do we help data consumers to help themselves, without hurting everyone else? In this talk, Alice shares lessons learned from taking an ‘extreme’ approach to self-service analytics: developing and sharing best practices, building a data community, encouraging SQL literacy, and constructing robust guard rails.

View Alice’s slides here.

Data Contracts in the Modern Data Stack, Zack Klein

Watch “Data Contracts in the Modern Data Stack | Whatnot” on YouTube

After two years of rapid growth, how does a small team keep up? Here, Zack explores data contracts—how we use Protobufs to serve as the source of truth for event definitions, govern event construction in production, automatically generate dbt models in the data warehouse, and more.

View slides here.

Keep an eye on the blog for more on this topic from Zack.

If data engineering opportunities like those shared at the conference interest you, check out the open roles on the Whatnot team!