Chronon, Airbnb's ML feature platform, is now open source
Airbnb has open-sourced Chronon, its internal machine learning feature platform designed to generate point-in-time-correct training data and serve low-latency features using a lambda-style architecture. Commenters contrast it with traditional OLAP databases and existing feature stores like Feast, Feathr, and Tecton, emphasizing Chronon’s focus on moving computation to the write path, composable aggregations and joins, and scalable “time travel” for backtesting features and prompts. The thread also touches on practical adoption questions, such as required team size, orchestration choices, and handling mutable OLTP data and complex, chained features.
Chronon’s purpose and architecture
- Chronon is a full rewrite of an earlier feature platform, built to improve scalability of time-travel and reduce serving latency via new algorithms and serialization/fetch strategies.
- Uses a lambda-style architecture: precomputed intermediate tables for batch/snapshot accuracy plus streaming updates for real-time features.
- Features are expressed via composable primitives (e.g., GroupBy aggregations, Join enrichments), which can be chained to handle multi-hop feature logic.
Adoption, scale, and operations
- Offline-only setup for new use cases is described as fast (under a week).
- Online serving requires more work: teams must validate that their key–value store can handle Chronon’s read/write patterns, typically over a month or more.
- Platform was motivated by:
- Inability to back-test real-time features without “log and wait” for months.
- High cost of building separate batch, streaming, indexing, and serving pipelines for each feature group; Chronon automates this via configuration.
Comparisons to other tools
- Other feature stores mentioned: Feast, Hopsworks, Feathr, Tecton.
- Some commenters argue Chronon isn’t a drop-in replacement for Feast; Feast maintainers say it does support some transforms and is still active.
- Chronon’s authors highlight differences from Tecton (time-travel scalability, composable primitives) and from “pure kappa” systems (need for lambda to fix upstream errors and support GDPR deletions).
Feature store vs OLAP/data warehouse debate
- One side sees feature stores as largely rebranded mixtures of existing DB technologies (relational, KV, OLAP, etc.).
- Others argue feature stores add:
- Point-in-time correct, windowed aggregations for millions of timestamps.
- Systematic sharing and reuse of partial aggregates.
- Push-compute-to-write for low-latency serving, rather than expensive read-side OLAP queries.
- StarRocks, ClickHouse, Rockset, and temporal DBs are cited; some report scalability limits with these for this use case.
Time-travel, mutable data, and caching
- “Time travel” means reconstructing feature values as of the exact time of past inferences for backtesting.
- Snapshots and timestamped tables are noted as partial solutions; defenders say they don’t cover the scale and precision Chronon targets.
- Chronon automatically decides which intermediate results to cache and claims sub-linear scaling of backfills by reusing intermediate state.
- There is support for mutable OLTP data via an “EntitySource” API, but details for specific patterns (like evolving status fields) are not fully clarified.
LLM and human-in-the-loop use
- Chronon can provide contextual features to LLM prompts and perform “time-travel evaluations” by replaying historical requests with historical context.
- A hypothetical refund-bot example sparked concern about delegating refund decisions to LLMs; others emphasize using LLMs mainly for context-gathering and triage, with humans in the loop.
Miscellaneous topics
- Airflow is the default orchestrator; the project is open to contributions for others (e.g., Cadence).
- Chronon reportedly powers many ML-guarded or ML-driven UI actions at Airbnb and is also used for some non-ML workloads.
- Several comments complain about Medium’s UX; alternatives like scribe.rip and the GitHub README are shared.