10 ms·
Chronon, Airbnb's ML feature platform, is now open source
- nikhilsimha 3y agoAuthor. Happy to answer any questions.
- dundun 3y agoHow does this relate to Zipline and Bighead? Does it replace those projects or is it a continuation of them?
- echrisinger 3y agoI'd imagine a continuation... he is also the author of Zipline
- nikhilsimha 3y agoBighead is the model training and inference platform. Chronon is a full re-write of zipline with 1) a different underlying algorithm for time-travel to address scalability concerns. 2) a different serde and fetching strategy to address latency concerns.
- morkalork 3y agoAt what size of team, features or number of models would you say the break even point is for investing time into using this platform?
- nikhilsimha 3y agoOffline is pretty easy to get started with. It should take less than a week to set it up for new use-cases across the company. (You can begin building training-sets if offline is setup) Online is a bit more involved - you need a month or more to test that your KV store scales against traffic coming from chronon for reads and writes.
- andscoop 3y agoI noticed airflow as the backing orchestration service. Was there any consideration for another orchestration tool? I know Airbnb has at least two internally, but also that airflow is the predominant one for the data org still.
- nikhilsimha 3y agoAirflow is the current implementation since it is the paved path at airbnb. But we are open to accepting contributions for other orchestrators. Someone mentioned they wanted to add cadence support.
- echrisinger 3y agoHow do you/AirBnB handle deeply linked features (2-hop+?) that are also latency sensitive? Maybe I'm missing something, but I don't imagine that with the transformation DSL described in Chronon. For our org, those are by far the most complicated to handle. Graph DBs are kind of scaling poorly, while storing state in stream processing jobs is way too large/expensive. Those would also be built on top of API sources, which then lead us to the unfortunate "log & wait" approach for our most important features
- nikhilsimha 3y agowe call this chaining. In the API itself - you could specify the chain links by specifying the source. To be precise - a GroupBy(aggregation primitive) can have a Join(enrichment primitive) as a source. To rephrase, you can enrich first and then aggregate and continue this chain indefinitely. > Graph DBs are kind of scaling poorly That makes sense. Since you scaling these on the read side it is much much harder than pre-computing on the write side. (That is what Chronon allows you to do)
- echrisinger 3y agoI'm also curious how you went from a non-platformatized approach to adopting this platform; what were the important insights for strategizing, prioritizing, motivating teams to lift existing pipelines into the new thing? Open ended question
- nikhilsimha 3y agoThere were two main drivers - - inability to back-test new real-time features. People were forced to log-and-wait to create training sets for months. Chronon reduces this to hours or days. - the difficulty of creating the lambda system (batch pipeline, streaming pipeline, index, serving endpoint) for every feature group. In chronon, you simply set a flag on your feature definition to spin up the lambda system.
- whiplash451 3y agoFirst of all, congrats on the release! Well done. A few questions: - Since the platform is designed to scale, it would be nice to see scalability benchmarks - Is the platform compatible with human-in-the-loop workflows? In my experience, those workflows tend to require vastly different needs than fully automated workflows (e.g. online advertising)
- nikhilsimha 3y agore: scalability benchmarks - we plan to publish more benchmark information against publicly available datasets in the near future. re: human-in-the-loop workflows - do you mean labeling?
- Reubend 3y agoLooks very useful. I'm not aware of any open source alternative (although I could just be ignorant here!)
- nikhilsimha 3y agoFeathr from linkedin is the closest. But there doesn't seem to be much recent activity on the project.
- dundun 3y agoThis is the biggest one: https://feast.dev/ https://feast.dev/
- echrisinger 3y agoThis isn't really a drop-in replacement; they don't offer transforms out of the box. Admittedly some of the transforms proposed in this article are a little simple & don't represent the full space of feature eng requirements for all large orgs
- econometrician 3y agoActually feast does support transformations depending upon the source. It supports transferring data on demand and via streaming. It does not support batch transformation only because technically it should just be an upload but we can revisit that decision.
- xLaszlo 3y agoI think feast is sunsetted
- econometrician 3y agoThere are new maintainers: https://feast.dev/blog/the-future-of-feast/ https://feast.dev/blog/the-future-of-feast/
- 3y ago
- syntaxing 3y agoWhy do major sites still use Medium as a blog platform.
- nikhilsimha 3y agoI am with you on this one.
- dartos 3y agoFree income?
- ttul 3y agoUgh yes. The first thing I see on clicking the link is an overwhelming login/join pop-over. I’m never visiting that blog again…
- appplication 3y agoSubstack is the same
- seattle_spring 3y agoDon’t let having to tap “x” a single time ruin your day. You’re missing out on a lot of good stuff.
- dumbo-octopus 3y agoWild that it's 2024 and you still don't have UBlock Origin.
- DandyDev 3y agoMaybe they opened the link on Safari on iOS like me?
- ayhanfuat 3y agoDisabling JavaScript helps with that (sometimes they don't show the full article if JS is disabled though).
- xiasongh 3y agoHow does Chronon handle mutable data when backfilling? Or does it make some assumptions on the underlying data?
- nikhilsimha 3y agoBy mutable data do you mean - change data coming from OLTP databases? If yes, we do this via the EntitySource api. https://www.chronon.ai/authoring_features/Source.html#streaming-entitysource https://www.chronon.ai/authoring_features/Source.html#stream...
- xiasongh 3y ago> By mutable data do you mean - change data coming from OLTP databases? Yes, exactly! I see there is some kind of support, but is it possible to use the OLTP database as an event source? For example, say I had a table in my OLTP database, `data.returns`, that had columns `ts`, the event time, and `status`, which can be PENDING or COMPLETED. I'd like to generate point-in-time correct training data, where the feature is the count of completed returns. It seems like all the necessary information to calculate this is there
- sfink 3y agoIt's refreshing to read something about ML and inference and have it not be anything related to a transformer architecture sending up fruit growing from a huge heap of rotten, unknown, mostly irrelevant data. With traditional ML, it's useful to talk about the sources of bias and error, and even measure some of them. You can do things that improve them without starting over on everything else. With LLMs, it's more like you buy a large pancake machine that you dump all of your compost into (and you suspect the installers might have hooked up to your sewage line as input too). It triples your electricity bill, it makes bizarre screeching noises as it runs, you haven't seen your cat in a week, but at the end out come some damn fine pancakes. I apologize. I'm talking about the thing that I was saying was a relief to be not talking about.
- nikhilsimha 3y agoI agree with you - about the sentiment around the GenAI megaphone. FWIW, Chronon does serve context within prompts to personalize LLM responses. It is also used to time-travel new prompts for evaluation.
- cactusplant7374 3y ago> time-travel new prompts for evaluation What does this mean?
- nikhilsimha 3y agoImagine you are building a customer support bot for a food delivery app. The user might say - I need a refund. The bot needs to know contextual information - order details, delivery tracking details etc. Now you have written a prompt template that needs to be rendered with contextual information. This rendered prompt is what the model will use to decide whether to issue a refund or not. Before you deploy this prompt to prod, you want to evaluate its performance - instances where it correctly decided to issue or decline a refund. To evaluate, you can “replay” historical refund requests. The issue is that the information in the context changes with time. You want to instead simulate the value of the context at a historical point in time - or time-travel.
- giovannibonetti 3y agoWhat is the difference between a ML feature store and a low-latency OLAP DB platform/data warehouse? I see many similarities between both, like the possibility of performing aggregation of large data sets in a very short time.
- uoaei 3y agoFeature stores are more for fast read and moderate write/update for ML training and inference flows. Good organization and fast query of relatively clean data. Data warehouse is more for relatively unstructured or blobby data with moderate read access and capacity for massive files. OLAP is mostly for feeding streaming and event-driven flows, including but not limited to ML.
- jamesblonde 3y agoYou need the columnar store for both training data and batch inference data. If you have a batch ML system that works with time series data, the feature store will help you create point in time correct training data snapshots from the mutable feature datab(no future data leakagae), as well as batch inference data. For real-time ml systems, it give uou row oriented retrival latencies for features. Most importantly, it helps modularize your ML system into feature pipelines training pipelines, and inference pipelines. No onolithic ML pipelines.
- nikhilsimha 3y agothe ability generate training sets against historical inferences to back-test new features another one is the focus on pushing as much compute to the write-side as possible (within Chronon) - specially joins and aggregations. OLAP databases and even graph databases don't scale well to high read traffic. Even when they do, the latencies are very high.
- giovannibonetti 3y agoYou may want to take a look at Starrocks [1]. It is an open-source DB [2] that competes with Clickhouse [3] and claims to scale well – even with joins – to handle use cases like real-time and user-facing analytics, where most queries should run in a fraction of a second. [1] https://www.starrocks.io/ https://www.starrocks.io/ [2] https://github.com/StarRocks/starrocks https://github.com/StarRocks/starrocks [3] https://www.starrocks.io/blog/starrocks-vs-clickhouse-the-quest-for-analytical-database-performance https://www.starrocks.io/blog/starrocks-vs-clickhouse-the-qu...
- travisporter 3y agoPaywalled for me
- nikhilsimha 3y agoIt opens for me in incognito mode - albeit with a large popup that I had to close.
- djaykay 3y agoThe downside is after you use the platform for a week, you have to delete all the expired models yourself and clean up all the labels or face a hefty housekeeping surcharge.
- deleted 3y ago[deleted]
- evolutionblues 3y agogreat work! When it comes to batched computations, why not leverage intermediate state much like streaming jobs. For example, if we need to calculate past 30 day sum for a value daily - it seems like this would compute so from scratch daily. Would it not make sense to model this as a sliding window that's updated daily?
- nikhilsimha 3y agoWe do this for training data generation already. We have plans to implement this behavior for computing the batch arm of feature serving.
- evolutionblues 3y agocan you point to documentation/ examples which talk about it - I could not find this while exploring the pages (or maybe I missed it altogether).
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- siquick 3y agoWhat does Airbnb use ML for?
- nikhilsimha 3y agoalmost every button click is either powered by a model or guarded by a model.
- jumpora 3y ago[dead]