4 ms·
We haven’t tried materialize - IIUC materialized is pure kappa. Since we need to correct upstream data errors and forget selective data(GDPR) automatically - we
by nikhilsimha 2y ago
We haven’t tried materialize - IIUC materialized is pure kappa. Since we need to correct upstream data errors and forget selective data(GDPR) automatically - we need a lambda system.
Tecton, we evaluated, but decided that the time-travel strategy wasn’t scalable for our needs at the time.
A philosophical difference with tecton is that, we believe the compute primitives (aggregation and enrichment) need to be composable. We don’t have a FeatureSet or a TrainingSet for that reason - we instead have GroupBy and Join.
This enables chaining or composition to handle normalization (think 3NF) / star-schema in the warehouse.
Side benefit is that, non ml use-cases are able to leverage functionality within Chronon.
- jamesblonde 2y agoFeatureSets are mutable data and TrainingSets are consistent snapshots of feature data (from FeatureSets). I fail to see what that has to do with composability. Join is still available for FeatureSets to enable composable feature views - join is resuse of feature data. GroupBy is just an aggregation in a feature pipeline, not sure your point here. You can still do star schema (and even snowflake schema if you have the right abstractions).
- jamesblonde 2y agoNormalization is a model-dependent transformation and happens after the feature store - needs to be consistent between training and inference pipelines.
- nikhilsimha 2y agoNormalization is overloaded. I was referring to schema normalization (3NF etc) not feature normalization - like standard scaling etc.
- jamesblonde 2y agoOk, but star schema is denormalized. Snowflake is normalized.
- nikhilsimha 2y agoTo be pedantic, even in star schema - the dim tables are denormalized, fact tables are not. I agree that my statement would be much better if used snowflake schema instead.
- jvican 2y agoThank you for sharing!
- throwaway2037 2y agoWhat is the meaning of pure kappa?
- pedrosorio 2y agohttps://learn.microsoft.com/en-us/azure/architecture/databases/guide/big-data-architectures#kappa-architecture https://learn.microsoft.com/en-us/azure/architecture/databas...
- _mdb 2y ago[dead]