3 ms·

Having recently worked on a data team at FAANG, all this is an ops nightmare for the team running the platform itself if you want to ensure data quality for eve

Having recently worked on a data team at FAANG, all this is an ops nightmare for the team running the platform itself if you want to ensure data quality for everyone querying the data. Im talking when you have hundreds of data sources and hundreds of query use cases. Anyone have any solutions you've tried?

bushbaba 5y ago
Checkout Apache Iceberg. Does a great job of handling many readers few writers. With data consistency and query consistency. It’s a great approach for your data lake and data warehousing needs.
- faizshah 5y ago
  This timetravel/rollback feature is really interesting: https://iceberg.apache.org/docs/latest/spark-queries/#time-travel https://iceberg.apache.org/docs/latest/spark-queries/#time-t...
atwebb 5y ago
FAANG seems to be an outlier but, it sounds a lot like the enterprise data mart strategy covered under a mix of stuff from principle #1. If you want quality, you need structure and review. Accessible data is helpful and needed to develop some of the mature processes, but for most day to day analysis/reporting, no one wants to create their own data model from scratch. Lots of FAANG doesn't apply to any other companies so it may just be a case of having a wholly unique use case. Though I'm surprised there isn't something already in place at this point (of course having very little knowledge of the case). For the dims/facts/marts, they tend to be business use case focused and not source/data which can reduce the targets down significantly since business use cases tend to repeat (or rhyme).
cgio 5y ago
Don’t have anyone querying the data but have everyone. Trying to line up definitions, dq requirements etc. ends up with internalisation of the proliferation of interfaces of explicit producer consumer relationships. Therefore, don’t be scared to go for point to point. That’s the spirit of elt after all, but we tend to always strive for am imaginary perfect data model.
deleted 5y ago
[deleted]

bushbaba 5y ago

Checkout Apache Iceberg. Does a great job of handling many readers few writers. With data consistency and query consistency. It’s a great approach for your data lake and data warehousing needs.

faizshah 5y ago
This timetravel/rollback feature is really interesting: https://iceberg.apache.org/docs/latest/spark-queries/#time-travel https://iceberg.apache.org/docs/latest/spark-queries/#time-t...

atwebb 5y ago

FAANG seems to be an outlier but, it sounds a lot like the enterprise data mart strategy covered under a mix of stuff from principle #1. If you want quality, you need structure and review. Accessible data is helpful and needed to develop some of the mature processes, but for most day to day analysis/reporting, no one wants to create their own data model from scratch. Lots of FAANG doesn't apply to any other companies so it may just be a case of having a wholly unique use case. Though I'm surprised there isn't something already in place at this point (of course having very little knowledge of the case). For the dims/facts/marts, they tend to be business use case focused and not source/data which can reduce the targets down significantly since business use cases tend to repeat (or rhyme).

cgio 5y ago

Don’t have anyone querying the data but have everyone. Trying to line up definitions, dq requirements etc. ends up with internalisation of the proliferation of interfaces of explicit producer consumer relationships. Therefore, don’t be scared to go for point to point. That’s the spirit of elt after all, but we tend to always strive for am imaginary perfect data model.

deleted 5y ago

[deleted]