3 ms·
Data Catalogs Are Dead; Long Live Data Discovery
- kfk 6y agoSome points are good but I challenge the assumption that our data is a mess because of “unstructured data”. In fact the one example the author makes is of highly structured data - salesforce. The issue I see on my end is that we keep buying solutions focusing mostly on functionality and not evaluating how such solutions store and manage the data they produce. Lack of integrated solutions results in complicated data pipelines due to having to jump through a lot of hoops to blend thing together.
- aynyc 6y agoI look after a lot of data. We have gone thru many different ideas and solutions. Data catalog is the only working solution we found after we POC ton of vendor solutions. I'm not AI scientist, but features generated by models are not structured?
- cratermoon 6y agoPerhaps this will shed some light: https://oc.ac.ge/file.php/16/_1_Shirky_2005_Ontology_is_Overrated.pdf https://oc.ac.ge/file.php/16/_1_Shirky_2005_Ontology_is_Over...
- alextheparrot 6y agoAnyone have a good data lineage story while still using mainstream data processing and ETL tools like Spark/Flink/etc.? I assume they may be selling something like that, but their "integration with my existing data stack" is pretty opaque. To me, that's the core part (As the article also mentions) that allows data discovery. Otherwise you're just guessing with heuristics [0]. It also allows someone to go from map [Input Table(s) => This Table's Fields] which I think is half the battle in understanding a table. [0] Not to say guessing is the worst, just that the information is already in the ETL code so why are we guessing?
- svordsmith918 6y agoHave you heard of Monte Carlo? Data lineage + data catalog + data anomaly detection: https://www.montecarlodata.com/product/ https://www.montecarlodata.com/product/
- alextheparrot 6y agoYeah, that's who the post was written by (Which I actually did read). > I assume they may be selling something like that, but their "integration with my existing data stack" is pretty opaque. The issue is that I have no clue what that actually looks like in practice and I'm not going to request a random demo to find out.
- inshadows 6y agoI gather that data catalogs, a rectangle on the diagram, is dead and data discovery, an ellipse on the diagram takes over. Is there anything more to this article than geometric shape of buzzword containers?