5 ms·
Data Warehousing 101
- billswift 15y ago>If you're building an archive, your only requirements are to minimize storage cost and to make sure the archive can keep up with the generation of data. And in some of the cases he mentions be really. really certain you don't lose data, since some of the laws impose criminal penalties on data loss, and not necessarily even on the most responsible parties (legislatures have been getting increasingly psychotic this way).
- pratikpatel 15y agoThe last stage of enterprise integration with the DW is through Data Marts, which are organized into Dimensions and Facts, and allow for dynamic interfaces for business users to mine their data. My current project is using Informatica CDC (Change Data Capture) to read multiple source databases through their logs and aggregate in real-time. Its really incredible and enables any level of intricate reporting requirements.
- r00fus 15y agoThe article talks about Informatica as an ETL tool provider... I wouldn't agree it's the "last stage" either, ETL is critical to any (E)DW.
- gaius 15y agoCan't mention MapReduce without Oracle Coherence (nee Tangosol)
- thibaut_barrere 15y agoOldies but goodies http://philip.greenspun.com/wtr/data-warehousing.html http://philip.greenspun.com/wtr/data-warehousing.html (data warehousing for cavemen)
- T_S_ 15y agoThis article seems a little out of date. It's missing most of the things I have looked at over the past year. E.g. Riak, MongoDB, Redis and so on.
- xal 15y agoShopify is pondering open-sourcing our internal tool called Tiller. It runs all the reporting for our considerable data warehouse efforts, yet it's lightweight and super fast to get running. Watch this space.
- mumrah 15y agoHive is a really slick DW tool built on top of Hadoop. It has a SQL-like language and supports typical DW techniques like table partitioning, key clustering, etc.
- eneveu 15y agoIf you are interested in Data Warehousing, you should read Ralph Kimball's "The Data Warehouse Toolkit": http://www.amazon.com/Data-Warehouse-Toolkit-Complete-Dimensional/dp/0471200247 http://www.amazon.com/Data-Warehouse-Toolkit-Complete-Dimens... When I started learning about BI (Business Intelligence), a few members of the Pentaho community advised me to read this book. I'm glad I did. Kimball is one of the "fathers" of data warehousing, and his book had a lot of great insights for dimensional modeling. It helped me avoid many design mistakes while building my DWH, and gave me insight I might have taken years to discover. It's a "theoretical" book, in the sense that it does not focus on any specific technology; it's also a "practical book", because he uses real-world scenarios (inventory management, e-commerce, CRM...) to demonstrate the various dimensional modeling techniques. I also liked the part about BI project management and encouraging BI in a company (= how to engage users and how to "sell" a BI project to management). He also has a newsletter with many DWH design tips (archives here: http://www.kimballgroup.com/html/07dt.html http://www.kimballgroup.com/html/07dt.html ).
- ajtaylor 15y agoAs a data warehouse newbie, this was an excellent introduction. I've heard a few of the names mentioned, but there are lots of new faces to explore.
- orenmazor 15y agoI'd like to learn more about this. can anybody recommend some cool projects (open source, or even just ideas) for me to explore?
- dgudkov 15y agoThis isn't a good article about data warehousing 101. I've been working in data warehousing since 2004. The core thing in DW is DWH data model because it's actually abstraction layer than converts raw transactional data into meaningful, consistent, correct and persistent representation of an organization's activity. Tools (including mentioned in the article) are just means to achieve that goal.