I wouldn't say it's a "lack of carpenters", but (to continue the metaphor) a lack of good carpenters who work in nice woodshops. In other words, it's failure t

I wouldn't say it's a "lack of carpenters", but (to continue the metaphor) a lack of good carpenters who work in nice woodshops. In other words, it's failure to hire high quality people to implement high quality ETL systems. I know plenty of teams are using Flume/Storm/etc, with unit tests, code reviews, etc. But that's like <0.01% of all ETL-related jobs on Earth. For everyone else, the biggest challenge with BI is doing basic, absolutely trivial stuff the right way. Why? (Forgive my generalizations, I'm too cynical to tone them down). ETL is usually done by inexperienced people because it's unrewarding, uncreative work that doesn't require a lot of skill to do. With remedial SQL skills and some Excel magic, a high school dropout can do ETL tasks that are up to par with businessy (read: not Engineering) standards. Why are Engineering standards not followed for ETL? Because ETL happens whenever EndangeredMiddleManagerX says "We need this new report because SomeGuy wants to feel relevant/smart at his next meeting." If they can produce that report, even once, and the numbers are believable, everyone gets a pat on the back. Repeat ad nauseam. In that kind of work environment, you can't hire a decent engineer even if you try. Best case, you get someone with ~2 yrs of experience who will leave the moment they find something better. Everyone else knows to stay the hell away from that bs. Big surprise when the same EndangeredMiddleManager types look at the past couple years of data and notice undocumented tables, columns with NULL+"N/a"+" " for missing values, databases that have disappeared and can't be rebuilt because the code was never checked in...Business schools are supposed to teach you that focusing on the short-term is dangerous. Guess that doesn't matter when your career plan is to keep switching companies/getting promoted every 1-3 years, while you leave untold trails of festering crap behind you.