4 ms·
Spot on. My favorite way to summarize this is: SQL is easy. Data is hard. If you aren't investing the work up front to make your data useful then no amount
by mulmen 7y ago
Spot on. My favorite way to summarize this is:
SQL is easy. Data is hard.
If you aren't investing the work up front to make your data useful then no amount of tooling or magic beans will make writing meaningful SQL easy. If your reports (SQL) are hard to write it's because you have bad data.
Having said that making data exploration easier is always a worthwhile exercise. The more business people can self-serve their questions the better. It falls to data people to make that possible not just with tools but with the data itself.
- ken 7y agoAgreed. I want to see more tools like OpenRefine.
- hcarvalhoalves 7y ago100x yes. Questions to be made on data are always an afterthought, after the systems to gather and produce this data have already been designed. The idea that it’s ok to just give a BI tool to business people later, and not involve them at the beginning to inform the system and data model design, is maybe why we’re in the tar pit. A really excellent process/tool would help business figure out what kind of questions they need answered, and work backwards to the data model and implementation.
- koreth1 7y agoSometimes it's not for lack of trying, though. I've done a decent amount of data model design in my career and as I got more experienced, I learned to make a point of asking the business people what specific questions they'd need the data to answer. Very often I got only very vague answers that were of little to no use in informing my design. I've seen this repeatedly at a variety of companies over the years. The business people are often winging it as much as the engineers are.
- rtpg 7y agoThe payoff from having some engineers doing the T of ETL (transforming the data to be more queryable, denormalizing values etc) is insane if you have problems in democratizing data analysis/reporting The dumb thing for people who don't know what this looks like: make a separate database. Create tables based on what you often want to query (but don't try to keep the same shape as the base data!). Then figure out how to fill in those tables from the base data
- boublepop 7y agoSQL is easy like solving differential equations is easy. If you know how, you just work througH it, but that’s only because you already learned all the prerequisites and then spend a year getting good at it. Anyways the point is that either way, you’ll never have an entire organization working at this level with SQL or solving equations and that’s fine. The CEO doesn’t need it, sales don’t need it. They do need numbers though, so having a dumbed down interface that can show some curves after clicking “net sales” and “country” let’s them move forwards with what matters to them, without having to learn the difference between an inner and outer join, which doesn’t matter to them. Not everyone needs to be a programmer even if they do need the value typically buried in programmer interfaces.
- cridal 7y agoWhat do you guys think about a system that would 'guide' the user through the process of creating a star schema data warehouse (Kimball style)? The advantage of this approach would be the fact that you do the dirty laundry upfront during the modeling & population phases. The end result is a data structure that is fool-proof, i.e. there is only one way to join facts/dimensions, it's self-documenting. In fact pre-joined views could automatically be created (and persisted if need be), giving the business user a clean structure to interact with.