3 ms·

I believe all data warehouses are limited by the quality of their data model. Most start with good relational intentions over a small domain, but eventually get

by jl6 6y ago

I believe all data warehouses are limited by the quality of their data model. Most start with good relational intentions over a small domain, but eventually get bogged down arguing how semantic angels might dance on ontological pins. The parts that work become ossified and impossible to change. The system starts to fragment into multiple federated datastores or unstructured file dumps (“big data!”) where you have to build your own integration every time you want to use the data. Someone comes along and proposes a unifying model (“everything is an event!”) and rebuilds the whole thing but with an extra layer of complexity. Someone suggests buying an industry data model instead - surely the data experts will have solved all these problems for us? A skunkworks project spins up and starts implementing the bought model with good relational intentions over a small domain... I don’t think personal data warehouses are immune to any of these forces.

simonw 6y ago
My implementation deliberately avoids any hint of a unified data model: each source of data (Twitter, Swarm, GitHub etc) gets a database schema that matches as closely as possible to the original JSON. Then you figure out and bookmark basic SQL queries for each one. If their data model changes, stuff will break and you will need to fix it. I'm fine with that.
iblaine 6y ago
>I believe all data warehouses are limited by the quality of their data model. This is common and most organizations overcome this problem with a higher effort downstream by Analysts & BI. I personally prefer 3NF tables, event streams or Kimball, in that order, as solutions to create a better DW. The reality is ugly behavior in data modeling can be overcome by Analysts & BI tools. And that seems to be the norm these days...use a DW to structure data to make data accessible, then let Analysts figure out the metrics. With the recent emphasis on tools like Looker or Tableau, this pattern seems to be working.
lyxsus 6y ago
> I believe all data warehouses are limited by the quality of their data model. wrt that, what do you think about Solid from Tim Berners-Lee? (solidproject.org)

by jl6 6y ago

simonw 6y ago

My implementation deliberately avoids any hint of a unified data model: each source of data (Twitter, Swarm, GitHub etc) gets a database schema that matches as closely as possible to the original JSON. Then you figure out and bookmark basic SQL queries for each one. If their data model changes, stuff will break and you will need to fix it. I'm fine with that.

iblaine 6y ago

>I believe all data warehouses are limited by the quality of their data model. This is common and most organizations overcome this problem with a higher effort downstream by Analysts & BI. I personally prefer 3NF tables, event streams or Kimball, in that order, as solutions to create a better DW. The reality is ugly behavior in data modeling can be overcome by Analysts & BI tools. And that seems to be the norm these days...use a DW to structure data to make data accessible, then let Analysts figure out the metrics. With the recent emphasis on tools like Looker or Tableau, this pattern seems to be working.

lyxsus 6y ago

> I believe all data warehouses are limited by the quality of their data model. wrt that, what do you think about Solid from Tim Berners-Lee? (solidproject.org)