5 ms·

I think we need to caveat _what_ is a database when we make statements like this: > Databases. They are the single source of truth for our most critical busine

I think we need to caveat _what_ is a database when we make statements like this: > Databases. They are the single source of truth for our most critical business data, yet as engineers we tend to overlook tooling with this in mind. Most folks, like myself, will assume that we're talking about traditional concepts of databases, like RDBMS or Key/Value stores etc. But over time, after building distributed systems, it is important to realize that _logs_ are also a form of a streaming database and we need it to treat it like such. This is the basic principle behind the venerable article by Jay Kreps (one of the authors of Kafka): The Log: What every software engineer should know about real-time data's unifying abstraction [1]. Once we start thinking this way, then we realize that everything ends up having a schema (protobuf, thrift, avro, etc.) -- just a "database" with a completely different view point. IMHO, and consequently this is true where I work, The Log is considered the single source of truth and what is traditionally referred to as a "database" is just a "just-in-time" view of data that is materialized from The Log. Anyway, that is just my 2c. [1]: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha...)

skybrian 6y ago
It seems like there's a fundamental distinction between systems where you control data entry (it's manually entered and you can refuse to accept data if it doesn't follow the rules, until the error is corrected) and systems where you are recording whatever you got. In the latter case, it's often better to keep data for debugging purposes than drop it on the floor. The raw data may not conform to any particular schema, but perhaps you transform it or only use a subset of it for more advanced queries, and do something else with the rejects. In older systems, this may be the difference between the paper form you got in the mail and what gets entered in the computer. Refusing to accept input has implications for power relations. Who loses when data is rejected? Who gets a chance to fix it, and how hard do they try? What error messages are there and long does it take to retry? Case study: vote by mail.
- mikemotherwell 6y ago
  >In older systems If only it were older systems! I get data from a variety of undocumented sources, and I started logging what I actually receive next to the normalised data, e.g. I either put raw JSON into the database next to the actual data it contains, or I put all downloaded data into a git repo and update the repo after every query. Why raw JSON? I've had data come back as JSON that did not validate like not escaping quotes, blank fields that were missing a value e.g. {"dog": , "cat": "yes"}, just plain wrong data (a number field that was "no number at present"). Putting it into the DB lets me fix the errors and reprocess. With non-JSON, I've had a raft of other errors/problems, and I find git works well here to show what changed and caused the errors. When dealing with anyone else's data, not just validating but also logging in a structured way that makes discovering errors easier is vital.

skybrian 6y ago

It seems like there's a fundamental distinction between systems where you control data entry (it's manually entered and you can refuse to accept data if it doesn't follow the rules, until the error is corrected) and systems where you are recording whatever you got. In the latter case, it's often better to keep data for debugging purposes than drop it on the floor. The raw data may not conform to any particular schema, but perhaps you transform it or only use a subset of it for more advanced queries, and do something else with the rejects. In older systems, this may be the difference between the paper form you got in the mail and what gets entered in the computer. Refusing to accept input has implications for power relations. Who loses when data is rejected? Who gets a chance to fix it, and how hard do they try? What error messages are there and long does it take to retry? Case study: vote by mail.

mikemotherwell 6y ago
>In older systems If only it were older systems! I get data from a variety of undocumented sources, and I started logging what I actually receive next to the normalised data, e.g. I either put raw JSON into the database next to the actual data it contains, or I put all downloaded data into a git repo and update the repo after every query. Why raw JSON? I've had data come back as JSON that did not validate like not escaping quotes, blank fields that were missing a value e.g. {"dog": , "cat": "yes"}, just plain wrong data (a number field that was "no number at present"). Putting it into the DB lets me fix the errors and reprocess. With non-JSON, I've had a raft of other errors/problems, and I find git works well here to show what changed and caused the errors. When dealing with anyone else's data, not just validating but also logging in a structured way that makes discovering errors easier is vital.