4 ms·

CRUD applications are developed brainlessly. They're more challenging than it seems. It's interesting that people develop CRUD applications in garbage-collecte

CRUD applications are developed brainlessly. They're more challenging than it seems. It's interesting that people develop CRUD applications in garbage-collected languages because manual memory management is a pain, but they'll gladly expose manual data management to the end user without worrying about analogous data bugs. If data bugs are important, you need your data management system to have some qualities of a garbage collection system (tracing, reference counting) or it needs to be self-repairing (fill in missing data on demand). This is actually a challenging problem, assuming your data is important. This is why cache invalidation is one of the Two Hard Things in programming according to Martin Fowler.

crispyambulance 11y ago
Thanks! that's a remarkably insightful way to look at it.
OpenDrapery 11y ago
Interesting. Would love to learn more about these kinds of ideas. Is this something you've been kicking around in your own mind? Or has someone put a more formal framework around these concepts? Where can I read more about it?
- humanrebar 11y ago
  Thanks. These are mostly my ideas (not that there are any truly new ideas), but they are more-or-less the result of trying to write idempotent services. That is, making sure the same request always returns the same response (modulo incremented sequences). Analogizing data issues to memory management is very useful to software developers that haven't been beat up by data availability and consistency issues in the past. It's also useful to study functional programming concepts, especially data immutability and data structures that work within that limitation, to identify possible design patterns for organizing things that themselves don't change. For more inspiration, learn about the plumbing of git if you haven't already. The fact that commits never change buys git all sorts of things. How do I model this in a database? I treat (table, sequence_id) pairs like I would memory addresses (or git objects). And I try to design append-only tables as much as possible. No UPDATE queries. Only INSERT (malloc) and DELETE (free). There are a lot of nice benefits here, including the ability to shard based on the age of data, being able to aggressively cache partial results and recalculate only when things change, etc. To that end, most data is best modeled as a time series if data consistency is an issue (and it is if you need to scale). There are exceptions for axiomatically true things, like mathematical calculations, but for the most part things like "how many users like sandwiches" change depending on when you ask them. So the right question is "how many users like sandwiches as of 11:32:23 today?". Or maybe "how many users liked sandwiches when you ran calculation job 432342?" Then you can collate timestamps or sequence IDs to see if some data is stale, if unused data can be cleaned up, etc.

crispyambulance 11y ago

Thanks! that's a remarkably insightful way to look at it.

OpenDrapery 11y ago

Interesting. Would love to learn more about these kinds of ideas. Is this something you've been kicking around in your own mind? Or has someone put a more formal framework around these concepts? Where can I read more about it?

humanrebar 11y ago
Thanks. These are mostly my ideas (not that there are any truly new ideas), but they are more-or-less the result of trying to write idempotent services. That is, making sure the same request always returns the same response (modulo incremented sequences). Analogizing data issues to memory management is very useful to software developers that haven't been beat up by data availability and consistency issues in the past. It's also useful to study functional programming concepts, especially data immutability and data structures that work within that limitation, to identify possible design patterns for organizing things that themselves don't change. For more inspiration, learn about the plumbing of git if you haven't already. The fact that commits never change buys git all sorts of things. How do I model this in a database? I treat (table, sequence_id) pairs like I would memory addresses (or git objects). And I try to design append-only tables as much as possible. No UPDATE queries. Only INSERT (malloc) and DELETE (free). There are a lot of nice benefits here, including the ability to shard based on the age of data, being able to aggressively cache partial results and recalculate only when things change, etc. To that end, most data is best modeled as a time series if data consistency is an issue (and it is if you need to scale). There are exceptions for axiomatically true things, like mathematical calculations, but for the most part things like "how many users like sandwiches" change depending on when you ask them. So the right question is "how many users like sandwiches as of 11:32:23 today?". Or maybe "how many users liked sandwiches when you ran calculation job 432342?" Then you can collate timestamps or sequence IDs to see if some data is stale, if unused data can be cleaned up, etc.