12 ms·
Applying “make invalid states unrepresentable”
- wa1987 6y agoRelated video on this subject: https://www.youtube.com/watch?v=IcgmSRJHu_8 https://www.youtube.com/watch?v=IcgmSRJHu_8
- matsemann 6y agoI thought about that video today when integrating with an old SOAP API. I need to find the name+some property of some persons. Instead of having a list with [(name1, prop1), (name2, prop2),...], I get two distinct lists of [name1, name2,..] and [prop1, prop2,..]. In practice I think the lists will always match. But there is nothing stopping them from not being the same length, or even worse: one having a gap..
- vbezhenar 6y agoPut an assertion. It's better to throw early rather than dealing with wrong data later.
- etripe 6y agoThis is a good introduction on a conceptual level. I think a large contributor to the problem is story-oriented development, where all that matters in the sprint is "getting it done" and not looking at the broader context. To make unrepresentable states practical, Scott Wlaschin has an excellent write-up here (0). His book (plugged in that article) is also excellent. [0] https://fsharpforfunandprofit.com/posts/designing-with-types-making-illegal-states-unrepresentable/ https://fsharpforfunandprofit.com/posts/designing-with-types...
- choeger 6y ago> I think a large contributor to the problem is story-oriented development, where all that matters in the sprint is "getting it done" and not looking at the broader context. I think you have a point here. This design offers much better safety, comparable to "parsing instead of validating". But it requires up-front design. And that is indeed "verboten" in modern software development management style. Why is it "verboten"? I think it has to do with two fundamental concepts of scrum et. al.: 1. Stories that are "ready" just need to be "implemented". The implication is that the developer does not design and everything is orthogonal. There are no interdependencies, maintenance effort or non-functional requirements. 2. A story that is "ready" focuses solely on the desired outcome for some selected examples. There is no generalization of the examples and consequently little to no abstraction. I think these issues stem from the fact that Scrum et. al. are intrinsically tools for managers to isolate them from the complexities of software engineering. Every metric of scrum, for instance, like "progress" or "definition of ready" is essentially empty of meaning for software engineering.
- chrisweekly 6y ago+1, favorited.
- aunty_helen 6y agoI like how you've said it here, but one thing that scrum doesn't have in it is relief from your professional duty as an engineer. If a system needs to be designed in a particular way, do so. That's how long it takes and that's why in planning you discuss how it will be designed. The design of how you're going to build the system is taken care of before the task is split into easily digestible bits that meet a definition of ready. If you need to build it before you know how to build it, scrum allows for research spikes into this. A focused, timeboxed interval that so you have the ability to estimate the actual difficulty of work to be completed.
- andrewflnr 6y agoI get the impression that anything management touches gets turned (one might say corrupted) into a tool for detaching from engineering details. That's not what sole was meant to be about, and it's not what waterfall was about, but it happens anyway. Details will always matter, but management's paycheck depends on not understanding that.
- lmm 6y ago> I think a large contributor to the problem is story-oriented development, where all that matters in the sprint is "getting it done" and not looking at the broader context. I think that's exactly backwards. This kind of overcomplicated representation usually happens because people put too much effort into designing their representations up front. If you follow story-oriented development and only implement the parts they actually need to get the current task done, you never end up with these wasteful extra states because you never actually needed them. But people think that planning before coding is somehow virtuous, and then they're tied to following those plans.
- acidbaseextract 6y agoThe representational debt the parent comment is referring to is akin to that quip, "if I had more time, I would have written a shorter letter." I find that both types of representational debt are common.
- jka 6y agoTo be fair, both are possible scenarios: - A team spends too much time over-engineering a representation for something that could be more easily maintained using a simple model - A team spends too little time considering the edge cases with a representation because they feel pressure to deliver the feature within a short space of time
- matthewmacleod 6y agoAnd they often happen at the same time, too. I've seen situations where developers over-plan a system in advance, then spend ages hacking smaller features and changes into it to meet quick turnaround times, instead of taking proper step back and being aware of when their original design needs to be revised.
- Aeolun 6y agoOur database gets shit added on an ‘as necessary’ basis, and I guarantee you it’s not a great thing.
- dtech 6y ago
- thsealienbstrds 6y agoIt doesn't sound to me like the kind of thing that any particular WoW is going to fix. As if it would suddenly naturally dawn on someone to do this given enough time. It's just lack of knowledge and/or discipline.
- maest 6y agoSum types are one of the main things I miss when working in Python. Is anyone aware of any good ways of adding sum types to Python?
- nybble41 6y agoI'm not sure you would consider this a good way, but one can implement (unchecked) sum and product types in any language with lexical closures via Scott encoding[1]: # data Pair a b = Pair a b def pair(x, y): return lambda f: f(x, y) # data Either a b = Left a | Right b def left(x): return lambda l, r: l(x) def right(y): return lambda l, r: r(y) v0 = pair(2, 3) v1 = left(7) v2 = right(9) # case v0 of { Pair x y -> x + y } print(v0(lambda x, y: x + y)) # case v1 of { Left x -> x + 1; Right y -> y * 2 } print(v1(lambda x: x + 1, lambda y: y * 2)) [1] https://en.wikipedia.org/wiki/Mogensen%E2%80%93Scott_encoding https://en.wikipedia.org/wiki/Mogensen%E2%80%93Scott_encodin...
- maest 6y agoWow, this looks clever. (maybe a bit too clever) I'll need to read more about it, thanks for the pointer!
- masklinn 6y agoThe entire point of sum types is that they be statically checked. Without static typing, they don't seem really useful: Erlang doesn't have sum types, and it doesn't really have a use for them until it gets a type system which can leverage them. Instead it models "sum types" as tagged tuples e.g. {ok, Ok} | {err, Err} however Erlang has very good pattern matching. Python… doesn't. There's a PEP but I stopped following it because the discussion was a mess. And it's apparently now split into 3 different PEP, I don't know whether that's an improvement or not though. Furthermore Python's dislike of HOFs means you can't really do "monadic" processing as you'd do in, say, smalltalk where your "variants" would really be subtypes with cool higher-order messages. So you're mostly just adding indirections.
- acidbaseextract 6y agoThere's a library: https://sumtypes.readthedocs.io/en/latest/ https://sumtypes.readthedocs.io/en/latest/ It's serviceable, and even does static checking of total coverage of cases. I will say that it makes linters freak out.
- joosters 6y agoThe time period example seems to miss an obvious weakness in the described Time Period 'object' - it's implied that the end date should be >= the start, but if you are representing a time period with ( Date, Date ) then you are still allowing invalid states to be represented - yet this is what the writer is trying to avoid. Likewise, a timeline split into contiguous periods can still represent out-of-order Dates. a Time Period object of ( Date, Duration ) would fix the first issue, and a TimeLine of ( Date, Duration, Duration, ... ) would fix the second one (assuming unsigned Durations!)
- jxf 6y agoI think you misunderstood the author's point: time periods aren't explicitly represented as (Date: start, Date: end). Instead they're a set of dates. The time periods are then implied by the set, making "end date before start date" impossible.
- VBprogrammer 6y agoIt also deals with the problem of the bounds on the ranges (open-closed, open-open etc) implicitly in a way which is harder to mess up. One complication it's potentially missing is exactly who's days we are talking about i.e. is it days starting in GMT or UTC or EST or whatever the 'suppliers' or the 'customers' timezone is, are we actually talking about some day concept perhaps from start of business. Representing this as a datetime start / end certainly makes it possible to represent these concepts, if perhaps not making anything else particularly easier.
- deleted 6y ago[deleted]
- zimpenfish 6y agoI was going to say that this has a weakness in testing because you can't `assert(end > start)` without knowing which is which because `assert(later > sooner)` will always be true but then I guess you'd have input (and other) validations before it got to that point anyway.
- choeger 6y ago
- nefitty 6y agoSoftware engineer here. Does anyone have any questions?
- leoc 6y agoI appreciate what people are getting at with “make invalid states unrepresentable”, but it can't be the best description of what the true objective is. After all, the first language to make invalid states unrepresentable was probably TECO https://en.wikipedia.org/wiki/TECO_(text_editor)#As_a_programming_language https://en.wikipedia.org/wiki/TECO_(text_editor)#As_a_progra... ...
- chrisweekly 6y agoRegarding "make invalid states unrepresentable", I'm curious what others think of FSM (Finite State Machines), eg use of XState (vs eg Redux) in the FE webapp state mgmt space.
- secondcoming 6y agoIt looks like a variation of Run Length Encoding to me, where the 'Run' is a duration instead of a count.
- judofyr 6y agoIn general I agree that it's nice to make invalid states unrepresentable, but I'm not sure if I agree that this counts as a fundamental "invalid state". There is nothing about contracts which require that you can only have one active at the same time, or that that current one must be open ended. From a practical point of view it might be advantageous if you maintain only a single contract with a customer at all times, but that is a business requirement which might be changed in the future. I mention this mostly from experience: Multiple times I've designed systems where I've reduced the representable states to the minimum, and when some requirements change I realize I have to re-design the full system. The new presented representation might make sense in this situation, but I'd be very wary of taking current business practices and make all other alternatives impossible to represent. It's a balancing act of course as you can go in the opposite direction and make it way too flexible. > This poor choice was not just a theoretical problem - gaps in contracts were found on more than one occasion, requiring hours of engineering effort to hunt down and fix. I'd like to hear more about what happened here. Was the problem that the default contract was not re-applied correctly? If so, changing the representation might not actually solve any problems — it make actually make it _worse_. A renewal of a contract typically involves some automated process where other services are involved (payment, invoicing, emails). The previous representation (with explicit start/end dates) made it possible for you to verify that everything was correct and lined up.
- ahoka 6y agoThen the code will be changed. Making up business requirements is the number one reason for instant legacy code. Code is not set in concrete, you can add that flexibility later when it is needed, but making everything overly generic to make it easier to "implement new requirements" only leads to code that is hard to change in my experience. Also don't forget that this is only an example.
- chii 6y agoThe YAGNI (you aint gonna need it) principle overrides the DRY principle imo.
- deleted 6y ago
- seer 6y agoWhile this is great if you know exactly what you want to achieve, it does “lock you in” those constraints on a more fundamental level. More times than I can count I’ve seen business requirements change to require those “unrepresentable” states, and since you’ve now designed you whole data model around it you need to add awful hacks to make it work. The timeline example is actually very telling. A lot of times you’d actually want to encode overlapping time periods at the edges. You’d be laughed out off a meeting where a business asks about this and you smugly explain how it is unrepresentable. I guess what I’m saying is that it might be worth over designing your system a bit to leave you some wiggle room, unless you have hard guarantees that something should be “impossible”
- TooCreative 6y agoAdditionally, turning (startDate, endDate) into a set of dates will make the code more complex in some places. Before: SELECT event FROM events WHERE endDate<"2021" After: Whatever additional complexity you add to your codebase to query end dates.
- dghf 6y agoIt's not that much more complex: if you do SELECT event FROM events WHERE startDate < "2021" then all but one of the results (the one with the greatest `startDate`) will also have an implicit end-date prior to 2021.
- peteradio 6y agoUntil business tells you that endDate is not necessarily greater than startDate. <- real world experience
- NegativeLatency 6y agoSounds interesting can you explain more about it?
- mobjack 6y ago
- ChrisMarshallNY 6y agoI'm not sure that this is what I think about, when I think of "Making Invalid States representable" (A concept that I practice). That said, it's an excellent, commonsense article that describes a highly usable approach to information architecture. I also agree that OOP programmers have always considered their designs to "represent the 'Real World'™." In my experience, I use OOP constructs to represent many things that should never be exposed to the user (like messages, adapters, states, and state transitions). There's the classic usability concept of the "Mental Model." That is the model that the user builds in their head, as they navigate the UX. These mental models can be drastically different from what happens internally, and a good UX designer can reinforce a desired model (which the user may then ignore).
- ChrisMarshallNY 6y agoUm..."unrepresentable". :P
- yxhuvud 6y agoNot certain I get the needless attack on OOP in this text. The error could just as well have happened in any alternative to OO. What is needed is the realization that there is that there is a schedule that needs to have full control over the times to not create coordination issues. That is a realization that is utterly independent of the OOP-ness of the eventual solution.
- kpmah 6y agoThe problem with a narrow OO mindset is that it encourages encapsulation and atomic objects with hidden state. Simply gluing those pieces together can create suboptimal representations - a more holistic thought process is better.
- jschwartzi 6y agoNot necessarily. Some people can't see the forest for the trees no matter what kind of programming language they're using. What's needed here is to sit down and think REALLY hard about how the system works and what some of the terms are that are used to describe things. And also to have an intution for when someone is using imprecise language. GP cited use of a "Schedule" object as a way to enforce the invariant. That might represent a block of intervals by a sequence of times. The less precise you are about what's needed and how the system should work, the more of a soupy mess you're going to build. People gravitate toward OO because OO makes it feel like you've got a lot of conceptual clarity. You can get back to that familiar "subject/object" dual we love in English. But the problem is actually that, although people can pick subjects and objects readily from a sentence, they might not have much luck picking the important subjects and objects from a sentence. And that's the problem we really need to solve.
- brbrodude 6y agoIt's as if OOP code negates putting thought in designing a solution, absolute nonsequitur for me.
- ajuc 6y agoThis is part of approach to programming that is more popular in functional world. You take requirements and make system exactly right to fit these requirements perfectly and don't bother with any other concerns. When you design this close to the requirements you get better, faster and more elegant code that's easier to understand - but when requirements change you have to do much more work to adapt. Suddenly a state which was previously invalid is valid, or a part of system that only needed one kind of input needs 4 different inputs from separate parts of your code. Have fun basically rewriting your program. That's IMHO the main motivation between differences in functional and OO programming - how close to the requirements you want to design your code.
- pjmlp 6y agoThis kind of approach (type driven development) was already popular in Algol derived languages, hence why the cowboy coder would call us on Ada, Modula-2, Object Pascal side of the fence, programming with straitjacket.
- ajuc 6y agoI don't think it maps 1-1 with strong vs weak typing. C (very loosely typed language) code is usually pretty close to the requirements and very strongly typed functional languages (like Haskell) are often "make DSL and write the specification in it, then run it". Meanwhile object oriented languages often have pretty strong typesystems and cultures of using them extensively, but they also encourage designing with margin for changes (and thus using lots of layers of abstractions instead of just implementing the specification as elegantly as possible).
- jl6 6y agoI like the concept but I’ve seen a fair few examples of where the developers and users clearly had differing opinions about which states are invalid! Dates are a rich vein of examples. Some users will happily consider “25th December” to be a date, without any year, because it might be the name of a folder in which they store their Christmas stuff. More seriously, genealogists or historians may want to record “25th December” (again without a year) as the date of a photograph because they can clearly see it was taken on a Christmas, they just don’t know which one. The naive developer would just slap a DateTime type onto the system and feel good about having avoided malformed input.
- isbvhodnvemrwvn 6y agoAlso in genealogy: - estimated dates - calculated dates (e.g. someone was 30 in 1870, so he was born in "calculated 1840") - unreadable or unavailable months or days (typically recorded as 1980-00-13) - time ranges with all of the above as boundaries e.g. "after 1760-03-00 and before calculated 1800" - plainly incorrect dates, but that's what the document says (1865-02-30) - no dates (some software tries to enforce putting in some data in for whatever reason) - dates with unknown calendar
- Terr_ 6y agoAll plausible and scary. It sounds like a recipe for a system where the different qualities of dates are their own entity, and anybody doing a date-search needs to provide some criteria on the degree of specificity or certainty they require. For that matter, someone might want to do a text-based search on dates: "This damaged photo shows 196_-_1, which could be at least 20 different months..."
- isbvhodnvemrwvn 6y agoThere's also another dimension to that uncertainty. Dates are typically attributes of events involving one or more people in various roles, and these events are (hopefully) attached to one or more sources. The type of sources you use affects the confidence of various attributes of the event (like dates or participants). In my area of research death records are typically a bad predictor for date and location of birth. When the records were managed by the churches (until after WWII pretty much), priests did not want to pester grieving family for the exact date of birth, so they relied on approximate age. Naturally the subject of the death record would not typically point out errors. On the other hand marriage records required looking at the actual birth records (sometimes mailed across the country), as these contained notes on all marriages of the individual (this was done to ensure monogamy). Is it on the genealogist to consider this confidence when specifying the date of the event, or should the software intervene? Due to complexity, the former is the industry standard. But maybe there are some brave (or stupid) people who will try to take it into account in the future.
- browsergrip 6y agoThis is good article. But the second example seems to suffer from the defect of the first. Removing default contracts and representing fixed contracts as intervals leaves it possible that these fixed contracts can overlap....which is probably...undesirable? In that case, applying the remedy of the first example (a set of dates, and inferring that every 2nd (even zero length, to account for adjacent fixed) interval will be default,) introduces another bug where if you lop off any random date in that set or list, you invert everything. I love the concept represented here, it is akin to normalization as in...simplify the representation so no redundancy is introduced as this usually leads to better results...but it seems it's no guarantee of better results. But maybe that's just because the "model" we are simplifying from was not an optimal representation. Perhaps there's a better model of the second example that doesn't end up with the defect of the first example. I really like this article but am struck by how something that I wanted to be almost a silver bullet trick for modeling, ends up being a mass of compromises mired in tradeoffs that doesn't show any clear way forward in the general case. Still, probably a good rule of thumb, but I guess this rule is not optimal...as it can have so many unworkable misinterpretations/misapplications. It would be cool to see a list of, like, "Programming Heuristics", ranked by decreasing general applicability, of which this rule was a member somewhere far down the list.
- chrisweekly 6y ago"It would be cool to see a list of, like, "Programming Heuristics", ranked by decreasing general applicability, of which this rule was a member somewhere far down the list." +1 for this! Anyone got good links to share?
- JangoSteve 6y ago> In that case, applying the remedy of the first example (a set of dates, and inferring that every 2nd (even zero length, to account for adjacent fixed) interval will be default,) introduces another bug where if you lop off any random date in that set or list, you invert everything. This is a good point, and something I usually describe recoverable problems versus non-recoverable problems. If I make start/end dates in the first example instead of just a set of start dates, then I can always create application-level or database-level constraints that don't allow either overlapping or incomplete segments. When business rules change, I can delete the constraints and update the business logic as necessary with no change to underlying data structures. However, if I miss implementing a constraint and it erroneously allows overlapping or incomplete segments, I can easily run a query to identify all such invalid entries. Then I can then investigate and decide how to fix them. However, if I go with the start-date-only set-based approach, and miss implementing a constraint, and it leads to a deleted date creating incomplete segments... I'm screwed. There will be no query you can do to identify incomplete segments to investigate or fix, because all segments are assumed to extend to the next start date. You can irreversibly lengthen one segment by deleting another, due to a forgotten constraint preventing you from making the change. These could both be errors on the developer's part, depending on the requirements at the time, but one data design may lead to more non-recoverable issues than the other. Add in the flexibility of the former approach, and I'd probably be more likely to implement the former approach than the one proposed.
- mlthoughts2018 6y ago“Make invalid states unrepresentable” is a type of fool’s gold. You don’t care that invalid states are unrepresentable, you only ever care that a specific instance of your running program is very unlikely to enter an invalid state - and the difference between formally disallowing invalid states vs. test coverage that proves a reasonable likelihood of avoiding invalid states is huge. The extra code and conceptual complexity spent to make type designs that disallow invalid cases is a liability, it comes with its own bugs, maintenance and huge risks of premature abstraction and brittleness in the face of changing requirements. If it takes anything more than a simple enum-style menu of permitted options, then it’s a code small. Things like Scala case classes (especially with sealed behavior), or pattern matching against type constructors, or phantom types - these are all very bad ideas, where the costs far outweigh the benefits. Most of the time you can just ignore enforcement of assumptions, and add a few assert statements plus lightweight unit tests and integration tests that generate an abundance of real world example cases - and achieve all the safety you need for a fraction of the code & conceptual complexity and tech debt incurred by false promises of enforcing correctness with type system designs.
- dudul 6y agoJust use the language of the business/domain to write your model/API. No need to reinvent names and rules that already exist. Do whatever you want with the underlying implementation, your DB, etc.
- a-dub 6y agothis is a nice idea, but as mentioned in other comments, i think the most important goal when designing a schema is to design to make the queries you'll frequently be making simple and efficient. extensibility comes second and if you can make some invalid states impossible to represent, then that's definitely a bonus.
- unwind 6y agoThis is sound advice for sure, and I think it applies much more broadly (or do I mean deeply?) than just for databases. For instance, one micro-application of it that makes a lot fo sense to me is the const-ness of variables in languages like C. Since a normal variable can be overwritten, and that affects the use and semantics of that variable, marking them as const whenever possible really helps in my opinion. For instance, take this micro-snippet of code from Redis [1]: int time_independent_strcmp(char *a, char *b) { char bufa[CONFIG_AUTHPASS_MAX_LEN], bufb[CONFIG_AUTHPASS_MAX_LEN]; /* The above two strlen perform len(a) + len(b) operations where either * a or b are fixed (our password) length, and the difference is only * relative to the length of the user provided string, so no information * leak is possible in the following two lines of code. */ unsigned int alen = strlen(a); unsigned int blen = strlen(b); unsigned int j; int diff = 0; Here, it seems quite important that the values of 'alen' and 'blen' do not change during the execution of the function, since it's iterating over them. The 'diff' variable on the other hand is intended to change as a function of all the characters in both strings, that's the whole purpose of the function. So, I think the middle two lines should be: const size_t alen = strlen(a); const size_t blen = strlen(b); That "locks" the values in, so you know that for the rest of the function at least these two values stay the same. Since changing either length mid-function would represent an invalid state, I think this is close to the OP's point. Also please note that I have massive amounts of respect for Redis and Antirez, I'm not trying to say that the code is bad or anything, it was simply the first file in the first high-profile open source project that came to mind. Obviously this code works and has probably been more tested than most things I've written, again I'm NOT trying to somehow paint that program(mer) in a bad light. Btw, changing the type (to me) to size_t is also an obvious, free, improvement since it frees the reader from having to worry about why the type was unsigned int to begin with. Also 'int' can be less wide than 'size_t', which again is probably not a problem in practice since the CONFIG_AUTHPASS_MAX_LEN is probably always going to be even less, but still. It's pointless complexity that triggers anxiety in people like me. :) [1]: https://github.com/redis/redis/blob/unstable/src/acl.c https://github.com/redis/redis/blob/unstable/src/acl.c
- H8crilA 6y agoSo Google's protocol buffers have this feature called "required" fields, which enforce schema in the type system. You should never use it. Never. It's one of those things that sound good until you're a few years into the project. Similar to how you should never be using meaningful IDs as primary keys for objects, always use meaningless fingerprint-like integers. Or how all integers should be signed unless you're dead sure the number is unsigned (like a fingerprint). And how many integers should actually be strings, unless you're dead sure this is a number (externally provided IDs, such as for example customer account IDs, are not numbers). Or how you should be careful to use bytes rather than unicode strings. Make your schema permissible and your code paranoid, it will pay off later. Build a data linter if necessary, but don't tie the schema.
- nemetroid 6y agoI feel that this advice is almost opposite to that given in the article. By making all fields optional, your data model no longer helps in making invalid states unrepresentable.
- H8crilA 6y agoExactly! Because today's invalid states are commonplace in the future.
- jdmichal 6y ago> And how many integers should actually be strings, unless you're dead sure this is a number. I phrase this as: If it doesn't make sense to do math on it, it's not a number. What does adding one to a customer account number mean? Absolutely nothing -- you get a completely different account number. So it's not a number, but a numeric string.
- lukasLansky 6y agoIt's not a string either though: in the same way integer addition (almost always) does not make sense, string concatenation (almost always) does not make sense either. The proper type would allow for equality check and explicit string (de)serialization only.
- legulere 6y agoYou could also see this as a form of single point of truth.
- teddyh 6y agoThis is very much like database normalization, in that it has the benefit of making invalid data impossible, but the drawback of often making queries into the data much more cumbersome and usually also inefficient. As with database normalization, it is a good idea to first do it as much as possible, and then denormalize again until it is fast enough.
- rocqua 6y agoI have always has the idea of a database that does the denormalizations you want automatically for you. Essentially, you keep the DB in a normalized state. You define views of the DB that you want. Then the DB keeps those views as tables for you, and the DB does all of the hard work of keeping those view tables consistent with the denormalized data. Essentially the DB does the atomicity, cache-invalidation, and cache-updating for you. You get performance, and you get the certainty that invalid states are un-representable. I guess the biggest blocker here is in automatically determining what fields do and do not need to be update automatically?
- Serow225 6y agoCheck out the Noria project/database from MIT, I think you'd like it :) https://github.com/mit-pdos/noria https://github.com/mit-pdos/noria https://corecursive.com/030-rethinking-databases-with-jon-gjengset/ https://corecursive.com/030-rethinking-databases-with-jon-gj... https://notamonadtutorial.com/interview-with-norias-creator-a-promising-dataflow-database-implemented-in-rust-352e2c3d9d95 https://notamonadtutorial.com/interview-with-norias-creator-... https://pdos.csail.mit.edu/papers/noria:osdi18.pdf https://pdos.csail.mit.edu/papers/noria:osdi18.pdf
- dllthomas 6y agohttps://en.wikipedia.org/wiki/Materialized_view https://en.wikipedia.org/wiki/Materialized_view
- moron4hire 6y agoThis violates one of my core, learned-the-hard-way, database design constraints: querying of rows should never have to rely on any linear dependence on any other row in the same table. If you ever have to do some sort of inner join of a table on itself to bring in the single "next" row to tell you information about the current row, then you are stuffed. Query complexity explodes. Performance takes a nose dive. The end date of a contract is a property of that contract, not any other. Forgetting all other contracts for a moment, what do you need to know about one contract? There should be a straight forward way to query that contract on its own, with a query that represents a tree through tables in the database. It should not become a graph, with the potential for cycles that graphs allow. And I get the business requirements could need no overlaps, but gaps are clearly possible if a customer leaves for a while and then comes back later. Does that person need to then become a new "customer", because you don't allow gaps? And then are your customers' PII only allowed to be registered to a single account? Comcast has been a huge pain in the ass in years past because of moving, gaps, and email address reuse.
- userbinator 6y agoAnother application of this principle is in data representation, and why I think text-based formats are horrible in general for communication between software that doesn't involve a human reading it the majority of the time: there, the "invalid states" not only cause complexity in the parser, but they also waste space (think of storing a 4-byte integer as the 4 bytes directly, vs. a string of variable-length ASCII text.)
- 1-more 6y agoA great talk on this subject is "Making Impossible States Impossible" by Richard Feldman from Elm Conf 2016 https://www.youtube.com/watch?v=IcgmSRJHu_8 https://www.youtube.com/watch?v=IcgmSRJHu_8
- UglyToad 6y agoI was pleased and a bit surprised to see this post talk about a database level approach to this problem. As important an idea it is at the application code level where most posts discuss it, especially in the context of type systems like Haskell, I think it gets neglected when it comes to persistence. For those of us developers who are mere CRUD peons I think it's the most important factor in system stability that is mostly negected; either in favour of speed of iteration (NoSQL) or checks at the application code layer. As I'm increasingly coming to appreciate, systems without enforced integrity at the database level are a breeding ground for bugs. You can add checks in application code but all it takes is 1 bad commit, or 1 check that slipped your notice and now you have bad data and all future code in the system needs to support and work around the bad data. With foundations of sand even the most elegant structure in application code is doomed to a short and catastrophic future. As other commenters mention hindsight is 20:20 and you won't always know what the constraints should have been until after the fact, or the constraints might be wrong. But the 'trendy' development practices treat good old fashioned SQL constraints and data integrity as decidedly unsexy, to the detriment of a lot of systems. MySQL didn't even have check constraints (well, actually apply them) until version 8 which shows how ignored these things are. I appreciate the post is more about the fundamental design of the stored data but people are also forgetting unique constraints, foreign keys and all the other tried and tested tools which protect the most important part of most CRUD systems, the data, from devolving into an awful mess.
- dahauns 6y agoThis line raised a huge red flag for me: "If the customer doesn’t have a fixed contract, it is assumed they are on a default contract" No. Don't assume, specify. Explicitly. If this is part of your specification, have a DefaultContract entity of some kind somewhere. And don't call this table just "Contracts", make it clear that those exist in addition to or overlay a default contract. It might sound like overkill, but in my experience in business application development, one of the single largest and most painful sources of errors and refactoring headaches are implicit assumptions in the data model.
- Justsignedup 6y agoYeah that's a lot to be said about being explicit. For example, what if even on a default contract you want to start tracking payment adherence, or maybe sales information and attributing them to sales' numbers. Or maybe you start reporting and want reports on contracts vs default contracts. And then things get messy. Because the real world gets messy.
- pdonis 6y ago> If this is part of your specification, have a DefaultContract entity of some kind somewhere. Yes, but not in the database table for contracts. That's how I read this part of the post. I would expect the assumption that if the customer doesn't have a fixed contract, they are on a default contract to be encoded in business logic somewhere in an application that uses this database.
- dahauns 6y ago> I would expect the assumption that if the customer doesn't have a fixed contract, they are on a default contract to be encoded in business logic somewhere in an application that uses this database. That's exactly the kind of harmful assumption I'm talking about. Harmful in that when people act on that assumption and actually implement it that way. How a default contract might be represented may vary, and of course it is in no way required or even sensible to be stored as a row in the contracts table. But to think that it is such a fundamentally different kind of data that it should be represented apart from the rest, in a different system, in a different layer, in a completely different form of representation - this is where madness lies.
- tomstuart 6y agoMy favourite real-world example is something like: you have a React component which accepts the boolean props `showFoo`, `showBar` and `showBaz` to control its mode. The intention is that exactly one of `foo`, `bar` or `baz` will be shown at any given time, but that invariant is maintained only loosely, e.g. by a parent component which holds three flags in its state and mutates them in tandem inside various event handlers. The obvious problem is that those three props can easily get out of sync if you make a mistake in updating them, and the equally obvious solution is to replace them with a single prop (e.g. `show` or `mode`) which contains a value from some enumeration — maybe just the name of the thing to show, or a numeric ID that is given meaning elsewhere, or (galaxy brain) a component. That way the invariant is maintained strongly and automatically by the representation itself. This example sounds vacuous — who would ever use three booleans in the first place? — but in practice it’s very easy for UI code to incrementally get into this mess over time without anyone noticing. The situation is also often more subtle, e.g. the invariant is more complex than “exactly one flag is true”, which makes it harder to spot that you can model all the valid states with a finite enumeration.
- gambler 6y ago>I think this happens because of atomistic, object-orientated thinking. If you think storing a list of date tuples is "OOP thinking", you have no clue what OOP really is. Educate yourself by listening to people who invented it, not Java consultants or FP zealots. OOP is about interacting with things via interfaces and messages, rather than data. An OOP solution to inconsistencies of this sort would be an interface that either automatically corrects inconsistencies or throws errors when you try to introduce them. The whole point of OOP approach is that you're not locked into a single data representation, so, for example, you can improve how you store data without re-engineering everything in your system that relies on that data.
- Kaze404 6y agoThere's a great talk from Richard Feldman that talks about this in the context of Elm. https://www.youtube.com/watch?v=IcgmSRJHu_8 https://www.youtube.com/watch?v=IcgmSRJHu_8
- wruza 6y agoAh, good old design for systems with no special business cases. Heard of them, but never seen one.
- JamesBarney 6y agoIf you enjoyed this blog post and want to go into more depth on how to model your domain using types check out https://fsharpforfunandprofit.com/series/designing-with-types.html https://fsharpforfunandprofit.com/series/designing-with-type...
- noisy_boy 6y agoI don't see how the contracts example is simplifying things while staying realistic. What if I need to have separate kind of default contracts for different classes of customers? What if I need to modify the details of a certain type of default contracts for all consumers that are using it? And on and on. If storing them in contracts table is not good (which I don't really agree with), then where should we store these variety of default contracts? How do I join them with contracts table to know which consumers have default contracts? Or groups consumers by type of default contracts? Keep the model sensible. Contracts belong to contracts. Add basic sanity to the model. The service that manages the data guards the data beyond basic data model sanity checks. Also, explicit is better than implicit.
- sakoht 6y agoPresuming if you are 100% sure what "invalid" is: It is possible that being _unable_ to represent a logic error might mean that a logic error is represented as a "valid but incorrect" value, which is even more dangerous. Take the example of storing a continuous series of date ranges. If I only store the first date of each pair, I can never accidentally have an overlap or gap. But if my code has a logic error that incorrectly calculates a range, being able to represent it could throw an error. If that code error translates to an incorrect break-point instead, I haven't prevented a bug, I've hidden it.
- tzs 6y agoA large fraction of the comments are about how if you do this and then someday your requirements change you might have to redo your underlying data structures or databases, with the implication being that you should therefore make those as general and flexible as possible. That reminds me of an interesting point I saw in a book whose title and author escape me. He said one of the reason you encounter so many bad designs in Java programs is that many new Java programmers look at the design of Swing to learn good design. He wasn't saying that Swing is badly designed--but Swing is a framework/library, not an application. What it takes to be a good framework/library is different than what it takes to be a good application. If you are writing an inventory management application you can design your tables and data structures and interfaces around things inventory management applications need. If you are writing a medical billing application you can design around what medical billing needs. If you are writing a framework or library that might be used by inventory management applications and medical billing applications and all the other nearly infinite kinds of applications people will write you need to keep it very general and flexible...but you also have to keep it fast and not too bloated. It's a much harder design problem, with different best practices for what is good design and what is not.
- hinkley 6y agoThere is a trick that I wish all senior staff knew but we find ourselves having to teach as a matter of routine. 1) Don't advertise what you're not selling 2) Don't sell everything that you've made It's possible to write software that has a public contract that allows only a subset of states that the internal system allows. You can use this to support one customer that has a requirement that is mutually incompatible with another customer's, but you can also use it for migrations. One should be able to create an API where the legal values in the system are strictly limited, but the data structures and storage format may have affordances to support migration. That doesn't necessarily solve the problem of communication between services and migrating these changes into a running system, but it's a useful tool. If you've ever had a coworker who insists on your service call diagrams looks like a tree or an acyclic graph, this problem is certainly one of many reasons they may be insisting on this. With a DAG there is an order in which I can deploy things that has a prayer (but not a guarantee) of letting all of the systems understand each other during each increment of deployment. People have come up with alternative solutions for this problem by employing sophisticated sets of feature toggles, and in some ways this is superior, but it trades the number of steps (each of which has a potential for human error, and consumes calendar time) for increased reliability on average.
- JoeAltmaier 6y agoSurely the data model is helpful, not only in keeping data integrity intact by disallowing invalid state. But also to help you think about your data, and discover simplifications and subtle rules to improve your model. I attacked the old 8-queens problem years ago as part of a contest in Byte Magazine. All the solutions published modelled the board as an 8X8 array with a 1 or zero to indicate the presence of a queen. They all ran slow and suffered from invalid game states confusing the algorithms. My solution was to observe that only 1 queen could be in each column (all solutions require queens to not be able to capture one another, and they can capture vertically). So I represented the board by an array of 8 values, the height of the queen in the column represented by indexing the array. Further, since only one queen can be in each row, the values were the numbers 1-8. My solution then, was to seed the array with the value 1, 2, 3, 4, 5, 6, 7, 8. Then test if array[i]-array[j] == (i-j) or (j-i), which would mean a diagonal capture. Simply permuting the values, searched a subset of board states that had to contain all possible solutions. And the permutation tree could be truncated as soon as for any (i,j) the test failed. Anyway, the program was tiny and finished in negligible time. A pity I didn't enter the contest!
- marshray 6y agoCustomer signed a document? Great, job one is to get all the data from those filled-in blanks into the database accurately. Whatever the customer signed is the authoritative contract, not some imaginary entry in a schema intentionally constructed to make it impossible to represent scenarios not envisioned by the developer.
- mr_gibbins 6y agoThe OP's conclusions are askew. If you remove the default contracts from the contracts table you are left with fixed term contracts. This means customers without fixed term contracts are assumed to have default contracts. Agree so far. But removing the default contracts removes the contract start date. Meaning the default start date HAS to be the customer start date and not a separate date. This changes the functionality of the system, not the state representation.