4 ms·

> In programming, we often come across the situation that there is not always a concrete value for something In database terms, the need for null values means

> In programming, we often come across the situation that there is not always a concrete value for something In database terms, the need for null values means that the data schema is denormalized. With a simpler approach there is seldom any need for these "not applicable" values. And that's the problem with all languages with special support for "nullable" values: They glorify bad data structure design. I've seen codebases where 30-50% of the lines is just handling of null values that SHOULD NOT BE THERE and where nothing meaningful (or even functional) happens when the value is actually null. Goes well together with OOP: In the name of isolation, each object is abstracted from all context necessary to construct a straightforward and correct program. The result is more and more meaningless boilerplate and burnt out developers. Personally I'm fine with not assigning any value to "not applicable" variables. Or putting a sentinel there, like -1 or NULL. But adding a physical case (i.e. changing the type) to support "not applicable" cases is just not a good idea.

marcosdumay 8y ago
> In database terms, the need for null values means that the data schema is denormalized. And nobody ever use a fully normalized schema in practice, because it's cumbersome and gets in the way of solving the problem. And that's for databases, that are optimized stable storage and data coherence, not for performance. A fully normalized schema is a performance disaster if applied to running applications.
- mianos 8y ago
  This is not really true of any mature system. There are many reasons to design a schema in normal form, including performance, maintainability and space optimisation. Database design outside start-ups is a fairly mature area and normal form is well proven good practice.
  - marcosdumay 8y ago
    I will call bullshit on that. I have never seen a database on the 6th normal form on the real world, and I don't believe any exist outside of a lab. Anything beyond 3rd normal form brings very sporadic gains to maintainability and space optimization. The usual case is that you lose those. About performance, I can imagine there exist situations where you'll gain some by going further than the 3rd normal form, but I don't believe anybody sees that as a normal situation.
  - klodolph 8y ago
    Yes, these are good reasons to use normal form. And it's also true that normal form is proven good practice. But in some cases, performance is important enough that adding a denormalized table or two makes sense. This isn't really "proven good practice" in any sense, but it's something that should be evaluated on a case by case basis when there's a compelling reason for it. It's kind of like the dream of having a sufficiently smart compiler. In theory, a relational database could operate in normal form and give you all the performance you need. In practice, they don't always do that.
  - magicalhippo 8y ago
    In our application we have a form that has at least 50 fields which are individually optional. This is in addition to the required fields, and currently it's all one table. Are you saying it would be better for performance and space to have all of those optional fields in separate tables, and that it wouldn't be a huge PITA to write queries against for say reporting and while debugging?
    - jstimpfle 8y ago
      Are these 50 fields "logically" relevant? In other words, do you do significant joining at these fields? Or in business logic, do you switch much on the presence of these fields? I'll assume "no" since they're so many of them and they're individually optional fields. Maybe most of them are strings and you could simply use empty strings. Or whatever, it doesn't matter. I'm not saying these cases are trivial, but they're leaf concerns and not really relevant from a "schema" point of view. But if the answer is really "yes", then I'll say that's at least a maintainability problem (bad coherence).
      - magicalhippo 8y ago
        They're fields in a digitized form, essentially. Multiple name, address etc sections representing people or companies, for example, as well as domain specific fields which shouldn't be used, are optional or are required depending on the value of a previous field. Most are strings of various lengths, but also plenty of numeric (weight etc) fields as well as dates. Expected arrival date for example, which is optional in most cases but required in some. Our UI would have to join in all the fields, as it displays them all in one window, and our customers would absolutely hate to split filling them up into multiple windows. Also, once the form is sent and approved by officials, the data has to be saved without change for 10 years, like saving the paper version of the form for 10 years in a filing cabinet. It also needs to be easily accessible for the users up to 3 years after approval in case one needs to make a correction, which is essentially done using a special copy of the form. I'm no db expert by any stretch, I've learned by doing. So not gonna claim this couldn't have been handled better.
        8y ago
- jstimpfle 8y ago
  What I was saying is that first-class nullable types encourage bad practice. I know that from lots of real world experiences. Flat tables work surprisingly well. Parent-pointers instead of child-lists work surprisingly well and bring a lot of benefits including modularity/coherence and cache efficiency. All it takes is rethinking the control flow. It's usually possible to iterate tables separately instead of walking trees (the hierarchy mindset).
  - seandougall 8y ago
    Doesn’t that also require that you implement indexing? Otherwise you’re iterating over the entire table for every access, which I’ve never found to work well in practice.
    - jstimpfle 8y ago
      There isn't really anything to implement. Basic "entities" get integer ids starting from 0. You just point into the flat array. Child lists or other structures that don't have natural ids can for example get "indexed" with first- and last-indices like this: struct Parent { some data; Child childFirst; Child childLast; }; struct Child { some childdata; Parent parent; }; struct Parent *parent; struct Child *child; int parentCnt; int childCnt; These indices are trivial to setup: SORT(child, childCnt, compare_child_by_parent); for (int i = 0; i < parentCnt; i++) { parent[i].childFirst = childCnt; parent[i].childLast = 0; } for (int i = 0; i < childCnt; i++) { if (parent[child[i].parent].childFirst > i) parent[child[i].parent].childFirst = i; if (parent[child[i].parent].childLast < i + 1) parent[child[i].parent].childLast = i + 1; } The important thing to see is that most operations are inside whole-table loops. Not in the O(n) sense - each step does something meaningful. Nested loops or complicated data manipulations are drastically decreased compared to OOP approaches.
mattnewport 8y ago
I think it's more helpful to think of C++ std::optional like Maybe in a functional language rather than as like a nullable type. Its primary use case is as a return value from functions, not as a data member. There are many functions where this can make good sense: parsing a string as a number, searching for an element in a collection, trying to load something from a file, taking a square root, etc.

marcosdumay 8y ago

> In database terms, the need for null values means that the data schema is denormalized. And nobody ever use a fully normalized schema in practice, because it's cumbersome and gets in the way of solving the problem. And that's for databases, that are optimized stable storage and data coherence, not for performance. A fully normalized schema is a performance disaster if applied to running applications.

mianos 8y ago
This is not really true of any mature system. There are many reasons to design a schema in normal form, including performance, maintainability and space optimisation. Database design outside start-ups is a fairly mature area and normal form is well proven good practice.
- marcosdumay 8y ago
  I will call bullshit on that. I have never seen a database on the 6th normal form on the real world, and I don't believe any exist outside of a lab. Anything beyond 3rd normal form brings very sporadic gains to maintainability and space optimization. The usual case is that you lose those. About performance, I can imagine there exist situations where you'll gain some by going further than the 3rd normal form, but I don't believe anybody sees that as a normal situation.
- klodolph 8y ago
  Yes, these are good reasons to use normal form. And it's also true that normal form is proven good practice. But in some cases, performance is important enough that adding a denormalized table or two makes sense. This isn't really "proven good practice" in any sense, but it's something that should be evaluated on a case by case basis when there's a compelling reason for it. It's kind of like the dream of having a sufficiently smart compiler. In theory, a relational database could operate in normal form and give you all the performance you need. In practice, they don't always do that.
- magicalhippo 8y ago
  In our application we have a form that has at least 50 fields which are individually optional. This is in addition to the required fields, and currently it's all one table. Are you saying it would be better for performance and space to have all of those optional fields in separate tables, and that it wouldn't be a huge PITA to write queries against for say reporting and while debugging?
  - jstimpfle 8y ago
    Are these 50 fields "logically" relevant? In other words, do you do significant joining at these fields? Or in business logic, do you switch much on the presence of these fields? I'll assume "no" since they're so many of them and they're individually optional fields. Maybe most of them are strings and you could simply use empty strings. Or whatever, it doesn't matter. I'm not saying these cases are trivial, but they're leaf concerns and not really relevant from a "schema" point of view. But if the answer is really "yes", then I'll say that's at least a maintainability problem (bad coherence).
    - magicalhippo 8y ago
      They're fields in a digitized form, essentially. Multiple name, address etc sections representing people or companies, for example, as well as domain specific fields which shouldn't be used, are optional or are required depending on the value of a previous field. Most are strings of various lengths, but also plenty of numeric (weight etc) fields as well as dates. Expected arrival date for example, which is optional in most cases but required in some. Our UI would have to join in all the fields, as it displays them all in one window, and our customers would absolutely hate to split filling them up into multiple windows. Also, once the form is sent and approved by officials, the data has to be saved without change for 10 years, like saving the paper version of the form for 10 years in a filing cabinet. It also needs to be easily accessible for the users up to 3 years after approval in case one needs to make a correction, which is essentially done using a special copy of the form. I'm no db expert by any stretch, I've learned by doing. So not gonna claim this couldn't have been handled better.
      - 8y ago
jstimpfle 8y ago
What I was saying is that first-class nullable types encourage bad practice. I know that from lots of real world experiences. Flat tables work surprisingly well. Parent-pointers instead of child-lists work surprisingly well and bring a lot of benefits including modularity/coherence and cache efficiency. All it takes is rethinking the control flow. It's usually possible to iterate tables separately instead of walking trees (the hierarchy mindset).
- seandougall 8y ago
  Doesn’t that also require that you implement indexing? Otherwise you’re iterating over the entire table for every access, which I’ve never found to work well in practice.
  - jstimpfle 8y ago
    There isn't really anything to implement. Basic "entities" get integer ids starting from 0. You just point into the flat array. Child lists or other structures that don't have natural ids can for example get "indexed" with first- and last-indices like this: struct Parent { some data; Child childFirst; Child childLast; }; struct Child { some childdata; Parent parent; }; struct Parent *parent; struct Child *child; int parentCnt; int childCnt; These indices are trivial to setup: SORT(child, childCnt, compare_child_by_parent); for (int i = 0; i < parentCnt; i++) { parent[i].childFirst = childCnt; parent[i].childLast = 0; } for (int i = 0; i < childCnt; i++) { if (parent[child[i].parent].childFirst > i) parent[child[i].parent].childFirst = i; if (parent[child[i].parent].childLast < i + 1) parent[child[i].parent].childLast = i + 1; } The important thing to see is that most operations are inside whole-table loops. Not in the O(n) sense - each step does something meaningful. Nested loops or complicated data manipulations are drastically decreased compared to OOP approaches.

mattnewport 8y ago

I think it's more helpful to think of C++ std::optional like Maybe in a functional language rather than as like a nullable type. Its primary use case is as a return value from functions, not as a data member. There are many functions where this can make good sense: parsing a string as a number, searching for an element in a collection, trying to load something from a file, taking a square root, etc.