3 ms·

I think I'm following this but on the other end, I've often thought that there is a need for a NA class/type hierarchy, so the Arrow approach discussed in this

I think I'm following this but on the other end, I've often thought that there is a need for a NA class/type hierarchy, so the Arrow approach discussed in this piece might be too simple at some level? I'm being really loose and am unfamiliar with Arrow but it seems as if these kinds of discussions are useful but also maybe heavily language-dependent. E.g., how Julia, Python, R, Rust, or whatever interfaces with Arrow is also relevant. Treatment of null and missing data in the context of data science has rarely gotten the full consideration it probably warrants in design. Interesting read though.

thriftwy 3y ago
Arrow-based systems are also notorious for completely broken UIs/displaying of blank pages without missing a beat. You definitely do want some error reporting, however not in the form of SIGSEGV.
ChrisRackauckas 3y ago
Julia has language-level support for missing values (https://docs.julialang.org/en/v1/manual/missing/ https://docs.julialang.org/en/v1/manual/missing/) which acts like the solution here, but generally for any value that can be put in an array by doing `Union{Missing,T}`. The array type will automatically optimize by having a bitmap of represented values (for any singleton type this works, not just Missing). There were lots of pre-v1.0 design discussions on this vs sentinels, such as https://discourse.julialang.org/t/representing-nullable-values/4729 https://discourse.julialang.org/t/representing-nullable-valu... which can be dug up. The conclusion was more about the safety than performance, since sentinel values are fundamentally unsafe in generally since they can take a valid value and make it "special" (like in the blog post, minimum integer). Thus Julia has the following 3 choices: * Use `nothing`, which throws an error at each operation like `nothing + 1`. This requires `if x === nothing` handling everywhere, is the safest option, but is generally considered clunky for statistics use. As `nothing` is a singleton type, it has the same optimizations as what's mentioned in the blog post. * Use `missing`, which propagates, i.e. `missing + 1 === missing`. This allows for any `Union{Missing,T}` to use a scheme similar to the blog post: as `nothing` is a singleton type, it has the same optimizations as what's mentioned in the blog post. * Use a sentinel of your choice, risks known and dependent on the type. And you can build your stats libraries however you so choose, though JuliaStats has strongly gravitated towards the middle option as something close in ergonomics to R's NA but while having a bit more safety guarantees and generality.
- derbOac 3y ago
  Interesting. I was familiar with Missing and nothing in Julia but had never really thought about other options from a safety angle. It makes sense though. I have run into issues where it would be nice to have some sort of missing type/class hierarchy though, in that the reason for missingness is coded somewhere and then how to represent that in those three categories gets complicated. At some level the different missing types get treated the same for many functions (such as your missing + 1 propagation example) but for other things not. Then you end up being tempted to use the third option which has its own issues. I suppose you could create another variable with type of missing information but then that also becomes complex quickly.

thriftwy 3y ago

Arrow-based systems are also notorious for completely broken UIs/displaying of blank pages without missing a beat. You definitely do want some error reporting, however not in the form of SIGSEGV.

ChrisRackauckas 3y ago

Julia has language-level support for missing values (https://docs.julialang.org/en/v1/manual/missing/ https://docs.julialang.org/en/v1/manual/missing/) which acts like the solution here, but generally for any value that can be put in an array by doing `Union{Missing,T}`. The array type will automatically optimize by having a bitmap of represented values (for any singleton type this works, not just Missing). There were lots of pre-v1.0 design discussions on this vs sentinels, such as https://discourse.julialang.org/t/representing-nullable-values/4729 https://discourse.julialang.org/t/representing-nullable-valu... which can be dug up. The conclusion was more about the safety than performance, since sentinel values are fundamentally unsafe in generally since they can take a valid value and make it "special" (like in the blog post, minimum integer). Thus Julia has the following 3 choices: * Use `nothing`, which throws an error at each operation like `nothing + 1`. This requires `if x === nothing` handling everywhere, is the safest option, but is generally considered clunky for statistics use. As `nothing` is a singleton type, it has the same optimizations as what's mentioned in the blog post. * Use `missing`, which propagates, i.e. `missing + 1 === missing`. This allows for any `Union{Missing,T}` to use a scheme similar to the blog post: as `nothing` is a singleton type, it has the same optimizations as what's mentioned in the blog post. * Use a sentinel of your choice, risks known and dependent on the type. And you can build your stats libraries however you so choose, though JuliaStats has strongly gravitated towards the middle option as something close in ergonomics to R's NA but while having a bit more safety guarantees and generality.

derbOac 3y ago
Interesting. I was familiar with Missing and nothing in Julia but had never really thought about other options from a safety angle. It makes sense though. I have run into issues where it would be nice to have some sort of missing type/class hierarchy though, in that the reason for missingness is coded somewhere and then how to represent that in those three categories gets complicated. At some level the different missing types get treated the same for many functions (such as your missing + 1 propagation example) but for other things not. Then you end up being tempted to use the third option which has its own issues. I suppose you could create another variable with type of missing information but then that also becomes complex quickly.