4 ms·

> syntax = "proto2" uses explicit presence by default > syntax = "proto3" used implicit presence by default (where cases 2 and 3 cannot be distinguished and ar

> syntax = "proto2" uses explicit presence by default > syntax = "proto3" used implicit presence by default (where cases 2 and 3 cannot be distinguished and are both represented by an empty string), but was later extended to allow opting into explicit presence with the optional keyword > edition = "2023", the successor to both proto2 and proto3, uses explicit presence by default The root of the problem seems to be go's zero-values. It's like putting makeup on a pig, your get rid of null-panics, but the null-ish values are still everywhere, you just have bad data creeping into every last corner of your code. There is no amount of validation that can fix the lack of decoding errors. And it's not runtime errors instead of compile-time errors, which can be kept in check with unit tests to some degree. It's just bad data and defaulting to carry on no matter what, like PHP back in the day.

delusional 2y ago
I don't think the reason for zero values has anything to do with "avoiding null panics". If you want to inline the types, that is avoid using most of your runtime on pointer chasing, you can't universally encode a null value. If I'm unclear, ask yourself: What would a null int look like? If what you wanted was to avoid null-panics, you can define the elementary operations on null. Generally null has always been defined as aggressively erroring, but there's nothing stopping a language definition from defining propagation rules like for float NaN.
- the_gipsy 2y ago
  Sorry, I don't follow you. If you don't have zero values, you either have nulls and panics, or you have some kind of sum-type á la Option<T> and cannot possibly construct null or zero-ish values. Is there a way to have your cake and eat it too, and are there real world examples of it?
  - masklinn 2y ago
    > or you have some kind of sum-type á la Option<T> and cannot possibly construct null or zero-ish values. Option types specifically allow defaulting (to none) even if the wrapped value is not default-able. You can very much construct null or zero-ish values in such a langage, but it’s not universal, types have to be opted into this capability.
    - the_gipsy 2y ago
      Exactly my point, you have to opt-in, and in practice you only do precisely where it's actually necessary. Which is completely different than "every single type can be a [null | zero value]". You cannot possibly construct some type A (that is not Option<A> or A@nullable or whatever) without populating it correctly. Of course you need some way to represent "absence of a value", the matter is how: simple but incorrect, or complex but correct. And, simple/complex here can mean both the language (so performance tradeoff), and (initial) programmer ergonomics. That's why I ask if you can have your cake and eat it too, the answer is no. Or you'll get sick sooner than later, in this case.
      - delusional 2y ago
        > You cannot possibly construct some type A (that is not Option<A> or A@nullable or whatever) without populating it correctly. Except you can. The language runtime is clearly doing it when it stores [None|Some(x)] inline in a fixed size struct.
        tsimionescu 2y ago
        There is no way to store None | Some(x) in sizeof(x) bytes, for simple information theory reasons. What you can do is store between 1 and 8 optional fields with only 1 byte of overhead, by using a single bit field to indicate which of the optional fields is set or not (since no commonly used processor supports bit-level addressing, storing 1 extra bit still needs an entire extra byte, so the other 7 bits in that byte are "free").
        delusional 2y ago
        > There is no way to store None | Some(x) in sizeof(x) bytes That's subtlety incorrect. Almost all languages with NULLs in fact already do this, including C. On my machine sizeof(void*)=8, and pointers can in fact express Some(x)|None. The cost of that None is neither a bit not a byte, it's a single value. A singular bit pattern. See the None that you talk about is grafted on. It wraps the original without interfacing with it. It extends the state by saying "whatever the value this thing has, its invalid". That's super wasteful. Instead of adding a single state, you've exploded the state space exponentially (in the literal sense).
  - bborud 2y ago
    No, there isn't. It is just other versions of the same problem with people pretending it is somehow different. People generally like to complain about NULL/nil whatever, but they rarely think about what the alternatives mean and what arrangements are completely equivalent. No matter what you do, you have to put some thought into design. Languages can't do the design work for programmers.
  - TheDong 2y ago
    There is a way to have your cake and eat it too: rust. In rust, you have: let s = S{foo: 42, ..Default::default()}; You just got all the remaining fields of 'S' set to "zero-ish" values, and there's no NPEs. The way you do this is by having types opt in to it, since zero values only make sense in some contexts. In go, the way to figure out if a type has a meaningful zero value is to read the docs. Every type has a zero value, but a lot of them just nil-pointer-exception or do something completely nonsensical if you try to use them. In rust, at compiletime you can know if something implements default or not, and so you can know if there's a sensible zero value, and you can construct it. Go doesn't give you your cake, it gives you doc comments saying "the zero value is safe to use" and "the zero value will cause unspecified behavior, please don't do it", which is clearly not _better_.
    - the_gipsy 2y ago
      I agree that rust, with Option and Default, is the only right choice - at least from what I've tried. Elm for example has Option but nothing like Default, so sometimes it's tedious that you have to repeat a lot of handmade defaults, or you're forced to use constructor functions everywhere. But at least the program is correct! Go is like PHP in regards to pushing errors forward. You simply cannot validate everything at every step. Decoding with invariants is the right alternative.
    - delusional 2y ago
      > There is a way to have your cake and eat it too: rust. Suppose my cake is that I have a struct A which holds a value, that doesn't have a default value, from your library B. Suppose that at the time I want to allocate A I don't yet have the information I need to initialize B, but I also know that I won't need B before I do have that information and can initialize it. In simple terms. I want to allocate A, which requires allocating B, but I don't want to initialize B, yet. What do I do? If you answer involves Option<B> then you're asking me to to grow my struct for no gain. That is clearly not _better_.
      - the_gipsy 2y ago
        Then you can't eat it too (or else you'll get very sick with NPEs/panics), sorry.
  - delusional 2y ago
    You're thinking in abstract terms, I'm talking about the concrete implementation details. If we, just as an example, take C. and int can never be NULL. It can be 0, compilers will sometimes tell you it's "uninitialized", but it can never be NULL. all possible combinations of bit patterns are meaningfully int. Pointers are different in that we've decided that the pattern where all bits are 0 is a value that indicates that it's not valid. Note that there's nothing in the definition of the underlying hardware that required this. 0 is an address just like any other, and we could have decided to just have all pointers mean the same thing, but we didn't. The NULL is just a language construct, and as a language construct it could be defined in any way you want it. You could defined your language such that dereferencing NULL would always return 0. You could decide that doing pointer arithmetic with NULL would yield another NULL. At the point you realize that it's just language semantics and not fundamental computer science, you realize that the definition is arbitrary, and any other definition would do. As for sum-types. You can't fundamentally encode any more information into an int. It's already completely saturated. What a sumtype does, at a fundamental level, is to bundle your int (which has a default value) with a boolean (which also has a default value) indicating if your int is valid. There's some optimizations you can do with a "sufficiently smart compiler" but like auto vectorization, that's never going to happen. I guess my point can be boiled down to the dual of the old C++ adage. Resource Allocation is NOT initialization. RAINI.
    - the_gipsy 2y ago
      Then your point is tangent to the question of zero values, and even more so to the abstract concept of zero values spilling over into protobuf.
tech132 2y ago
From what I remember, proto3 behavior happened to map to objective c since iOS maps coincidentally happened at around the same time so they could be loud. It was partially reverted with proto3 optional and fully reverted finally. Go's implementation happened to come around the same time as proto3 so allowed struct access, despite behaving quite differently when accessing nil fields. That is also finally reverted. Hopefully more lessons already learned from the Java days will come sooner than later going forward...
9rx 2y ago
> It's like putting makeup on a pig, your get rid of null-panics How so? In Go, nil is the zero value for a pointer and is ripe for panic just like null. Zero values do not avoid that problem at all, nor do they intend to.
- the_gipsy 2y ago
  That's just that they picked a worse case of zero value for slices and maps, presumably for performance gains.
  - tsimionescu 2y ago
    It should be noted that slices and maps are completely opposite ends of how they behave in relation to nil in Go. A nil slice is just an empty slice, there is no operation you could do with one that will fail if done with the other. In contrast, a nil map doesn't support any operation whatsoever, it will panic on doing anything with it.
    - the_gipsy 2y ago
      You're quite mistaken. You can use len() on both nil maps and slices, and it will return zero (as with empty ones). Panic occurs on both nil assignments. But then, access only panics on nil slices - nil maps produce the zero value. It's horrible. https://go.dev/play/p/2KFOoJ0oyWB https://go.dev/play/p/2KFOoJ0oyWB
      - mxey 2y ago
        Access on an empty slice would also panic because of out of bounds. So there’s no useful distinction there. However, nil slices work just fine when calling append on them, while writing to a nil map in any way will panic.
      - tsimionescu 2y ago
        Right, forgot that len() works on nil maps, and I was really not aware that reading from a nil map is not an error, that's crazy. For the nil slice though what I said remains true: a nil slice is the same thing as an empty slice. Of course reading from or writing to its first element panics, given that it doesn't have any elements. The same would have happened if you had initialized it as `var ns []int = make([]int, 0)` or `ns := []int{}`.
knodi 2y ago
Yes, as much as I love Go and love working with it every day. This inner workings of Go with zero-values has been an design issue that comes up again and again and again.
deleted 2y ago
[deleted]

delusional 2y ago

I don't think the reason for zero values has anything to do with "avoiding null panics". If you want to inline the types, that is avoid using most of your runtime on pointer chasing, you can't universally encode a null value. If I'm unclear, ask yourself: What would a null int look like? If what you wanted was to avoid null-panics, you can define the elementary operations on null. Generally null has always been defined as aggressively erroring, but there's nothing stopping a language definition from defining propagation rules like for float NaN.

the_gipsy 2y ago
Sorry, I don't follow you. If you don't have zero values, you either have nulls and panics, or you have some kind of sum-type á la Option<T> and cannot possibly construct null or zero-ish values. Is there a way to have your cake and eat it too, and are there real world examples of it?
- masklinn 2y ago
  > or you have some kind of sum-type á la Option<T> and cannot possibly construct null or zero-ish values. Option types specifically allow defaulting (to none) even if the wrapped value is not default-able. You can very much construct null or zero-ish values in such a langage, but it’s not universal, types have to be opted into this capability.
  - the_gipsy 2y ago
    Exactly my point, you have to opt-in, and in practice you only do precisely where it's actually necessary. Which is completely different than "every single type can be a [null | zero value]". You cannot possibly construct some type A (that is not Option<A> or A@nullable or whatever) without populating it correctly. Of course you need some way to represent "absence of a value", the matter is how: simple but incorrect, or complex but correct. And, simple/complex here can mean both the language (so performance tradeoff), and (initial) programmer ergonomics. That's why I ask if you can have your cake and eat it too, the answer is no. Or you'll get sick sooner than later, in this case.
    - delusional 2y ago
      > You cannot possibly construct some type A (that is not Option<A> or A@nullable or whatever) without populating it correctly. Except you can. The language runtime is clearly doing it when it stores [None|Some(x)] inline in a fixed size struct.
      - tsimionescu 2y ago
        There is no way to store None | Some(x) in sizeof(x) bytes, for simple information theory reasons. What you can do is store between 1 and 8 optional fields with only 1 byte of overhead, by using a single bit field to indicate which of the optional fields is set or not (since no commonly used processor supports bit-level addressing, storing 1 extra bit still needs an entire extra byte, so the other 7 bits in that byte are "free").
        delusional 2y ago
        > There is no way to store None | Some(x) in sizeof(x) bytes That's subtlety incorrect. Almost all languages with NULLs in fact already do this, including C. On my machine sizeof(void*)=8, and pointers can in fact express Some(x)|None. The cost of that None is neither a bit not a byte, it's a single value. A singular bit pattern. See the None that you talk about is grafted on. It wraps the original without interfacing with it. It extends the state by saying "whatever the value this thing has, its invalid". That's super wasteful. Instead of adding a single state, you've exploded the state space exponentially (in the literal sense).
- bborud 2y ago
  No, there isn't. It is just other versions of the same problem with people pretending it is somehow different. People generally like to complain about NULL/nil whatever, but they rarely think about what the alternatives mean and what arrangements are completely equivalent. No matter what you do, you have to put some thought into design. Languages can't do the design work for programmers.
- TheDong 2y ago
  There is a way to have your cake and eat it too: rust. In rust, you have: let s = S{foo: 42, ..Default::default()}; You just got all the remaining fields of 'S' set to "zero-ish" values, and there's no NPEs. The way you do this is by having types opt in to it, since zero values only make sense in some contexts. In go, the way to figure out if a type has a meaningful zero value is to read the docs. Every type has a zero value, but a lot of them just nil-pointer-exception or do something completely nonsensical if you try to use them. In rust, at compiletime you can know if something implements default or not, and so you can know if there's a sensible zero value, and you can construct it. Go doesn't give you your cake, it gives you doc comments saying "the zero value is safe to use" and "the zero value will cause unspecified behavior, please don't do it", which is clearly not _better_.
  - the_gipsy 2y ago
    I agree that rust, with Option and Default, is the only right choice - at least from what I've tried. Elm for example has Option but nothing like Default, so sometimes it's tedious that you have to repeat a lot of handmade defaults, or you're forced to use constructor functions everywhere. But at least the program is correct! Go is like PHP in regards to pushing errors forward. You simply cannot validate everything at every step. Decoding with invariants is the right alternative.
  - delusional 2y ago
    > There is a way to have your cake and eat it too: rust. Suppose my cake is that I have a struct A which holds a value, that doesn't have a default value, from your library B. Suppose that at the time I want to allocate A I don't yet have the information I need to initialize B, but I also know that I won't need B before I do have that information and can initialize it. In simple terms. I want to allocate A, which requires allocating B, but I don't want to initialize B, yet. What do I do? If you answer involves Option<B> then you're asking me to to grow my struct for no gain. That is clearly not _better_.
    - the_gipsy 2y ago
      Then you can't eat it too (or else you'll get very sick with NPEs/panics), sorry.
- delusional 2y ago
  You're thinking in abstract terms, I'm talking about the concrete implementation details. If we, just as an example, take C. and int can never be NULL. It can be 0, compilers will sometimes tell you it's "uninitialized", but it can never be NULL. all possible combinations of bit patterns are meaningfully int. Pointers are different in that we've decided that the pattern where all bits are 0 is a value that indicates that it's not valid. Note that there's nothing in the definition of the underlying hardware that required this. 0 is an address just like any other, and we could have decided to just have all pointers mean the same thing, but we didn't. The NULL is just a language construct, and as a language construct it could be defined in any way you want it. You could defined your language such that dereferencing NULL would always return 0. You could decide that doing pointer arithmetic with NULL would yield another NULL. At the point you realize that it's just language semantics and not fundamental computer science, you realize that the definition is arbitrary, and any other definition would do. As for sum-types. You can't fundamentally encode any more information into an int. It's already completely saturated. What a sumtype does, at a fundamental level, is to bundle your int (which has a default value) with a boolean (which also has a default value) indicating if your int is valid. There's some optimizations you can do with a "sufficiently smart compiler" but like auto vectorization, that's never going to happen. I guess my point can be boiled down to the dual of the old C++ adage. Resource Allocation is NOT initialization. RAINI.
  - the_gipsy 2y ago
    Then your point is tangent to the question of zero values, and even more so to the abstract concept of zero values spilling over into protobuf.

tech132 2y ago

From what I remember, proto3 behavior happened to map to objective c since iOS maps coincidentally happened at around the same time so they could be loud. It was partially reverted with proto3 optional and fully reverted finally. Go's implementation happened to come around the same time as proto3 so allowed struct access, despite behaving quite differently when accessing nil fields. That is also finally reverted. Hopefully more lessons already learned from the Java days will come sooner than later going forward...

9rx 2y ago

> It's like putting makeup on a pig, your get rid of null-panics How so? In Go, nil is the zero value for a pointer and is ripe for panic just like null. Zero values do not avoid that problem at all, nor do they intend to.

the_gipsy 2y ago
That's just that they picked a worse case of zero value for slices and maps, presumably for performance gains.
- tsimionescu 2y ago
  It should be noted that slices and maps are completely opposite ends of how they behave in relation to nil in Go. A nil slice is just an empty slice, there is no operation you could do with one that will fail if done with the other. In contrast, a nil map doesn't support any operation whatsoever, it will panic on doing anything with it.
  - the_gipsy 2y ago
    You're quite mistaken. You can use len() on both nil maps and slices, and it will return zero (as with empty ones). Panic occurs on both nil assignments. But then, access only panics on nil slices - nil maps produce the zero value. It's horrible. https://go.dev/play/p/2KFOoJ0oyWB https://go.dev/play/p/2KFOoJ0oyWB
    - mxey 2y ago
      Access on an empty slice would also panic because of out of bounds. So there’s no useful distinction there. However, nil slices work just fine when calling append on them, while writing to a nil map in any way will panic.
    - tsimionescu 2y ago
      Right, forgot that len() works on nil maps, and I was really not aware that reading from a nil map is not an error, that's crazy. For the nil slice though what I said remains true: a nil slice is the same thing as an empty slice. Of course reading from or writing to its first element panics, given that it doesn't have any elements. The same would have happened if you had initialized it as `var ns []int = make([]int, 0)` or `ns := []int{}`.

knodi 2y ago

Yes, as much as I love Go and love working with it every day. This inner workings of Go with zero-values has been an design issue that comes up again and again and again.

deleted 2y ago

[deleted]