5 ms·

It's staggering to me that people keep making these "rich" data formats without sum types. At least to me, the "ors" are just as important as the "ands" in doma

It's staggering to me that people keep making these "rich" data formats without sum types. At least to me, the "ors" are just as important as the "ands" in domain modeling. Apart from that, while you can always sort of fake it with a bunch of optional fields I believe that you kind of need a native encoding to a tagged union if you want to avoid bloating your messages.

the_girabbit 5y ago
Genuine question—why would you need a sum type in a self-describing data format?
- valenterry 5y ago
  Well, there are already sumtypes, just only specific builtin ones, not custom ones. E.g. booleans are sumtypes (true | false). Everything else that is nullable is also a sumtype (e.g. number | null). I think it should be pretty obvious how these are helpful and why they are needed no?
  - the_girabbit 5y ago
    Yeah, but it’s a schema-less, self-describing data format. It’s not like a specific position in a data stream has a requirement to be a specific type. I can see why sum types would be useful in a schema or for the elements of a collection that is required to be homogeneous (ie. List<Foo|Bar>). For what use case would one use custom sum types in a schema-less data format?
dastbe 5y ago
data interchange formats try to encode as little backwards incompatible information as possible. in this case, it would be the restriction that something is a sum type when it could have multiple fields set in the future. another example is protobuf moving to all fields being optional by default. as for the wire format, a variant struct where you've only instantiated a single field will encode down to just about the minimum amount of information required.
- vlovich123 5y ago
  Have you looked at cap’n’proto. It does sum types in a very sane way.
- valenterry 5y ago
  That's not contradicting though. One can always choose not to use (native) sumtypes if they are interested in extreme performance or compatibility. But logically speaking, it is _good_ that it's a restriction that a sumtype can't just turn into a multiple-fields type. Because while my software (as the consumer) might still be able to deserialize it, the assumption that only one field is set would be broken and my logic would now potentially broken. Much better if that happens at deserialization time then later one when I find out that my data is incorrect/corrupt.
- nly 5y ago
  Avro went the opposite way to most and just makes the concept of an optional field implementable via a union with null Non union fields can even be upgraded to unions later Personally I find the protobufs "everything is optional!" Behaviour fucking insane and awful to deal with, but it is true to the semantics of its underlying wire format.
yakkityyak 5y ago
You should look into https://cuelang.org https://cuelang.org
vlovich123 5y ago
Cap’n’proto has native sum types.
joshlemer 5y ago
Doesn't it trivially have "sum types" since it's just arbitrary self-describing data? i.e. nobody is stopping you from passing around objects in such a way: {a:1} {a:{b:2}} {a:4} {a:{b:4}} There's no static type layer over top of this, so it's inherently up to interpretation and whatever type system you want to use to describe this data, to be able to express that the values of `a` can be of type `number | {b: number}`
- valenterry 5y ago
  > There's no static type layer over top of this Yeah, that's the problem. I mean, hey, why json? We could just use unstructured plaintext for everything and now we are free to do everything. But obviously that has its own drawbacks. Having built-in support for sumtypes means better and more ergonomic support from libraries, it means there is one standard and not different ways to encode things and it also means better performance and tooling.
  - joshlemer 5y ago
    The point is that there's no reason to single out sumtypes here. Insofar as ions/json has support for arrays/objects/strings/numbers, it has exactly the same support for sumtypes, as in the example I showed above. Here is a list of "sumtype" `string | number | object`: [{}, "hi", 1, 2, 3, "yo", {a: "bc"}]
    - valenterry 5y ago
      No, that is not a sumtype, that's an array. In the same sense "1e-12" is not a number, it's a string. Yes, it's a string that encodes a number in a certain notion, but for alle the tooling, the IDE, the libraries, etc. it will stay a string.
      - joshlemer 5y ago
        What I mean is, it is an array of a sumtype `number | string | object`. So precisely, you could call it a `list<number | string | object>`
jsolson 5y ago
Protobuf supports sum types in the higher-level generated descriptors and languages -- on the wire they're just encoded as, well... oneof a number of possible options.
- ricardobeat 5y ago
  Which results in very painful inconsistencies when you’re dealing with the same schema on different platforms.
  - xyzzy_plugh 5y ago
    Are you referring to different language implementations/runtimes? I don't follow your point about inconsistencies.
    - deleted 5y ago
      [deleted]
spenczar5 5y ago
Others have mentioned Protobuf and Capnproto's support. Avro has them too, they're called Union. It seems that sum types are the norm, actually.
- clhodapp 5y ago
  Those do now but I believe that all of them added support years after their initial versions
  - spenczar5 5y ago
    I think you're incorrect: Avro had unions in version 1.0 [0], which is from 2012. Capnproto had unions back in 2013 [1]. That's from the v0.1 days, or maybe even earlier. Protobuf has had oneof support for about 7 years. They were added in version 2.6.0, from 2014-08-15 [2]. That's still 6 years after the initial public release in 2008, though, so this is maybe what you were thinking of? I don't know too many people who were using protobuf in those days outside of Google, though. --- [0] https://avro.apache.org/docs/1.0.0/spec.html#Unions https://avro.apache.org/docs/1.0.0/spec.html#Unions [1] https://github.com/capnproto/capnproto/commit/eb8404a157e07460d95ec2b6571d2239e851a65f https://github.com/capnproto/capnproto/commit/eb8404a157e074... [2] https://github.com/protocolbuffers/protobuf/blob/master/CHANGES.txt https://github.com/protocolbuffers/protobuf/blob/master/CHAN...
    - clhodapp 5y ago
      Thanks for the references, friend! And yes, I definitely am primarily thinking of protobuf, as I struggled with this back with version 2.5. I had the (apparently mistakenly) impression that Avro and Cap'n Proto (which I think actually first came out in this timeframe) were about on par.
seniorsassycat 5y ago
ion schema is a type system that can validate ion values and it supports sum types. https://amzn.github.io/ion-schema/docs/spec.html#union https://amzn.github.io/ion-schema/docs/spec.html#union The ion data model doesn't describe a schema or type system. It's a data structure where values are of a known type. In the binary format values are preceded by a type id, in the text format the syntax declares the type - "" for string, {} for struct. The data model doesn't declare what types a value could have, only the type it does have.

the_girabbit 5y ago

Genuine question—why would you need a sum type in a self-describing data format?

valenterry 5y ago
Well, there are already sumtypes, just only specific builtin ones, not custom ones. E.g. booleans are sumtypes (true | false). Everything else that is nullable is also a sumtype (e.g. number | null). I think it should be pretty obvious how these are helpful and why they are needed no?
- the_girabbit 5y ago
  Yeah, but it’s a schema-less, self-describing data format. It’s not like a specific position in a data stream has a requirement to be a specific type. I can see why sum types would be useful in a schema or for the elements of a collection that is required to be homogeneous (ie. List<Foo|Bar>). For what use case would one use custom sum types in a schema-less data format?

dastbe 5y ago

data interchange formats try to encode as little backwards incompatible information as possible. in this case, it would be the restriction that something is a sum type when it could have multiple fields set in the future. another example is protobuf moving to all fields being optional by default. as for the wire format, a variant struct where you've only instantiated a single field will encode down to just about the minimum amount of information required.

vlovich123 5y ago
Have you looked at cap’n’proto. It does sum types in a very sane way.
valenterry 5y ago
That's not contradicting though. One can always choose not to use (native) sumtypes if they are interested in extreme performance or compatibility. But logically speaking, it is _good_ that it's a restriction that a sumtype can't just turn into a multiple-fields type. Because while my software (as the consumer) might still be able to deserialize it, the assumption that only one field is set would be broken and my logic would now potentially broken. Much better if that happens at deserialization time then later one when I find out that my data is incorrect/corrupt.
nly 5y ago
Avro went the opposite way to most and just makes the concept of an optional field implementable via a union with null Non union fields can even be upgraded to unions later Personally I find the protobufs "everything is optional!" Behaviour fucking insane and awful to deal with, but it is true to the semantics of its underlying wire format.

yakkityyak 5y ago

You should look into https://cuelang.org https://cuelang.org

vlovich123 5y ago

Cap’n’proto has native sum types.

joshlemer 5y ago

Doesn't it trivially have "sum types" since it's just arbitrary self-describing data? i.e. nobody is stopping you from passing around objects in such a way: {a:1} {a:{b:2}} {a:4} {a:{b:4}} There's no static type layer over top of this, so it's inherently up to interpretation and whatever type system you want to use to describe this data, to be able to express that the values of `a` can be of type `number | {b: number}`

valenterry 5y ago
> There's no static type layer over top of this Yeah, that's the problem. I mean, hey, why json? We could just use unstructured plaintext for everything and now we are free to do everything. But obviously that has its own drawbacks. Having built-in support for sumtypes means better and more ergonomic support from libraries, it means there is one standard and not different ways to encode things and it also means better performance and tooling.
- joshlemer 5y ago
  The point is that there's no reason to single out sumtypes here. Insofar as ions/json has support for arrays/objects/strings/numbers, it has exactly the same support for sumtypes, as in the example I showed above. Here is a list of "sumtype" `string | number | object`: [{}, "hi", 1, 2, 3, "yo", {a: "bc"}]
  - valenterry 5y ago
    No, that is not a sumtype, that's an array. In the same sense "1e-12" is not a number, it's a string. Yes, it's a string that encodes a number in a certain notion, but for alle the tooling, the IDE, the libraries, etc. it will stay a string.
    - joshlemer 5y ago
      What I mean is, it is an array of a sumtype `number | string | object`. So precisely, you could call it a `list<number | string | object>`

jsolson 5y ago

Protobuf supports sum types in the higher-level generated descriptors and languages -- on the wire they're just encoded as, well... oneof a number of possible options.

ricardobeat 5y ago
Which results in very painful inconsistencies when you’re dealing with the same schema on different platforms.
- xyzzy_plugh 5y ago
  Are you referring to different language implementations/runtimes? I don't follow your point about inconsistencies.
  - deleted 5y ago
    [deleted]

spenczar5 5y ago

Others have mentioned Protobuf and Capnproto's support. Avro has them too, they're called Union. It seems that sum types are the norm, actually.

clhodapp 5y ago
Those do now but I believe that all of them added support years after their initial versions
- spenczar5 5y ago
  I think you're incorrect: Avro had unions in version 1.0 [0], which is from 2012. Capnproto had unions back in 2013 [1]. That's from the v0.1 days, or maybe even earlier. Protobuf has had oneof support for about 7 years. They were added in version 2.6.0, from 2014-08-15 [2]. That's still 6 years after the initial public release in 2008, though, so this is maybe what you were thinking of? I don't know too many people who were using protobuf in those days outside of Google, though. --- [0] https://avro.apache.org/docs/1.0.0/spec.html#Unions https://avro.apache.org/docs/1.0.0/spec.html#Unions [1] https://github.com/capnproto/capnproto/commit/eb8404a157e07460d95ec2b6571d2239e851a65f https://github.com/capnproto/capnproto/commit/eb8404a157e074... [2] https://github.com/protocolbuffers/protobuf/blob/master/CHANGES.txt https://github.com/protocolbuffers/protobuf/blob/master/CHAN...
  - clhodapp 5y ago
    Thanks for the references, friend! And yes, I definitely am primarily thinking of protobuf, as I struggled with this back with version 2.5. I had the (apparently mistakenly) impression that Avro and Cap'n Proto (which I think actually first came out in this timeframe) were about on par.

seniorsassycat 5y ago

ion schema is a type system that can validate ion values and it supports sum types. https://amzn.github.io/ion-schema/docs/spec.html#union https://amzn.github.io/ion-schema/docs/spec.html#union The ion data model doesn't describe a schema or type system. It's a data structure where values are of a known type. In the binary format values are preceded by a type id, in the text format the syntax declares the type - "" for string, {} for struct. The data model doesn't declare what types a value could have, only the type it does have.