5 ms·

It's not just a problem with data movement inside the machine. Data movement between processes and machines is also horribly wasteful. Text encoding/decoding w

It's not just a problem with data movement inside the machine. Data movement between processes and machines is also horribly wasteful. Text encoding/decoding wastes massive amounts of energy, which is why I've been developing a format over the past 2 years with the benefits of binary (smaller, more efficient) as well as text (human readability/editability): https://github.com/kstenerud/concise-encoding https://github.com/kstenerud/concise-encoding

aidos 7y ago
Seems like a good effort. I couldn’t easily find an example of the text encoding looking at your readme though.
- Reelin 7y ago
  Text: https://github.com/kstenerud/concise-encoding/blob/master/cte-specification.md#example https://github.com/kstenerud/concise-encoding/blob/master/ct... Binary: https://github.com/kstenerud/concise-encoding/blob/master/cbe-specification.md#decimal-floating-point https://github.com/kstenerud/concise-encoding/blob/master/cb...
Reelin 7y ago
How does this compare to MessagePack? (https://github.com/msgpack/msgpack https://github.com/msgpack/msgpack)
- kstenerud 7y ago
  MessagePack: * Doesn't have a sister text format for human editing. * Uses raw uncompressed types for integers and floats, which wastes space * Doesn't support arbitrary sized/precision numeric types * Timestamp format uses seconds instead of Gregorian fields (which means that your implementation must keep a leap second database, and time conversions are complicated) * No time zones * Doesn't have metadata, comments, or references * Container types have an unnecessary length field * Stores data in big endian byte order (an unnecessary expense since all popular modern chips are little endian) * Wastes an extra bit for negative integers * 32-bit limitations on length (so max 4gb payload sizes) * No versioning of documents
  - Reelin 7y ago
    That's actually a pretty convincing list! You might want to consider adding a feature comparison matrix for a few of the more popular formats in a prominent location. It also probably wouldn't hurt to lead the page with a small example of encoded text, the way MessagePack does (it's what got me to give their website a closer look, which led to GitHub, and then I started using it). Regarding the point about having a sister text format, I suppose I'd tend to view JSON as filling that role. Not to bother you for a second comparison, but if you happen to have it handy seeing how CTE compares to JSON in terms of encoding efficiency would be interesting to me. Edit: I found a list of MessagePack <--> JSON limitations that I hadn't really thought about! (https://github.com/ludocode/msgpack-tools#differences-between-messagepack-and-json https://github.com/ludocode/msgpack-tools#differences-betwee...)
    - kstenerud 7y ago
      Thanks for the feedback! I'm not exactly the greatest at marketing, so any help in this regard is most appreciated :)
satanspastaroll 7y ago
Waste can be very subjective. Exchanging text makes the format easier to debug and accessible to humans, which can be a purpose in itself.
- kstenerud 7y ago
  100% agreed, which is why Concise Encoding has 1:1 type-compatible binary [1] and text [2] formats, which can be automatically converted when needed with no data loss. Human accessibility is an absolute must for any modern format. [1] https://github.com/kstenerud/concise-encoding/blob/master/cbe-specification.md https://github.com/kstenerud/concise-encoding/blob/master/cb... [2] https://github.com/kstenerud/concise-encoding/blob/master/cte-specification.md https://github.com/kstenerud/concise-encoding/blob/master/ct...
- 9c8675a8 7y ago
  It should only be easy to debug and accessible to humans when needed IMO. I really think it should be possible to load a "production message", log event or whatever into a debug parser which only then spits out human readable output.
  - satanspastaroll 7y ago
    Even switching between modes may introduce new kinds of undefined behavior, which may not be visible on the text-only side
    - kstenerud 7y ago
      Every extra step in the process adds another potential failure point. But we're fast approaching a data and energy crunch that will push the industry towards binary formats once more. This is my attempt to keep that shift sane, and avoid the mess of the 80s and 90s. The implementations are almost done now, and my first tool will be a command line utility that reads one format and spits out the other, so that you can take a binary dump from your production system using tcpdump or wireshark or whatever, and then convert it to a human readable format to see what's going on. I'll probably even put in a hex-reader so that you can log the raw message and then read it back: 2019-10-02:15:00:32: Received message [01 76 85 6e 75 6b 65 73 88 6c 61 75 6e 64 68 65 64 79] $ ceconv --hex 01 76 85 6e 75 6b 65 73 88 6c 61 75 6e 64 68 65 64 79 v1 { nukes = launched }
userbinator 7y ago
I've worked with lots of binary formats, and they are equally "human-readable" (and far less unambiguous in a lot of cases) once you get used to it. As a coworker once said, "everything is human-readable if you have a hex editor." It seems your format is yet another attempt at reinventing ASN.1.
- kstenerud 7y ago
  But the point is that binary formats shouldn't require looking at hex data just to make sense of them. Especially not nowadays. Once you're stuck parsing binary data by hand (which is tedious), you've lost the amazing benefit that text formats gave us. And editing binary formats is even worse. That's why I split it into compatible text and binary formats. You transmit/store in binary, and read/edit in text, and the computer deals with the conversion automatically.
  - kortex 7y ago
    I think this is the direction things are heading. Currently, I use yaml for tons of configs and the like because it's super easy to modify. I use jq to process because jq is the bee's knees. There is a yq out there which can read most config formats, but it doesn't support the full jq syntax, so instead I pipe everything through `yq read -j -` to convert to json and then `| jq 'query'`. works a treat. I can see with a few more small parts an application which persists as bson, streams / tempfile / and/or mmaps to yaml/hcl/whatever you like, opens that in your editor of choice, and translates back to bson.
Const-me 7y ago
> Data movement between processes and machines is also horribly wasteful I agree, and in .NET that’s a solved problem: http://const.me/articles/net-tcp/ http://const.me/articles/net-tcp/ (2010). Their text format is XML. Their binary version takes small fraction of bandwidth, especially when using a custom pre-shared XML dictionary. Binary is much cheaper to produce or parse, the built-in DataContractSerializer understands both formats. Supported by all modern editions of .NET including .NET Core and is compatible across them, e.g. I’ve been recently using it for a network protocol of a Linux ARM device.
kortex 7y ago
This is pretty cool! I really like that dates (and timezone info) are first class types. Often neglected, time ser/de has been a monkeywrench in many a project. Protobufs are neat but `protoc` is an absolute nightmare. I would warn against going down the path of transpiling to lang-specific message files if at all possible (I don't think you are taking that approach, but github freezes my phone browser, I'll look more on my dev machine) Great stuff! I would love to see this get more traction. Interchange formats are a classic victim of critical mass necessity.

aidos 7y ago

Seems like a good effort. I couldn’t easily find an example of the text encoding looking at your readme though.

Reelin 7y ago
Text: https://github.com/kstenerud/concise-encoding/blob/master/cte-specification.md#example https://github.com/kstenerud/concise-encoding/blob/master/ct... Binary: https://github.com/kstenerud/concise-encoding/blob/master/cbe-specification.md#decimal-floating-point https://github.com/kstenerud/concise-encoding/blob/master/cb...

Reelin 7y ago

How does this compare to MessagePack? (https://github.com/msgpack/msgpack https://github.com/msgpack/msgpack)

kstenerud 7y ago
MessagePack: * Doesn't have a sister text format for human editing. * Uses raw uncompressed types for integers and floats, which wastes space * Doesn't support arbitrary sized/precision numeric types * Timestamp format uses seconds instead of Gregorian fields (which means that your implementation must keep a leap second database, and time conversions are complicated) * No time zones * Doesn't have metadata, comments, or references * Container types have an unnecessary length field * Stores data in big endian byte order (an unnecessary expense since all popular modern chips are little endian) * Wastes an extra bit for negative integers * 32-bit limitations on length (so max 4gb payload sizes) * No versioning of documents
- Reelin 7y ago
  That's actually a pretty convincing list! You might want to consider adding a feature comparison matrix for a few of the more popular formats in a prominent location. It also probably wouldn't hurt to lead the page with a small example of encoded text, the way MessagePack does (it's what got me to give their website a closer look, which led to GitHub, and then I started using it). Regarding the point about having a sister text format, I suppose I'd tend to view JSON as filling that role. Not to bother you for a second comparison, but if you happen to have it handy seeing how CTE compares to JSON in terms of encoding efficiency would be interesting to me. Edit: I found a list of MessagePack <--> JSON limitations that I hadn't really thought about! (https://github.com/ludocode/msgpack-tools#differences-between-messagepack-and-json https://github.com/ludocode/msgpack-tools#differences-betwee...)
  - kstenerud 7y ago
    Thanks for the feedback! I'm not exactly the greatest at marketing, so any help in this regard is most appreciated :)

satanspastaroll 7y ago

Waste can be very subjective. Exchanging text makes the format easier to debug and accessible to humans, which can be a purpose in itself.

kstenerud 7y ago
100% agreed, which is why Concise Encoding has 1:1 type-compatible binary [1] and text [2] formats, which can be automatically converted when needed with no data loss. Human accessibility is an absolute must for any modern format. [1] https://github.com/kstenerud/concise-encoding/blob/master/cbe-specification.md https://github.com/kstenerud/concise-encoding/blob/master/cb... [2] https://github.com/kstenerud/concise-encoding/blob/master/cte-specification.md https://github.com/kstenerud/concise-encoding/blob/master/ct...
9c8675a8 7y ago
It should only be easy to debug and accessible to humans when needed IMO. I really think it should be possible to load a "production message", log event or whatever into a debug parser which only then spits out human readable output.
- satanspastaroll 7y ago
  Even switching between modes may introduce new kinds of undefined behavior, which may not be visible on the text-only side
  - kstenerud 7y ago
    Every extra step in the process adds another potential failure point. But we're fast approaching a data and energy crunch that will push the industry towards binary formats once more. This is my attempt to keep that shift sane, and avoid the mess of the 80s and 90s. The implementations are almost done now, and my first tool will be a command line utility that reads one format and spits out the other, so that you can take a binary dump from your production system using tcpdump or wireshark or whatever, and then convert it to a human readable format to see what's going on. I'll probably even put in a hex-reader so that you can log the raw message and then read it back: 2019-10-02:15:00:32: Received message [01 76 85 6e 75 6b 65 73 88 6c 61 75 6e 64 68 65 64 79] $ ceconv --hex 01 76 85 6e 75 6b 65 73 88 6c 61 75 6e 64 68 65 64 79 v1 { nukes = launched }

userbinator 7y ago

I've worked with lots of binary formats, and they are equally "human-readable" (and far less unambiguous in a lot of cases) once you get used to it. As a coworker once said, "everything is human-readable if you have a hex editor." It seems your format is yet another attempt at reinventing ASN.1.

kstenerud 7y ago
But the point is that binary formats shouldn't require looking at hex data just to make sense of them. Especially not nowadays. Once you're stuck parsing binary data by hand (which is tedious), you've lost the amazing benefit that text formats gave us. And editing binary formats is even worse. That's why I split it into compatible text and binary formats. You transmit/store in binary, and read/edit in text, and the computer deals with the conversion automatically.
- kortex 7y ago
  I think this is the direction things are heading. Currently, I use yaml for tons of configs and the like because it's super easy to modify. I use jq to process because jq is the bee's knees. There is a yq out there which can read most config formats, but it doesn't support the full jq syntax, so instead I pipe everything through `yq read -j -` to convert to json and then `| jq 'query'`. works a treat. I can see with a few more small parts an application which persists as bson, streams / tempfile / and/or mmaps to yaml/hcl/whatever you like, opens that in your editor of choice, and translates back to bson.

Const-me 7y ago

> Data movement between processes and machines is also horribly wasteful I agree, and in .NET that’s a solved problem: http://const.me/articles/net-tcp/ http://const.me/articles/net-tcp/ (2010). Their text format is XML. Their binary version takes small fraction of bandwidth, especially when using a custom pre-shared XML dictionary. Binary is much cheaper to produce or parse, the built-in DataContractSerializer understands both formats. Supported by all modern editions of .NET including .NET Core and is compatible across them, e.g. I’ve been recently using it for a network protocol of a Linux ARM device.

kortex 7y ago

This is pretty cool! I really like that dates (and timezone info) are first class types. Often neglected, time ser/de has been a monkeywrench in many a project. Protobufs are neat but `protoc` is an absolute nightmare. I would warn against going down the path of transpiling to lang-specific message files if at all possible (I don't think you are taking that approach, but github freezes my phone browser, I'll look more on my dev machine) Great stuff! I would love to see this get more traction. Interchange formats are a classic victim of critical mass necessity.