5 ms·

Even JSON is going to be "bloated". This cries out for a well-done, well-thought out binary format. While binary formats have certain disadvantages that never g

by jerf 6y ago

Even JSON is going to be "bloated". This cries out for a well-done, well-thought out binary format. While binary formats have certain disadvantages that never go away, a lot of the worst ones that we associate with them are the result of older standards that failed to contain enough extensibility. But there's a lot of good prior art to look at now. Heck, nowadays you could probably just define something as Protobuf (or your choice of similar standard, don't want to go too far into the weeds on the exact choice here) and be 95% of the way towards done. Even just a linearization of most of SVG into the obvious layout, simply dropping any problematic features (script, XLink, namespace extensions) would probably be a pretty good start that you could churn out in a week as a prototype. You'd get an efficient binary format, an obvious mechanism for translating existing SVG into it (with equally obvious mechanisms for warning users about the existence of untranslated features in the output; also an easy mechanism for scanning sample SVGs and getting a statistical idea of how many fit into your new scheme), an obvious mechanism for translating your format back into SVG, and serialization support as cross-platform as your choice of serialization mechanism right out of the box. If this is your goal, this has a pretty hard to beat bang-for-the-buck.

chrisseaton 6y ago
> that failed to contain enough extensibility Or too much abstraction and extensibility leading to excessive complexity!
hinkley 6y ago
There are tools for shrinking SVG that do some of what you say. One advantage of the text format is that you can reduce precision of the path data to a couple of significant figures. That would be hard to do with a binary format. You'd have to decide up front on 8 bytes or 16 bytes per coordinate pair.
- jerf 6y ago
  There's existing variable-length binary number formats [1] that already exist. SVG already has so many variable-length things in it that the downsides would probably be insignificant. I don't know if any of the existing Protobuf-like things have these built in, though. I know some of them have variable integers in them but I don't know about variable floats. This is kind of what I was getting at in that we have a lot of prior art now. Someone putting variable ints or floats into SVG no longer has to create their own bespoke format, and proceed to fall into the various traps themselves and accidentally write it forever into the spec; there's a lot more of these sorts of pieces that you can pick up off the shelf than there used to be. [1] https://github.com/regexident/var_float https://github.com/regexident/var_float
  - hinkley 6y ago
    I don't see how var_float lets you detect the length of a number field from the input. It seems to be designed to take arbitrary input from another function (which has already decided how big the data is out-of-band). Am I missing something, or can I not read 2 bytes from a file and detect if that's the end of the number or I need to read the next 2 bytes?
    - jerf 6y ago
      I dunno if that specific library has that feature, but it is certainly something that can be implemented. It's just a matter of programming.
    - afiori 6y ago
      yes, that is half of how utf8 works as an encoding: the first byte of a unit is either 0xxxxxxx or 11xxxxxx, in the first it is a single ascii byte and in the second case all the following bytes that start with 10xxxxxx are part of the same unit. for this purpose it would be enough to revert the order of the stream.
- Someone 6y ago
  Binary formats need not use fixed-length fields. You could use a format that stores small integers in 8 bits, yet allows for larger ones. See for example https://developers.google.com/protocol-buffers/docs/encoding#varints https://developers.google.com/protocol-buffers/docs/encoding.... You don’t even have to use byte-sized fields, but doing that makes encoding and decoding harder, and the overhead of also storing the actual lengths in bits of variable sized fields may be too much to make it worth that.
  - hinkley 6y ago
    I think you missed my point. SVG coordinates are effectively arbitrary precision. 3.5 bits per byte of arbitrary precision, but arbitrary precision nonetheless. There are no small integers, only scalar or vector values.
    - kevincox 6y ago
      varint style encoding is also arbitrary precision (although protobuf doesn't support arbitrary precision) It is trivial to get 7 bits per byte and you can do even better for large values by encoding the length upfront (this also makes decoding faster).
Const-me 6y ago
A format like this? https://en.wikipedia.org/wiki/Extensible_Binary_Meta_Language https://en.wikipedia.org/wiki/Extensible_Binary_Meta_Languag...
DemocracyFTW 6y ago
> Even JSON is going to be "bloated". This cries out for a well-done, well-thought out binary format Well unless it doesn't. I've recently read more than one blog post that claimed JSON was actually borderline faster—sometimes more than borderline—than some popular binary formats like protobufs. I won't link those benchmarks here because none were well enough conducted or presented to warrant a link; I myself treat those reports as plausible but anecdotal hints. It's not like saying "binary format" makes all your troubles go poof. You still have to parse it. Yes, you can control it better when you create it, but when you've created it and it sees a lot of adoption, then updating it will be just as troublesome as updating any popular format, meaning many clients won't support some of the new features for at least some time. Numbers of course are central in any actual SVG, and it turns out using a textual format to send numbers is not as stupid as it sounds, space-wise: `1` is just one byte, but it's 8 bytes (I think) of RAM when turned into a JS IEEE754 number. You can represent arbitrary precision or restrict yourself to integers on a whim when using text, something probably not true of most binary formats. One of my qualms with SVG are that XML itself is a needlessly complex format. As sort-of-proof for that claim I present you the fact that the [namespace on the `href` attribute has been elided from the standard](https://developer.mozilla.org/en-US/docs/Web/SVG/Element/use https://developer.mozilla.org/en-US/docs/Web/SVG/Element/use). This of course is just the tip of the iceberg of XML complexity. The other big Qualm is that for some reason the authors of SVG felt it necessary to open the doors for a lot of variation on how to write things like e.g. the `<path>` element's `d` attribute, like you can add commas or leave them out, you don't need whitespace in `"45-67"` because that can be parsed as `(45,-67)` and so on. Not useless stuff, but in the end more than minimal complexity. I find text-based formats so much better in many regards. Just look at the rich appearance of web pages today, almost all of that has been done using text-based formats. They're really powerful, and where needed, may be compressed using a generic algorithm.

by jerf 6y ago

chrisseaton 6y ago

> that failed to contain enough extensibility Or too much abstraction and extensibility leading to excessive complexity!

hinkley 6y ago

There are tools for shrinking SVG that do some of what you say. One advantage of the text format is that you can reduce precision of the path data to a couple of significant figures. That would be hard to do with a binary format. You'd have to decide up front on 8 bytes or 16 bytes per coordinate pair.

jerf 6y ago
There's existing variable-length binary number formats [1] that already exist. SVG already has so many variable-length things in it that the downsides would probably be insignificant. I don't know if any of the existing Protobuf-like things have these built in, though. I know some of them have variable integers in them but I don't know about variable floats. This is kind of what I was getting at in that we have a lot of prior art now. Someone putting variable ints or floats into SVG no longer has to create their own bespoke format, and proceed to fall into the various traps themselves and accidentally write it forever into the spec; there's a lot more of these sorts of pieces that you can pick up off the shelf than there used to be. [1] https://github.com/regexident/var_float https://github.com/regexident/var_float
- hinkley 6y ago
  I don't see how var_float lets you detect the length of a number field from the input. It seems to be designed to take arbitrary input from another function (which has already decided how big the data is out-of-band). Am I missing something, or can I not read 2 bytes from a file and detect if that's the end of the number or I need to read the next 2 bytes?
  - jerf 6y ago
    I dunno if that specific library has that feature, but it is certainly something that can be implemented. It's just a matter of programming.
  - afiori 6y ago
    yes, that is half of how utf8 works as an encoding: the first byte of a unit is either 0xxxxxxx or 11xxxxxx, in the first it is a single ascii byte and in the second case all the following bytes that start with 10xxxxxx are part of the same unit. for this purpose it would be enough to revert the order of the stream.
Someone 6y ago
Binary formats need not use fixed-length fields. You could use a format that stores small integers in 8 bits, yet allows for larger ones. See for example https://developers.google.com/protocol-buffers/docs/encoding#varints https://developers.google.com/protocol-buffers/docs/encoding.... You don’t even have to use byte-sized fields, but doing that makes encoding and decoding harder, and the overhead of also storing the actual lengths in bits of variable sized fields may be too much to make it worth that.
- hinkley 6y ago
  I think you missed my point. SVG coordinates are effectively arbitrary precision. 3.5 bits per byte of arbitrary precision, but arbitrary precision nonetheless. There are no small integers, only scalar or vector values.
  - kevincox 6y ago
    varint style encoding is also arbitrary precision (although protobuf doesn't support arbitrary precision) It is trivial to get 7 bits per byte and you can do even better for large values by encoding the length upfront (this also makes decoding faster).

Const-me 6y ago

A format like this? https://en.wikipedia.org/wiki/Extensible_Binary_Meta_Language https://en.wikipedia.org/wiki/Extensible_Binary_Meta_Languag...

DemocracyFTW 6y ago

> Even JSON is going to be "bloated". This cries out for a well-done, well-thought out binary format Well unless it doesn't. I've recently read more than one blog post that claimed JSON was actually borderline faster—sometimes more than borderline—than some popular binary formats like protobufs. I won't link those benchmarks here because none were well enough conducted or presented to warrant a link; I myself treat those reports as plausible but anecdotal hints. It's not like saying "binary format" makes all your troubles go poof. You still have to parse it. Yes, you can control it better when you create it, but when you've created it and it sees a lot of adoption, then updating it will be just as troublesome as updating any popular format, meaning many clients won't support some of the new features for at least some time. Numbers of course are central in any actual SVG, and it turns out using a textual format to send numbers is not as stupid as it sounds, space-wise: `1` is just one byte, but it's 8 bytes (I think) of RAM when turned into a JS IEEE754 number. You can represent arbitrary precision or restrict yourself to integers on a whim when using text, something probably not true of most binary formats. One of my qualms with SVG are that XML itself is a needlessly complex format. As sort-of-proof for that claim I present you the fact that the [namespace on the `href` attribute has been elided from the standard](https://developer.mozilla.org/en-US/docs/Web/SVG/Element/use https://developer.mozilla.org/en-US/docs/Web/SVG/Element/use). This of course is just the tip of the iceberg of XML complexity. The other big Qualm is that for some reason the authors of SVG felt it necessary to open the doors for a lot of variation on how to write things like e.g. the `<path>` element's `d` attribute, like you can add commas or leave them out, you don't need whitespace in `"45-67"` because that can be parsed as `(45,-67)` and so on. Not useless stuff, but in the end more than minimal complexity. I find text-based formats so much better in many regards. Just look at the rich appearance of web pages today, almost all of that has been done using text-based formats. They're really powerful, and where needed, may be compressed using a generic algorithm.