5 ms·
You're missing a major feature. Text formats don't have any natural integer size limits. That's been a big win for HTTP. It's easy to imagine the original HT
by wmil 11y ago
You're missing a major feature. Text formats don't have any natural integer size limits.
That's been a big win for HTTP. It's easy to imagine the original HTTP authors thinking that 24 bits is plenty of space for file size (anything bigger should use FTP) or using seconds since 1970 in 32 bits instead of text dates.
Those are a bit obvious, but a developer could easily put in something like a 255 chunk limit without thinking too hard about it.
Even John Carmack accidentally screwed over OpenGL driver developers by copying the GL_EXTENSIONS list into a 1000 char buffer in GL Quake (that's way too small).
A binary format makes hard limits too easy.
- bdarnell 11y ago> You're missing a major feature. Text formats don't have any natural integer size limits. Sure they do. Where do you think Y2K came from? RFC 822 said "years are two digits" and people wrote parsers to that spec. Even now the official rule (AFAIK) is two or four digits, not an arbitrary number of digits. And even when the spec has no limit, implementations often intrude. For a more current example, JSON has no "natural" numeric size limits, but interoperability of integers greater than 2^53 is problematic because some implementations assume double-precision floating point while others can parse larger numbers into a 64-bit integer. Limits are a part of protocol design; a good protocol should make them clearly defined, suitably sized, and feasible to change.
- ThrustVectoring 11y agoEvery format has limits, sometimes people just choose to not document them. For example, try transmitting 3↑↑↑3 as an integer over HTTP.
- dnr 11y agoIf you use protocol buffers or similar libraries, you get variable-length integers (usually a 7 bits per byte encoding), which have no natural size limit. (You can go out of your way to get fixed-length integers in protocol buffers, but presumably you'd have a good reason because the default is variable-length.)
- kentonv 11y agoAll good formats -- text or binary -- are capable of being extended over time in a backwards-compatible way, in order to fix mistakes. Formats that don't allow this are bad and should not be used. Protobufs and Cap'n Proto in particular have very strong support for forwards- and backwards-compatibility. Choosing a too-small fixed integer size is one way that a format can screw up and need correction -- one which happens to be (somewhat) exclusive to binary formats. But there are plenty of ways text formats can screw up too, like trying to pack fundamentally structured data into an ad-hoc hard-to-parse string format in order to make it look more pleasant to humans (I'm looking at you, MIME type specifications), or failing to use consistent parsing rules (cookie and user-agent headers totally flout otherwise-consistent HTTP grammar rules, confounding well-written parsers).
- deleted 11y ago[deleted]
- mcv 11y agoIs this an unbiased argument for binary formats in general, or is it a sales pitch for Protobufs and Cap'n Proto? If a fairly recent tool is the only way to get binary formats right, then it's entirely understandable that they've fallen out of use.
- kentonv 11y agoObviously I am biased, but my intent is to make a technical argument, not a sales pitch. I don't make any money from these (and I no longer have anything to do with protobufs). Protobufs was open sourced in 2008 (after being in use inside Google since around 2001). I suspect other binary formats prior to that had a solution for compatibility as well, but I haven't done a survey.