4 ms·
> Naive people find liberation in text-based protocols that don't require tools to understand. I don't think text is necessarily the problem here. Other proto
by CivBase 6y ago
> Naive people find liberation in text-based protocols that don't require tools to understand.
I don't think text is necessarily the problem here. Other protocols might allow you to use less memory, but they aren't running into a memory limitation in this instance.
You can divide parsers into two groups: parsers designed to read data which follows a specific schema and parsers designed to read arbitrary data.
Schema-aware parsers have additional context which allows them to process the data much more efficiently. The stricter the schema, the more efficient the parser can be. Meanwhile, parsers for arbitrary data have no such context and must rely completely on syntax to parse the data. A poorly optimized schema-aware parser may still outperform a well optimized parser for arbitrary data.
You can still encode data as text and even use popular syntaxes like JSON or YAML with schema-aware parsers. I've personally helped design a schema-aware parser for XML data. But you should keep in mind that most third-party parsers for those syntaxes will be designed to parse arbitrary data. Parsers for arbitrary data can still be very fast - fast enough for many purposes. However, schema-aware parsers have a significant advantage when it comes to computational performance.
- minitech 6y agoText is the problem, here and everywhere else. Text encourages looking for delimiters, escaping, and flexibility (optional whitespace, case insensitivity, leading zeros, redundant signs, newline style compatibility). Compared to binary with length prefixes, that’s really slow, prone to injection vulnerabilities and subtle incompatibility, requires copying, introduces more side channels (the length of your message now depends on the values of numbers inside it!), ….
- deathanatos 6y agoAnd once you get down to it, text still requires tools to parse. I routinely run stuff through `jq`, because despite JSON being a text format, actually consuming it as a human requires a tool. (Often to filter it, but sometimes just to colorize the output to make it legible.)
- CivBase 6y agoText =/= Variable Length I can store fixed-length data using text. A strict schema can specify things like fixed lengths, value padding, orders for maps, exact whitespace rules, etc which can entirely remove the need for delimiters. A schema could even require length prefixes, just like you might use for a non-text format. These concepts do not necessitate abandoning text. If you need the data to be readable/writable by a human with minimal tooling, you can still leverage text without sacrificing much parsing efficiency.
- minitech 6y agoIn theory, sure. In practice, nobody does, and length prefixes stop a format like that from being conveniently writable with a plain text editor.
- CivBase 6y agoI work on avionics and we definitely use fixed-length text all the time. Text is often much more convenient than blobs, even when adhering to strict schemas. Syntaxes like XML, JSON, YAML, etc have become popular for their convenience and versatility, but they do not represent the whole of what is possible with text.
- renox 6y agoYou've not used enough Ada: its variable length string is so clunky that fixed sized text is used instead..
- c-cube 6y agoThat's orthogonal. Msgpack is basically json but easier to parse from C, and less inefficient on floats and integers.