15 ms·

Protocols and formats absolutely should not require decoding strings. I think you are mistaken. Can you name any well-established protocol or format that does n

• adgjlsfhk1 5y ago
  CSV?
• ironmagma 5y ago
  HTTP for one. It does something like assume ASCII-like until it encounters odd looking bytes and/or a meta encoding tag.
  • wyldfire 5y ago
    I think HTML tags like 'meta' are merely payload to HTTP, right? Presumably a markup language doesn't fit Andrew's criteria for "protocol"?
• 10000truths 5y ago
  Protobuf string fields are expected to be in UTF-8. Although I'd expect a sane implementation of a protobuf decoder to throw an exception or otherwise indicate an error if it receives a malformed protobuf encoding.
  • haberman 5y ago
    Protobuf decoders are expected to validate UTF-8 strings for syntax="proto3" files, but not syntax="proto2". The behavior diverges mostly for historical reasons. This is a validation pass only and it doesn't make any meaning of the code points, except to validate that none of them are surrogate code points (disallowed in UTF-8).
• _rend 5y ago
  JSON[1], for instance, specifies: 1. "JSON syntax describes a sequence of Unicode code points. JSON also depends on Unicode in the hex numbers used in the \u escapement notation." 2. "A JSON text is a sequence of tokens formed from Unicode code points that conforms to the JSON value grammar." 3. "A string is a sequence of Unicode code points wrapped with quotation marks (U+0022). All code points may be placed within the quotation marks except for the code points that must be escaped: quotation mark (U+0022), reverse solidus (U+005C), and the control characters U+0000 to U+001F. There are two-character escape sequence representations of some characters." 4. "Any code point may be represented as a hexadecimal escape sequence. The meaning of such a hexadecimal number is determined by ISO/IEC 10646. If the code point is in the Basic Multilingual Plane (U+0000 through U+FFFF), then it may be represented as a six-character sequence: a reverse solidus, followed by the lowercase letter u, followed by four hexadecimal digits that encode the code point." 5. "Note that the JSON grammar permits code points for which Unicode does not currently provide character assignments." JSON does require Unicode awareness, both for general parsing, and for correctly interpreting strings. Backslashes are allowed for special escape characters, which means that you must be aware of the format (and the encoding of the text) in order to be able to decode. Note that JSON also doesn't specify a required encoding, only Unicode correctness, so a parser may need to be able to handle multiple Unicode encodings and differentiate between them. The spec doesn't specify what to do with code points which are not understood as Unicode (given especially the allowance for unassigned characters), but explicitly-invalid Unicode should be rejected. [1] https://www.ecma-international.org/wp-content/uploads/ECMA-404_2nd_edition_december_2017.pdf https://www.ecma-international.org/wp-content/uploads/ECMA-4...
  • Spex_guy 5y ago
    Parsing this out of utf-8 encoding requires no knowledge of unicode or even utf-8. All of the relevant characters (reverse solidus, quotation mark, and control characters) are single byte characters in the ascii subset. These characters cannot be found inside multi-byte characters in utf-8 due to the design of the encoding. Converting the unicode character escape codes to utf-8 would require knowledge of utf-8 encoding, but this unescaping is not a feature that would be provided by the language regardless.
• pcwalton 5y ago
  By "decode a string" do you mean that you must apply some sort of Unicode transformation to it? If so, the HFS+ file system, which .dmg files often are, is an example: Unicode NFD normalization must be performed on all filenames because of case insensitivity requirements [1]. [1]: https://eclecticlight.co/2021/05/08/explainer-unicode-normalization-and-apfs/ https://eclecticlight.co/2021/05/08/explainer-unicode-normal...
• tdeck 5y ago
  The problem is you've invented your own definition of the terms "string" and "string handling" which doesn't seem to agree with the generally accepted usage of these words. You're fixated on the word "decode" despite that not being the original focus of the discussion, not being present in my comment or the original comment you're replying to. I could create a standard library called "arrayofbytes" that lets you, for example: - Search an array of bytes for a smaller array of bytes - Split up an array of bytes based on an array of delimiter bytes - Selectively convert an array of bytes in range 41-5A to bytes in the range 61-7A But wouldn't it be more appropriate to describe these as "string handling" functions using the generally accepted terminology of our profession?
  • somethingor 5y ago
    In Zig, those functions are in std.mem (distinct from std.ascii and std.unicode)
• joe_guy 5y ago
  Anything with case insensitivity, like domain names and email addresses, require knowledge of strings as strings. Case sensitivity, ordering, and case changing rules, are reasons not only to have strings, but extensive culture support built into them. String types also let you compare utf16 bytes to utf8 bytes, for example. To think that strings are opaque bytes is massively naive. It presumes one encoding exists, the input is always valid, in addition to what is said above. An example of what you're setting yourself up for, is an exploit based around differences in handling invalid utf8.
• mamcx 5y ago
  What??? > the spec does not tell you to decode any strings Which "spec" is that? My user ask for csv, see csv, upload csv, do SEARCH on CSV, transform them, etc. Even do some scripting on them. Nowhere, EVER "opaque encoded bytes" is on the spec of the users!
• lmm 5y ago
  The HTTP specification does tell you to behave differently depending on the presence or absence of specific values for case-insensitive headers. That requires decoding the header name strings.
  • coldtea 5y ago
    >That requires decoding the header name strings. Which should be ascii though no?
    • lmm 5y ago
      Maybe, but it's not "treating strings as opaque encoded bytes" which is what was asked.
      • lhorie 5y ago
        ASCII is kinda special in that it's both a way of encoding strings and it has character-to-byte equivalence. You can specify an HTTP header as being any number of characters between `\n\r` characters or any number of bytes between `0x0A 0x0D`, and the parsing code is going to compile to pretty much the same thing regardless of whether you know what `0x0A` means or not.
        • lmm 5y ago
          Right, but to do case-insensitive comparisons of header names you need to decode them as strings, not just byte sequences.
• chrismorgan 5y ago
  > * CSV: wrong. the spec does not tell you to decode any strings, nor is it necessary to have any unicode awareness in order to properly read and parse the data or deal with the delimiters. CSV is defined in terms of characters, not bytes, so CSV parsing does require you to be encoding-aware: if your CSV file is encoded as UTF-16, bytewise parsing will destroy the data.
  • Akira1364 5y ago
    That's not really true at all. The single byte / single character comma separators are all that matters there. As long as you directly acknowledge / exactly replicate whatever blob of data happens to be in between each set of commas (even if it is nonsense garbage text) then you're correctly parsing the CSV.
    • rini17 5y ago
      Well let's try :grinningface:,:grinningface: which is d83d de00 002c d83d de00 in UTF16BE. If you extract first column by naively cutting the blob bytewise before the comma you end up with d83d de00 00 with extra NUL byte, which is a problem. With UTF16LE you'd prepend NUL which is even worse.
      • Akira1364 5y ago
        I think you somewhat misunderstood the manner I was suggesting you'd be approaching things there, though nothing you've said is incorrect.
        • chrismorgan 5y ago
          Then I’m not sure what you were suggesting. The crux is that CSV operates upon strings, not bytes.
      • lhorie 5y ago
        > cutting the blob bytewise That doesn't make sense. If you're working with utf16, why would you slice bytewise? That's like slicing a zip file bytewise and wondering why it got corrupted. The whole point of the argument for string support at a library level (rather than assuming some sort of equivalence between stringness and its underlying byte buffer at the language level) is that fixed width bytes fundamentally cannot model human language characters unambiguously because what a "character" is depends on the encoding/decoding contract of the program manipulating the byte buffer. Assuming equivalence between 0x2C and `,` stems from the ancient history of ASCII, english and usage of C `char` as a mechanism to squeeze performance out of string operations by not properly supporting the full gamut of valid human language characters. For a low level language that might be used to implement protocols, it totally makes sense that foo.len is length in bytes, because you're pretty much never going to want to know number of grapheme clusters at a protocol level. It doesn't make sense for a language level .len to be length in terms of codepoint count because that assumes encoding, which is fundamentally a business logic level concern.
• alophawen 5y ago
  The tar / star format requires decoding strings of octal numbers. https://en.wikipedia.org/wiki/Tar_(computing)#Header https://en.wikipedia.org/wiki/Tar_(computing)#Header