6 ms·
I suppose you could saw that parsing any text-based protocol in general "Is a Minefield". They look so simple and "readable", which is why they're appealing ini
by userbinator 7y ago
I suppose you could saw that parsing any text-based protocol in general "Is a Minefield". They look so simple and "readable", which is why they're appealing initially, but parsing text always involves lots of corner-cases and I've always thought it a huge waste of resources to use text-based protocols for data that's not actually meant for human consumption the vast majority of the time.
Consider something as simple as parsing an integer in a text-based format; there may be whitespace to skip, an optional sign character, and then a loop to accumulate digits and convert them (itself a subtraction, multiply, and add), and there's still the questions of all the invalid cases and what they should do. In contrast, in a binary format, all that's required is to read the data, and the most complex thing which might be required is endianness conversion. Length-prefixed binary formats are almost trivial to parse, on par with reading a field from a struture.
- juliusmusseau 7y agoConsider only this: "1.001" I'll use JavaScript numeric literals here as my translation medium (ironic!): Norway locale parses it to: 1001 USA locale parses it to: 1.001 France locale parses it to: NaN https://docs.oracle.com/cd/E19455-01/806-0169/overview-9/index.html https://docs.oracle.com/cd/E19455-01/806-0169/overview-9/ind...
- mort96 7y agoNo. In a programming context, any norwegian or french programmer would expect that to evaluate to 1.001, not 1001 or NaN.
- eitland 7y agoSweet summer child, isn't that what people say these days? I and you can agree that the reasonable thing to do is to accept American encoding. (The below is somewhat simplified, the way I remember it on a late Saturday night 10+ years later.) The outsourced team of programmers from our software vendor did not. They (mostly) used the built in regional settings in Windows (best practice, don't reinvent the wheel) meaning we had to come up with ways to make sure the machines ran with wrong regional settings (since a lot of stuff was already serialized that way && critical parts of the software was hardcoded to use US standard.) Fun times ;-)
- userbinator 7y agoI remember Excel also has(had?) some very disturbing behaviour around the interaction of CSV files and the regional decimal point separator settings.
- kozziollek 7y agoNot sure about CSV files, but every time I'm pasting numbers generated by Python or Java I had to use Notepad or Excel itself to replace dots to commas because of (Windows'!) regional settings.
- deleted 7y ago[deleted]
- HelloNurse 7y agoIn a somebody sent this input to my program context, designing some means to choose between these three results (or possibly others) is the main concern. Actual parsing is the easy part.
- juliusmusseau 7y agoDo you get the user to choose? Great, now the UI is 2x more complex. Do you just go with what the browser or client locale says? (User lives in Canada but their laptop is set to Turkish locale). Go with the locale tied to the geo location of the user instead? (User above stubbornly enters all amounts using Norwegian conventions after completing high school in Norway). If it's important, a confirmation page presented to the user and formatted in their presumed locale can help a lot.
- robocat 7y agoBinary formats have their own serious problems. > Length-prefixed binary formats are almost trivial to parse They definitely are not, as displayed by the fact that binary lengths are the root cause of a huge number of security flaws. JSON mostly avoids that. > the most complex thing which might be required is endianness conversion That is a gross simplification. When you look at the details of binary representations, things get complex, and you end up with corner cases. Let's look at floating point numbers: with a binary format you can transmit NaN, Infinity, -infinity, and -0. You can also create two NaN numbers that do not have the same binary representation. You have to choose single or double precision (maybe a benefit, not always). Etc. Similarly in JSON integers or arrays of integers are nothing special. It is mostly a benefit not to have to specify UInt8Array. JSON is one of many competitors within an ecology of programming, including binary formats, and yet JSON currently dominates large parts of that ecology. So far a binary format mutation hasn't beaten JSON, which is telling since binary had the early advantage (well: binary definitely wins in parts of the ecology, just as JSON wins in other parts).
- magila 7y ago> They definitely are not, as displayed by the fact that binary lengths are the root cause of a huge number of security flaws. JSON mostly avoids that. I assume you're mainly referring to buffer overflows, which are a problem with text based formats too. See for example the series of overflow vulnerabilities in IIS's HTTP parser which lead to some of the most disruptive worms in history like Code Red. Really this is more of a problem with memory-unsafe languages than serialization formats. > Let's look at floating point numbers: with a binary format you can transmit NaN, Infinity, -infinity, and -0 Depending on the use case being able to encode these values may be a requirement, in which case binary is no worse than text. > You can also create two NaN numbers that do not have the same binary representation. This is specific to IEEE 754, not all binary representations have this issue. Text based formats also have far more pervasive problems with lacking a canonical representation so it's hard to count this as a point against binary. > JSON is one of many competitors within an ecology of programming, including binary formats, and yet JSON currently dominates large parts of that ecology. This is just an appeal to popularity fallacy.
- skywhopper 7y agoThis is a little too generous about the benefits of binary formats vs text formats. Ultimately, any data exchange between disparate systems is going to be a challenging task, no matter what format you choose. Both sides have to implement it in a compatible way. And ultimately, every format is a binary format. Encoding machine-level data structures direct on the wire sounds good, but it quickly gets complicated when you have to deal with multiple architectures and languages. And you don't have the benefit of the gradually accreted de-facto conventions like using UTF-8 encoding for text-based formats to fall back on, much less the ability for humans to troubleshoot by being able to read the wire protocol. With sufficient discipline and rigor, and a good suite of tests, developed over years of practical experience, you can evolve a good general binary wire protocol, but by then it will turn out to be so complicated and heavyweight to use, that some upstart will come up with a NEW FANTASTIC format that doesn't have any of the particular annoyances of your rigorous protocol, and developers will flock to this innovative and efficient new format because it will help them get stuff done much faster, and most of them won't run into the edge cases the new format doesn't cover for years, and then some of them will write articles like this one and comments like yours and we can repeat the cycle every 10-20 years, just like we've been doing.
- pnx 7y ago>Consider something as simple as parsing an integer in a text-based format; there may be whitespace to skip, an optional sign character, and then a loop to accumulate digits and convert them (itself a subtraction, multiply, and add), and there's still the questions of all the invalid cases and what they should do. ^[ *][-?][0-9][0-9]*[ *]$ You're welcome. Anything that passes that regex is a valid number. Now using that as a basis of a lexer means that you can store any int in whatever precision you feel like. It's unfortunate that the majority of programmers these days are so computer illiterate that they can't write a parser for matching parens and call you an elitist for pointing out this is something anyone with a year of programming should be able to do in their sleep.
- userbinator 7y agoJust matching that regex alone is going to take a pretty large number of instructions (at least dozens.) That's not "simple" by any measure.
- pnx 7y agoProbably tens of thousands. And it will still be a faster operation than fetching the next json file from disk by an order of magnitude, or two.
- deleted 7y ago[deleted]
- IloveHN84 7y agoWait wait. XML with XSD Schemas are a easy problem. You can't fail with a XSD schema on place