4 ms·

> On the bright side, at least IP, TCP, etc. weren't designed in the same way... And yet almost anyone can implement an HTTP server with their own parser, that

by zzzcpan 11y ago

> On the bright side, at least IP, TCP, etc. weren't designed in the same way... And yet almost anyone can implement an HTTP server with their own parser, that works for every client out there in just a few weeks, but not TCP stack or DNS server. It is incorrect to assume, that parsing text is hard. It never was and never will be. It's the other way around.

userbinator 11y ago
And yet almost anyone can implement an HTTP server with their own parser, that works for every client out there in just a few weeks, but not TCP stack or DNS server. That has little to do with binary vs. text. It's because [1] TCP is itself a very complex protocol, especially to get all the corner cases right, and [2] DNS servers are not popular to write because there is usually very little reason to. If HTTP was a simple binary protocol you'd probably make the same observation.
- zzzcpan 11y ago
  I think it has everything to do with it. Because text protocols share a single textual encoding with lots of tools available and usually don't force you to parse anything you don't need. Put a few rules in your parser generator and you are done. Binary protocols, on the other hand, each use their own encoding and usually force you to decode every little thing before they can be used correctly. Which always makes them much more complicated and error-prone, than textual protocols.
  - andrewflnr 11y ago
    Text protocols "each use their own encoding" in exactly the same way as binary ones. The only difference you've really mentioned is that tools for handling them aren't as mature. There's also a difference in design, but I suspect that's because binary protocols are used in places where the complexity pays for itself, and anything else, including a text format, would be too fat.
dunkelheit 11y ago
I don't quite understand. TCP is hard because the state machine itself is complex. Parsing actual messages is trivial in comparison. One can just compare the length of the segment format description (a couple of pages in the original RFC) with the combined length of numerous RFCs detailing TCP behavior in various corner cases.
- andrewflnr 11y ago
  Actually, I think you understand perfectly.
MichaelGG 11y ago
This is simply incorrect. Most clients/servers will get edge cases wrong. Things like line endings. Or line folding. Comments in headers. Not even getting into sillier things that aren't just parsing, like absolute vs relative URI in request line vs host header. Or how some headers have different parse semantics. SIP uses the same format as HTTP and the number of basic interop (and security!) issues due to parsing is astounding. Parsing human writeable text is hard, proven via such interop issues. The key here is that this is "human writeable". If HTTP, for instance, eliminated free formatting and CRLF in favour of just LF that'd be a huge improvement. You could actually parse all headers, correctly, just doing: Var parts = msg.split(\n\n) Var hdrs = parts[0].split(\n).map(|x| { Var xs = x.split(:) xs.[0], xs.[1] } Instead, people write similar code, thinking it works, and might achieve interop with some popular implementations that handle things well enough. If they used a tight text protocol (which really is basically a bloated binary protocol, just eg with newline instead of null as a terminator), none of these encoding issues would exist. It'd still be pointlessly inefficient, but reduce basic parse issues. In fact the IETF has belatedly realised this for HTTP and the latest RFC kills spacing around colon and line folding. In the name of security, since it was causing actual issues since implementations weren't reading messages the same way. And funny enough, this updated HTTP1.1 RFC breaks preciously working implementations.
- developer2 11y ago
  Right. The problem isn't the text protocol, it's the loose RFCs and implementations. Ultimately, "text" and "binary" are not different things; text is a subset of binary. The difference is that for some reason, many developers wind up allowing human-friendly text protocols to be flexible in terms of parsing. There's no technical reason for this, as a text protocol can easily be defined with requirements just as strict as any binary protocol. I think with HTTP specifically, the problem is that implementations came first. There were already multiple implementations in the wild before the RFCs were written. Thus, the spec was written to accommodate the implementations rather than the other way around.
- zzzcpan 11y ago
  You are making a big deal out of nothing. Skip empty lines in front of a request and after its content, look for tabs/spaces after \n for multiline headers and that's it, your HTTP parser is ok. No point in parsing comments. And it's not wrong, it's just not RFC compliant. In this case it would even be wrong to comply with RFC. Actually, we've been parsing human writable text for half of a century. It's kind of a solved problem. We have good enough tools and DSLs to make binary protocols much harder to deal with.

by zzzcpan 11y ago

userbinator 11y ago

And yet almost anyone can implement an HTTP server with their own parser, that works for every client out there in just a few weeks, but not TCP stack or DNS server. That has little to do with binary vs. text. It's because [1] TCP is itself a very complex protocol, especially to get all the corner cases right, and [2] DNS servers are not popular to write because there is usually very little reason to. If HTTP was a simple binary protocol you'd probably make the same observation.

zzzcpan 11y ago
I think it has everything to do with it. Because text protocols share a single textual encoding with lots of tools available and usually don't force you to parse anything you don't need. Put a few rules in your parser generator and you are done. Binary protocols, on the other hand, each use their own encoding and usually force you to decode every little thing before they can be used correctly. Which always makes them much more complicated and error-prone, than textual protocols.
- andrewflnr 11y ago
  Text protocols "each use their own encoding" in exactly the same way as binary ones. The only difference you've really mentioned is that tools for handling them aren't as mature. There's also a difference in design, but I suspect that's because binary protocols are used in places where the complexity pays for itself, and anything else, including a text format, would be too fat.

dunkelheit 11y ago

I don't quite understand. TCP is hard because the state machine itself is complex. Parsing actual messages is trivial in comparison. One can just compare the length of the segment format description (a couple of pages in the original RFC) with the combined length of numerous RFCs detailing TCP behavior in various corner cases.

andrewflnr 11y ago
Actually, I think you understand perfectly.

MichaelGG 11y ago

This is simply incorrect. Most clients/servers will get edge cases wrong. Things like line endings. Or line folding. Comments in headers. Not even getting into sillier things that aren't just parsing, like absolute vs relative URI in request line vs host header. Or how some headers have different parse semantics. SIP uses the same format as HTTP and the number of basic interop (and security!) issues due to parsing is astounding. Parsing human writeable text is hard, proven via such interop issues. The key here is that this is "human writeable". If HTTP, for instance, eliminated free formatting and CRLF in favour of just LF that'd be a huge improvement. You could actually parse all headers, correctly, just doing: Var parts = msg.split(\n\n) Var hdrs = parts[0].split(\n).map(|x| { Var xs = x.split(:) xs.[0], xs.[1] } Instead, people write similar code, thinking it works, and might achieve interop with some popular implementations that handle things well enough. If they used a tight text protocol (which really is basically a bloated binary protocol, just eg with newline instead of null as a terminator), none of these encoding issues would exist. It'd still be pointlessly inefficient, but reduce basic parse issues. In fact the IETF has belatedly realised this for HTTP and the latest RFC kills spacing around colon and line folding. In the name of security, since it was causing actual issues since implementations weren't reading messages the same way. And funny enough, this updated HTTP1.1 RFC breaks preciously working implementations.

developer2 11y ago
Right. The problem isn't the text protocol, it's the loose RFCs and implementations. Ultimately, "text" and "binary" are not different things; text is a subset of binary. The difference is that for some reason, many developers wind up allowing human-friendly text protocols to be flexible in terms of parsing. There's no technical reason for this, as a text protocol can easily be defined with requirements just as strict as any binary protocol. I think with HTTP specifically, the problem is that implementations came first. There were already multiple implementations in the wild before the RFCs were written. Thus, the spec was written to accommodate the implementations rather than the other way around.
zzzcpan 11y ago
You are making a big deal out of nothing. Skip empty lines in front of a request and after its content, look for tabs/spaces after \n for multiline headers and that's it, your HTTP parser is ok. No point in parsing comments. And it's not wrong, it's just not RFC compliant. In this case it would even be wrong to comply with RFC. Actually, we've been parsing human writable text for half of a century. It's kind of a solved problem. We have good enough tools and DSLs to make binary protocols much harder to deal with.