4 ms·

I much prefer bit packed protocols. It's easy to process them on either side and if you really need them to be human readable you can dump the readable to a lo

I much prefer bit packed protocols. It's easy to process them on either side and if you really need them to be human readable you can dump the readable to a log or write a tool to let you see it. But to be honest, after a little while, it's like the Matrix, you can just see what you're looking for.

axaxs 8y ago
There is a beauty to simple bit packed protocols. That said, they seem to always end up not a great idea. Wholesale changing protocols is hard, so people tend to try to tack features onto what exists. Look at the clusterfk that DNS has become, with various degrees of support, depending on the server.
- JustSomeNobody 8y ago
  > Wholesale changing protocols is hard... In my dozen plus years of working with them, I've not seen a large change like this. And TBH, there are way to structure them to make sure they handle changes. I mostly deal with hardware that has to be stable in the field for years, so maybe that's the difference.
- mhandley 8y ago
  On the other hand, you could look at SIP for an example of what happens when you've an extensible ASCII-based protocol, and everyone and their dog decides to extend it. It's not pretty, that's for sure. I've come to regard it as my success disaster.
walshemj 8y ago
One of my bosses could read the raw x.400 OSI networking packets and tell which ADMD it was from their botched implementation cough sprint cough I had to resort to a 409 dump
- JustSomeNobody 8y ago
  Well, I’m not that good but when I see our raw data I can pick out the fields and read the values easily enough. You really do just get used to a system and how to read it. There’re patterns to it.
mjevans 8y ago
Offhand, I agree but also want to suggest. Error replies should also include at least a short-text response (often along side a numeric one). Initial connection strings might also have a text-string that says something useful to humans, like what the protocol is; just like the various multimedia container formats that were developed on the Internet rather than by 'media companies'. Push complexity as far up the protocol stack as possible; but don't sacrifice useful extensiblity at lower levels if it makes sense. At the same time don't depend on that data staying the same (if it exists in another layer, it should be modifiable without breaking your actual protocol). FTP is a great example of a protocol that (because of connection multiplexing limitations) embeds data which should be low level in to a higher level. If an RFC exists that describes the protocol then packed is probably OK. If no RFC described protocol or method exists, try to get as close as possible with off the shelf stuff and prototype with human readable things where possible on top of that until solid requirements for a new RFC are refined.
- Gibbon1 8y ago
  > Error replies should also include at least a short-text response (often along side a numeric one). I always try and design in a debug mode. Turn it on and the destination will try to tell you exactly what you did wrong instead of stonewalling you. Painful experience has taught me that you really want unique start and stop tokens, message type, message length, rev field and checksum/mac always. That at least allows you to mechanically validate and dispatch packets/messages.
  - mjevans 8y ago
    Cannot upvote enough building it //in//; and shipping it. Conformant implementations must include it and allow admins to enable it. You reminded me of the horror of talking to closed source things where there isn't even useful server-side debugging data. Stuff just fails or gets dropped without informing anyone why it went bad. This also applies to things like my credit card - I would love to have the last week or two of even /failed/ attempts at using my card in my online statement. That would really, really help with figuring out if someone was trying to use my card, or if a given service that rejected use of my card even tried to hit the CC company. (This happened with a major travel site which probably did it's own processing; having a firm direction to push and solid data might have helped.)
ci5er 8y ago
IDK if you would say that ASN.1 (DER) is 'bit packed', but I still kind of like it. I had some young feller tell me some time back why ASN.1 was the anti-christ, but I can't remember (for the life of me why). Do you happen to know/remember why ASN.1 is 'bad'? I can write 8-bit assembler to it - I can write 32- or 64- bit compiled C to it, and it's pretty easy to create an FPGA pre-processor (and router) for it, and it certainly is more constrained than random JSON strings. What's wrong with ASN.1? Too old?
- wmf 8y ago
  ASN.1 has multiple ways to serialize the same message which opens it up to bugs in rarely-used code paths. And every ASN.1 implementation was rife with security holes. But mostly ASN.1 comes from the same "bad neighborhood" as CORBA, X.400, X.509, OSI protocols, etc.
  - ci5er 8y ago
    Fair enough - there were bugs in specs (like the X.509 letting you attach arbitrarily large image or blobs), but mostly I found that the compilers were buggy. Whenever (way back in the day) we (our/my company) did SET (secure 3P "secure" credit card protocol) with competitors (MS/HP/RSA/IBM/Netscape/etc.), because we compiled an interpreter from the spec, we were able to put in code-path switches depending on the counter party and adapt. Since they had a buggy compiler from a 3rd party - they could not. Was that an issue with ASN.1? Or crappy tooling that people used?

axaxs 8y ago

There is a beauty to simple bit packed protocols. That said, they seem to always end up not a great idea. Wholesale changing protocols is hard, so people tend to try to tack features onto what exists. Look at the clusterfk that DNS has become, with various degrees of support, depending on the server.

JustSomeNobody 8y ago
> Wholesale changing protocols is hard... In my dozen plus years of working with them, I've not seen a large change like this. And TBH, there are way to structure them to make sure they handle changes. I mostly deal with hardware that has to be stable in the field for years, so maybe that's the difference.
mhandley 8y ago
On the other hand, you could look at SIP for an example of what happens when you've an extensible ASCII-based protocol, and everyone and their dog decides to extend it. It's not pretty, that's for sure. I've come to regard it as my success disaster.

walshemj 8y ago

One of my bosses could read the raw x.400 OSI networking packets and tell which ADMD it was from their botched implementation cough sprint cough I had to resort to a 409 dump

JustSomeNobody 8y ago
Well, I’m not that good but when I see our raw data I can pick out the fields and read the values easily enough. You really do just get used to a system and how to read it. There’re patterns to it.

mjevans 8y ago

Offhand, I agree but also want to suggest. Error replies should also include at least a short-text response (often along side a numeric one). Initial connection strings might also have a text-string that says something useful to humans, like what the protocol is; just like the various multimedia container formats that were developed on the Internet rather than by 'media companies'. Push complexity as far up the protocol stack as possible; but don't sacrifice useful extensiblity at lower levels if it makes sense. At the same time don't depend on that data staying the same (if it exists in another layer, it should be modifiable without breaking your actual protocol). FTP is a great example of a protocol that (because of connection multiplexing limitations) embeds data which should be low level in to a higher level. If an RFC exists that describes the protocol then packed is probably OK. If no RFC described protocol or method exists, try to get as close as possible with off the shelf stuff and prototype with human readable things where possible on top of that until solid requirements for a new RFC are refined.

Gibbon1 8y ago
> Error replies should also include at least a short-text response (often along side a numeric one). I always try and design in a debug mode. Turn it on and the destination will try to tell you exactly what you did wrong instead of stonewalling you. Painful experience has taught me that you really want unique start and stop tokens, message type, message length, rev field and checksum/mac always. That at least allows you to mechanically validate and dispatch packets/messages.
- mjevans 8y ago
  Cannot upvote enough building it //in//; and shipping it. Conformant implementations must include it and allow admins to enable it. You reminded me of the horror of talking to closed source things where there isn't even useful server-side debugging data. Stuff just fails or gets dropped without informing anyone why it went bad. This also applies to things like my credit card - I would love to have the last week or two of even /failed/ attempts at using my card in my online statement. That would really, really help with figuring out if someone was trying to use my card, or if a given service that rejected use of my card even tried to hit the CC company. (This happened with a major travel site which probably did it's own processing; having a firm direction to push and solid data might have helped.)

ci5er 8y ago

IDK if you would say that ASN.1 (DER) is 'bit packed', but I still kind of like it. I had some young feller tell me some time back why ASN.1 was the anti-christ, but I can't remember (for the life of me why). Do you happen to know/remember why ASN.1 is 'bad'? I can write 8-bit assembler to it - I can write 32- or 64- bit compiled C to it, and it's pretty easy to create an FPGA pre-processor (and router) for it, and it certainly is more constrained than random JSON strings. What's wrong with ASN.1? Too old?

wmf 8y ago
ASN.1 has multiple ways to serialize the same message which opens it up to bugs in rarely-used code paths. And every ASN.1 implementation was rife with security holes. But mostly ASN.1 comes from the same "bad neighborhood" as CORBA, X.400, X.509, OSI protocols, etc.
- ci5er 8y ago
  Fair enough - there were bugs in specs (like the X.509 letting you attach arbitrarily large image or blobs), but mostly I found that the compilers were buggy. Whenever (way back in the day) we (our/my company) did SET (secure 3P "secure" credit card protocol) with competitors (MS/HP/RSA/IBM/Netscape/etc.), because we compiled an interpreter from the spec, we were able to put in code-path switches depending on the counter party and adapt. Since they had a buggy compiler from a 3rd party - they could not. Was that an issue with ASN.1? Or crappy tooling that people used?