12 ms·

I admire these old protocols that are intentionally built to be usable both by machines and humans. Like the combination of a response code, and a human readabl

by im3w1l 2y ago

I admire these old protocols that are intentionally built to be usable both by machines and humans. Like the combination of a response code, and a human readable explanation. A help command right in the protocol. Makes me think it's a shame that making a json based protocol is so much easier to whip up than a textual ebnf-specified protocol. Like imagine if in python there was some library that you gave an ebnf spec maybe with some extra features similar to regex (named groups?) and you could compile it to a state machine and use it to parse documents, getting a Dict out.

orf 2y ago
Unfortunately, in practice they are a nightmare. Look at the WHOIS protocol for an example. Humans don’t look at responses very much, so you should optimise for machines. If you want a human-readable view, then turn the JSON response into something readable.
- poincaredisk 2y ago
  Whois protocol has no grammar to speak of (the response body is not defined at all, just a text blob) which makes it a nightmare to parse. Having a proper response format would solve this. Though I agree, I prefer my responses in JSON.
  - tephra 2y ago
    If you want WHOIS with a better defined format we have a protocol for that! It's called RDAP https://en.wikipedia.org/wiki/Registration_Data_Access_Protocol https://en.wikipedia.org/wiki/Registration_Data_Access_Proto...
- astrobe_ 2y ago
  The next logical step is to use a machine-friendly format instead; that is a binary protocol. Even HTML and XML which were designed for readability and manual writing eventually became 'not usable enough" ("became" because I think part of it is that their success made them exposed to less technical populations), and now we have markdown everywhere which most of the times is converted to HTML. So if you are going to use a tool more sophisticated than Ed/Edlin to read and write (rich) text in a certain format, it could be more efficient to focus on making the job of the machine - and of the programmer, easier. If you look at a binary protocol such as NTP, the binary format leaves very little room for Postel's principle [1], so it is straightforward to make a program that queries a server and display the result. [1] https://en.wikipedia.org/wiki/Robustness_principle https://en.wikipedia.org/wiki/Robustness_principle
kragen 2y ago
maybe we could have a format that was more human-readable than json (or especially xml) but still reliably emittable and parseable? yaml, maybe, or toml, although i'm not that enthusiastic about them. another proposal for such a thing was ogdl (https://ogdl.org/ https://ogdl.org/), a notation for what i think are called rose trees > OGDL: Ordered Graph Data Language > A simple and readable data format, for humans and machines alike. > OGDL is a structured textual format that represents information in the form of graphs, where the nodes are strings and the arcs or edges are spaces or indentation. their example: network eth0 ip 192.168.0.10 mask 255.255.255.0 gw 192.168.0.1 hostname crispin another possibility is jevko; https://jevko.org/ https://jevko.org/ describes it and http://canonical.org/~kragen/rose/ http://canonical.org/~kragen/rose/ are some of my notes about the possibilities of similar rose-tree data formats
- hnlmorg 2y ago
  Formats like TOML are horrible for heavily nested data (even XML does a better job here) and the last time I checked, TOML didn’t support arrays at the top level. YAML is nicer than JSON to write, but I wouldn’t say it’s any nicer to read. If you want something that’s less punctuation heavy, then I’d prefer we go full Wirth and have something more akin to Pascal.
  - kragen 2y ago
    arrays at the top level are probably a bad idea for protocols that need to evolve in a backward-compatible way what do you mean about heavily nested data? do the other formats i linked do a better job there? i'm not sure it's possible to come up with a data format that will work well for such a wide range of use cases, but it sure would be nice to have. json is pretty great in terms of being able to load it into the browser, or visidata, or python, or js, or whatever
    - hnlmorg 2y ago
      > arrays at the top level are probably a bad idea for protocols that need to evolve in a backward-compatible way Depends on the protocol. It might be preferable to version the end point. Or if it’s a specific function, eg list-synonyms” then having a dictionary just to reference an array could be argued as unnecessary protocol bloat. Particularly given the aim of this exercise is readability. > what do you mean about heavily nested data? do the other formats i linked do a better job there? I mean a tree like structure. JSON and YAML are probably the best in class here. XML, for all of its warts, is good at handling nested data in a readable way too. TOML was more based around a flatter structure. > i'm not sure it's possible to come up with a data format that will work well for such a wide range of use cases, It’s not. The moment that happens, that format then becomes unwieldy and people then feel the urge to invent yet another new format to simplify things. It’s a vicious circle that happens over and over again in the tech sector.
donatj 2y ago
In my department's (we were formerly our own company) internal framework throwing .html on the end of any JSON response outputs it in nested HTML tables. I personally find it very helpful.
- hnlmorg 2y ago
  At that point you might as well drop JSON altogether and use an XHTML subset so your rendered output is also valid XML (instead of having two different and incompatible markups merged together)
  - zeven7 2y ago
    I’m assuming they use the .html trick for human reading of the data by developers rather than it being used in production
    - donatj 2y ago
      That's exactly what it's for
    - hnlmorg 2y ago
      Ahhh that makes more sense.
  - theamk 2y ago
    That sounds like a bad idea. Unlike JSON, XML is (1) non-trivial to parse safely, (2) hard to reliably extract information and (3) verbose. Leave the markup languages for intended purpose: text markup. Don't force them to carry data.
    - hnlmorg 2y ago
      I’m not generally a fan of XML either but what you posted there is just factually incorrect in just about every conceivable way. 1. There’s plenty of XML parsers already available for most languages. Yeah there have been high profile exploits based from XML but given the scale of XMLs usage, it’s fair to say those exploits are atypical usage where XML can be user supplied. And as long as you’re not allowing users to upload their own XML, then you get to control the schema so there isn’t any risks in using XML. 2. XMLs entire purpose is a data store. I’m not someone who likes to blame the developers for using their tools wrong but honestly, if you can’t unmarshal an XML schema you have control over then you’re not going to succeed with JSON either. 3. It is. But it’s also highly compressible because of its repetitive tags. So for HTTP endpoints, it actually doesn’t work out any different to JSON. > Leave the markup languages for intended purpose: text markup. Don't force them to carry data. You do realise the entire point of XML is to carry data? It might have fallen out of favour in recent years but those of us old enough to remember a time before JSON will talk about how JSON is just a simplified reimplementation of XML. And with things like JSON schemas, JSON is continuing to copy XML features.
dspillett 2y ago
> Makes me think it's a shame that making a json based protocol is so much… Maybe I'm not the human you are thinking of, being a techie, but I find a well structured JSON response, as long as it isn't overly verbose and is presented in open form rather than minified, to be a good compromise of human readable and easy to digest programmatically.
- marcosdumay 2y ago
  The legibility is probably one of the main reasons JSON got adopted. XML can be made to not look too bad, but in SOAP it must be unreadable, so everybody was looking into fixing this.
  - xmlmann 2y ago
    XML has a sweet advantage though. It can be styled in the browser. For example a sitemap that works for Google, OpenAI, &c. and is human readable looking like a web page. Example: https://www.wpbeginner.com/sitemap.xml https://www.wpbeginner.com/sitemap.xml
    - immibis 2y ago
      This is fun, but in reality, not all that useful. Except you can impress other nerds when they "view source" and there's not an HTML tag in sight.
  - immibis 2y ago
    XML is a language designed for markup (i.e. text formatting), and fitting structured documents into it creates an impedance mismatch. Dictionary definitions may be considered as marked-up documents, so it may work. The overall structure of the dictionary is not.
    - deleted 2y ago
      [deleted]
- keepamovin 2y ago
fouc 2y ago
textual ebnf-specified protocol > json
somat 2y ago
REST (REpresentational State Transfer) as a concept is very human orientated. The idea was a sort of academic abstraction of html. but it can be boiled down to: when you send a response, also send the entire application needed to handle that response. It is unfortunate that collectively we had a sort of brain fart and said "ok, REST == http, got it" and lost the rest of the interesting discussion about what it means to send the representational state of the process.
zzo38computer 2y ago
> I admire these old protocols ... The protocols that have a response code with an explanation is helpful. A help command is also helpful. So, I had written NNTP server that does that, and the IRC and NNTP client software I use can display them. > Makes me think it's a shame that making a json based protocol is so much easier to whip up ... I personally don't; I find I can easily work with text-based protocols if the format is easily enough. I think there are problems with JSON. Some of the problems are: it requires parsing escapes and keys/values, does not properly support character sets other than Unicode, cannot work with binary data unless it is encoded using base64 or hex or something else (which makes it inefficient), etc. There are other problems too. > Like imagine if in python there was some library that you gave an ebnf spec ... Maybe it is possible to add such a library in Python, if there is not already such things.
sixdimensional 2y ago
May I humbly submit “parsing expression grammars”[1] for your consideration? Fairly simple and somewhat fun.. Python has PEG parsing built in, but also the pyparsing or parsimonious modules too. I have built EDI X12 parsers and toy languages with this. [1] https://en.wikipedia.org/wiki/Parsing_expression_grammar https://en.wikipedia.org/wiki/Parsing_expression_grammar
- sixdimensional 2y ago
  Also lark in Python too

by im3w1l 2y ago

orf 2y ago

Unfortunately, in practice they are a nightmare. Look at the WHOIS protocol for an example. Humans don’t look at responses very much, so you should optimise for machines. If you want a human-readable view, then turn the JSON response into something readable.

poincaredisk 2y ago
Whois protocol has no grammar to speak of (the response body is not defined at all, just a text blob) which makes it a nightmare to parse. Having a proper response format would solve this. Though I agree, I prefer my responses in JSON.
- tephra 2y ago
  If you want WHOIS with a better defined format we have a protocol for that! It's called RDAP https://en.wikipedia.org/wiki/Registration_Data_Access_Protocol https://en.wikipedia.org/wiki/Registration_Data_Access_Proto...
astrobe_ 2y ago
The next logical step is to use a machine-friendly format instead; that is a binary protocol. Even HTML and XML which were designed for readability and manual writing eventually became 'not usable enough" ("became" because I think part of it is that their success made them exposed to less technical populations), and now we have markdown everywhere which most of the times is converted to HTML. So if you are going to use a tool more sophisticated than Ed/Edlin to read and write (rich) text in a certain format, it could be more efficient to focus on making the job of the machine - and of the programmer, easier. If you look at a binary protocol such as NTP, the binary format leaves very little room for Postel's principle [1], so it is straightforward to make a program that queries a server and display the result. [1] https://en.wikipedia.org/wiki/Robustness_principle https://en.wikipedia.org/wiki/Robustness_principle

kragen 2y ago

maybe we could have a format that was more human-readable than json (or especially xml) but still reliably emittable and parseable? yaml, maybe, or toml, although i'm not that enthusiastic about them. another proposal for such a thing was ogdl (https://ogdl.org/ https://ogdl.org/), a notation for what i think are called rose trees > OGDL: Ordered Graph Data Language > A simple and readable data format, for humans and machines alike. > OGDL is a structured textual format that represents information in the form of graphs, where the nodes are strings and the arcs or edges are spaces or indentation. their example: network eth0 ip 192.168.0.10 mask 255.255.255.0 gw 192.168.0.1 hostname crispin another possibility is jevko; https://jevko.org/ https://jevko.org/ describes it and http://canonical.org/~kragen/rose/ http://canonical.org/~kragen/rose/ are some of my notes about the possibilities of similar rose-tree data formats

hnlmorg 2y ago
Formats like TOML are horrible for heavily nested data (even XML does a better job here) and the last time I checked, TOML didn’t support arrays at the top level. YAML is nicer than JSON to write, but I wouldn’t say it’s any nicer to read. If you want something that’s less punctuation heavy, then I’d prefer we go full Wirth and have something more akin to Pascal.
- kragen 2y ago
  arrays at the top level are probably a bad idea for protocols that need to evolve in a backward-compatible way what do you mean about heavily nested data? do the other formats i linked do a better job there? i'm not sure it's possible to come up with a data format that will work well for such a wide range of use cases, but it sure would be nice to have. json is pretty great in terms of being able to load it into the browser, or visidata, or python, or js, or whatever
  - hnlmorg 2y ago
    > arrays at the top level are probably a bad idea for protocols that need to evolve in a backward-compatible way Depends on the protocol. It might be preferable to version the end point. Or if it’s a specific function, eg list-synonyms” then having a dictionary just to reference an array could be argued as unnecessary protocol bloat. Particularly given the aim of this exercise is readability. > what do you mean about heavily nested data? do the other formats i linked do a better job there? I mean a tree like structure. JSON and YAML are probably the best in class here. XML, for all of its warts, is good at handling nested data in a readable way too. TOML was more based around a flatter structure. > i'm not sure it's possible to come up with a data format that will work well for such a wide range of use cases, It’s not. The moment that happens, that format then becomes unwieldy and people then feel the urge to invent yet another new format to simplify things. It’s a vicious circle that happens over and over again in the tech sector.

donatj 2y ago

In my department's (we were formerly our own company) internal framework throwing .html on the end of any JSON response outputs it in nested HTML tables. I personally find it very helpful.

hnlmorg 2y ago
At that point you might as well drop JSON altogether and use an XHTML subset so your rendered output is also valid XML (instead of having two different and incompatible markups merged together)
- zeven7 2y ago
  I’m assuming they use the .html trick for human reading of the data by developers rather than it being used in production
  - donatj 2y ago
    That's exactly what it's for
  - hnlmorg 2y ago
    Ahhh that makes more sense.
- theamk 2y ago
  That sounds like a bad idea. Unlike JSON, XML is (1) non-trivial to parse safely, (2) hard to reliably extract information and (3) verbose. Leave the markup languages for intended purpose: text markup. Don't force them to carry data.
  - hnlmorg 2y ago
    I’m not generally a fan of XML either but what you posted there is just factually incorrect in just about every conceivable way. 1. There’s plenty of XML parsers already available for most languages. Yeah there have been high profile exploits based from XML but given the scale of XMLs usage, it’s fair to say those exploits are atypical usage where XML can be user supplied. And as long as you’re not allowing users to upload their own XML, then you get to control the schema so there isn’t any risks in using XML. 2. XMLs entire purpose is a data store. I’m not someone who likes to blame the developers for using their tools wrong but honestly, if you can’t unmarshal an XML schema you have control over then you’re not going to succeed with JSON either. 3. It is. But it’s also highly compressible because of its repetitive tags. So for HTTP endpoints, it actually doesn’t work out any different to JSON. > Leave the markup languages for intended purpose: text markup. Don't force them to carry data. You do realise the entire point of XML is to carry data? It might have fallen out of favour in recent years but those of us old enough to remember a time before JSON will talk about how JSON is just a simplified reimplementation of XML. And with things like JSON schemas, JSON is continuing to copy XML features.

dspillett 2y ago

> Makes me think it's a shame that making a json based protocol is so much… Maybe I'm not the human you are thinking of, being a techie, but I find a well structured JSON response, as long as it isn't overly verbose and is presented in open form rather than minified, to be a good compromise of human readable and easy to digest programmatically.

marcosdumay 2y ago
The legibility is probably one of the main reasons JSON got adopted. XML can be made to not look too bad, but in SOAP it must be unreadable, so everybody was looking into fixing this.
- xmlmann 2y ago
  XML has a sweet advantage though. It can be styled in the browser. For example a sitemap that works for Google, OpenAI, &c. and is human readable looking like a web page. Example: https://www.wpbeginner.com/sitemap.xml https://www.wpbeginner.com/sitemap.xml
  - immibis 2y ago
    This is fun, but in reality, not all that useful. Except you can impress other nerds when they "view source" and there's not an HTML tag in sight.
- immibis 2y ago
  XML is a language designed for markup (i.e. text formatting), and fitting structured documents into it creates an impedance mismatch. Dictionary definitions may be considered as marked-up documents, so it may work. The overall structure of the dictionary is not.
  - deleted 2y ago
    [deleted]
keepamovin 2y ago

fouc 2y ago

textual ebnf-specified protocol > json

somat 2y ago

REST (REpresentational State Transfer) as a concept is very human orientated. The idea was a sort of academic abstraction of html. but it can be boiled down to: when you send a response, also send the entire application needed to handle that response. It is unfortunate that collectively we had a sort of brain fart and said "ok, REST == http, got it" and lost the rest of the interesting discussion about what it means to send the representational state of the process.

zzo38computer 2y ago

> I admire these old protocols ... The protocols that have a response code with an explanation is helpful. A help command is also helpful. So, I had written NNTP server that does that, and the IRC and NNTP client software I use can display them. > Makes me think it's a shame that making a json based protocol is so much easier to whip up ... I personally don't; I find I can easily work with text-based protocols if the format is easily enough. I think there are problems with JSON. Some of the problems are: it requires parsing escapes and keys/values, does not properly support character sets other than Unicode, cannot work with binary data unless it is encoded using base64 or hex or something else (which makes it inefficient), etc. There are other problems too. > Like imagine if in python there was some library that you gave an ebnf spec ... Maybe it is possible to add such a library in Python, if there is not already such things.

sixdimensional 2y ago

May I humbly submit “parsing expression grammars”[1] for your consideration? Fairly simple and somewhat fun.. Python has PEG parsing built in, but also the pyparsing or parsimonious modules too. I have built EDI X12 parsers and toy languages with this. [1] https://en.wikipedia.org/wiki/Parsing_expression_grammar https://en.wikipedia.org/wiki/Parsing_expression_grammar

sixdimensional 2y ago
Also lark in Python too