6 ms·

It’s a pyramid. At the bottom you have CSV which is popular beyond belief, and has no real specification, with common cases wildly handled differently across l

It’s a pyramid. At the bottom you have CSV which is popular beyond belief, and has no real specification, with common cases wildly handled differently across libraries. In the middle you have JSON which isn’t 100% interoperable, but goes 98% of the way. And you have XML and protobufs at the top tip, who have strong mechanisms available for interoperability but at an operational cost that rarely justifies the upgrade from JSON. I suspect it will take a lot more that “well designed” and “better specified” to justify moving away from JSON as the default stepup from chaotic CSV like formats.

heresie-dabord 5y ago
I would describe it as a _pyramid of bounded viability_ , from the minimally viable to the feature-burdened maximum. CSV excels (ha) as a minimally viable exchange format for data. Combined with awk, grep, sed, bash, and Perl, and some simple SVG or D3 with SVG, the analytical solution is fast, scalable, and automatable. But CSV has limits. The column headers are the schema. Beyond these bounds, we have the other formats. JSON is messier. Its strength is in network/browser encapsulations and operations. As I have seen it used, people insert an array as a kind of schema, and they stay away from complex nesting where scalability starts failing and other difficult tooling must be summoned to compensate (parsing tools such as jq). Beyond JSON's bounds, we have XML and associated tooling. XML is versatile and expressive. XML and JSON can be written simply but both can be abused by programmers who aren't thinking beyond their own cursor. This is a rich set of tooling for data representation. In the end, one of the main advantages of CSV is that it remains a format that brings little tooling baggage ("ecosystem") to the task.
maple3142 5y ago
I am not sure if parsing XML is better than parsing JSON. Many languages or libraries' XML parser are dangerous by default. You usually need to manually configure your XML parser to be secure from XML-related attacks. Fortunately, some languages and libraries are going to make XML have a securer defaults, this is a good change. IMO, I think XML shouldn't have include many questionable features from security perspective.
- Sohcahtoa82 5y ago
  > Many languages or libraries' XML parser are dangerous by default. Seriously, XML External Entities is an incredibly dumb feature. To have it enabled by default makes it even worse.
- pdimitar 5y ago
  Agreed, and I like the libraries I saw in the past that deliberately only support a small subset of all XML extensions (sadly now I can't remember the names). Reducing attack surface and increasing sanity in one stroke is a policy that much more open-source software has to adopt.
- ievans 5y ago
  For those unfamiliar with these attack vectors, there code injection and denial-of-service issues that in previous version of Python, were exploitable by default. Projects like https://pypi.org/project/defusedxml/ https://pypi.org/project/defusedxml/ were designed to be secure against these issues by default, rather than requiring the library user to opt in. The defusedxml project has an excellent matrix showing viability of the attack types against various python XML implementations: https://pypi.org/project/defusedxml/#python-xml-libraries https://pypi.org/project/defusedxml/#python-xml-libraries
- makeitdouble 5y ago
  You are right, and XML parsers can have a very large attack surface due to the sheer amount of specs to adhere to. I see XML as better in the expressiveness it has, and more mature out of the box options to validate and transform it. Security and bugs remain an issue, but at the scale it can be used, there is a fighting chance to have experts dealing with the hardening of it all. Swagger like format definitions are still pretty lax in my option in comparison. Now I wouldn't want to get back to XML land, I just think it occupies a pretty solid niche that is hard to match with anything more simple.
- tannhaeuser 5y ago
  To be fair, XML wasn't intended as a data exchange format but as simplified SGML subset for use as delivery format on the web. While that largely hasn't happened, XML with XSD (sans rarely used feats) remains a strong exchange format for coarsely-grained inter-party traffic such as payment systems, taxes and other public/private data, etc. I'm guessing the security deficits you mention are XML entity attacks. Well, SGML has CAPACITY ENTLVL in the SGML declaration to limit expansion depth. And a markup authoring or delivery format without entities/text macros is quite useless, even though HTML, when seen as a stand-alone markup language rather than SGML vocabulary, lacks it.
petschge 5y ago
I'd argue the top of the pyramid is actually formats such as HDF5. That format was started to store voyager data and we can still read it after more than 40 years. It makes the format of entries extremely clear ("this is an 3d array of floating point numbers in IEEE755 format, with 64, 2 and 17 entries per dimension") and encourages further meat data ("it is in statV per centimeter and came out of channel of of the intrument") in addition to naming the data set "electric field". Compared to horrible piles of binary data that used to be common (and still are!), it's a breeze of fresh air to work with.
recursive 5y ago
> you have CSV which is popular beyond belief, and has no real specification What about RFC 4180? Works for me.
- deleted 5y ago
  [deleted]
- fomine3 5y ago
  Not work for rest of the world (called Excel)
jedimastert 5y ago
>CSV...has no real specification Bite your tongue sir! It has a GLORIOUS specification! https://datatracker.ietf.org/doc/html/rfc4180 https://datatracker.ietf.org/doc/html/rfc4180
- makeitdouble 5y ago
  I started reading with a light of hope I thought was long extinct, expecting this long awaited standardization of some kind, then on the first page: > Status of This Memo > This memo provides information for the Internet community. > It does not specify an Internet standard of any kind. My disappointment is immeasurable and my day is ruined.
  - deleted 5y ago
    [deleted]
mumblemumble 5y ago
I would argue that, in the long run, gRPC/protobuf has a lower operational cost than JSON as long as you don't need to talk to it from a browser. (Consuming it from a browser is a hassle because client-side JavaScript code is unable to speak the full gRPC API, so you need to fuss with reverse proxies to get everything working.) What it doesn't have is a short learning curve. In order to get started, you need to learn the *.proto format, and how to use the code generator, and all the design implementations of the different data types it supports, and all of that. But, once you get over that hump, it makes a lot of the hard stuff much, much easier to get right. What I keep wishing for is some sort of "gRPC-lite" that doesn't include quite as many questionable micro-optimizations as protobuf/gRPC, but does include all of the really good ideas like specification-first API development, service reflection, and a clean logical separation between HTTP semantics and the semantics of the API that's being implemented on top of it.
- nawgz 5y ago
  > gRPC/protobuf has a lower operational cost than JSON as long as you don't need to talk to it from a browser. So, reading this the other way (browsers are king)... you claim gRPC is so useless as to not be able to power your entire system and requires you to standup duplicate interchange systems for different use cases? Yikes.
  - mumblemumble 5y ago
    Browsers are king for some people, not others. For the stuff I'm working on, the browser is just the tip of the iceberg. For everything below the waterline, the operational benefits (strong static typing, well-defined backward- and forward-compatibility semantics, better throughput and latency characteristics) greatly outweigh the, "but we have to use a lightweight Envoy reverse proxy to expose some things to the browser," problem. I also have a tendency to consider that Envoy proxy to be more of a feature than a bug, anyway. It's pretty easy to set up, all told. We want to gatekeep the edge, anyway, for various reasons, so it's not like there was ever a reality in which we weren't going to be fussing with a reverse proxy. And it serves as a nice opportunity to stop and be thoughtful about exactly what we're choosing to expose to the Internet. Speaking purely as a developer, I do find it to be an annoyance. But I also acknowledge that inconveniencing developers for the sake of the greater good can be a wise move.
mst 5y ago
When faced with a case where SV is a natural fit, I've long been in the habit of (ass-u-ming I get to make that call, of course) specifying PostgreSQL COPY style TSV as the interchange format, and using more 'normal' TSV to make it easy to get data exports into Excel and friends. That's turned out to be rather less annoying than any other approach to SV I've tried over the years.

heresie-dabord 5y ago

I would describe it as a _pyramid of bounded viability_ , from the minimally viable to the feature-burdened maximum. CSV excels (ha) as a minimally viable exchange format for data. Combined with awk, grep, sed, bash, and Perl, and some simple SVG or D3 with SVG, the analytical solution is fast, scalable, and automatable. But CSV has limits. The column headers are the schema. Beyond these bounds, we have the other formats. JSON is messier. Its strength is in network/browser encapsulations and operations. As I have seen it used, people insert an array as a kind of schema, and they stay away from complex nesting where scalability starts failing and other difficult tooling must be summoned to compensate (parsing tools such as jq). Beyond JSON's bounds, we have XML and associated tooling. XML is versatile and expressive. XML and JSON can be written simply but both can be abused by programmers who aren't thinking beyond their own cursor. This is a rich set of tooling for data representation. In the end, one of the main advantages of CSV is that it remains a format that brings little tooling baggage ("ecosystem") to the task.

maple3142 5y ago

I am not sure if parsing XML is better than parsing JSON. Many languages or libraries' XML parser are dangerous by default. You usually need to manually configure your XML parser to be secure from XML-related attacks. Fortunately, some languages and libraries are going to make XML have a securer defaults, this is a good change. IMO, I think XML shouldn't have include many questionable features from security perspective.

Sohcahtoa82 5y ago
> Many languages or libraries' XML parser are dangerous by default. Seriously, XML External Entities is an incredibly dumb feature. To have it enabled by default makes it even worse.
pdimitar 5y ago
Agreed, and I like the libraries I saw in the past that deliberately only support a small subset of all XML extensions (sadly now I can't remember the names). Reducing attack surface and increasing sanity in one stroke is a policy that much more open-source software has to adopt.
ievans 5y ago
For those unfamiliar with these attack vectors, there code injection and denial-of-service issues that in previous version of Python, were exploitable by default. Projects like https://pypi.org/project/defusedxml/ https://pypi.org/project/defusedxml/ were designed to be secure against these issues by default, rather than requiring the library user to opt in. The defusedxml project has an excellent matrix showing viability of the attack types against various python XML implementations: https://pypi.org/project/defusedxml/#python-xml-libraries https://pypi.org/project/defusedxml/#python-xml-libraries
makeitdouble 5y ago
You are right, and XML parsers can have a very large attack surface due to the sheer amount of specs to adhere to. I see XML as better in the expressiveness it has, and more mature out of the box options to validate and transform it. Security and bugs remain an issue, but at the scale it can be used, there is a fighting chance to have experts dealing with the hardening of it all. Swagger like format definitions are still pretty lax in my option in comparison. Now I wouldn't want to get back to XML land, I just think it occupies a pretty solid niche that is hard to match with anything more simple.
tannhaeuser 5y ago
To be fair, XML wasn't intended as a data exchange format but as simplified SGML subset for use as delivery format on the web. While that largely hasn't happened, XML with XSD (sans rarely used feats) remains a strong exchange format for coarsely-grained inter-party traffic such as payment systems, taxes and other public/private data, etc. I'm guessing the security deficits you mention are XML entity attacks. Well, SGML has CAPACITY ENTLVL in the SGML declaration to limit expansion depth. And a markup authoring or delivery format without entities/text macros is quite useless, even though HTML, when seen as a stand-alone markup language rather than SGML vocabulary, lacks it.

petschge 5y ago

I'd argue the top of the pyramid is actually formats such as HDF5. That format was started to store voyager data and we can still read it after more than 40 years. It makes the format of entries extremely clear ("this is an 3d array of floating point numbers in IEEE755 format, with 64, 2 and 17 entries per dimension") and encourages further meat data ("it is in statV per centimeter and came out of channel of of the intrument") in addition to naming the data set "electric field". Compared to horrible piles of binary data that used to be common (and still are!), it's a breeze of fresh air to work with.

recursive 5y ago

> you have CSV which is popular beyond belief, and has no real specification What about RFC 4180? Works for me.

deleted 5y ago
[deleted]
fomine3 5y ago
Not work for rest of the world (called Excel)

jedimastert 5y ago

>CSV...has no real specification Bite your tongue sir! It has a GLORIOUS specification! https://datatracker.ietf.org/doc/html/rfc4180 https://datatracker.ietf.org/doc/html/rfc4180

makeitdouble 5y ago
I started reading with a light of hope I thought was long extinct, expecting this long awaited standardization of some kind, then on the first page: > Status of This Memo > This memo provides information for the Internet community. > It does not specify an Internet standard of any kind. My disappointment is immeasurable and my day is ruined.
- deleted 5y ago
  [deleted]

mumblemumble 5y ago

I would argue that, in the long run, gRPC/protobuf has a lower operational cost than JSON as long as you don't need to talk to it from a browser. (Consuming it from a browser is a hassle because client-side JavaScript code is unable to speak the full gRPC API, so you need to fuss with reverse proxies to get everything working.) What it doesn't have is a short learning curve. In order to get started, you need to learn the *.proto format, and how to use the code generator, and all the design implementations of the different data types it supports, and all of that. But, once you get over that hump, it makes a lot of the hard stuff much, much easier to get right. What I keep wishing for is some sort of "gRPC-lite" that doesn't include quite as many questionable micro-optimizations as protobuf/gRPC, but does include all of the really good ideas like specification-first API development, service reflection, and a clean logical separation between HTTP semantics and the semantics of the API that's being implemented on top of it.

nawgz 5y ago
> gRPC/protobuf has a lower operational cost than JSON as long as you don't need to talk to it from a browser. So, reading this the other way (browsers are king)... you claim gRPC is so useless as to not be able to power your entire system and requires you to standup duplicate interchange systems for different use cases? Yikes.
- mumblemumble 5y ago
  Browsers are king for some people, not others. For the stuff I'm working on, the browser is just the tip of the iceberg. For everything below the waterline, the operational benefits (strong static typing, well-defined backward- and forward-compatibility semantics, better throughput and latency characteristics) greatly outweigh the, "but we have to use a lightweight Envoy reverse proxy to expose some things to the browser," problem. I also have a tendency to consider that Envoy proxy to be more of a feature than a bug, anyway. It's pretty easy to set up, all told. We want to gatekeep the edge, anyway, for various reasons, so it's not like there was ever a reality in which we weren't going to be fussing with a reverse proxy. And it serves as a nice opportunity to stop and be thoughtful about exactly what we're choosing to expose to the Internet. Speaking purely as a developer, I do find it to be an annoyance. But I also acknowledge that inconveniencing developers for the sake of the greater good can be a wise move.

mst 5y ago

When faced with a case where SV is a natural fit, I've long been in the habit of (ass-u-ming I get to make that call, of course) specifying PostgreSQL COPY style TSV as the interchange format, and using more 'normal' TSV to make it easy to get data exports into Excel and friends. That's turned out to be rather less annoying than any other approach to SV I've tried over the years.