5 ms·

The reason CSV is popular is because it is (1) super simple, and (2) the simplicity leads to ubiquity. It is extremely easy to add CSV export and import capabil

The reason CSV is popular is because it is (1) super simple, and (2) the simplicity leads to ubiquity. It is extremely easy to add CSV export and import capability to a data tool, and that has come to mean that there are no data tools that don't support CSV format. Parquet is the opposite of simple. Even when good libraries are available (which it usually isn't), it is painful to read a Parquet file. Try reading a Parquet file using Java and Apache Parquet lib, for example. Avro is similar. Last I checked there are two Avro libs for C# and each has its own issues. Until there is a simple format that has ubiquitous libs in every language, CSV will continue to be the best format despite the issues caused by under-specification. Google Protobuf is a lot closer than Parquet or Avro. But Protobuf is not a splitable format, which means it is not Big Data friendly, unlike Parquet, Avro and CSV.

wetpaws 3y ago
[dead]
msla 3y ago
> there are no data tools that don't support CSV format. They support CSV but not your CSV. For example, how does quoting work? Does quoting work?
- rovr138 3y ago
  I was working with a vendor’s csv recently… They had never had a customer do X on Y field, so they never quoted it nor added code to quote it if needed.. Of course, we did X in one entry. Took me too long to find that which obviously messed up everything after.
kerkeslager 3y ago
> Google Protobuf is a lot closer than Parquet or Avro. But Protobuf is not a splitable format, which means it is not Big Data friendly, unlike Parquet, Avro and CSV. Eh, I don't think that's the problem. If that was the problem, there are a zillion ways to chunk files; .tar is probably the most ubiquitous but there are others. The bigger problem is that Protobuf is way harder to use. Part of the reason CSV is underspecified is it's simple enough it feels, at first glance, like it doesn't need specification. Protobuf has enough dark corners that I definitely don't know all of it, despite having used it pretty extensively. I think Unicode Separated Values (USV) is a much better alternative and as another poster mentioned, I hope it takes off.
neonsunset 3y ago
Avro is a terrible serialization format. Well, not necessarily the spec but all tooling around it is. It should never be picked over multitude of other, better options unless the company is, maybe, a Java shop.
arp242 3y ago
> Parquet is the opposite of simple. Even when good libraries are available (which it usually isn't), it is painful to read a Parquet file. Try reading a Parquet file using Java and Apache Parquet lib, for example. I skimmed their docs a bit: https://parquet.apache.org/docs/ https://parquet.apache.org/docs/ I would not look forward to implementing that. It all seems rather complex, and even worse: not actually all that well described. I suppose all the information is technically there, but it's really not a well-design well-written specification that's easy to implement. The documentation seems like an afterthought. This is probably why good libraries are rare.
- munchler 3y ago
  Thanks for the link. I couldn't even get past this part: > Apache Parquet is a columnar storage format available to any project in the Hadoop ecosystem Nope and nope.
  - arp242 3y ago
    Yeah, it all seems very application specific. I mean, for starters columnar storage isn't really appropriate for a lot of data. That's perfectly fine! Nothing wrong with any of this. Just means it's not a great candidate for a general application-agnostic data-exchange format.
Jensson 3y ago
> But Protobuf is not a splitable format, which means it is not Big Data friendly, unlike Parquet, Avro and CSV. What, when I worked at Google concatenating protobuf strings was a common way to concatenate protobufs, they are absolutely splittable. People might not know it but there is a reason they are designed like they are, it is to handle big data as you say. If you mean you can't split a single protobuf, sure, but you don't need to do that, it is like trying to split a single csv line, doesn't mean csv isn't splittable. Edit: Maybe some of that tooling to work with protobufs are internal only or not a part of the external protobuf packages though.
- mordae 3y ago
  > designed like they are I laughed. We don't call somebody writing a poor varint serializer for in-house use, then discovering that it doesn't handle negative numbers and floats that well so slapping a couple of hotfixes on top of it that make it necessary to have a protocol specification file, "design".
- breadwinner 3y ago
  Protobuf does not have built-in delimiters or sync markers between records, which makes it not possible to start reading from an arbitrary point in the middle of a Protobuf-encoded file and correctly interpret the data. That makes Protobuf not a splitable format.

wetpaws 3y ago

[dead]

msla 3y ago

> there are no data tools that don't support CSV format. They support CSV but not your CSV. For example, how does quoting work? Does quoting work?

rovr138 3y ago
I was working with a vendor’s csv recently… They had never had a customer do X on Y field, so they never quoted it nor added code to quote it if needed.. Of course, we did X in one entry. Took me too long to find that which obviously messed up everything after.

kerkeslager 3y ago

> Google Protobuf is a lot closer than Parquet or Avro. But Protobuf is not a splitable format, which means it is not Big Data friendly, unlike Parquet, Avro and CSV. Eh, I don't think that's the problem. If that was the problem, there are a zillion ways to chunk files; .tar is probably the most ubiquitous but there are others. The bigger problem is that Protobuf is way harder to use. Part of the reason CSV is underspecified is it's simple enough it feels, at first glance, like it doesn't need specification. Protobuf has enough dark corners that I definitely don't know all of it, despite having used it pretty extensively. I think Unicode Separated Values (USV) is a much better alternative and as another poster mentioned, I hope it takes off.

neonsunset 3y ago

Avro is a terrible serialization format. Well, not necessarily the spec but all tooling around it is. It should never be picked over multitude of other, better options unless the company is, maybe, a Java shop.

arp242 3y ago

> Parquet is the opposite of simple. Even when good libraries are available (which it usually isn't), it is painful to read a Parquet file. Try reading a Parquet file using Java and Apache Parquet lib, for example. I skimmed their docs a bit: https://parquet.apache.org/docs/ https://parquet.apache.org/docs/ I would not look forward to implementing that. It all seems rather complex, and even worse: not actually all that well described. I suppose all the information is technically there, but it's really not a well-design well-written specification that's easy to implement. The documentation seems like an afterthought. This is probably why good libraries are rare.

munchler 3y ago
Thanks for the link. I couldn't even get past this part: > Apache Parquet is a columnar storage format available to any project in the Hadoop ecosystem Nope and nope.
- arp242 3y ago
  Yeah, it all seems very application specific. I mean, for starters columnar storage isn't really appropriate for a lot of data. That's perfectly fine! Nothing wrong with any of this. Just means it's not a great candidate for a general application-agnostic data-exchange format.

Jensson 3y ago

> But Protobuf is not a splitable format, which means it is not Big Data friendly, unlike Parquet, Avro and CSV. What, when I worked at Google concatenating protobuf strings was a common way to concatenate protobufs, they are absolutely splittable. People might not know it but there is a reason they are designed like they are, it is to handle big data as you say. If you mean you can't split a single protobuf, sure, but you don't need to do that, it is like trying to split a single csv line, doesn't mean csv isn't splittable. Edit: Maybe some of that tooling to work with protobufs are internal only or not a part of the external protobuf packages though.

mordae 3y ago
> designed like they are I laughed. We don't call somebody writing a poor varint serializer for in-house use, then discovering that it doesn't handle negative numbers and floats that well so slapping a couple of hotfixes on top of it that make it necessary to have a protocol specification file, "design".
breadwinner 3y ago
Protobuf does not have built-in delimiters or sync markers between records, which makes it not possible to start reading from an arbitrary point in the middle of a Protobuf-encoded file and correctly interpret the data. That makes Protobuf not a splitable format.