3 ms·

It's become a very common interchange format, even internally; it's also easy to deflate. I have had to work on codebases where CSV was being pumped out at basi

It's become a very common interchange format, even internally; it's also easy to deflate. I have had to work on codebases where CSV was being pumped out at basically the speed of a NIC card (its origin was Netflow, and then aggregated and otherwise processed, and the results sent via CSV to a master for further aggregation and analysis). I really don't get, though, why people can't just use protocol buffers instead. Is protobuf really that hard?

nobleach 2y ago
Extremely hard to tell an HR person, "Right-click on here in your Workday/Zendesk/Salesforce/etc UI and export a protobuf". Most of these folks in the business world LIVE in Excel/Spreadsheet land so a CSV feels very native. We can agree all day long that for actual data TRANSFER, CSV is riddled with edge cases. But it's what the customers are using.
- heavenlyblue 2y ago
  It's extremely unlikely they need to load spreadsheets large enough for 21Gb/s speed to matter
  - nobleach 2y ago
    Oh absolutely! I'm just mentioning why CSV is chosen over Protobufs.
  - SteveNuts 2y ago
    You’d be surprised. Big telcos use CSV and SFTP for CDR data, and there’s a lot of it.
matja 2y ago
Kind of, there isn't a 1:1 mapping of protobuf wire types to schema types, so you need to package the protobuf schema with the data and compile it to parse the data, or decide on the schema before-hand. So now you need to decide on a file format to bundle the schema and the data.
bombela 2y ago
protobuf is more friction, and actually slow to write and read. For better or worse, CSV is easy to produce via printf. Easy to read by breaking lines and splitting by the delimiter. Escaping delimiters part of the content is not hard, though often added as an afterthought. Protobuf requires to install a library. Understand how it works. Write a schema file. Share the shema to others. The API is cumbersome. Finally to offer this mutable struct via setter and getter abstraction, with variable length encoded numbers, variable length strings etc. The library ends up quite slow. In my experience protobuf is slow and memory hungry. The generated code is also quite bloated, which is not helping. See https://capnproto.org/ https://capnproto.org/ for details from the original creator of protobuf. Is CSV faster than protobuf? I don't know, and I haven't tested. But I wouldn't be surprised if it is.
- raron 2y ago
  > For better or worse, CSV is easy to produce via printf. Easy to read by breaking lines and splitting by the delimiter. Escaping delimiters part of the content is not hard, though often added as an afterthought. Based on the amount of software I seen producing broken CSV or can't parse (more-or-less) valid CSV, I don't think that is true. It seems to be easy, because just printf("%s,%d,%d\n", ...) but it is full of edge cases most programmers don't think about.
  - elteto 2y ago
    Not an issue when you control both ends of the pipe. CSV is a great interchange format for tabular data, especially so if it's only/mostly numeric. If you need to pass tabular data from internal service X to internal service Y it's great. And it's really fast.
    - Neywiny 2y ago
      Hmmm if they're just internal tools, why not just an array of structs? No parsing needed. Can have optionals. Can't go faster than nothing.
      - 2y ago
to11mtm 2y ago
I'm not the biggest fan of Protobuf, mostly around the 'perhaps-too-minimal' typing of the system and the performance differentials present on certain languages in the library. e.x. I know in .NET space, MessagePack is usually faster than proto, I think similar is true for JVM. Main disadvantage is there's not good schema based tooling around it.

nobleach 2y ago

Extremely hard to tell an HR person, "Right-click on here in your Workday/Zendesk/Salesforce/etc UI and export a protobuf". Most of these folks in the business world LIVE in Excel/Spreadsheet land so a CSV feels very native. We can agree all day long that for actual data TRANSFER, CSV is riddled with edge cases. But it's what the customers are using.

heavenlyblue 2y ago
It's extremely unlikely they need to load spreadsheets large enough for 21Gb/s speed to matter
- nobleach 2y ago
  Oh absolutely! I'm just mentioning why CSV is chosen over Protobufs.
- SteveNuts 2y ago
  You’d be surprised. Big telcos use CSV and SFTP for CDR data, and there’s a lot of it.

matja 2y ago

Kind of, there isn't a 1:1 mapping of protobuf wire types to schema types, so you need to package the protobuf schema with the data and compile it to parse the data, or decide on the schema before-hand. So now you need to decide on a file format to bundle the schema and the data.

bombela 2y ago

protobuf is more friction, and actually slow to write and read. For better or worse, CSV is easy to produce via printf. Easy to read by breaking lines and splitting by the delimiter. Escaping delimiters part of the content is not hard, though often added as an afterthought. Protobuf requires to install a library. Understand how it works. Write a schema file. Share the shema to others. The API is cumbersome. Finally to offer this mutable struct via setter and getter abstraction, with variable length encoded numbers, variable length strings etc. The library ends up quite slow. In my experience protobuf is slow and memory hungry. The generated code is also quite bloated, which is not helping. See https://capnproto.org/ https://capnproto.org/ for details from the original creator of protobuf. Is CSV faster than protobuf? I don't know, and I haven't tested. But I wouldn't be surprised if it is.

raron 2y ago
> For better or worse, CSV is easy to produce via printf. Easy to read by breaking lines and splitting by the delimiter. Escaping delimiters part of the content is not hard, though often added as an afterthought. Based on the amount of software I seen producing broken CSV or can't parse (more-or-less) valid CSV, I don't think that is true. It seems to be easy, because just printf("%s,%d,%d\n", ...) but it is full of edge cases most programmers don't think about.
- elteto 2y ago
  Not an issue when you control both ends of the pipe. CSV is a great interchange format for tabular data, especially so if it's only/mostly numeric. If you need to pass tabular data from internal service X to internal service Y it's great. And it's really fast.
  - Neywiny 2y ago
    Hmmm if they're just internal tools, why not just an array of structs? No parsing needed. Can have optionals. Can't go faster than nothing.
    - 2y ago

to11mtm 2y ago

I'm not the biggest fan of Protobuf, mostly around the 'perhaps-too-minimal' typing of the system and the performance differentials present on certain languages in the library. e.x. I know in .NET space, MessagePack is usually faster than proto, I think similar is true for JVM. Main disadvantage is there's not good schema based tooling around it.