5 ms·

I was just thinking how lousy a data format JSON is for tabular data. Would sending it down in CSV improve things at all?

by pbreit 4y ago

ZeroCool2u 4y ago
If anything you'd probably want to send it in Arrow[1] format. CSV's don't even preserve data types. [1]: https://arrow.apache.org/ https://arrow.apache.org/
- alchemist1e9 4y ago
  arrow/feather is really the best format these days for tabular data transmission. anyone who disagrees I’d be very interested to hear your thoughts on alternatives.
  - pletnes 4y ago
    What about compression - is this part of arrow itself?
    - adgjlsfhk1 4y ago
      It's not part of Arrow, but Arrow is columnar so just a basic LZ4/ZSTD will work pretty well.
- pletnes 4y ago
  Or parquet, for compression?
  - FridgeSeal 4y ago
    Arrow is meant as the “in-memory” dual to Parquet, which is meant as the “on disk serialisation format”. Many parquet supporting libs will load Parquet files into an Arrow structure in memory for example.
- pbreit 4y ago
  Arrow looks super complicated. Are data types useful for data to/fro web/mobile clients? Encode type into the column header?
  - adgjlsfhk1 4y ago
    data types are absolutely helpful. when you know a column stores Float64 data, you don't have to write out float to base 10 and parse it back. You just dump the bytes.
ashvardanian 4y ago
Yes, we also constantly think about that! In the document collections of UKV, for example, we have interoperability between JSON, BSON, and MessagePack objects [1]. CSV is another potential option, but text-based formats aren't ideal for large scale transmissions. One thing people do - use two protocols. That is the case with Apache Arrow Flight RPC = gRPC for tasks, Arrow for data. It is a viable path, but compiling gRPC is a nightmare, and we don't want to integrate it into our other libraries, as we generally compile everything from sources. Seemingly, UJRPC can replace gRPC, and for the payload we can continue using Arrow. We will see :) [1]: https://github.com/unum-cloud/ukv/blob/main/src/modality_docs.cpp https://github.com/unum-cloud/ukv/blob/main/src/modality_doc...
- sally_glance 4y ago
  Are there any synergies with capnproto [1] or is the focus here purely on huge payloads? I'm just an interested hobbyist when it comes to performant RPC frameworks but had some fun benchmarking capnproto for a small gamedev-related project and it was pretty awesome. [1] https://capnproto.org/ https://capnproto.org/
loeg 4y ago
No, parsing CSV is also pretty slow. You want some sort of length prefixed and ideally fixed width column format.
- pbreit 4y ago
  Slower than JSON?
- mattewong 4y ago
  Parsing CSV doesn't have to be slow if you use something like xsv or zsv (https://github.com/liquidaty/zsv https://github.com/liquidaty/zsv) (disclaimer: I'm an author). The speed of CSV parsers is fast enough that unless you are doing something ultra-trivial such as "count rows", your bottleneck will be elsewhere. The benefits of CSV are: - human readable - does not need to be typed (sometimes, data in the raw such as date-formatted data is not amenable to typing without introducing a pre-processing layer that gets you further from the original data) - accessible to anyone: you don't need to be a data person to dbl-click and open in Excel or similar The main drawback is that if your data is already typed, CSV does not communicate what the type is. You can alleviate this through various approaches such as is described at https://github.com/liquidaty/zsv/blob/main/docs/csv_json_sqlite.md https://github.com/liquidaty/zsv/blob/main/docs/csv_json_sql..., though I wouldn't disagree that if you can be assured that your starting data conforms to non-text data types, there are probably better formats than CSV. The main benefit of Arrow, IMHO, is less as a format for transmitting / communicating but rather as a format for data at rest, that would benefit from having higher performance column-based read and compression
  - hermitcrab 4y ago
    If a CSV has quoting (e.g. because the data contains comma or quote chars) aren't you effectively forced to parse it in a single thread? See also: 'Why isn’t there a decent file format for tabular data?' https://news.ycombinator.com/item?id=31220841 https://news.ycombinator.com/item?id=31220841
    - mattewong 4y ago

I was just thinking how lousy a data format JSON is for tabular data. Would sending it down in CSV improve things at all?

by pbreit 4y ago

I was just thinking how lousy a data format JSON is for tabular data. Would sending it down in CSV improve things at all?

ZeroCool2u 4y ago

If anything you'd probably want to send it in Arrow[1] format. CSV's don't even preserve data types. [1]: https://arrow.apache.org/ https://arrow.apache.org/

alchemist1e9 4y ago
arrow/feather is really the best format these days for tabular data transmission. anyone who disagrees I’d be very interested to hear your thoughts on alternatives.
- pletnes 4y ago
  What about compression - is this part of arrow itself?
  - adgjlsfhk1 4y ago
    It's not part of Arrow, but Arrow is columnar so just a basic LZ4/ZSTD will work pretty well.
pletnes 4y ago
Or parquet, for compression?
- FridgeSeal 4y ago
  Arrow is meant as the “in-memory” dual to Parquet, which is meant as the “on disk serialisation format”. Many parquet supporting libs will load Parquet files into an Arrow structure in memory for example.
pbreit 4y ago
Arrow looks super complicated. Are data types useful for data to/fro web/mobile clients? Encode type into the column header?
- adgjlsfhk1 4y ago
  data types are absolutely helpful. when you know a column stores Float64 data, you don't have to write out float to base 10 and parse it back. You just dump the bytes.

ashvardanian 4y ago

Yes, we also constantly think about that! In the document collections of UKV, for example, we have interoperability between JSON, BSON, and MessagePack objects [1]. CSV is another potential option, but text-based formats aren't ideal for large scale transmissions. One thing people do - use two protocols. That is the case with Apache Arrow Flight RPC = gRPC for tasks, Arrow for data. It is a viable path, but compiling gRPC is a nightmare, and we don't want to integrate it into our other libraries, as we generally compile everything from sources. Seemingly, UJRPC can replace gRPC, and for the payload we can continue using Arrow. We will see :) [1]: https://github.com/unum-cloud/ukv/blob/main/src/modality_docs.cpp https://github.com/unum-cloud/ukv/blob/main/src/modality_doc...

sally_glance 4y ago
Are there any synergies with capnproto [1] or is the focus here purely on huge payloads? I'm just an interested hobbyist when it comes to performant RPC frameworks but had some fun benchmarking capnproto for a small gamedev-related project and it was pretty awesome. [1] https://capnproto.org/ https://capnproto.org/

loeg 4y ago

No, parsing CSV is also pretty slow. You want some sort of length prefixed and ideally fixed width column format.

pbreit 4y ago
Slower than JSON?
mattewong 4y ago
Parsing CSV doesn't have to be slow if you use something like xsv or zsv (https://github.com/liquidaty/zsv https://github.com/liquidaty/zsv) (disclaimer: I'm an author). The speed of CSV parsers is fast enough that unless you are doing something ultra-trivial such as "count rows", your bottleneck will be elsewhere. The benefits of CSV are: - human readable - does not need to be typed (sometimes, data in the raw such as date-formatted data is not amenable to typing without introducing a pre-processing layer that gets you further from the original data) - accessible to anyone: you don't need to be a data person to dbl-click and open in Excel or similar The main drawback is that if your data is already typed, CSV does not communicate what the type is. You can alleviate this through various approaches such as is described at https://github.com/liquidaty/zsv/blob/main/docs/csv_json_sqlite.md https://github.com/liquidaty/zsv/blob/main/docs/csv_json_sql..., though I wouldn't disagree that if you can be assured that your starting data conforms to non-text data types, there are probably better formats than CSV. The main benefit of Arrow, IMHO, is less as a format for transmitting / communicating but rather as a format for data at rest, that would benefit from having higher performance column-based read and compression
- hermitcrab 4y ago
  If a CSV has quoting (e.g. because the data contains comma or quote chars) aren't you effectively forced to parse it in a single thread? See also: 'Why isn’t there a decent file format for tabular data?' https://news.ycombinator.com/item?id=31220841 https://news.ycombinator.com/item?id=31220841
  - mattewong 4y ago