4 ms·
Not sure, but seems like it will be producing apache arrow data and carrying it across the data stack end to end from OTEL. This would be great for creating dat
by theLiminator 1y ago
Not sure, but seems like it will be producing apache arrow data and carrying it across the data stack end to end from OTEL. This would be great for creating data without a bunch of duplication/redundant processing steps and exporting it in a form that's ready to query.
- piterrro 1y agoUnless I dont understand that fully (which could be the case). This idea could fly if downstream readers will be able to read it. Json is great because anything can read it, process, transform and serialize without having to know the intrisics of the protocol. Whats the point of using binary, columnar format for data in transit?
- arccy 1y agobetter compression https://opentelemetry.io/blog/2023/otel-arrow/ https://opentelemetry.io/blog/2023/otel-arrow/ You don't do high performance without knowing the data schema.
- odie5533 1y agoIs Arrow better than Parquet or Protobuf?
- theLiminator 1y agoArrow is an in-memory columnar format, kinda orthogonal to parquet (which is an at-rest format). Protobuf is a better comparison, but it's more message oriented and not suited for analytics.
- datadrivenangel 1y agoNot having to write to disk is great, and zero-copy in memory access is instant...
- arccy 1y agothe blog post comparison is against OTLP which is protobuf
- lquerel 1y ago[dead]