4 ms·

Not that the optimisation isn't interesting but if the query is column-based, wouldn't it make more sense to use a binary column-oriented format like Apache Par

by Youden 5y ago

Not that the optimisation isn't interesting but if the query is column-based, wouldn't it make more sense to use a binary column-oriented format like Apache Parquet [0]? Or perhaps a binary format like Protobuf, Webpack, Cap'n'proto etc.? Are there other concerns that cause JSON to make sense? [0]: https://parquet.apache.org/documentation/latest/ https://parquet.apache.org/documentation/latest/

BerislavLopac 5y ago
Our Amazon Ion [0], which is compatible with JSON. [0] https://amzn.github.io/ion-docs/ https://amzn.github.io/ion-docs/
tomnipotent 5y ago
Seq seems to be targeting application logs, where JSON (and jsonl) is a common format and it makes sense.
- jsnell 5y ago
  There's three data formats involved: 1. The one used for ingesting data. 2. The one used for storage. 3. The one used for exporting data. There is no reason for all three formats to be the same, and there are good reason for them not to be: the actual use cases are requiring very different properties from the data format. Since you'd expect the average log record to be ingested once, processed a gazillion times, and returned as output <<1 times, any data conversion costs would get amortized over a lot of processing. Even minor efficiency gains in the at-rest storage format should pay off quickly. Your answer is addressing format 1: the clients are using JSON anyway, so it makes sense for the ingestion format to be JSON. That's making the bet that any data conversion costs can't be paid back over the lifetime of the data. But they are already not using the raw client input as the storage format! At a minimum they're validating / minifying / canonicalizing the JSON, so they're already paying the cost of doing a format conversion. (But all of this is obvious, and clearly the people writing this service are smart and would have thought of it. So it seems like there's a non-obvious answer for using minified JSON for storage at rest.)

by Youden 5y ago

BerislavLopac 5y ago

Our Amazon Ion [0], which is compatible with JSON. [0] https://amzn.github.io/ion-docs/ https://amzn.github.io/ion-docs/

tomnipotent 5y ago

Seq seems to be targeting application logs, where JSON (and jsonl) is a common format and it makes sense.

jsnell 5y ago
There's three data formats involved: 1. The one used for ingesting data. 2. The one used for storage. 3. The one used for exporting data. There is no reason for all three formats to be the same, and there are good reason for them not to be: the actual use cases are requiring very different properties from the data format. Since you'd expect the average log record to be ingested once, processed a gazillion times, and returned as output <<1 times, any data conversion costs would get amortized over a lot of processing. Even minor efficiency gains in the at-rest storage format should pay off quickly. Your answer is addressing format 1: the clients are using JSON anyway, so it makes sense for the ingestion format to be JSON. That's making the bet that any data conversion costs can't be paid back over the lifetime of the data. But they are already not using the raw client input as the storage format! At a minimum they're validating / minifying / canonicalizing the JSON, so they're already paying the cost of doing a format conversion. (But all of this is obvious, and clearly the people writing this service are smart and would have thought of it. So it seems like there's a non-obvious answer for using minified JSON for storage at rest.)