2 ms·

One thing that really gets under my skin when I think about observability data is the abject waste we incur by shipping all this crap around as UTF-8 bytes. Thi

One thing that really gets under my skin when I think about observability data is the abject waste we incur by shipping all this crap around as UTF-8 bytes. This post (from 1996!) puts us all to shame: https://lists.w3.org/Archives/Public/www-logging/1996May/0000.html https://lists.w3.org/Archives/Public/www-logging/1996May/000... Knowing the type of each field unlocks some interesting possibilities. If we can classify fields as STRING, INTEGER, UUID, FLOAT, TIMESTAMP, IP, etc we could store (and transmit!) them optimally. In particular, knowing whether we can delta-encode is important--if you have a timestamp column, storing the deltas (with varint or vbyte encoding) is way cheaper than storing each and every timestamp. Only store each string once, in a compressed way, and refer to it by ID (with smaller IDs for more frequent strings). It's sickening to imagine how much could be saved by exploiting redundancy in these data if we could just know the type of each field. You get some of this with formats like protocol buffers, but not enough. Another thing, as you mention, is optimizing for search. Indexing everything seems like the wrong move. Maybe some partial indexing strategy? Rollups? Just do everything with mapreduce jobs? I don't know what the right answer is but fully indexing data which are mostly write-only is definitely wrong.

heyoni 3y ago
That second part of only storing reference ids sounds like db normalizing but for logs. Seems reasonable though!
fnordpiglet 3y ago
In my analysis of some exabyte scale log analytic systems the top two compute spend buckets was json serde and utf8 conversions.
ablob 3y ago
Storing by delta can bite you quite hard in the event of data corruption. Instead of 1 data point being affected it would cascade down. Selecting specific ranges where the concrete bottom/top as in "give me everything between 1-2 pm from last Saturday" might also become problematic. I'm sure there's a tradeoff to be had here; Weaving data-dependencies throughout your file certainly leaves a redundancy hole not everyone is willing to have.
- jcgrillo 3y ago
  I think we could limit the blast radius by working in reasonably sized chunks--like O(10-100MB)--and possibly replicating (which becomes much more attractive when the data set gets a lot smaller). But you're right, it's a good point that redundancy can be a feature.

heyoni 3y ago

That second part of only storing reference ids sounds like db normalizing but for logs. Seems reasonable though!

fnordpiglet 3y ago

In my analysis of some exabyte scale log analytic systems the top two compute spend buckets was json serde and utf8 conversions.

ablob 3y ago

Storing by delta can bite you quite hard in the event of data corruption. Instead of 1 data point being affected it would cascade down. Selecting specific ranges where the concrete bottom/top as in "give me everything between 1-2 pm from last Saturday" might also become problematic. I'm sure there's a tradeoff to be had here; Weaving data-dependencies throughout your file certainly leaves a redundancy hole not everyone is willing to have.

jcgrillo 3y ago
I think we could limit the blast radius by working in reasonably sized chunks--like O(10-100MB)--and possibly replicating (which becomes much more attractive when the data set gets a lot smaller). But you're right, it's a good point that redundancy can be a feature.