2 ms·
CloudWatch Logs Insights, Snowflake, Grafana Loki all do minimally indexed queries with highly parallel brute force scans of tons and tons of smallish s3 object
by fnordpiglet 3y ago
CloudWatch Logs Insights, Snowflake, Grafana Loki all do minimally indexed queries with highly parallel brute force scans of tons and tons of smallish s3 objects. A lot of data, especially data like logs, the ratio of “retrieved:ingested” can be in excess of 1:100. That makes it very much worth while to not index up front but to pay the retrieval cost for that rare amount of data that is actually retrieved. The minimal indexing mentioned is typically time and sometimes bloom filteresque term existence indices at the object level.
- londons_explore 3y ago> CloudWatch Logs Insights, Snowflake, Grafana Loki And the common thing in this list of tools? They're all laggy and not a nice experience to use. Even if queries are rare, there is often an actual human sitting waiting for the results of a query. Whereas while ingesting logs, there is no human waiting. I would prefer to burn more CPU time to save some human time.
- fnordpiglet 3y agoI use all three and for large scales they’re considerably faster than the alternative; which at some scales (peta/exabyte) is “none.”