2 ms·
In the case i mention above of log data where data access patterns dominate storage without retrieval computing indices up front is absurdly expensive and chall
by fnordpiglet 3y ago
In the case i mention above of log data where data access patterns dominate storage without retrieval computing indices up front is absurdly expensive and challenging at extreme scales. In that case minimal time and perhaps probabilistic indices computed on ingestion and extremely parallel brute force search on search works out both in terms of unbounded scale and cost - by like one to two orders of magnitude.
- electroly 3y agoIndeed, we also use an unindexed store for our logs (having migrated to that from a system that was indexed). This is clearly not the "data is so trivially small that nothing matters" situation that the OP is describing in 4 of the 5 of their examples. I hope you can see this isn't at all incompatible with my post.
- fnordpiglet 3y agoI can. I’m just pointing out there are definitely cases where indexing sucks. I agree the original post doesn’t touch those cases.