2 ms·
Exactly! I've been asked the followup question: "now assume that you have 8 petabytes of log files to search", which leads into distributed computing, realtime
by kstrauser 11y ago
Exactly! I've been asked the followup question: "now assume that you have 8 petabytes of log files to search", which leads into distributed computing, realtime vs on-demand collection, eventually consistent counters, the pros and cons of sampling vs exact counts, and so on.
The unbounded form of that question can be astonishingly hard, and for the kinds of jobs I'd be interviewing for I'd expect the challenging next questions.
I'd still answer along the lines of "for a quick solution I'd grep for the error level and count the number of lines, but that's not scalable".
- TheLoneWolfling 11y agoDitto. (Also, no need to pipe + count: `grep -c` works.) Also, problem with sampling: you oversample long lines. I'm not actually sure if there is any way to count probabilistically that doesn't. I suppose if you had some prior of how long the types were you may be able to do better, but in general, good luck.