4 ms·

> If anyone wants to pay me to write a sane log storage and searching system... send me an email ;) You can pry lingo/sawzall from my cold, dead hands.

• jrockway 7y ago
  I am thinking more along the lines of interactive querying, not analysis. Show me all log lines for this request ID, or show me all log lines for the last 5 minutes from every instance of the job, etc. Google has a system for that... but when I was there, it was awful. Meanwhile in the real world, we have ELK... and it's even worse. People stop looking at logs once the kubelet rotates it. It's just too slow and flaky. (But yes... one key aspect to lingo/sawzall's design is that logs are sharded. And naturally, logs are sharded. Each program produces a log file over a period of time, and so (time, pod) forms a natural shard. Introduce something like ELK, and your sharding is thrown away, so you can never properly parallelize searching. A properly designed logging system would maintain shards and ensure that workers have replicas of those shards, so that you can use lots of computers to quickly get you the result you want. Of course, as much should be indexed as practical, so you can find the shard you're looking for without looking at every shard. Lots of work that could be done here, and it's all super easy. That's why it makes me mad that nobody has done this.)
  • jsmeaton 7y ago
    Have you used sumologic or splunk? I feel like they both have the capabilities you’re talking about.