3 ms·
This is a good post (and the regex crate in rust is _fantastic_), but I'm always a bit surprised when I find these tools were not built for general log searchin
by erulabs 8y ago
This is a good post (and the regex crate in rust is _fantastic_), but I'm always a bit surprised when I find these tools were not built for general log searching and analytics, but:
> What we want out of those files is incredibly tiny—a few thousand integers, labelled with names and version numbers. For example, “2018-10-25 Bundler 1.16.2 123456”, or “2018-10-25 platform x86_64-darwin17 9876”.
I'm almost positive that something like Prometheus could do this just as cheaply (run a single Prometheus node on free tier EC2 with a grafana container alongside) - there would be no massive log files to parse, just integers for the desired download counts, collected regularly and viewable in near-real-time (instead of waiting for the day's log-and-parse job to be completed).
I recently replaced a massively expensive "view count" system which was built using bloom filters and hadoop map-reduce with a very small piece of code that called `INCR` and `GET`
against a Redis Cluster - this allowed real-time view counts, for far less money, with exponentially less log storage, massively fewer in-house lines-of-code, and an API that could not possibly be made simpler (no Java or Hadoop or AWS knowledge required)
Why is it developers so badly want to parse lots and lots of access log data, rather than just increment a metric somewhere as an event happens?
AWS Glue feels like a product meant for people who refuse to take a step back and think about what they're building. We also evaluated that and found it would be hilariously expensive, had poor documentation, was a brand new product, had OOM issues, and _still_ people on my team wanted to use it rather than a system actually designed for metrics in the first place.
- nostrademons 8y agoOftentimes the interesting data isn't what the metric is, but how it correlates with different user populations and actions taken. That can be as simple as knowing that all of your pageviews with zero clickthroughs happened on IE and were because clicktracking was broken on that browser. You need to store the individual events to do that sort of analysis, and once you're in that situation you run into single-source-of-truth consistency issues that strongly encourage you to store them as a log. If that's not the situation your business is in, by all means just increment a counter. It's faster, simpler, and takes much less storage space. It sounds like the RubyGems situation in the article requires new metrics to be added on request, though, and it's really handy if those can be backfilled from previous logs even if you hadn't thought to compute that metric beforehand.
- gilfoyle 8y agoThe usecase in the article was to parse logs from a CDN serving cached content not an app server which serves every request.
- erulabs 8y agoRubygems.org uses Fastly which supports streaming logs (as should any CDN worth their salt - there is no reason whatsoever to wait a whole day to download data that is by definition event-based). FWIW, Fastly CDN log streaming is exactly how I built the view-count system I mentioned above. At the end of the day a computer does a thing and counts the thing it did. Why write down what it did in a bloated format that only needs to be picked apart later?
- zerd 8y agoFrom the post it looks like that's what he does now. Fastly uploads files to S3 (you can specify how often), and then a Lambda function is run per file to aggregate the files into view-counts. How is this different from yours?