4 ms·

Well, to me "applications export interesting signals directly in a way understood by monitoring service" feels like a legacy approach. It places the burden of

Well, to me "applications export interesting signals directly in a way understood by monitoring service" feels like a legacy approach. It places the burden of decision "what is an interesting alert signal" and burden of structuring the log file output on the software developer! And it places that burden at an inconvinient time, when the system is still in the making. On the other hand, by logging everything as text, and then running intellegent/structurizing real time search engine over logs one can make/modify these decisions at a later time. And it can be done both by devs/ops, without touching the source code!

thrownaway2424 12y ago
That seems silly though. I can replace stats on a thing that normally takes 50 usecs in a log line because it will take more than that long just to log the fact and an insane amount of cpu to analyze such a thing. The large scale systems that I personally operate produce a few KB per minute in structured stats, a few MB per second in structured logs, and hundreds of MB per second in unstructured text logs. I know which of these I'd rather use for monitoring.
- dchichkov 12y ago
  To thrownaway2424. What seems silly is that processing of a few of MB per second of unstructured text logs by a real time search engine seems impossible to you. Think web-crawlers. Search engines are efficient....
  - donavanm 12y ago
    youre thinking too small. Try hundreds of KB to a couple MB per second per host. And tens of thousands of hosts. Data streams at (tens of) gigabits per second are not trivial.
    - dchichkov 12y ago
      I don't know. In my experience, one big elasticsearch box can cope with a few months of 2-3 MB/sec log data. I guess that the entropy of log file information is quite low and the search engine is being able to take advantage of that and keep its indexes rather small. But gigabits per second... I just don't know.
  - thrownaway2424 12y ago
    What do you use to monitoring the "real time search engine"?
    - dchichkov 12y ago
      Is that a joke-question? The one that I've used is the elasticsearch / kibana. And usually one would be using elasticsearch to monitor the elasticsearch :) That's the good thing about this setup, you have all the logs from all your applications (think like custom text logs from your routers, your custom applications, temperature sensors, syslogs, windows servers) aggregated in one place. And when something happens (at a particular moment in time, or with a particular machine, or with a particular key) suddenly you are able to search/drill down and locate the actual cause. And maybe even configure a dashboard or make a plot that would show when this problem was showing up. Scalable real time search engines with the ability to create trends/dashboards is one powerfull toy ;) It is ridiculuos and silly. But it is an immensely powerfull approach.

thrownaway2424 12y ago

That seems silly though. I can replace stats on a thing that normally takes 50 usecs in a log line because it will take more than that long just to log the fact and an insane amount of cpu to analyze such a thing. The large scale systems that I personally operate produce a few KB per minute in structured stats, a few MB per second in structured logs, and hundreds of MB per second in unstructured text logs. I know which of these I'd rather use for monitoring.

dchichkov 12y ago
To thrownaway2424. What seems silly is that processing of a few of MB per second of unstructured text logs by a real time search engine seems impossible to you. Think web-crawlers. Search engines are efficient....
- donavanm 12y ago
  youre thinking too small. Try hundreds of KB to a couple MB per second per host. And tens of thousands of hosts. Data streams at (tens of) gigabits per second are not trivial.
  - dchichkov 12y ago
    I don't know. In my experience, one big elasticsearch box can cope with a few months of 2-3 MB/sec log data. I guess that the entropy of log file information is quite low and the search engine is being able to take advantage of that and keep its indexes rather small. But gigabits per second... I just don't know.
- thrownaway2424 12y ago
  What do you use to monitoring the "real time search engine"?
  - dchichkov 12y ago
    Is that a joke-question? The one that I've used is the elasticsearch / kibana. And usually one would be using elasticsearch to monitor the elasticsearch :) That's the good thing about this setup, you have all the logs from all your applications (think like custom text logs from your routers, your custom applications, temperature sensors, syslogs, windows servers) aggregated in one place. And when something happens (at a particular moment in time, or with a particular machine, or with a particular key) suddenly you are able to search/drill down and locate the actual cause. And maybe even configure a dashboard or make a plot that would show when this problem was showing up. Scalable real time search engines with the ability to create trends/dashboards is one powerfull toy ;) It is ridiculuos and silly. But it is an immensely powerfull approach.