3 ms·
What usage could possibly need 5M log/s on a single system? Are you trying to log every TCP and UDP packet? To give a rough point of comparison, CloudFlare is
by user5994461 10y ago
What usage could possibly need 5M log/s on a single system? Are you trying to log every TCP and UDP packet?
To give a rough point of comparison, CloudFlare is taking 4M page views per second and that's 5% of the global internet traffic.
- vardump 10y ago> Are you trying to log every TCP and UDP packet? No. Although I've done that too... To debug timing critical code. Or Heisenbugs that disappear when you log. Ever had a bug that vanishes when you enable logging? Not 5M sustained, just short sub-millisecond bursts here and there.
- user5994461 10y agoSure I've had my share of heisenbug. Still remember one not long ago where the dev enabled debug logs in all our applications at once over the week end. It turns out that a week end of log is 1 TB :D Back to the topic. I suppose that saying "5M/s" to refer to 5000/ms during a single millisecond is misleading. There are many implicit "buffers" (raw CPU power, intermediate hard drives, TCP queues, syslog/fluentd processing) that may smooth that kind of peak very aggressively long before it reaches graylog. Interesting talk by the way. I'm doing similar things, at a smaller scale though.