4 ms·
CloudFlare generates 50gb/s of logs globally and have handled collecting this volume in two ways. Historically the logs are sent to a local syslog-ng through t
by dknecht 13y ago
CloudFlare generates 50gb/s of logs globally and have handled collecting this volume in two ways. Historically the logs are sent to a local syslog-ng through the use of a PIPE and the forwarded to central logger. This can be done with nginx with no patches by just treating the PIPE as file. Just make sure you do a little buffering inside nginx.
access_log /dev/nginx_access log_format_name buffer=64k flush=10s;
Since this is a pipe there is still some blocking IO, but no worse off then writing to local file.
The way we are migrating will be non-blocking IO through the use of Lua Resty log module ( https://github.com/cloudflare/lua-resty-logger-socket https://github.com/cloudflare/lua-resty-logger-socket ).
We will write up a blog post at some point but hopefully this is useful to you. In the future we are going to be going one step further and have NGINX emit protobuff files. Feel free to email dane AT cloudflare.com if you have any questions.
- justincormack 13y agoyes Openresty is the big part of the Nginx ecosystem that is outside the Nginx company and provides a completely different way of adding features.
- kevinastone 13y ago+1 on that eventual blog post.
- chrissnell 13y agoApologies everyone for the somewhat o/t question here but, what do you guys do with your 50Gb of logs every second? Where do they go after they leave nginx?
- sbierwagen 13y agoTo the NSA, presumably: http://exiledonline.com/read-yasha-levines-introduction-to-the-crypto-spy-service-cloudflare-isucker-big-brother-internet-culture/ http://exiledonline.com/read-yasha-levines-introduction-to-t...
- grey-area 13y agoNot a customer of Cloudflare, but that seems a bit unlikely given the CEO has called NSA gag orders 'insane': http://www.washingtonpost.com/blogs/the-switch/wp/2013/09/12/cloudflare-ceo-says-insane-nsa-gag-order-is-costing-u-s-tech-firms-customers/ http://www.washingtonpost.com/blogs/the-switch/wp/2013/09/12...
- ars_technician 13y agoA CEO disagreeing with gag orders has not been a good indicator in the past.
- eastdakota 13y agoTo be 100% clear: we have never been asked or ordered to share log data with the NSA. We've not participated in any program like PRISM and would fight vigorously if we ever were. We do receive law enforcement requests on occasion, typically to determine who owns or hosts a site behind our network. When we've received law enforcement requests for customer data (e.g., account information like the email address of an account) which we determine are abusive or do not follow the principles of Due Process we have and will continue to go to court to fight for the rights of our users. Whenever possible, even if the legal request meets our standards, we also notify customers of legal requests and allow them to challenge the requests themselves before turning over any data. We take this extremely seriously and spend significant technical, legal, and public policy resources to ensure law enforcement's job is neither easier nor harder by the mere existence of CloudFlare.
- sbierwagen 13y agoSure, but why should I have any reason to believe you? If you had been served with a national security letter, you would be obligated to lie about it. Plus, you have a pretty significant financial incentive to lie about how great Cloudflare is, with no downside, since you're not under oath on HN. And even if you were, officials who have lied about the extent of surveillance programs while under oath haven't been prosecuted. Representatives of Facebook lied. Ditto with Google. Etc, etc, etc.
- egwor 13y agodefinitely interested in this. Where do you send this for analysis? Elastic search? Can that keep up?
- crdoconnor 13y agoNSA :)
- dknecht 13y agoFor most customer they are stored just long enough to provide support, create aggregates for the analytics dashboard, and update threat profiles. Our enterprise customers have the option collect raw logs through SFTP for up to 3 days.
- eastdakota 13y agoHere's a blog post on exactly what we log: http://blog.cloudflare.com/what-cloudflare-logs http://blog.cloudflare.com/what-cloudflare-logs TL;DR: we generate logs at the edge, those are turned into aggregates to display analytics data (e.g., page views, hits, bandwidth), then logs are discarded unless you're an Enterprise customer in which case we allow you to download the raw log data for 3 days.
- colechristensen 13y agoI was ready to suggest just this if I understand you correctly. You mean creating a http://linux.die.net/man/7/pipe http://linux.die.net/man/7/pipe and configure syslog-ng to read from it? A very comparable configuration for apache https://peter.blogs.balabit.com/2010/02/how-to-collect-apache-logs-by-syslog-ng/ https://peter.blogs.balabit.com/2010/02/how-to-collect-apach...
- rurounijones 13y agogb is GByte or GBit?
- rdtsc 13y ago> CloudFlare generates 50gb/s of logs globally Wow, mind blowing. That's 4pb/day, 1+eb/year.
- earino 13y ago50gb/s ofCloudFlare logging data to "data science" at sounds like the most interesting damned problem I've ever heard. Well, probably second to how you manage that flow into a pipeline in the first place.
- deleted 13y ago[deleted]
- samcrawford 13y agoThat is a mind blowing amount of data. Is this all access/error log data, or are there other logs being generated? If it _were_ access log only, and your log lines average 1KB in length (pretty generous), that's 52m qps. If we take that further and assume each object is 93KB [1] then your outbound traffic is almost 40Tb/s (terabits per second). So I assume it must be more than just access logging! 1. https://developers.google.com/speed/articles/web-metrics https://developers.google.com/speed/articles/web-metrics
- dknecht 13y agoSorry. I wrote this after a long day and replied above that this should read per minute and not per second.
- dknecht 13y agoEDIT: Sorry wrote this after a long day. That should read per minute and not per second.