4 ms·

Apologies everyone for the somewhat o/t question here but, what do you guys do with your 50Gb of logs every second? Where do they go after they leave nginx?

sbierwagen 13y ago
To the NSA, presumably: http://exiledonline.com/read-yasha-levines-introduction-to-the-crypto-spy-service-cloudflare-isucker-big-brother-internet-culture/ http://exiledonline.com/read-yasha-levines-introduction-to-t...
- grey-area 13y ago
  Not a customer of Cloudflare, but that seems a bit unlikely given the CEO has called NSA gag orders 'insane': http://www.washingtonpost.com/blogs/the-switch/wp/2013/09/12/cloudflare-ceo-says-insane-nsa-gag-order-is-costing-u-s-tech-firms-customers/ http://www.washingtonpost.com/blogs/the-switch/wp/2013/09/12...
  - ars_technician 13y ago
    A CEO disagreeing with gag orders has not been a good indicator in the past.
- eastdakota 13y ago
  To be 100% clear: we have never been asked or ordered to share log data with the NSA. We've not participated in any program like PRISM and would fight vigorously if we ever were. We do receive law enforcement requests on occasion, typically to determine who owns or hosts a site behind our network. When we've received law enforcement requests for customer data (e.g., account information like the email address of an account) which we determine are abusive or do not follow the principles of Due Process we have and will continue to go to court to fight for the rights of our users. Whenever possible, even if the legal request meets our standards, we also notify customers of legal requests and allow them to challenge the requests themselves before turning over any data. We take this extremely seriously and spend significant technical, legal, and public policy resources to ensure law enforcement's job is neither easier nor harder by the mere existence of CloudFlare.
  - sbierwagen 13y ago
    Sure, but why should I have any reason to believe you? If you had been served with a national security letter, you would be obligated to lie about it. Plus, you have a pretty significant financial incentive to lie about how great Cloudflare is, with no downside, since you're not under oath on HN. And even if you were, officials who have lied about the extent of surveillance programs while under oath haven't been prosecuted. Representatives of Facebook lied. Ditto with Google. Etc, etc, etc.
    - true_religion 13y ago
      No. National security letters can act as gag orders, obligating you not to talk about their existence but you're never required to lie and say you have never been served a letter. In that case, you can be cagy (as Google has I believe) and say something on the lines of "I can neither confirm nor deny receiving a NS letter" which of course is double speak for "I have one, and I can't talk about it directly".
egwor 13y ago
definitely interested in this. Where do you send this for analysis? Elastic search? Can that keep up?
- crdoconnor 13y ago
  NSA :)
dknecht 13y ago
For most customer they are stored just long enough to provide support, create aggregates for the analytics dashboard, and update threat profiles. Our enterprise customers have the option collect raw logs through SFTP for up to 3 days.
eastdakota 13y ago
Here's a blog post on exactly what we log: http://blog.cloudflare.com/what-cloudflare-logs http://blog.cloudflare.com/what-cloudflare-logs TL;DR: we generate logs at the edge, those are turned into aggregates to display analytics data (e.g., page views, hits, bandwidth), then logs are discarded unless you're an Enterprise customer in which case we allow you to download the raw log data for 3 days.

sbierwagen 13y ago

To the NSA, presumably: http://exiledonline.com/read-yasha-levines-introduction-to-the-crypto-spy-service-cloudflare-isucker-big-brother-internet-culture/ http://exiledonline.com/read-yasha-levines-introduction-to-t...

grey-area 13y ago
Not a customer of Cloudflare, but that seems a bit unlikely given the CEO has called NSA gag orders 'insane': http://www.washingtonpost.com/blogs/the-switch/wp/2013/09/12/cloudflare-ceo-says-insane-nsa-gag-order-is-costing-u-s-tech-firms-customers/ http://www.washingtonpost.com/blogs/the-switch/wp/2013/09/12...
- ars_technician 13y ago
  A CEO disagreeing with gag orders has not been a good indicator in the past.
eastdakota 13y ago
To be 100% clear: we have never been asked or ordered to share log data with the NSA. We've not participated in any program like PRISM and would fight vigorously if we ever were. We do receive law enforcement requests on occasion, typically to determine who owns or hosts a site behind our network. When we've received law enforcement requests for customer data (e.g., account information like the email address of an account) which we determine are abusive or do not follow the principles of Due Process we have and will continue to go to court to fight for the rights of our users. Whenever possible, even if the legal request meets our standards, we also notify customers of legal requests and allow them to challenge the requests themselves before turning over any data. We take this extremely seriously and spend significant technical, legal, and public policy resources to ensure law enforcement's job is neither easier nor harder by the mere existence of CloudFlare.
- sbierwagen 13y ago
  Sure, but why should I have any reason to believe you? If you had been served with a national security letter, you would be obligated to lie about it. Plus, you have a pretty significant financial incentive to lie about how great Cloudflare is, with no downside, since you're not under oath on HN. And even if you were, officials who have lied about the extent of surveillance programs while under oath haven't been prosecuted. Representatives of Facebook lied. Ditto with Google. Etc, etc, etc.
  - true_religion 13y ago
    No. National security letters can act as gag orders, obligating you not to talk about their existence but you're never required to lie and say you have never been served a letter. In that case, you can be cagy (as Google has I believe) and say something on the lines of "I can neither confirm nor deny receiving a NS letter" which of course is double speak for "I have one, and I can't talk about it directly".

egwor 13y ago

definitely interested in this. Where do you send this for analysis? Elastic search? Can that keep up?

crdoconnor 13y ago
NSA :)

dknecht 13y ago

For most customer they are stored just long enough to provide support, create aggregates for the analytics dashboard, and update threat profiles. Our enterprise customers have the option collect raw logs through SFTP for up to 3 days.

eastdakota 13y ago

Here's a blog post on exactly what we log: http://blog.cloudflare.com/what-cloudflare-logs http://blog.cloudflare.com/what-cloudflare-logs TL;DR: we generate logs at the edge, those are turned into aggregates to display analytics data (e.g., page views, hits, bandwidth), then logs are discarded unless you're an Enterprise customer in which case we allow you to download the raw log data for 3 days.