4 ms·
I've faced DoS attacks for years as I run internet forums. The simple advice for layer 7 (application) attacks: 1. Design your web app to be incredibly cachea
by buro9 10y ago
I've faced DoS attacks for years as I run internet forums.
The simple advice for layer 7 (application) attacks:
1. Design your web app to be incredibly cacheable
2. Use your CDN to cache everything
3. When under attack seek to identify the site (if you host more than one) and page that is being attacked. Force cache it via your CDN of choice.
4. If you cannot cache the page then move it.
5. If you cannot cache or move it, then have your CDN/security layer of choice issue a captcha challenge or similar.
The simple advice for layer 3 (network) attacks:
1. Rely on the security layer of choice, if it's not working change vendor.
On the L3 stuff, when it comes to DNS I've had some bad experiences (Linode, oh they suffered) some pretty good experiences (DNS Made Easy) and some great experiences (CloudFlare).
On L7 stuff, there's a few things no-one tells you about... like if you have your application back onto AWS S3 and serve static files, that the attack can be on your purse as the bandwidth costs can really add up.
It's definitely worth thinking of how to push all costs outside of your little realm. A Varnish cache or Nginx reverse proxy with file system cache can make all the difference by saving your bandwidth costs and app servers.
I personally put CloudFlare in front of my service, but even then I use Varnish as a reverse proxy cache within my little setup to ensure that the application underneath it is really well cached. I only have about 90GB of static files in S3, and about 60GB of that is in my Varnish cache, which means when some of the more interesting attacks are based on resource exhaustion (and the resource is my pocket), they fail because they're probably just filling caches and not actually hurting.
The places you should be ready to add captchas as they really are uncacheable:
* Login pages
* Shopping Cart Checkout pages
* Search result pages
Ah, there's so much one can do, but generally... designing to be highly cacheable and then using a provider who routinely handles big attacks is the way to go.
- partycoder 10y agoCaching not necessarily means more speed. Sometimes it can make things slower. 1. Get from cache 2. Determine if cached value is valid 3. Query data store 4. Put data store value in cache 5. Return data Instead of just getting it directly. In order to be able to cache you need to think about good cache invalidation. And client side caching won't work against malicious users.
- benchaney 10y agoThose are problems the CDN solves for you.
- partycoder 10y agoCDN is only for static content and I was assumed all your static content would be already on a CDN... it's a standard practice.
- modoc 10y agoCDN is NOT only for static content. Minimally Cacheable, or catchable based on cookie value, content can be cached on a CDN. Also running ALL traffic through a CDN (like CloudFlare or Akamai) allows you to do traffic optimization, FEO, DDOS protection, and much more.
- jaypaulynice 10y agoHave you heard about cache busting? Someone just needs to request a page that's not cached and the request will always hit your web servers.
- buro9 10y agoYou'd be surprised how few attacks I've personally seen vary that much. But yes, it happens, and good applications put their identifiers in their paths and ignore the querystrings, most CDN/security providers allow the configuring of their layer to ignore querystrings entirely. Of course, this is precisely the attack that works on a search page, hence the advice above to be ready to captcha that if you haven't. Anything GETable cache, everything else you need to think about how to validate the good traffic (trivially computable CSRF tokens help) and captcha the rest. 404s, 401s, etc... they should cost the underlying server as little resource as possible and also cache their result at an applicable cache layer (404s at the edge and 401s internally, 403s at the edge if possible, etc).
- samplonius 10y agoIf there are 90GB of static files, and 60GB are in the Varnish cache, cache busting will be pretty ineffective.
- ryanlol 10y agoIf there's any dynamic content and the request hits that, Varnish cache will be pretty ineffective.
- buro9 10y agoActually Varnish is great here, one normalises the requests and retains only the querystrings that are valid for your application filtering out (removing) all those that are not valid. The key thing is, you know your application, and you know what the valid keys are and the valid value ranges. If you can express that in your HTTP server and discard requests then it can be very cheaply done. A forum really doesn't have that many distinct URLs, and so this is easily done. It would be harder on a much more complex application, but the original question related to these smaller side-project applications.
- Kenji 10y agoUh, stupid question but how do you cache a website like for example this comment thread on hackernews? Suppose a DDoSer calls this comment thread a lot of times. The request has to go through to the server because when I hit F5 or post a comment myself, I see the comments in realtime. How do you handle that exactly? Does caching for a few seconds help already, or does the backbone push updated sites to the CDN server? I have no experience in DDos mitigation.
- voltagex_ 10y agoWhat you can do with a site like HN will be different than if you're a shopping getting DDoSed on Black Friday by a competitor. You can put the whole of HN into read only mode if needed and it'll have no real impact; disallowing purchases on MyAmazonCompetitor.com would be catastrophic.
- owlish 10y agoIf you're getting more traffic than 1 request/s, it's less work to generate a static cached version on the cadence of ~1 second than to dynamically generate the content for each request.
- jdcarr 10y agoThe important thing you need to assess is how critical is it that clients receive fresh data. You can imagine that for a real time service it would be better to provide a timeout immediately rather than providing stale data. HN is an example of a near on-line site where some delay is perfectly acceptable. No one cares that they're receiving a 2 second old page, it's better for the site users to reveive old data fast rather than new data slow. If you use nginx the following commands would help out significantly (if I remember them correctly) proxy_cache_use_stale updating proxy_cache_lock on proxy_cache_lock_timeout 1s This config allows nginx to fetch cache updates while serving clients and when fresh data is received from the upstream application server it'll use that immediately. If that's wrong hopefully someone can correct the conf.
- greenleafjacob 10y agoMake your comments system a static site generator, so that each comment generates a static HTML page and you serve that statically. 4chan does this.