6 ms·
HTTP Post Denial Of Service: More dangerous than initially thought
- nodata 16y ago(This has been known about for ages.)
- andrewvc 16y agoYep, and this is why evented io is pretty much mandatory for a modern webserver (like nginx). If you need Apache you can always proxy nginx in front of it.
- tedunangst 16y agofile descriptors aren't free, even for evented webservers. you've only moved the limit higher, not removed it.
- tptacek 16y agoThis is true but not really responsive to the parent comment's point; the nginx architecture is better suited to handling this kind of attack; contexts are much more expensive than file descriptors. File descriptors are themselves an artificial construct that can be coded around. In the unlikely event that this attack ever became super relevant, a relatively simple kernel module could drastically amp up the number of connections a host could absorb --- although a RED queue would probably keep you from having to go even that far.
- tedunangst 16y agoI'm not sure what you mean by context. Clearly nginx also has some context, such as the data read so far, associated with each handle. I guess you mean thread stack?
- tptacek 16y agoApache has an entire execution context associated with it (either a thread or a process).
- jws 16y agoI agree. I've been killing servers with it since the '90s. The current interest comes from it (possibly) arriving as an attack tool in botnets and not being mitigated by current anti-DDOS methods. It is time to cope with it.
- tptacek 16y agoYeah, because existing anti-DDoS methods are so effective at "conventional" DDoS attacks. Uh-huh.
- sabat 16y agoAnd it would be easy enough to defend against if it really became a problem -- just time out. Most web sites don't need have a need to receive gigantic POST data.
- JoachimSchipper 16y agoIt's difficult to distinguish between a persistent user trying to upload a large image over a spotty satellite link and a hacker.
- sabat 16y agoYou're right, but I still don't believe this creates an impossible situation: * most webapps don't allow giant user uploads of any kind * if you have a persistent user with a net connection that's that slow then you have no choice but to time him out
- JoachimSchipper 16y agoA sufficiently-slow connection will make even a reasonable amount of data take a long time. I'm sure Flickr has users that upload large photos over satellite links... There's also no reason why you need to time this person out. If you configure kernel and application for small (or auto-sizing) buffers and use an event-based structure for the application, you should be just fine.
- sabat 16y agoIt's really a matter of one's approach and level of paranoia. Me? If I wasn't running Flickr (say I was running getfreckle.com), I'd probably opt for more conservative settings. I may not need to time that guy out, but I'd probably do it so I wouldn't have to worry later. * If you configure kernel and application for small (or auto-sizing) buffers and use an event-based structure for the application* This would be all the better.
- andrewmccall 16y agoTrue. Though it's a safe assumption that if you suddenly have a large number of spotty satellite users and they're DOSing your site it's probably something malicious.
- chair6 16y agoYep .. one example of a similar attack here - http://ha.ckers.org/slowloris/ http://ha.ckers.org/slowloris/ - with references to others.
- jws 16y agoOriginal paper: http://www.owasp.org/images/4/43/Layer_7_DDOS.pdf http://www.owasp.org/images/4/43/Layer_7_DDOS.pdf (well, slide deck) § Summary ➊ The slowloris attacks of last year consumed all of a host's available HTTP server connections with minimal bandwidth by trickling out HTTP headers forever at a slow pace. Countermeasures were developed and applied. ➋ This exploit consumes connections similarly, but does so by trickling out data in the content section of a POST request. Countermeasures targeting slowloris are ineffective. ➌ Countermeasures for this will include HTTP request timeouts, but to support distant users operating over potentially saturated links you will have to set the timeout high enough that any well connected attacker will still be able to kill your server. (That has some extrapolation from me.) § Discussion The real solution will be to not consume a process or other expensive resource until the entire POST request has been received by the server. (Expand to PUT and any other method with content in the request.) This becomes difficult in HTTP/1.1. To really support 1.1 you have to give the request the option to reject before the content is sent. (Yes you other old farts, the client doesn't necessarily blindly send the content, but waits for your server to say "ok, the method, resource, and headers look good, send me the content".) I suppose you could force all those decisions to be made by the HTTP server, but that would be sad.
- moe 16y agoTo really support 1.1 you have to give the request the option to reject before the content is sent. This is actually already specified, at least for PUT (see 100-continue).
- pyre 16y agoThis seems easy to defend against for sites that don't accept a lot of POST data. You could set a really low threshold for content-length. You could maybe even get fancy and only allow large POST data on certain URLs (e.g. low POST content-length threshold on www.example.com, but a higher threshold on admin.example.com; assuming that you can reject the connection as unauthenticated before receiving the data).
- andrewmccall 16y agoThis makes perfect sense. If for example you're accepting 140 characters there is no reason you should have an excessive content-length for that URL.
- eis 16y agoThis is not as dangerous as the authors make it sound like. Just limit the maximum connections per IP (e.g. via mod_evasive in Lighty or firewall like iptables) and voila - attack voided. This is very similar to what Slowloris does but Slowloris is much more intelligent because it slows the connection down before it all request headers are sent and therefor many modules will not execute their limit checks just yet. "What’s special about this denial of service attack is that it’s very hard to fix" "Therefore, to properly fix it would mean to break the protocol" Not at all as outlined above. From the paper: "In HTTP/1.1 where chunked encoding is supported and there is no “Content-Length” HTTP header, the lethality is amplified." No, it changes exactly nothing regarding this attack. Why would it? I do not know why these guys make such a fuss about something so trivial and obvious to anyone dealing with HTTP servers. But sadly this is quite common in the security industry.
- dangrossman 16y agoAll they need is a botnet of at least 256 computers, which can be rented for a small fee off any number of shady forums. Now what are you going to do? A much better solution is to put a proxy in front that can hold lots of connections very cheaply. nginx and other evented servers can hold 10,000 trickling connections without using significant resources.
- eis 16y agoIf you have a botnet attacking you, then it's a whole different story anyways. Those can get your services down easily regardless of what attack they use or what protocol is in play (HTTP,FTP,SMTP, you name it) so the botnet argument does not really hold here. A proxy in front does not help if it does not limit per IP connections as an attacker can just use up all filedescriptors or connection slots. It just ups the ante a bit but not much. 10k connections can very easily be created from any DSL/cable connection let alone a server in a datacenter.
- btilly 16y agoFor most websites, sure. But the big boys have to deal with botnets all the time, and (mostly) manage to remain up. They need to figure this sort of thing out.
- snorkel 16y agoUse Apache's LimitRequest* directives http://httpd.apache.org/docs/2.0/mod/core.html#limitrequestbody http://httpd.apache.org/docs/2.0/mod/core.html#limitrequestb...
- eis 16y agoThis suggestion is impractical as the attack works with even fairly small Content-Length. Say 10000 and sending 1 byte per second will limit you to 10kb uploads but still give each connection hours to run. Since most IO timeouts are fairly large you can even with a limit of 1000 bytes run a connection for hours until you hit this limit. The only solution is to limit connections per IP and maybe defining a minimum speed can help too.
- chc 16y agoIf this became a problem, couldn't we just have the server do triage? Like, monitor the speed of open connections, and when we're nearing saturation, tell some percentage of the absurdly slow connections (maybe with some heuristics to detect more likely attack patterns), "Sorry, we can't support you at this time." If we're going to deny service to someone, it may as well be the problem clients.
- tptacek 16y agoThis attack seems just as straightforward to address in a reverse proxy as "slowloris" (which was, despite the publicity, itself an attack known for many years prior to its announcement). Two things remain true: It is straightforwardly possible to bring a conventional full-featured web servers or app server to its knees with leveraged attacks (in which target pain greatly exceeds attacker effort). and This doesn't much matter, since even stupider attacks do just as much damage. For instance, during the 2002 Winter Olympics, which Arbor Networks was contracted to do anti-DDoS for, attackers simply got tens of thousands of machines to make vanilla browser HTTP requests to targets. If anything, things like this "more dangerous than initially thought" request is easier to defend against than the dumbest attacks; it involves unusual patterns of POST requests, unusually large content-lengths, and unusually long-lived connections, all of which can easily make connections a candidate for a random early drop queue. The fact that tools may not immediately exist to counter this kind of attack is probably not an indication of how hard those countermeasures are to build.
- eis 16y agoCouldn't agree more. Anything that deviates from usual usage patterns can be relatively easily detected and negative impact mitigated. But nowerdays botnets are the real threat with every 13 year old kid being able to get hundreds of machines infected with ease which is enough to pose a real threat to most websites when they launch absolutely valid HTTP requests. Botnet sizes of tens of thousands of zombies are not uncommon. It is a tough problem and the reason why DDoS attacks are so effective and therefor popular. They work. When your site gets beyond a certain size, chances are you don't even have accesslogs for performance reasons because you get dozens of thousands of requests per second. So finding the malicious requests and singling them out from the rest to be able to filter them is hard to say the least. Plus every false positive could really hurt. Imagine you accidently filter some check-out requests in some online shop - that's a lost sale. Attacks like the one outlined in the "paper" (it's really more a bunch of slides) can be easily defended against. They are not new just because someone alters the request method from GET to POST and wants to sell it as dangerous just to grab some fame. The fact that they suggest a change in the protocol would be needed just reinforces my suspicion. Announcing a flaw in a hugely popular protocol like HTTP gets you a sh*tload of attantion and in the end jobs. I know a few people working for companies in the web-security field, some also member of OWASP. Exagerations are very very common. The more fear a customer has the more protections you can sell them.
- xxi 16y agoacunetix.com