36 ms·

Say Cheese: a snapshot of the massive DDoS attacks coming from IoT cameras

336 points by jgrahamc 10y ago

danielcid 10y ago
That's a problem that will only get worse. To give an example, last year, Incapsula recorded ~9,000 IoT cameras attacking them. A few months ago, Sucuri recorded ~25,000. CloudFlare is seeing close to 50k. And that's the attackers just using a small portion of their real power for http floods. Our report from a few months ago breaking down the types of cameras and networking doing the attack - very similar to what CloudFlare saw: https://blog.sucuri.net/2016/06/large-cctv-botnet-leveraged-ddos-attacks.html https://blog.sucuri.net/2016/06/large-cctv-botnet-leveraged-... *I work at Sucuri.
kingosticks 10y ago
The first attacks were described as only lasting minutes. Is this the perpetrators stopping? If so, why? In case people start to notice their 'smart' lightbulbs are glowing unusually bright? Otherwise what else stops the attack?
- tehmaco 10y ago
  Proof of ability to potential customer? Checking that their command and control software works? I doubt you'd notice a difference on the device unless you were doing something network/CPU intensive on it, even then, it'd just seem slow.
  - kingosticks 10y ago
    Ah OK that makes sense, thanks. It's handy for companies like Cloudflare to get a heads up on what's around the corner.
- TorKlingberg 10y ago
  If you have failed to take a site down in 15 minutes, continuing for longer is not going to make a difference. Also the attacker may be renting time on the botnet.
zzleeper 10y ago
If things get too bad, can cloudfare pool with google, etc. and just block the IPs of those affected? (until someone fills a captcha)
- sofaofthedamned 10y ago
  AFAIK Cloudflare only deal with HTTP and HTTPS, so nothing else goes through their proxy. Which doesn't matter anyway because the traffic is still going through the edge networks (the ISP and a little bit upstream from there), so is still bandwidth impacting at some point. We really need a way of getting the ISPs to talk to the providers upstream, and a way of notifying back that a particular host is causing problems. Unfortunately ISPs don't want to do this because unhappy customers = revenue. Also I don't want my traffic analysed on the way through, so there's a freedom issue on this too. So it's not an easy problem to solve, but the ISPs could do it. But they won't. On the flip side dealing with this sort of stuff is why i'd like to work for Cloudflare, Google, Facebook or others. It's fascinating how technical solutions can be bypassed by customer or provider inertia.
- deleted 10y ago
  [deleted]
- deleted 10y ago
  [deleted]
- deleted 10y ago
  [deleted]
- majke 10y ago
  There are researchers in many companies that are actively fighting the botnets. Some links: - https://krebsonsecurity.com/2016/10/who-makes-the-iot-things-under-attack/ https://krebsonsecurity.com/2016/10/who-makes-the-iot-things... - http://blog.malwaremustdie.org/ http://blog.malwaremustdie.org/ - https://www.akamai.com/us/en/our-thinking/information-security/information-security-research.jsp https://www.akamai.com/us/en/our-thinking/information-securi...
bhouston 10y ago
Does Cloudflare still host (edit: proxy) the companies that do DDoS attacks thus serving to justify the use of Cloudflare services?
- jgrahamc 10y ago
  We don't do hosting.
  - bhouston 10y ago
    Quoting Brian Krebs: > These two services, like most booters, are hidden behind CloudFlare, a content distribution service that lets sites obscure their true Internet address. In case anyone cares, Lizardstresser’s real Internet address currently is 217.71.50.57, at a hosting facility in Bosnia. https://krebsonsecurity.com/2014/12/lizard-kids-a-long-trail-of-fail/ https://krebsonsecurity.com/2014/12/lizard-kids-a-long-trail...
    - aurbano 10y ago
      So they don't do hosting... What you've quoted just explains how they act as a layer in front of the actual server
      - bhouston 10y ago
        Host/proxy - a non-relevant detail. I misspoke. Either way, Cloudflare appears to both help booters and then also sells services to protect you from booters.
        jgrahamc 10y ago
        Cloudflare's CEO spoke about this here: https://www.youtube.com/watch?v=SWFX-zEYwN0 https://www.youtube.com/watch?v=SWFX-zEYwN0
        e40 10y ago
riskable 10y ago
I've been warning about this kind of thing for years and years. Both professionally and in my personal life. On forums and anywhere people will listen... IoT is a security disaster I hate to suggest things like this but we may need legal remedies to solve the problem before it becomes an Internet Zombie Armageddon (IZA). It's that bad. There's going to be tens of billions of IoT devices installed at businesses and homes in just a few short years. If they're not receiving regular, timely security updates the Internet as we know it may just plain collapse from the sheer weight of all the zombies. I propose a mandatory "nutrition label" of sorts for IoT device packaging. Something like this: * Updates itself regularly * Conforms to NIST security standards * Public penetration test results: https://ptr.iot.nist.gov/A2L7P9D23345 * Shares data with 3rd party: No * Administration by vendor: No * Requires Internet connection to administer: No * Requires 3rd party service to function: No * Notifies of intrusion: Yes * Logs all access: Yes * Encrypts all network traffic: Yes Something like a publicly-posted penetration test would be incredibly valuable. Especially if it was performed after each update of the device and at regular intervals. Even if it was an automated test (which in theory could be 'gamed') it would still be better than what we have now which is nothing!
- elthran 10y ago
  All very well in theory, but I'm sure it will suffer the same fate as ordinary nutrition labels - the general populace just doesn't care/isn't motivated enough to care
  - tjr 10y ago
    Bundle all of those points up into some sort of standard level of acceptance and let conforming products have a big shiny sticker saying so on the front of the package? As a bit of an analogy, while I may enjoy poring over computer system security details, I'm pretty novice at a lot of more real-world things. Yesterday I was buying a new dryer exhaust hose. I read some reviews online, and looked at boxes in the store. I managed to learn enough to realize I wanted an all-metal construction so it wouldn't catch on fire. But I really wanted some big shiny signal to let me know "HEY YES IF YOU HAVE NO IDEA WHAT YOU'RE DOING, BUY THIS ONE!"...
akerro 10y ago
> * Requires 3rd party service to function: No > * Public penetration test results: No vibrator manufacturer will agree to this.
- jgrahamc 10y ago
  Seriously? That comment is worthy of Beavis and Butthead. And not in a good way.
- sctb 10y ago
  We detached this subthread from https://news.ycombinator.com/item?id=12684368 https://news.ycombinator.com/item?id=12684368 and marked it off-topic.
falcolas 10y ago
Strangely enough, I think the solution is in devices like OnHub, Google WiFi, or other third party managed routers (yes, another IOT device). Why? When attacks like this are first detected, they can be shut down at the source. Known bad devices can be preemptively locked down behind a firewall. Traffic to specific IP addresses, or packets which spoof headers to cause amplification attacks could all be detected and dropped before they even reach the ISP. Yes, there will be problems and bad diagnoses, no, I don't want one of these managing my home network; but it would be perfect for my mother, for my non-technical friends, and for the average Joe with a subverted IP camera.
- Mister_Snuggles 10y ago
  This is actually a really neat idea. In another comment[0] I described how I keep my cameras away from the internet. Someone like Google could easily build something into their routers and firewall them off as they're added to the network. The firewall could allow them to talk to the appropriate cloud service, and nothing else, in order to better secure them without the user even knowing it's happening. This doesn't solve all of the IoT problems, but it does help some for the one described in the article. [0] https://news.ycombinator.com/item?id=12685381 https://news.ycombinator.com/item?id=12685381
- hathawsh 10y ago
  I like this idea a lot. Perhaps every IoT device should publish a profile of its expected behavior to the router. For example, a wifi-enabled thermostat might tell the router that it only intends to connect to a particular range of servers, by DNS name. The router could then set up an automatic firewall rule that stays in sync with DNS changes. The user should be able to disable that firewall rule (to overcome technical issues), but most users will never have a reason to do so. We could bootstrap this idea by having the router fingerprint IoT devices and apply matching firewall policies from a database. If only we could install "apps" on routers, like we do on Android and iOS devices, someone could get started with this idea right now.
  - anchpop 10y ago
    The only issue is that after that point, the device can never connect anywhere else.
Mister_Snuggles 10y ago
I've got a bunch of cheap Wifi cameras and I don't worry about this kind of thing at all. All of the cameras live on a separate Wifi network that is unable to connect to the internet. I use Zoneminder to handle all of the recording, motion detection, etc. For remote access, I've got a VPN server running. Fortunately, the cameras I use are perfectly happy to work without being able to talk to the manufacturer's servers (though they do try!). In a nutshell, I get almost all of the "benefits" of the cloud service (the only missing thing is off-site storage, but a fix for that is in the works) with almost none of the drawbacks. Of course, this is NOT easy for the average consumer to set up. They expect to be able to plug the camera in and have things Just Work, which is why we've got cameras that can open ports on routers/firewalls, register themselves with cloud services, etc, with minimal effort. As for IoT in general, my future plans are to use this separate WiFi network for all IoT-like devices that don't really need to connect to the internet. The Philips Hue is a good example - it has a bunch of cloud-enabled functionality, but I don't actually need any of it.
- joblessjunkie 10y ago
  It sounds like you actually worry about his kind of thing a great deal, considering the amount of effort you put in.
  - Mister_Snuggles 10y ago
    Believe it or not, it was a whole bunch of little things that added up to this state. When I started with the cameras I was just using the motion detection and email features that were built into the firmware. I never bothered setting up the cloud stuff because I didn't need it. Slowly I transitioned to having the cameras dump images onto a local NAS via FTP. Eventually I started using the cameras for more than watching the cats, so I needed something with better motion detection and recording capabilities. I started with 'motion', then used a fork of 'motion' that seemed to work better, then finally ended up with ZoneMinder. On the VPN side, I decided many years ago that I wanted remote access to my "stuff" - file shares, etc. This started out with the PPTP VPN built into my router, which is terrible but sort of worked. The next step was writing a script that would run SSH and set up a couple dozen tunnels, which is differently terrible but sort of worked. Eventually I got tired of maintaining that and ended up with an IPSec VPN using racoon -- split DNS and split tunneling are truly amazing. The extra Wifi network actually started as a misguided attempt to prevent other stuff (Netflix, mainly) from interfering with the cameras. I figured that putting them on a separate Wifi network would isolate them and give them dedicated bandwidth. My first attempt involved using a Raspberry Pi as an AP. It worked, but not very well at all. The second attempt used the Virtual AP functionality on my router -- it wasn't until I set this up that I understood that Virtual APs don't quite work this way. Around this time I started to see articles about cloud-enabled cameras doing bad things and decided that this setup was worth keeping. The camera and VPN stuff both went on over the last 5-10 years or so, with the Wifi changes being something I did within the last year. You're right that there was a lot of effort, but as you can see it wasn't really driven by privacy/security until the end. Most of these things were driven by trying to find better ways to scratch various itches. The next planned change is to re-address my network so that it doesn't use the 192.168.1.0/24 range that every consumer router seems to use by default. This way I will be able to use VPN when I'm visiting friends without having to tether to my cell phone. Like everything else, it's just scratching an itch.
pbnjay 10y ago
I have a few cheap chinese ip cameras and I noticed that they have telnet open (I subsequently rooted them to fix this) and of course have them behind my home firewall. However this and other IoT devices got me thinking that maybe it's time to start an underground device hacking organization solely committed to not only rooting and reverse engineering these devices but also releasing new firmware for all of them with open source internals. I'm sure this would have to be "underground" due to all the legal restrictions around the embedded device drivers. Who wants to do this? I wouldn't know where to start besides a .onion site...
- rasz_pl 10y ago
  >not only rooting and reverse engineering these devices but also releasing new firmware for all of them with open source internals I had the same idea, except in my version automated scan finds exploitable IOT devices and updates them with new firmware straight from /dev/null. Problem SOLVED.
  - emp_zealoth 10y ago
    Actually it would be a nice thing to do. Wonder how long would it take manufacturers of all this crap fix their shit after massive warranty claim flood came in
- wil421 10y ago
  People already release custom firmware for routers. I'm not sure what the manufacturers think about this but I'm sure the same community could be built for IoT devices. The manufacturers would probably be upset they don't get my data to analyze anymore.
bogomipz 10y ago
Does anyone if there's a commonality amongst these IoT security cameras that is facilitating this? Is it just non-sensible defaults or is something more like the embedded webserver/SoC?
- blacksmith_tb 10y ago
  It's also unclear to me if the devices are being connected directly to the public net - I would have expected most homes and businesses to have some kind of firewall. I suppose it's possible that UPnP is exposing a web interface and/or telnet.
- c22 10y ago
  I think the biggest shared threat is the ability to punch holes in firewalls with upnp. That and the fact that they're running ancient unpatched software stacks.
  - bogomipz 10y ago
    Indeed, that gets back to not sane defaults though I suppose. The filet-of-firewall was particularly troubling, Flash and DNS rebinding is all it takes if UPnP is running on your router. This worth read if anyone is interested: http://www.kb.cert.org/vuls/id/361684 http://www.kb.cert.org/vuls/id/361684
polynomial 10y ago
Seems somewhat ironic that an article about DDoD is published on CloudFlare's blog.
- marcosdumay 10y ago
  CLoudFare seems honestly interested in making the Internet a better place, one that does not need its current services. I guess they can pivot easily into a cheap bandwidth provider, so this doesn't hurt much. I salute their help and everything, but it's always important to keep in mind that helpful companies are just an acquisition away from antagonizing us.
- xxdesmus 10y ago
  I'm not so sure "ironic" means what you think it means.
- jlgaddis 10y ago
  Ironic? To me, it seems quite fitting that a company who has "protecting you from DDoS attacks" as one of their major selling points would publish an article about (recent and relevant) DDoS attacks.
  - hueving 10y ago
    Ironic in the sense that they protect companies offering DDoS as a service.
jwmullally 10y ago
A general solution to IoT security would be for all IoT devices to only communicate to your personally owned home gateway, which would run open-source drivers for each device to provide the networking/external communication functionality. The IoT device could even be assigned its own isolated network link to the router (i.e. sandboxed). That approach seems like the only logical conclusion for the device owners. I hope this is how the field will eventually evolve. While its against the commercial interests of "Internet data" companies (who might subsidize devices), in the long term regular consumer appliance manufacturers will implement it if theres enough market pull for it.
- mtgx 10y ago
  That is kind of what Nest's Thread protocol allows IoT devices to do. But strangely enough, Google is going the opposite direction with its devices, including coming up with such stupid ideas of controlling your (OnHub) Wi-Fi router over the Internet through an app.
  - jwmullally 10y ago
    Thread is just an IPv6 wireless mesh routing protocol right [1][2]?. On its own it doesn't solve 2 key security problems: * Any device can send arbitrary data to somewhere on the internet * Internally, any devices can send arbitrary data to any other device ----------------------- Another less-critical but important usability limitation is the short sighted nature of how device logic + APIs will evolve long-term for Device-to-Device communication. Device-to-Device implies the application protocols are running on the devices themselves, which can't be bugfixed or upgraded by the user and in the majority of cases will likely never be updated by the manufacturer after release. In [2] they give examples of devices like fridges and thermostats talking to each other. There's 3 long-term ways I see of handling device-to-device application APIs: * Option 1: Raw device-to-device (Short term): How the hell will those APIs be agreed upon and evolve across multiple years and device products? Bluetooth now barely handles relatively simple interop use cases (files, contacts, audio...). Will Thermostat X support Fridge Protocol 8? How will the user handle more advanced use cases? * Option 2: Vendor managed (Currently the most likely): The devices will talk to the vendor servers which will talk to other vendor servers which will talk to your other devices. Terrible for the user for so many reasons but great for the vendor: its the simplest to implement, keeps protocols secret and locks user into giving that specific vendor endpoint all that juicy data. * Option 3: User managed (IMHO Preferable): Computer drivers are a much better model for this. Drivers sit over devices to transform the weird device API (fixed target) into a consistent interop API (community driven moving target) that does what people want. This still supports device-to-device: device A -> API call on gateway driver A -> gw driver interop API -> API call gateway driver B -> device B. This is what Option 2 (vendor managed) will actually have to do under the hood anyway. This could be implemented in a simple way e.g. with per-device(type) containers exposing a TCP/UDP port to the device. I predict Option 3 to become mainstream when the year of the Linux desktop lands :D ----------------- [1] https://www.threadgroup.org/portals/0/documents/resources/Thread_Technical_Overview.pdf https://www.threadgroup.org/portals/0/documents/resources/Th... [2] https://www.infoq.com/articles/thread-protocol-for-home-automation https://www.infoq.com/articles/thread-protocol-for-home-auto... (Thread is a poorly chosen name for a new computer protocol:P)
Animats 10y ago
Interesting. This set of attacks is coming from real IP addresses. (SYN floods and single-packet UDP-based attacks use a totally bogus source IP address.) The device can be talked to and potentially located. Somebody needs to be finding some of those devices and analyzing them. How are they controlled? IRC? Some set of known domains or IP addresses? A self-healing mesh network? All of those have been used, and most modern botnets have a quite sophisticated control network. This provides both robustness and conceals the actual controller. In modern botnets, everything is encrypted, so it's hard to take over the botnet.[1] Attacks from real IP addresses are potentially blockable upstream. That's only a temporary fix. The device, if behind a NAT, can request a new IP address and evade the block. It may do so periodically, just to confuse the issue. At least you can block a hostile IP at a firewall, so it can't use server resources. That's what Cloudflare is doing. On the legal front, it's worth suing makers of vulnerable devices for negligence. Their EULA will not protect them, because the victim isn't their customer and isn't a party to the EULA. A few court orders to recall a product might get the attention of the IoT industry. [1] https://www.damballa.com/downloads/r_pubs/WP_Botnet_Communications_Primer.pdf https://www.damballa.com/downloads/r_pubs/WP_Botnet_Communic...
- exhilaration 10y ago
  On the legal front, it's worth suing makers of vulnerable devices for negligence. Where does one sue the no-name Chinese manufacturers of these devices? I have a feeling a lawsuit won't get very far in Shenzhen Municipal Court. There's absolutely no consequences to them building these devices and you cannot expect a solution from them.
  - nitrogen 10y ago
    Sue the importers then, and/or file ITC complaints.
    - flukus 10y ago
      Who says there is an importer and not a chinese retail site or ebay seller?
supergeek133 10y ago
Seriously? I get the cameras scenario keeps coming up, but most of these (if not all) are CCTV systems with a DVR with open remote management that either: - Is turned on by default (bad) - Is turned on without enabling even the slightest of security by the end user (stupid but still bad) - If a or b aren't true, it has some sort of exploit that the company will never fix (if they are still in business) The "rest" of IoT in many cases doesn't have this problem at this scale. You're either talking about proximal networks with no outside internet (z-wave, 15.4, etc) or something with secure communication with the home service (PKI for example). I'm more and more frustrated with IoT fearmongering, if you want to talk about CCTV fearmongering? OK. But these aren't Nest cams, or Arlo cams. They're off the shelf DVR CCTV systems.
t3ra 10y ago
Time probably time to name and shame brands. CP Plus is doing so much marketing in India with TV slots. They should probably invest that in doing some research and patching .
merb 10y ago
> Connection: close well if they would have keep-alive it would be probably even more disastrous. especially if the botnet would have some intelligence that it would try to detect the keep alive timeout and send requests exactly in this time frame. -- also a lot of these devices are probably built with linux. and here is something that is probably bad with linux, the lifespan of most linux iot devices are probably outspanning the official lifecycle of any distribution/kernel. this may lead to problems that the vendor will just abadonned it or doesn't care anymore, if he even provided a way to upgrade. on windows this is probably as bad as well since most vendors will probably not pay additional fee's after the next windows iot gets out of live. it's probably time to abadon most os's for these devices and create a new open source project for these kind of devices, hardended from scratch. this would at least help with the unmaintained part of the os parts, which would make the situation at least 1% better. well it won't fix the laziness of certain vendors.. but it would be a beginning.
DenisM 10y ago
How about this for a fix? Just thinking out loud here... Let's make a protocol that allows a server to specify to its upstream ISP to cut off all traffic from a particular IP address. The ISP can either soak up the traffic, or go further up to its own upstream ISP and relay the request, and so on until the source of the attack is reached. Now it's between an IoT device owner and his residential ISP, it's up to them to sort it out. To motivate all ISPs to play along let's make a law for each ISP to be financially responsible for failure to honor such requests, enough for a victim to pay for defense plus some %% in penalties. Sort of an automated "Ceases and Desist" letter, and a failure to abide is an automatic tort. Hm?
- pixl97 10y ago
  Let's be a hacker and abuse the protocol and shutdown the internet
- anonymousDan 10y ago
  But how do you know which ips? To request for in the first place without banning legitimate users?
  - DenisM 10y ago
    It shouldn't be hard to tell a person from a dos robot. Person can authorize to get whitelisted, or solve captcha. Persons rate of requests is much lower than that of a bot. The problem is what to do once abusive IPs are identified. Right now you will pay for all the bandwidth used.
- zzzcpan 10y ago
  BGP flowspec works kind of like that. Also routers don't have unlimited capacity to hold everyone's filtering rules, don't forget that.
  - DenisM 10y ago
    Your typical large-scale attack has a few thousand IPs at most, so it's technically doable. All that's missing is the motivation for everyone to abide. With enough incentive hardware makers might accommodate, too.
throw2016 10y ago
At the moment a ddos is like an open weapon that can be deployed by nearly anybody to further any agenda; governments, extortionists, and anyone with an axe to grind. But putting the burden on all devices that connect to the network is idealistic. Sure it may work for the 99 billion devices, but what about the 1 billion devices that may not be properly secured. This wack a mole strategy rarely works. There must be a way for edge nodes to quickly detect a ddos and shut it down. We should be extremely wary security hysteria often used by people with agendas to gain control and lock things down. Only 'approved' devices can connect to the network or everyone on the internet must surrender their passport to the nearest authority and be surveiled 24/7 to prempt and prevent criminal operations.
gohwell 10y ago
Their graphs mention Mrps and Krps, any idea what the R stands for?
- kbuck 10y ago
  Requests (i.e. HTTP requests).
MrMullen 10y ago
The only way I see IoT security being taken serious is when someone makes a bot that hunts down insecure IoT devices and permanently disables (bricks) them. Only then will customers will force manufacturers to take the subject of security seriously.
boulos 10y ago
Nobody seemed to mention it, but the large payload query is designed to also DoS the hell out of old/insecure PHP binaries that will blindly accept (and keep appending) as many items as you want. (In this case the array $a). Similar attacks probably work against other CGI parsing things that don't expect to have hundreds of thousands of qps each with long querystrings.
farhanhubble 10y ago
It's high time we start fixing criminal liability for companies building such poor quality software and hardware.
dx034 10y ago
The OVH write-up [0] is very interesting combined with this, as they had to withstand an even bigger attack. Unfortunately, they don't publish as many statistics. But apparently their attack came mostly from Spain, so different AS than Cloudflare. Their plans to upgrade the Ddos mitigation sound very interesting, will be interesting to see how long it takes until they need the 5Tbps: >will be replaced by a new technology developed in-house and based on FGPA (programmable integrated circuits) and codes that have been in development for the past 18 months. This will allow us to offer up a VAC capable of withstanding DDoS attacks with peaks up to 5 Tbps without slowing down our network. [0] https://www.ovh.com/us/news/articles/a2367.the-ddos-that-didnt-break-the-camels-vac https://www.ovh.com/us/news/articles/a2367.the-ddos-that-did...
codedokode 10y ago
Internet is decentralized so there is no authority that could stop DDoS attack. We need a protocol that would allow a host to require peer or upstream network to block/allow traffic from specific IPs or subnets. And of course the denied traffic should not be paid for.

336 points by jgrahamc 10y ago

That's a problem that will only get worse. To give an example, last year, Incapsula recorded ~9,000 IoT cameras attacking them. A few months ago, Sucuri recorded ~25,000. CloudFlare is seeing close to 50k. And that's the attackers just using a small portion of their real power for http floods. Our report from a few months ago breaking down the types of cameras and networking doing the attack - very similar to what CloudFlare saw: https://blog.sucuri.net/2016/06/large-cctv-botnet-leveraged-ddos-attacks.html https://blog.sucuri.net/2016/06/large-cctv-botnet-leveraged-... *I work at Sucuri.

kingosticks 10y ago

The first attacks were described as only lasting minutes. Is this the perpetrators stopping? If so, why? In case people start to notice their 'smart' lightbulbs are glowing unusually bright? Otherwise what else stops the attack?

tehmaco 10y ago
Proof of ability to potential customer? Checking that their command and control software works? I doubt you'd notice a difference on the device unless you were doing something network/CPU intensive on it, even then, it'd just seem slow.
- kingosticks 10y ago
  Ah OK that makes sense, thanks. It's handy for companies like Cloudflare to get a heads up on what's around the corner.
TorKlingberg 10y ago
If you have failed to take a site down in 15 minutes, continuing for longer is not going to make a difference. Also the attacker may be renting time on the botnet.

zzleeper 10y ago

If things get too bad, can cloudfare pool with google, etc. and just block the IPs of those affected? (until someone fills a captcha)

sofaofthedamned 10y ago
AFAIK Cloudflare only deal with HTTP and HTTPS, so nothing else goes through their proxy. Which doesn't matter anyway because the traffic is still going through the edge networks (the ISP and a little bit upstream from there), so is still bandwidth impacting at some point. We really need a way of getting the ISPs to talk to the providers upstream, and a way of notifying back that a particular host is causing problems. Unfortunately ISPs don't want to do this because unhappy customers = revenue. Also I don't want my traffic analysed on the way through, so there's a freedom issue on this too. So it's not an easy problem to solve, but the ISPs could do it. But they won't. On the flip side dealing with this sort of stuff is why i'd like to work for Cloudflare, Google, Facebook or others. It's fascinating how technical solutions can be bypassed by customer or provider inertia.
deleted 10y ago
[deleted]
deleted 10y ago
[deleted]
deleted 10y ago
[deleted]
majke 10y ago
There are researchers in many companies that are actively fighting the botnets. Some links: - https://krebsonsecurity.com/2016/10/who-makes-the-iot-things-under-attack/ https://krebsonsecurity.com/2016/10/who-makes-the-iot-things... - http://blog.malwaremustdie.org/ http://blog.malwaremustdie.org/ - https://www.akamai.com/us/en/our-thinking/information-security/information-security-research.jsp https://www.akamai.com/us/en/our-thinking/information-securi...

bhouston 10y ago

Does Cloudflare still host (edit: proxy) the companies that do DDoS attacks thus serving to justify the use of Cloudflare services?

jgrahamc 10y ago
We don't do hosting.
- bhouston 10y ago
  Quoting Brian Krebs: > These two services, like most booters, are hidden behind CloudFlare, a content distribution service that lets sites obscure their true Internet address. In case anyone cares, Lizardstresser’s real Internet address currently is 217.71.50.57, at a hosting facility in Bosnia. https://krebsonsecurity.com/2014/12/lizard-kids-a-long-trail-of-fail/ https://krebsonsecurity.com/2014/12/lizard-kids-a-long-trail...
  - aurbano 10y ago
    So they don't do hosting... What you've quoted just explains how they act as a layer in front of the actual server
    - bhouston 10y ago
      Host/proxy - a non-relevant detail. I misspoke. Either way, Cloudflare appears to both help booters and then also sells services to protect you from booters.
      - jgrahamc 10y ago
        Cloudflare's CEO spoke about this here: https://www.youtube.com/watch?v=SWFX-zEYwN0 https://www.youtube.com/watch?v=SWFX-zEYwN0
        e40 10y ago

riskable 10y ago

I've been warning about this kind of thing for years and years. Both professionally and in my personal life. On forums and anywhere people will listen... IoT is a security disaster I hate to suggest things like this but we may need legal remedies to solve the problem before it becomes an Internet Zombie Armageddon (IZA). It's that bad. There's going to be tens of billions of IoT devices installed at businesses and homes in just a few short years. If they're not receiving regular, timely security updates the Internet as we know it may just plain collapse from the sheer weight of all the zombies. I propose a mandatory "nutrition label" of sorts for IoT device packaging. Something like this: * Updates itself regularly * Conforms to NIST security standards * Public penetration test results: https://ptr.iot.nist.gov/A2L7P9D23345 * Shares data with 3rd party: No * Administration by vendor: No * Requires Internet connection to administer: No * Requires 3rd party service to function: No * Notifies of intrusion: Yes * Logs all access: Yes * Encrypts all network traffic: Yes Something like a publicly-posted penetration test would be incredibly valuable. Especially if it was performed after each update of the device and at regular intervals. Even if it was an automated test (which in theory could be 'gamed') it would still be better than what we have now which is nothing!

elthran 10y ago
All very well in theory, but I'm sure it will suffer the same fate as ordinary nutrition labels - the general populace just doesn't care/isn't motivated enough to care
- tjr 10y ago
  Bundle all of those points up into some sort of standard level of acceptance and let conforming products have a big shiny sticker saying so on the front of the package? As a bit of an analogy, while I may enjoy poring over computer system security details, I'm pretty novice at a lot of more real-world things. Yesterday I was buying a new dryer exhaust hose. I read some reviews online, and looked at boxes in the store. I managed to learn enough to realize I wanted an all-metal construction so it wouldn't catch on fire. But I really wanted some big shiny signal to let me know "HEY YES IF YOU HAVE NO IDEA WHAT YOU'RE DOING, BUY THIS ONE!"...

akerro 10y ago

> * Requires 3rd party service to function: No > * Public penetration test results: No vibrator manufacturer will agree to this.

jgrahamc 10y ago
Seriously? That comment is worthy of Beavis and Butthead. And not in a good way.
sctb 10y ago
We detached this subthread from https://news.ycombinator.com/item?id=12684368 https://news.ycombinator.com/item?id=12684368 and marked it off-topic.

falcolas 10y ago

Strangely enough, I think the solution is in devices like OnHub, Google WiFi, or other third party managed routers (yes, another IOT device). Why? When attacks like this are first detected, they can be shut down at the source. Known bad devices can be preemptively locked down behind a firewall. Traffic to specific IP addresses, or packets which spoof headers to cause amplification attacks could all be detected and dropped before they even reach the ISP. Yes, there will be problems and bad diagnoses, no, I don't want one of these managing my home network; but it would be perfect for my mother, for my non-technical friends, and for the average Joe with a subverted IP camera.

Mister_Snuggles 10y ago
This is actually a really neat idea. In another comment[0] I described how I keep my cameras away from the internet. Someone like Google could easily build something into their routers and firewall them off as they're added to the network. The firewall could allow them to talk to the appropriate cloud service, and nothing else, in order to better secure them without the user even knowing it's happening. This doesn't solve all of the IoT problems, but it does help some for the one described in the article. [0] https://news.ycombinator.com/item?id=12685381 https://news.ycombinator.com/item?id=12685381
hathawsh 10y ago
I like this idea a lot. Perhaps every IoT device should publish a profile of its expected behavior to the router. For example, a wifi-enabled thermostat might tell the router that it only intends to connect to a particular range of servers, by DNS name. The router could then set up an automatic firewall rule that stays in sync with DNS changes. The user should be able to disable that firewall rule (to overcome technical issues), but most users will never have a reason to do so. We could bootstrap this idea by having the router fingerprint IoT devices and apply matching firewall policies from a database. If only we could install "apps" on routers, like we do on Android and iOS devices, someone could get started with this idea right now.
- anchpop 10y ago
  The only issue is that after that point, the device can never connect anywhere else.

Mister_Snuggles 10y ago

I've got a bunch of cheap Wifi cameras and I don't worry about this kind of thing at all. All of the cameras live on a separate Wifi network that is unable to connect to the internet. I use Zoneminder to handle all of the recording, motion detection, etc. For remote access, I've got a VPN server running. Fortunately, the cameras I use are perfectly happy to work without being able to talk to the manufacturer's servers (though they do try!). In a nutshell, I get almost all of the "benefits" of the cloud service (the only missing thing is off-site storage, but a fix for that is in the works) with almost none of the drawbacks. Of course, this is NOT easy for the average consumer to set up. They expect to be able to plug the camera in and have things Just Work, which is why we've got cameras that can open ports on routers/firewalls, register themselves with cloud services, etc, with minimal effort. As for IoT in general, my future plans are to use this separate WiFi network for all IoT-like devices that don't really need to connect to the internet. The Philips Hue is a good example - it has a bunch of cloud-enabled functionality, but I don't actually need any of it.

joblessjunkie 10y ago
It sounds like you actually worry about his kind of thing a great deal, considering the amount of effort you put in.
- Mister_Snuggles 10y ago
  Believe it or not, it was a whole bunch of little things that added up to this state. When I started with the cameras I was just using the motion detection and email features that were built into the firmware. I never bothered setting up the cloud stuff because I didn't need it. Slowly I transitioned to having the cameras dump images onto a local NAS via FTP. Eventually I started using the cameras for more than watching the cats, so I needed something with better motion detection and recording capabilities. I started with 'motion', then used a fork of 'motion' that seemed to work better, then finally ended up with ZoneMinder. On the VPN side, I decided many years ago that I wanted remote access to my "stuff" - file shares, etc. This started out with the PPTP VPN built into my router, which is terrible but sort of worked. The next step was writing a script that would run SSH and set up a couple dozen tunnels, which is differently terrible but sort of worked. Eventually I got tired of maintaining that and ended up with an IPSec VPN using racoon -- split DNS and split tunneling are truly amazing. The extra Wifi network actually started as a misguided attempt to prevent other stuff (Netflix, mainly) from interfering with the cameras. I figured that putting them on a separate Wifi network would isolate them and give them dedicated bandwidth. My first attempt involved using a Raspberry Pi as an AP. It worked, but not very well at all. The second attempt used the Virtual AP functionality on my router -- it wasn't until I set this up that I understood that Virtual APs don't quite work this way. Around this time I started to see articles about cloud-enabled cameras doing bad things and decided that this setup was worth keeping. The camera and VPN stuff both went on over the last 5-10 years or so, with the Wifi changes being something I did within the last year. You're right that there was a lot of effort, but as you can see it wasn't really driven by privacy/security until the end. Most of these things were driven by trying to find better ways to scratch various itches. The next planned change is to re-address my network so that it doesn't use the 192.168.1.0/24 range that every consumer router seems to use by default. This way I will be able to use VPN when I'm visiting friends without having to tether to my cell phone. Like everything else, it's just scratching an itch.

pbnjay 10y ago

I have a few cheap chinese ip cameras and I noticed that they have telnet open (I subsequently rooted them to fix this) and of course have them behind my home firewall. However this and other IoT devices got me thinking that maybe it's time to start an underground device hacking organization solely committed to not only rooting and reverse engineering these devices but also releasing new firmware for all of them with open source internals. I'm sure this would have to be "underground" due to all the legal restrictions around the embedded device drivers. Who wants to do this? I wouldn't know where to start besides a .onion site...

rasz_pl 10y ago
>not only rooting and reverse engineering these devices but also releasing new firmware for all of them with open source internals I had the same idea, except in my version automated scan finds exploitable IOT devices and updates them with new firmware straight from /dev/null. Problem SOLVED.
- emp_zealoth 10y ago
  Actually it would be a nice thing to do. Wonder how long would it take manufacturers of all this crap fix their shit after massive warranty claim flood came in
wil421 10y ago
People already release custom firmware for routers. I'm not sure what the manufacturers think about this but I'm sure the same community could be built for IoT devices. The manufacturers would probably be upset they don't get my data to analyze anymore.

bogomipz 10y ago

Does anyone if there's a commonality amongst these IoT security cameras that is facilitating this? Is it just non-sensible defaults or is something more like the embedded webserver/SoC?

blacksmith_tb 10y ago
It's also unclear to me if the devices are being connected directly to the public net - I would have expected most homes and businesses to have some kind of firewall. I suppose it's possible that UPnP is exposing a web interface and/or telnet.
c22 10y ago
I think the biggest shared threat is the ability to punch holes in firewalls with upnp. That and the fact that they're running ancient unpatched software stacks.
- bogomipz 10y ago
  Indeed, that gets back to not sane defaults though I suppose. The filet-of-firewall was particularly troubling, Flash and DNS rebinding is all it takes if UPnP is running on your router. This worth read if anyone is interested: http://www.kb.cert.org/vuls/id/361684 http://www.kb.cert.org/vuls/id/361684

polynomial 10y ago

Seems somewhat ironic that an article about DDoD is published on CloudFlare's blog.

marcosdumay 10y ago
CLoudFare seems honestly interested in making the Internet a better place, one that does not need its current services. I guess they can pivot easily into a cheap bandwidth provider, so this doesn't hurt much. I salute their help and everything, but it's always important to keep in mind that helpful companies are just an acquisition away from antagonizing us.
xxdesmus 10y ago
I'm not so sure "ironic" means what you think it means.
jlgaddis 10y ago
Ironic? To me, it seems quite fitting that a company who has "protecting you from DDoS attacks" as one of their major selling points would publish an article about (recent and relevant) DDoS attacks.
- hueving 10y ago
  Ironic in the sense that they protect companies offering DDoS as a service.

jwmullally 10y ago

A general solution to IoT security would be for all IoT devices to only communicate to your personally owned home gateway, which would run open-source drivers for each device to provide the networking/external communication functionality. The IoT device could even be assigned its own isolated network link to the router (i.e. sandboxed). That approach seems like the only logical conclusion for the device owners. I hope this is how the field will eventually evolve. While its against the commercial interests of "Internet data" companies (who might subsidize devices), in the long term regular consumer appliance manufacturers will implement it if theres enough market pull for it.

mtgx 10y ago
That is kind of what Nest's Thread protocol allows IoT devices to do. But strangely enough, Google is going the opposite direction with its devices, including coming up with such stupid ideas of controlling your (OnHub) Wi-Fi router over the Internet through an app.
- jwmullally 10y ago
  Thread is just an IPv6 wireless mesh routing protocol right [1][2]?. On its own it doesn't solve 2 key security problems: * Any device can send arbitrary data to somewhere on the internet * Internally, any devices can send arbitrary data to any other device ----------------------- Another less-critical but important usability limitation is the short sighted nature of how device logic + APIs will evolve long-term for Device-to-Device communication. Device-to-Device implies the application protocols are running on the devices themselves, which can't be bugfixed or upgraded by the user and in the majority of cases will likely never be updated by the manufacturer after release. In [2] they give examples of devices like fridges and thermostats talking to each other. There's 3 long-term ways I see of handling device-to-device application APIs: * Option 1: Raw device-to-device (Short term): How the hell will those APIs be agreed upon and evolve across multiple years and device products? Bluetooth now barely handles relatively simple interop use cases (files, contacts, audio...). Will Thermostat X support Fridge Protocol 8? How will the user handle more advanced use cases? * Option 2: Vendor managed (Currently the most likely): The devices will talk to the vendor servers which will talk to other vendor servers which will talk to your other devices. Terrible for the user for so many reasons but great for the vendor: its the simplest to implement, keeps protocols secret and locks user into giving that specific vendor endpoint all that juicy data. * Option 3: User managed (IMHO Preferable): Computer drivers are a much better model for this. Drivers sit over devices to transform the weird device API (fixed target) into a consistent interop API (community driven moving target) that does what people want. This still supports device-to-device: device A -> API call on gateway driver A -> gw driver interop API -> API call gateway driver B -> device B. This is what Option 2 (vendor managed) will actually have to do under the hood anyway. This could be implemented in a simple way e.g. with per-device(type) containers exposing a TCP/UDP port to the device. I predict Option 3 to become mainstream when the year of the Linux desktop lands :D ----------------- [1] https://www.threadgroup.org/portals/0/documents/resources/Thread_Technical_Overview.pdf https://www.threadgroup.org/portals/0/documents/resources/Th... [2] https://www.infoq.com/articles/thread-protocol-for-home-automation https://www.infoq.com/articles/thread-protocol-for-home-auto... (Thread is a poorly chosen name for a new computer protocol:P)

Animats 10y ago

Interesting. This set of attacks is coming from real IP addresses. (SYN floods and single-packet UDP-based attacks use a totally bogus source IP address.) The device can be talked to and potentially located. Somebody needs to be finding some of those devices and analyzing them. How are they controlled? IRC? Some set of known domains or IP addresses? A self-healing mesh network? All of those have been used, and most modern botnets have a quite sophisticated control network. This provides both robustness and conceals the actual controller. In modern botnets, everything is encrypted, so it's hard to take over the botnet.[1] Attacks from real IP addresses are potentially blockable upstream. That's only a temporary fix. The device, if behind a NAT, can request a new IP address and evade the block. It may do so periodically, just to confuse the issue. At least you can block a hostile IP at a firewall, so it can't use server resources. That's what Cloudflare is doing. On the legal front, it's worth suing makers of vulnerable devices for negligence. Their EULA will not protect them, because the victim isn't their customer and isn't a party to the EULA. A few court orders to recall a product might get the attention of the IoT industry. [1] https://www.damballa.com/downloads/r_pubs/WP_Botnet_Communications_Primer.pdf https://www.damballa.com/downloads/r_pubs/WP_Botnet_Communic...

exhilaration 10y ago
On the legal front, it's worth suing makers of vulnerable devices for negligence. Where does one sue the no-name Chinese manufacturers of these devices? I have a feeling a lawsuit won't get very far in Shenzhen Municipal Court. There's absolutely no consequences to them building these devices and you cannot expect a solution from them.
- nitrogen 10y ago
  Sue the importers then, and/or file ITC complaints.
  - flukus 10y ago
    Who says there is an importer and not a chinese retail site or ebay seller?

supergeek133 10y ago

Seriously? I get the cameras scenario keeps coming up, but most of these (if not all) are CCTV systems with a DVR with open remote management that either: - Is turned on by default (bad) - Is turned on without enabling even the slightest of security by the end user (stupid but still bad) - If a or b aren't true, it has some sort of exploit that the company will never fix (if they are still in business) The "rest" of IoT in many cases doesn't have this problem at this scale. You're either talking about proximal networks with no outside internet (z-wave, 15.4, etc) or something with secure communication with the home service (PKI for example). I'm more and more frustrated with IoT fearmongering, if you want to talk about CCTV fearmongering? OK. But these aren't Nest cams, or Arlo cams. They're off the shelf DVR CCTV systems.

t3ra 10y ago

Time probably time to name and shame brands. CP Plus is doing so much marketing in India with TV slots. They should probably invest that in doing some research and patching .

merb 10y ago

> Connection: close well if they would have keep-alive it would be probably even more disastrous. especially if the botnet would have some intelligence that it would try to detect the keep alive timeout and send requests exactly in this time frame. -- also a lot of these devices are probably built with linux. and here is something that is probably bad with linux, the lifespan of most linux iot devices are probably outspanning the official lifecycle of any distribution/kernel. this may lead to problems that the vendor will just abadonned it or doesn't care anymore, if he even provided a way to upgrade. on windows this is probably as bad as well since most vendors will probably not pay additional fee's after the next windows iot gets out of live. it's probably time to abadon most os's for these devices and create a new open source project for these kind of devices, hardended from scratch. this would at least help with the unmaintained part of the os parts, which would make the situation at least 1% better. well it won't fix the laziness of certain vendors.. but it would be a beginning.

DenisM 10y ago

How about this for a fix? Just thinking out loud here... Let's make a protocol that allows a server to specify to its upstream ISP to cut off all traffic from a particular IP address. The ISP can either soak up the traffic, or go further up to its own upstream ISP and relay the request, and so on until the source of the attack is reached. Now it's between an IoT device owner and his residential ISP, it's up to them to sort it out. To motivate all ISPs to play along let's make a law for each ISP to be financially responsible for failure to honor such requests, enough for a victim to pay for defense plus some %% in penalties. Sort of an automated "Ceases and Desist" letter, and a failure to abide is an automatic tort. Hm?

pixl97 10y ago
Let's be a hacker and abuse the protocol and shutdown the internet
anonymousDan 10y ago
But how do you know which ips? To request for in the first place without banning legitimate users?
- DenisM 10y ago
  It shouldn't be hard to tell a person from a dos robot. Person can authorize to get whitelisted, or solve captcha. Persons rate of requests is much lower than that of a bot. The problem is what to do once abusive IPs are identified. Right now you will pay for all the bandwidth used.
zzzcpan 10y ago
BGP flowspec works kind of like that. Also routers don't have unlimited capacity to hold everyone's filtering rules, don't forget that.
- DenisM 10y ago
  Your typical large-scale attack has a few thousand IPs at most, so it's technically doable. All that's missing is the motivation for everyone to abide. With enough incentive hardware makers might accommodate, too.

throw2016 10y ago

At the moment a ddos is like an open weapon that can be deployed by nearly anybody to further any agenda; governments, extortionists, and anyone with an axe to grind. But putting the burden on all devices that connect to the network is idealistic. Sure it may work for the 99 billion devices, but what about the 1 billion devices that may not be properly secured. This wack a mole strategy rarely works. There must be a way for edge nodes to quickly detect a ddos and shut it down. We should be extremely wary security hysteria often used by people with agendas to gain control and lock things down. Only 'approved' devices can connect to the network or everyone on the internet must surrender their passport to the nearest authority and be surveiled 24/7 to prempt and prevent criminal operations.

gohwell 10y ago

Their graphs mention Mrps and Krps, any idea what the R stands for?

kbuck 10y ago
Requests (i.e. HTTP requests).

MrMullen 10y ago

The only way I see IoT security being taken serious is when someone makes a bot that hunts down insecure IoT devices and permanently disables (bricks) them. Only then will customers will force manufacturers to take the subject of security seriously.

boulos 10y ago

Nobody seemed to mention it, but the large payload query is designed to also DoS the hell out of old/insecure PHP binaries that will blindly accept (and keep appending) as many items as you want. (In this case the array $a). Similar attacks probably work against other CGI parsing things that don't expect to have hundreds of thousands of qps each with long querystrings.

farhanhubble 10y ago

It's high time we start fixing criminal liability for companies building such poor quality software and hardware.

dx034 10y ago

The OVH write-up [0] is very interesting combined with this, as they had to withstand an even bigger attack. Unfortunately, they don't publish as many statistics. But apparently their attack came mostly from Spain, so different AS than Cloudflare. Their plans to upgrade the Ddos mitigation sound very interesting, will be interesting to see how long it takes until they need the 5Tbps: >will be replaced by a new technology developed in-house and based on FGPA (programmable integrated circuits) and codes that have been in development for the past 18 months. This will allow us to offer up a VAC capable of withstanding DDoS attacks with peaks up to 5 Tbps without slowing down our network. [0] https://www.ovh.com/us/news/articles/a2367.the-ddos-that-didnt-break-the-camels-vac https://www.ovh.com/us/news/articles/a2367.the-ddos-that-did...

codedokode 10y ago

Internet is decentralized so there is no authority that could stop DDoS attack. We need a protocol that would allow a host to require peer or upstream network to block/allow traffic from specific IPs or subnets. And of course the denied traffic should not be paid for.