10 ms·

I got hit with a ~40Gbps DDoS last week. These attacks are on the rise. Some responses to folks above: Success working with upstreams is quite varied. Some car

• CircleSpokes 5y ago
  DNS based amplification has been very popular for many years now. By this point if a DNS resolver is still being used by amplification no email or contacting the ISP will do anything as they have received many other similar emails.
  • flatiron 5y ago
    Dns is so broken in many different ways. It’s crazy it’s still the back bone of the internet when it’s insecure awful nonsense. It’s just ingrained.
    • worik 5y ago
      It is not the back bone of the Internet. It is the back bone of most naming services. The internet is IP. It is very robust. DNS is an application over the top of the Internet.
      • sroussey 5y ago
        The public won’t be able to use the internet without it.
        • Y_Y 5y ago
          This is an interesting proposal. After some thought I must say I'm also in favour of it.
          • wizzwizz4 5y ago
            The public would still use internet applications, like Facebook. They just wouldn't be able to use the web any more.
            • mitigating 5y ago
              Apps use hostsnames 99% of the time.
            • toast0 5y ago
              I don't think Facebook's apps work when there's no access to DNS. At least it didn't seem like it when I was working to keep that capability for WhatsApp as it moved into FB datacenters. I don't think very many other applications will work without DNS either, although I never did much competitive testing.
              • xrisk 5y ago
                Telegram does AFAIK.
              • wizzwizz4 5y ago
                Facebook could (relatively) easily modify their system to do its own IP handling; they probably have enough money to rebuild the entire stack.
                • toast0 5y ago
    • zrm 5y ago
      Modern UDP-based protocols handle this in two ways. First, prefer to make responses no larger than the request, so there is no amplification. Second, if the response has to be larger than the request, send the requestor an address-specific value in a small initial response, e.g. HMAC of a secret and the sender IP. Then any request that incurs a large response has to contain that value. If the sender is spoofing the IP address and can't receive the small response sent to that address, they can't cause a large response to be sent there. This can't be done with DNS because of "security" middleboxes. They ossify the protocol because they reject anything they don't understand, and they don't understand new versions of the protocol even if both of the endpoints do. So the protocol gets frozen in time and no security improvements can be made because of the things that claim to be there to improve security.
      • anakaine 5y ago
        That sounds like its time to push standards forward, announce deprecations in advance, and have as many end services as possible adopt erroring if what they are receiving isn't standards compliant. There is little actual reason for security middleware to not keep up.
        • jfrunyon 5y ago
          Except that in practice, deprecations don't work, no matter how much warning you give.
          • anakaine 5y ago
            The point of deprecations is to eventually force a bad experience for those who are not keeping up. They definitely do work, but the time periods to affect change can be long. In the tech sphere many seem to interpret a long transition period as not working granted the usual pace of change.
            • NullPrefix 5y ago
              Is IPv4 deprecated already?
• pilsetnieks 5y ago
  Can you give more detail on what you mean by the "recent Mikrotik CVE?" As far as I understood, the recent botnet was utilizing devices still unpatched since the original issue (2018?), as well as credentials gathered then even for patched devices that were affected but passwords weren't rotated.
  • ryuuchin 5y ago
    You are correct. It's not really "new" just that it's still being used. https://blog.mikrotik.com/security/meris-botnet.html https://blog.mikrotik.com/security/meris-botnet.html
• alcover 5y ago
  I fail to understand how DNS reflection attacks are possible. Isn't it in the interest of any ISP to block outgoing spoofed IP packets ? So as not to be accused of letting those attacks originate from them ?
  • toast0 5y ago
    If you're thinking of residential ISPs which own the IPs that their customers use, it's mostly pretty simple to prevent spoofing (or at least, prevent spoofing outside their address range), but once you get customers that can bring their own IPs, it can be more cumbersome and the default is to not care. There are, of course, proposals and protocols to make things better, but not caring is easier, and strict enforcement is hard, especially as the bandwidth goes up. There's also enough ISPs that don't do egress filtering that name and shame isn't effective, and anyway, what am I going to do if I'm connected to an ISP that doesn't filter? I have exactly one meaningful offer of connectivity, so that's the one I've accepted, regardless of its merits. Many ISPs have a similar hold on customers.
    • alcover 5y ago
      Thanks. > but once you get customers that can bring their own IPs (...) You know these IPs though, since you route to your customer. So you could drop the offending packets. (I may be getting over my head here).
      • vgb2k18 5y ago
        > I may be getting over my head here) It may be so, however my own need for clarification was the same as yours. If ISP xyz operates a /16 ip block, and only forwards source packets from its /16... Then, I guess, my question: how do spoofed packers usually travel past their first hops on their route to the target? [edit] after-thought: I imagine attackers might easily spoof their source as any of their ISPs /16 or /8. Feels like that's not entire story here though.
        • tw04 5y ago
• temptemptemp111 5y ago
  What is the lowest level hardware that can help defend against DDoS on the backbone level? Or is that not a thing?
  • deleted 5y ago
    [deleted]
• olalonde 5y ago
  Stupid question but can't you just proxy your servers behind Cloudflare (while the attack last)?
  • tw04 5y ago
    That works for a website. Not so much if it's the core router of your business campus.
    • lucb1e 5y ago
      Or if you want your site to be accessible for everyone, value privacy, or remember this guy from yesterday: https://news.ycombinator.com/item?id=28552948 https://news.ycombinator.com/item?id=28552948
      • DenseComet 5y ago
        Thats not really a solution. If you're getting hit with a 40Gbps attack and are running a website and you can simply use Cloudflare, that's a perfectly valid solution. Sure it means that some folks can't crawl your sites and a small number of people might get hit with captchas, but it's better than having no site. I think people who make this argument tend to forget that Cloudflare is an infrastructure provider that the website owner employs. It's not really MITM if the site owner explicitly asked them to terminate TLS so that CF can provide load balancing, tunneling, and a number of other services. It's the exact same as using an AWS ELB. Yeah it terminates TLS, but you can't really say its doing MITM since the site owner specifically configured it for that purpose.
        • lucb1e 5y ago
          > Sure it means that some folks can't crawl your sites and a small number of people might get hit with captchas, but it's better than having no site. Yeah, I just feel like people overreact to news. Moms of 2021: this person in the news had a very bad case of X (covid, covid vaccine, hazelnuts, idk), I should avoid X preemptively Nerds of 2021: this website in the news had a very bad DDoS attack, I should avoid DDoS attacks preemptively The vast majority of people do not need to break the internet[1] for DDoS protection. It is really not that common. I know exactly nobody whose personal website got DDoS'ed. I do know people whose personal website is behind Cloudflare to preemptively avoid this problem. I run a website myself where people can host all sorts of contents, I can totally imagine not everyone is happy with that. Never been on the receiving end of any kind of abuse though (people even ask me if I'm not afraid of that!). And if I were, I'd talk to my ISP -- they were previously involved in lawsuits for internet freedoms (i.e. on the good side), perhaps they are also happy to help me keep a site hosted with them before I need to consider moving to big brother corp for protection. > It's not really MITM if the site owner explicitly asked them to terminate TLS Nobody means MITM in the attacker sense when the service being MITM'd literally asked the proxy to proxy their traffic. Obviously. Saying that Cloudflare MITMs connections is a way to carry both meaning and judgement, similar to how I will talk about middleboxes on corporate networks that block evil haxxor tools that I need for my daily work (y'know, wireshark and such). I call those MITM boxes because that's what they do but also because I think they're more evil than good and the term reflects that (even if there are obviously pros and cons, same with Cloudflare). > It's the exact same as using an AWS ELB Hmm, if I understand what AWS does correctly, their load balancing service just routes traffic internally. It's not a transparent proxy where you think you're talking to one company but really you're talking to another. The manager at BigBank also understands intuitively that if they host their data at ExampleCorp, that ExampleCorp needs to not have data breaches. But if Cloudflare it just removing malicious traffic, it's not immediately obvious that they are in just as sensitive a position. The privacy policy rarely if ever mentions such proxying services. I take your point though that it's not that different. This is also why I'd never host with Amazon or configure my email servers to be Google's, but yeah Cloudflare proxying gets more comments than hosting the whole thing at what some people perceive as an evilcorp. Not sure if that's for the aforementioned reasons or not. [1] https://en.wikipedia.org/wiki/End-to-end_principle https://en.wikipedia.org/wiki/End-to-end_principle
• r1ch 5y ago
  According to Mikrotik the recent botnet of hacked routers are only used for proxying traffic, not generating it. If this is true then they're only useful for hiding source addresses, an attacker actually loses power if using them for a volumetric attack.
  • judge2020 5y ago
    If that were true, it would be trivial to trace back the traffic to the origin of the attack since you would see 40Gbps incoming at the target, one set of intermediaries, and 40Gbps to a single source (which is also unlikely given 40Gbps uplinks are quite a big monthly expense). They might be using the Tor network (or creating a makeshift proxy network), but it would seemingly be a waste of bandwidth on the target routers. The regular C&C approach seems more practical since it can make use of available bandwidth and leaves less of a trail.
    • deleted 5y ago
      [deleted]
    • Sebb767 5y ago
      It's quite unlikely there's a single link sending the traffic; that would be super easy to block. Most likely this is used to either hide the command server or the actually compromised servers. While symmetrical load can be quite obvious, it's less so if we're talking about 2000 links sending at 20 mb/s each. Especially when those links also have legitimate traffic.
• omginternets 5y ago
  > recent microtik CVE I bought a mikrotik router recently, for my home network. How can I tell if I am impacted by this, and are there any mitigation strategies?
  • ryuuchin 5y ago
    https://blog.mikrotik.com/security/meris-botnet.html https://blog.mikrotik.com/security/meris-botnet.html It's not really recent (2018) just that it's still being used after all this time. If you're using a default config on the home router you're basically already fine (save for changing the default login).
  • Kemejii 5y ago
    Microtik DDOS attacks are using CVE-2018-14847. [1] If your router is up-to-date and not previously compromised, you should be fine. [1] https://blog.mikrotik.com/security/meris-botnet.html https://blog.mikrotik.com/security/meris-botnet.html
• Jerry2 5y ago
  > As a result, I'm going to add recurring testing for this capability and burn a couple IPs to make sure upstreams are listening to our rtbh announcements. Could you or someone else expand on this? How do you coordinate rtbh with your ISP? And how do you check for whether it's working? I'd love to learn more on this topic. Thanks!
  • deleted 5y ago
    [deleted]
  • AdamJacobMuller 5y ago
    Many ISPs (better called transit providers in this context) offer a service whereby you announce to them a route (over BGP) with a specific BGP community, sometimes over a special session, sometimes inband with your normal transit sessions and they will blackhole (route to discard, null0, /dev/null) all traffic to that IP. Unlike normal internet announcements these are generally (exclusively with the providers i've worked with) available down to the smallest IP unit (v4/32, v6/64) so you can blackhole an IP which is being attacked without impacting other IPs inside the same subnet. How do you test it? Very simple. Announce an IP (or a few) as blackhole and test to make sure things don't work (from that IP). Very simply could setup something to ping something on that provider's infrastructure from that IP and... if it starts to work, alert!
    • kbenson 5y ago
      To my knowledge there's also ways to integrate network observability tools (like kentik) to automate this to a degree, for those that are big enough or DDOS events are common enough that is useful to do so. I imagine getting paged because of a DDOS is slightly easier when it's telling you it already null routed a few IPs so the rest of your network isn't screwed and you just have to identify how problematic those specific IPs being out of service is and whether you need to take action or wait for them to get bored.
    • cft 5y ago
      That sort of encourages the attackers, because from their point of view, the attack is succeeding while the IP is being blackholed.
• ryuuchin 5y ago
  > Very concerned about the recent microtik CVE, as that is going to make for some very large botnets. To be pedantic there is technically no recent Mikrotik CVE WRT Meris. It was patched in 2018(?) shortly after discovery. From their response to the Meris botnet[1]: > As far as we have seen, these attacks use the same routers that were compromised in 2018, when MikroTik RouterOS had a vulnerability, that was quickly patched. > Unfortunately, closing the vulnerability does not immediately protect these routers. If somebody got your password in 2018, just an upgrade will not help. You must also change password, re-check your firewall if it does not allow remote access to unknown parties, and look for scripts that you did not create. It goes into more detail to further check/harden the device in the blog post. A lot of issues stem from having Winbox or other admin access not properly firewalled off and open to the world. Blessing and a curse of the power you have with these devices I guess. [1] https://blog.mikrotik.com/security/meris-botnet.html https://blog.mikrotik.com/security/meris-botnet.html
• silisili 5y ago
  I work for a DDoS prevention provider, and business is booming at the moment. I'm not a salesman nor do I care that much about the company, but seriously, let someone handle this stuff for you. There are at least a handful of us left, and we are good at what we do, generally.
  • LunaSea 5y ago
    I don't think it's a question of service quality rather than a question of technical independence and sovereignty.
    • silisili 5y ago
      I get that, but it's one of those things a lot of people go through, and blow a ton of effort on solved problems. Because it's not one of the things that's so obvious and commoditized...yet. Peace of mind is worth a lot. If this person were with my company for example, this wouldn't even be a comment. I say my company meaning my employer. I have exactly zero stake in mentioning them by name or advertising them.
      • tediousdemise 5y ago
        The best name in the DDoS protection business is Cloudflare. I’m sure they’re going to acquire or snuff out all competition within a few years. It’s probably a liability to go with anyone else, due to the lack of network resources.
        • sporkland 5y ago
          Don't Google, Amazon and Msft have their own CDN products with DDoS protections? I know they probably aren't at cloudflare's level of sophistication but in a few years time you don't think it will be a native offering from the big three as well as cloudflare duking it out in this space?
          • silisili 5y ago
• dr-detroit 5y ago
  Big tech is in bed with these slavs (read:slobs) so I don't know why you're fighting it so hard. Welcome to the nu normal.
• ilaksh 5y ago
  Or decentralize towards content-centric services.