23 ms·

Belgian ISP under 250 Gbps DDoS for days on end

497 points by laurensr 5y ago

ericbarrett 5y ago
> EDIT 16/09/2021 10:12*: During the night we had two more attacks. We are working with the authorities, who have confirmed they are looking into it and are doing everything in their power to find the responsible individuals. We were contacted by an individual who verified he was behind the attacks, asking for a ransom. Roughly how many criminal groups are active in DDoS ransoms (as opposed to data crimes, like cryptolockers and exfiltration)? How common is this nowadays? Clearly it happens, but I've no idea the general scale of the problem in 2021.
- masklinn 5y ago
  > Roughly how many criminal groups are active in DDoS ransoms These days there are groups providing ddos as a service. They control huge iot botnets and for a sum they’ll point whatever fraction you pay for to whatever target you want.
  - ericbarrett 5y ago
    Put more concretely, how many can front 100+ Gbps? E.g. 2-3, or dozens, hundreds, any teenager, etc.
    - sva_ 5y ago
      Depends what kind of attack they use. A few years back, reflection attacks were all the rage, and it was cheap as fuck. But I can't say how many Gbps they delivered. They'd usually market this as "stresstest" service, and you could pay by Paypal etc. Curiously they were all behind a Cloudflare-anti-DDoS wall. I still wonder if Cloudflare knew/knows about them, probably good for their business.
stingraycharles 5y ago
Since this is HN, it’s 2021 and DDoS’es are still a thing: why are they still a thing? Is there some fundamental “anonymity” to the Internet that makes it impossible to structurally prevent DDoS attacks? Apart from CloudFlare-like approaches, are there any R&D in the pipeline that may kill this type of attack once and for all? To me it’s incredibly infuriating to see the damage that still happens with these extremely simple techniques. Will it ever end? Edit: to elaborate, I know that there are tons of insecure Internet devices and whatnot. I’m more interested in standards, and core protocol improvements that can fundamentally rid the world of these types of attacks.
- foobiekr 5y ago
  DDOS events typically use distributed botnets. Unless you analyze the CNC you’re probably asking for wide deployment of acls (flow spec or otherwise); and such agreements happen occasionally, as they operationally brutal but are slow (otherwise they would be subject to abuse). You, right now, may be participating in a DDOS attack without knowing it. Should your ISP cut you off based on information from some Belgian ISP?
- spenczar5 5y ago
  DDoSes come from asymmetries in the difficulty of making a request and providing a response. For example, a classic is the SYN flood. In this, the attacker makes many requests to open TCP connections (with a SYN packet) but does none of the rest of the handshaking. The server does the initial bookkeeping for setting up TCP connections, before eventually timing out and freeing the resources. The SYN packets are very cheap and easy to form (and the attacker can immediately free all the resources after firing them away), but the connection state information is a little bigger. This asymmetry builds up, making the server overloaded. I believe that asymmetries in resource consumption are, at some point, unavoidable. You also asked: > are there any R&D in the pipeline that may kill this type of attack once and for all? Note that DDoS attacks are a class of attacks. They're a description for a broad genre. It's kind of like how "bronchitis" actually just means "enflamed bronchial tubes" - it's a symptom, not a cause. Even if we permanently defeat SYN floods, there are a gazillion other DDoS-type attacks (UDP flooding, messing with ARP tables - its an almost endless list).
mfkp 5y ago
I wonder if this is related to the attack on voip.ms (ongoing for multiple days) https://twitter.com/voipms https://twitter.com/voipms
vadfa 5y ago
A 250 Gbps attack is bringing an ISP down? How is this possible in the age of 1 Gbps connections in every home?
- BenjiWiebe 5y ago
  50 1Gbps connections usually peak at less than 50Gbps. Depending on your demographic and area, it can be a lot less. 250 Gbps will service much more than 250 1Gbps connections. Also, not nearly every ISP offers 1 Gbps to all their customers. Some ISPs don't offer it at all.
  - tyingq 5y ago
    There's also things like local Netflix/Google/etc caching boxes where you can serve downsteam traffic without touching your upstream as much.
- KindOne 5y ago
  They seem to only offer 20Mbps - 500Mbps. https://www.edpnet.be/nl/thuis/internet.html https://www.edpnet.be/nl/thuis/internet.html https://www.edpnet.be/nl/business/internet.html https://www.edpnet.be/nl/business/internet.html
  - laurensr 5y ago
    They are mainly offering VDSL2 connections over phone lines. They also offer Fiber over a monopolist's (called Proximus) GPON infrastructure, but the rollout of that GPON network is still in early stages. However, that monopolist has pledged to accelerate that rollout in light of the increasing work-from-home statistics following the pandemic. https://www.proximus.com/news/2020/20201208-proximus-brings-fiber-network-to-brussels.html https://www.proximus.com/news/2020/20201208-proximus-brings-...
- NullPrefix 5y ago
  overselling
- MeinBlutIstBlau 5y ago
tigerlily 5y ago
How can I check my home network for signs I have any devices participating in a botnet?
- mfkp 5y ago
  Generally large spikes in upload data usage is the best indicator. Many consumer grade routers are starting to build in per-device bandwidth usage, which helps narrow down the culprit.
- system2 5y ago
  Invest in something like a cheap sonicwall. Without bunch of licenses, it can still show you all the traffic information and act as a very powerful firewall with all types of blocking. You can search "sonicwall soho". If you don't want to invest, you can also use pfsense. You can use an old computer to turn it into a perfect firewall.
  - antattack 5y ago
    I used both pfsense, opnsense and none had good facilities to keep track where your packets are going.
    - system2 5y ago
      Then you might want to check sonicwall DPI.
s800 5y ago
I got hit with a ~40Gbps DDoS last week. These attacks are on the rise. Some responses to folks above: Success working with upstreams is quite varied. Some care, some don't, and it can be difficult to get to folks that can help- even if their networks are impacted as well. Some carriers immediately turn this into a sales opp. - buy more bandwidth, buy more services. In our case it was based on DNS reflection from a large number of hosts. I've contacted the top sources (ISPs hosting the largest number of attackers) and provided IPs and timestamps. I've received zero responses. Geo-based approaches yielded no helpful reduction in source traffic. Also, during this event we discovered an upstream of ours had misconfigured our realtime blackhole capability. As a result, I'm going to add recurring testing for this capability and burn a couple IPs to make sure upstreams are listening to our rtbh announcements. Very concerned about the recent microtik CVE, as that is going to make for some very large botnets. Personally this all is very disappointing because it creates an incentive to centralize / de-distribute applications to a few extremely large infrastructure providers who can survive an attack of these magnitudes.
- CircleSpokes 5y ago
  DNS based amplification has been very popular for many years now. By this point if a DNS resolver is still being used by amplification no email or contacting the ISP will do anything as they have received many other similar emails.
  - flatiron 5y ago
    Dns is so broken in many different ways. It’s crazy it’s still the back bone of the internet when it’s insecure awful nonsense. It’s just ingrained.
    - worik 5y ago
      It is not the back bone of the Internet. It is the back bone of most naming services. The internet is IP. It is very robust. DNS is an application over the top of the Internet.
pstuart 5y ago
I'm assuming that eBPF would be a potent tool for dealing with this, and it looks like CloudFlare agrees: https://blog.cloudflare.com/l4drop-xdp-ebpf-based-ddos-mitigations/ https://blog.cloudflare.com/l4drop-xdp-ebpf-based-ddos-mitig...
- viraptor 5y ago
  Software solutions at the receiving end are meaningless for stateless DDoS. It doesn't matter how quickly you drop/filter packets if your physical connection cannot receive any more of them. You need to ensure the traffic never saturates your connection in the first place to avoid issues - this can only be done on the sending end or somewhere on the path to you.
  - pstuart 5y ago
    Good to know
- kortilla 5y ago
  Ebpf only matters if you have the bandwidth to receive it in the first place.
ineedasername 5y ago
I'm very much not a network engineer, but I'd like to understand the magnitude of this issue because my intuition is wrong: 250 Gbps seems like it would definitely be a lot for a server or website but it also seem like a drop in the bucket for an ISP providing broadband for many customers. Clearly I'm wrong because it is an issue here. I'd like to understand why I'm wrong, and I hope that here, on HN, that's taken in the spirit of curiosity intended and not negativism. So, what am I missing? Maybe Belgian broadband is lower capacity than what I'm used to in a US metropolitan area? Maybe this particular ISP served a population too small to have a... um... "fat pipe"? I'd like to understand.
- xhrpost 5y ago
  250Gbps is still a lot of bandwidth for a network even today. An enterprise ISP like Verizon or Spectrum may have less of a capacity issue with it but for many other networks this is a lot. Quick example, though not an ISP, but the entire Internet archive operates on 60Gbps of bandwidth. So an attack of 250 would be quite a problem. http://blog.archive.org/2020/05/11/thank-you-for-helping-us-increase-our-bandwidth/ http://blog.archive.org/2020/05/11/thank-you-for-helping-us-...
  - ineedasername 5y ago
    the entire Internet archive operates on 60Gbps of bandwidth Wow, yes, that definitely puts the magnitude of this into perspective. Thank you.
  - Groxx 5y ago
    Though that's a substantial amount of traffic: isn't this currently going through one ISP, which (I would hope) has many more customers than just archive.org? "top 160 site" is significant, but still minuscule compared to all internet traffic.
    - therealdrag0 5y ago
      I think I’m thinking of it reverse from you. A top 160 site could have more concurrent users than a small regional ISP.
pt_PT_guy 5y ago
and no one accountable :-)
- lucb1e 5y ago
  Yeah this is the part I don't get. Ok the source IP is spoofed, so you need to use another technique. The packets come in on some physical line, so you can ask the person on the other end which line they're getting it from and so forth. Relatively labor-intensive (I imagine a few hours' worth of time from each network hop), but you get there. Next, you find that it's coming from across some border, so you hand it off to the local authority. As I understand it, this is where people drop the ball. The local authorities can't be arsed, so the criminals there just keeps doing it. Why then not block that connection? If the provider in that country wants to be able to continue to operate their services, they'll need to fix this problem or customers around the world just won't be able to receive responses from their network. Same with the DNS resolvers: if someone has an open DNS resolver that is constantly being abused, as all of them are, just cut them off right? You don't need that on your IP ranges. Or blackhole that source IP on the edge of your network if you see abuse coming from that resolver. It's not rocket science to figure out which part of DNS traffic is abuse if you see a constant volume of responses to queries that the client in your network never actually asked for. Neither of these things seem to happen. Why can you be an ass on the internet and never be held accountable?
  - TobTobXX 5y ago
    Possible for DoS attacks. Practically impossible for DDoS attacks.
    - lucb1e 5y ago
      Huh? I wrote this: > Same with the DNS resolvers: if someone has an open DNS resolver that is constantly being abused, as all of them are, just cut them off right? [etc.] That's what I'm not understanding why it is not being applied to easily solve those reflective DDoS attacks.
forty 5y ago
Why are the people doing those attacks doing them? Ransom? Or are they just doing that to prove and advertise/demo their capacity to harm?
- schleck8 5y ago
  someone here said the attacker asked for a ransom
JackMcMack 5y ago
I'm an Edpnet subscriber currently suffering the effects of the DDOS. It's annoying to say the least. Last year's DDOS attack was relatively simple, pointed at the DNS servers. Simply using other dns servers was good enough to get back online. Edpnet also joined NaWas [0] at that time, a non-profit for ISPs to be able to redirect all trafic through big pipes when needed to deal with large attacks. Because the current attacks are rapidly shifting targets, it's a game of cat and mouse to properly filter the ddos. In practice, this means that some sites such as google and youtube keep working, but other services might not be available. It is extremely annoying when all of a sudden AWS api calls time out, or a Teams or Slack call suddenly drops to very low bandwith, and then drops entirely. I've had to resort to my phone's hotspot multiple times in the last few days. Yes, I pay for SLA, but what's the point in that? I've got priority in case of a cable break, and failover to 4G connection, but that's no use if the upstream is congested. The sad part is that the attack works because it it a small isp, 45000 customers. [1] It is the main reason I'm a customer, they offer good service for great prices. Kudos for not paying the ransom. If the attacks continue for much longer, I will probably switch to the bigger, more expensive, less customer friendly isp. I'm happy to support a local company instead of a big multinational coorporation. But if my clients can't depend on me when working from home, I've got no choice but to pick the ISP with the bigger and more expensive pipes. [0] https://www.nbip.nl/en/nawas/ https://www.nbip.nl/en/nawas/ [1] https://datanews.knack.be/ict/nieuws/edpnet-al-dagen-getroffen-door-zware-ddos/article-news-1779585.html https://datanews.knack.be/ict/nieuws/edpnet-al-dagen-getroff...
- lucb1e 5y ago
  If you give them kudos for not paying the ransom, it's also worth considering not switching away from the small ISP you chose because they were doing things right. Otherwise they might as well have not done the morally right thing and given y'all better service. Though, of course, I do understand your conundrum. Perhaps there's some middle ground, where for anything that really needs the uptime you have a (mobile) fallback? Still annoying if there are outages on non-essential lines but perhaps better than forcing them into paying the criminals.
  - JackMcMack 5y ago
blunte 5y ago
How often do we ever find the actual individual humans resonsible for these bad actions? I would imagine if the individuals were identified, some harmed parties would spend money eliminating them to dissuade others doing the same thing.
- U8dcN7vx 5y ago
  They are all too often in another country, so they can sometimes be determined but seldom accountable.
Havoc 5y ago
Unfortunately this works. Kills small ISPs on all fronts. Customer perception, technical and financial
hda2 5y ago
Spamhus bad ISPs. Give them low QoS until they fix their problem. If all major exchanges and service providers agree to punish IP blocks, ISPs will have no choice but to better police their networks.
ThinkBeat 5y ago
What is the motivation? Extortion? Retribution? Censorship? Marketing? QA? Are they asking for money stop? Are they being paid by some actor to do this because they wishes to protest / punish the company? Are they demonstrating what they can do, in order to drum up business? Are they testing what their system can accomplish? and iron out any bugs and learn how to best utilize it? So what mitigations they face and how to overcome them?
- stardenburden 5y ago
  The attackers are asking for ransom. So as usual, money.
riofoxx 5y ago
I can help you earn $ 6.0,0.0.0 with $ 5.0.0.0 in a w.e.e.k t.h.r.o.u.g.h M.R h.a r.l.a.n.d. w.h.o h.e.l.p.e.d m.e g.e.t the b.e.s.t in c.r.y.p.t.o t.r.a.d.i.n.g.. Contact information via WhatsApp +13052395906)
Aeolun 5y ago
It there no way to just completely block every IP sending you so much traffic? You might end up blocking a few million, but your network remains uncongested.
- kazen44 5y ago
  the only way to block this traffic effectively is by asking your upstream peers to blackhole the traffic. dropping this with a firewall or other device is nearly impossible because of the high load. remote triggered blackholing is done by BGP. which between peers only allows a /24 in terms of the smallest announcable prefix. blocking every /24 from which an IP in a ddos originates kills most if not all of your reachability.
  - Aeolun 5y ago
    Thanks for that explanation. Wouldn’t some device still need to blackhole the stuff though? I’d still argue that blackholing /24 blocks and retaining some reachability is preferable to losing all of it. Now I’m kind of curious how fast you can make hardware drop stuff, but even inside my local PC I probably cannot reach 250Gbps of data flow (Hmm, 20 Gbps for DDR4, guess not).
edoceo 5y ago
Don't the peering agreements have to pay for bandwidth? Isn't there a bigger cost for those flooding this one? Could they charge-back and encourage peers to throttle? Can peers fix this with pricing model changes?
dgdfg 5y ago
https://www.kaowarsom.be/sites/default/files/webform/yr2020/Yogibo-presents-japan-pk3.pdf https://www.kaowarsom.be/sites/default/files/webform/yr2020/... https://www.kaowarsom.be/sites/default/files/webform/yr2020/Yogibo-presents-japan-pk2.pdf https://www.kaowarsom.be/sites/default/files/webform/yr2020/... https://www.kaowarsom.be/sites/default/files/webform/yr2020/Yogibo-presents-japan-pk1.pdf https://www.kaowarsom.be/sites/default/files/webform/yr2020/... https://www.kaowarsom.be/sites/default/files/webform/yr2020/Yogibo-presents-japan-pk0.pdf https://www.kaowarsom.be/sites/default/files/webform/yr2020/... https://www.kaowarsom.be/sites/default/files/webform/yr2020/Yogibo-presents-japan-pk.pdf https://www.kaowarsom.be/sites/default/files/webform/yr2020/... https://twitter.com/Ert2541/status/1439482365491834885 https://twitter.com/Ert2541/status/1439482365491834885 https://twitter.com/i/events/1439482429031403520 https://twitter.com/i/events/1439482429031403520 https://twitter.com/FHMJyv2yqViFD7n/status/1439483132550402048 https://twitter.com/FHMJyv2yqViFD7n/status/14394831325504020... https://www.vikkislade.org.uk/23588/_2021_rizin_30 https://www.vikkislade.org.uk/23588/_2021_rizin_30 https://www.vikkislade.org.uk/23588/_live_rizin_30_t3envrdltuwultnefczc9w https://www.vikkislade.org.uk/23588/_live_rizin_30_t3envrdlt... https://www.vikkislade.org.uk/23588/_30_rizin_30_live https://www.vikkislade.org.uk/23588/_30_rizin_30_live
r_singh 5y ago
I worked at an ISP AAA/Radius provider and since we started offering a cloud offering to our customers we were DDoSed twice. The source IPs were a Turkish university, I wrote to them a few times but they seemed clueless. Cloudflare's protection suite was beyond our budget and our engineers really started moving the system between 5 different IP blocks thanks to the help of our customers. We still don't know who's responsible, but we suspect it could be done by our competitor... (PS - We're based in India)
teitoklien 5y ago
https://gnunet.org/en/gns.html https://gnunet.org/en/gns.html
- doublepg23 5y ago
  I've tried getting into gnunet on and off for a couple years now (granted the last time was 2+ years now) and it was always inscrutable to me. Nowhere near as easy as Tor or even FreeNet (which is a dead Java app used only by the worst people as far as I could tell).
bullen 5y ago
Why not just block the offending IPs in iptables? Edit: as a Swede it's very funny that customer is called klant in dutch!
leoc 5y ago
Mods, can the URL please be corrected to https://issues.edpnet.be/?p=3507 https://issues.edpnet.be/?p=3507 ? The current link to https://issues.edpnet.be/ https://issues.edpnet.be/ is likely to rot badly over the years.

497 points by laurensr 5y ago

> EDIT 16/09/2021 10:12*: During the night we had two more attacks. We are working with the authorities, who have confirmed they are looking into it and are doing everything in their power to find the responsible individuals. We were contacted by an individual who verified he was behind the attacks, asking for a ransom. Roughly how many criminal groups are active in DDoS ransoms (as opposed to data crimes, like cryptolockers and exfiltration)? How common is this nowadays? Clearly it happens, but I've no idea the general scale of the problem in 2021.

masklinn 5y ago
> Roughly how many criminal groups are active in DDoS ransoms These days there are groups providing ddos as a service. They control huge iot botnets and for a sum they’ll point whatever fraction you pay for to whatever target you want.
- ericbarrett 5y ago
  Put more concretely, how many can front 100+ Gbps? E.g. 2-3, or dozens, hundreds, any teenager, etc.
  - sva_ 5y ago
    Depends what kind of attack they use. A few years back, reflection attacks were all the rage, and it was cheap as fuck. But I can't say how many Gbps they delivered. They'd usually market this as "stresstest" service, and you could pay by Paypal etc. Curiously they were all behind a Cloudflare-anti-DDoS wall. I still wonder if Cloudflare knew/knows about them, probably good for their business.

stingraycharles 5y ago

Since this is HN, it’s 2021 and DDoS’es are still a thing: why are they still a thing? Is there some fundamental “anonymity” to the Internet that makes it impossible to structurally prevent DDoS attacks? Apart from CloudFlare-like approaches, are there any R&D in the pipeline that may kill this type of attack once and for all? To me it’s incredibly infuriating to see the damage that still happens with these extremely simple techniques. Will it ever end? Edit: to elaborate, I know that there are tons of insecure Internet devices and whatnot. I’m more interested in standards, and core protocol improvements that can fundamentally rid the world of these types of attacks.

foobiekr 5y ago
DDOS events typically use distributed botnets. Unless you analyze the CNC you’re probably asking for wide deployment of acls (flow spec or otherwise); and such agreements happen occasionally, as they operationally brutal but are slow (otherwise they would be subject to abuse). You, right now, may be participating in a DDOS attack without knowing it. Should your ISP cut you off based on information from some Belgian ISP?
spenczar5 5y ago
DDoSes come from asymmetries in the difficulty of making a request and providing a response. For example, a classic is the SYN flood. In this, the attacker makes many requests to open TCP connections (with a SYN packet) but does none of the rest of the handshaking. The server does the initial bookkeeping for setting up TCP connections, before eventually timing out and freeing the resources. The SYN packets are very cheap and easy to form (and the attacker can immediately free all the resources after firing them away), but the connection state information is a little bigger. This asymmetry builds up, making the server overloaded. I believe that asymmetries in resource consumption are, at some point, unavoidable. You also asked: > are there any R&D in the pipeline that may kill this type of attack once and for all? Note that DDoS attacks are a class of attacks. They're a description for a broad genre. It's kind of like how "bronchitis" actually just means "enflamed bronchial tubes" - it's a symptom, not a cause. Even if we permanently defeat SYN floods, there are a gazillion other DDoS-type attacks (UDP flooding, messing with ARP tables - its an almost endless list).

mfkp 5y ago

I wonder if this is related to the attack on voip.ms (ongoing for multiple days) https://twitter.com/voipms https://twitter.com/voipms

vadfa 5y ago

A 250 Gbps attack is bringing an ISP down? How is this possible in the age of 1 Gbps connections in every home?

BenjiWiebe 5y ago
50 1Gbps connections usually peak at less than 50Gbps. Depending on your demographic and area, it can be a lot less. 250 Gbps will service much more than 250 1Gbps connections. Also, not nearly every ISP offers 1 Gbps to all their customers. Some ISPs don't offer it at all.
- tyingq 5y ago
  There's also things like local Netflix/Google/etc caching boxes where you can serve downsteam traffic without touching your upstream as much.
KindOne 5y ago
They seem to only offer 20Mbps - 500Mbps. https://www.edpnet.be/nl/thuis/internet.html https://www.edpnet.be/nl/thuis/internet.html https://www.edpnet.be/nl/business/internet.html https://www.edpnet.be/nl/business/internet.html
- laurensr 5y ago
  They are mainly offering VDSL2 connections over phone lines. They also offer Fiber over a monopolist's (called Proximus) GPON infrastructure, but the rollout of that GPON network is still in early stages. However, that monopolist has pledged to accelerate that rollout in light of the increasing work-from-home statistics following the pandemic. https://www.proximus.com/news/2020/20201208-proximus-brings-fiber-network-to-brussels.html https://www.proximus.com/news/2020/20201208-proximus-brings-...
NullPrefix 5y ago
overselling
MeinBlutIstBlau 5y ago

tigerlily 5y ago

How can I check my home network for signs I have any devices participating in a botnet?

mfkp 5y ago
Generally large spikes in upload data usage is the best indicator. Many consumer grade routers are starting to build in per-device bandwidth usage, which helps narrow down the culprit.
system2 5y ago
Invest in something like a cheap sonicwall. Without bunch of licenses, it can still show you all the traffic information and act as a very powerful firewall with all types of blocking. You can search "sonicwall soho". If you don't want to invest, you can also use pfsense. You can use an old computer to turn it into a perfect firewall.
- antattack 5y ago
  I used both pfsense, opnsense and none had good facilities to keep track where your packets are going.
  - system2 5y ago
    Then you might want to check sonicwall DPI.

s800 5y ago

I got hit with a ~40Gbps DDoS last week. These attacks are on the rise. Some responses to folks above: Success working with upstreams is quite varied. Some care, some don't, and it can be difficult to get to folks that can help- even if their networks are impacted as well. Some carriers immediately turn this into a sales opp. - buy more bandwidth, buy more services. In our case it was based on DNS reflection from a large number of hosts. I've contacted the top sources (ISPs hosting the largest number of attackers) and provided IPs and timestamps. I've received zero responses. Geo-based approaches yielded no helpful reduction in source traffic. Also, during this event we discovered an upstream of ours had misconfigured our realtime blackhole capability. As a result, I'm going to add recurring testing for this capability and burn a couple IPs to make sure upstreams are listening to our rtbh announcements. Very concerned about the recent microtik CVE, as that is going to make for some very large botnets. Personally this all is very disappointing because it creates an incentive to centralize / de-distribute applications to a few extremely large infrastructure providers who can survive an attack of these magnitudes.

CircleSpokes 5y ago
DNS based amplification has been very popular for many years now. By this point if a DNS resolver is still being used by amplification no email or contacting the ISP will do anything as they have received many other similar emails.
- flatiron 5y ago
  Dns is so broken in many different ways. It’s crazy it’s still the back bone of the internet when it’s insecure awful nonsense. It’s just ingrained.
  - worik 5y ago
    It is not the back bone of the Internet. It is the back bone of most naming services. The internet is IP. It is very robust. DNS is an application over the top of the Internet.

pstuart 5y ago

I'm assuming that eBPF would be a potent tool for dealing with this, and it looks like CloudFlare agrees: https://blog.cloudflare.com/l4drop-xdp-ebpf-based-ddos-mitigations/ https://blog.cloudflare.com/l4drop-xdp-ebpf-based-ddos-mitig...

viraptor 5y ago
Software solutions at the receiving end are meaningless for stateless DDoS. It doesn't matter how quickly you drop/filter packets if your physical connection cannot receive any more of them. You need to ensure the traffic never saturates your connection in the first place to avoid issues - this can only be done on the sending end or somewhere on the path to you.
- pstuart 5y ago
  Good to know
kortilla 5y ago
Ebpf only matters if you have the bandwidth to receive it in the first place.

ineedasername 5y ago

I'm very much not a network engineer, but I'd like to understand the magnitude of this issue because my intuition is wrong: 250 Gbps seems like it would definitely be a lot for a server or website but it also seem like a drop in the bucket for an ISP providing broadband for many customers. Clearly I'm wrong because it is an issue here. I'd like to understand why I'm wrong, and I hope that here, on HN, that's taken in the spirit of curiosity intended and not negativism. So, what am I missing? Maybe Belgian broadband is lower capacity than what I'm used to in a US metropolitan area? Maybe this particular ISP served a population too small to have a... um... "fat pipe"? I'd like to understand.

xhrpost 5y ago
250Gbps is still a lot of bandwidth for a network even today. An enterprise ISP like Verizon or Spectrum may have less of a capacity issue with it but for many other networks this is a lot. Quick example, though not an ISP, but the entire Internet archive operates on 60Gbps of bandwidth. So an attack of 250 would be quite a problem. http://blog.archive.org/2020/05/11/thank-you-for-helping-us-increase-our-bandwidth/ http://blog.archive.org/2020/05/11/thank-you-for-helping-us-...
- ineedasername 5y ago
  the entire Internet archive operates on 60Gbps of bandwidth Wow, yes, that definitely puts the magnitude of this into perspective. Thank you.
- Groxx 5y ago
  Though that's a substantial amount of traffic: isn't this currently going through one ISP, which (I would hope) has many more customers than just archive.org? "top 160 site" is significant, but still minuscule compared to all internet traffic.
  - therealdrag0 5y ago
    I think I’m thinking of it reverse from you. A top 160 site could have more concurrent users than a small regional ISP.

pt_PT_guy 5y ago

and no one accountable :-)

lucb1e 5y ago
Yeah this is the part I don't get. Ok the source IP is spoofed, so you need to use another technique. The packets come in on some physical line, so you can ask the person on the other end which line they're getting it from and so forth. Relatively labor-intensive (I imagine a few hours' worth of time from each network hop), but you get there. Next, you find that it's coming from across some border, so you hand it off to the local authority. As I understand it, this is where people drop the ball. The local authorities can't be arsed, so the criminals there just keeps doing it. Why then not block that connection? If the provider in that country wants to be able to continue to operate their services, they'll need to fix this problem or customers around the world just won't be able to receive responses from their network. Same with the DNS resolvers: if someone has an open DNS resolver that is constantly being abused, as all of them are, just cut them off right? You don't need that on your IP ranges. Or blackhole that source IP on the edge of your network if you see abuse coming from that resolver. It's not rocket science to figure out which part of DNS traffic is abuse if you see a constant volume of responses to queries that the client in your network never actually asked for. Neither of these things seem to happen. Why can you be an ass on the internet and never be held accountable?
- TobTobXX 5y ago
  Possible for DoS attacks. Practically impossible for DDoS attacks.
  - lucb1e 5y ago
    Huh? I wrote this: > Same with the DNS resolvers: if someone has an open DNS resolver that is constantly being abused, as all of them are, just cut them off right? [etc.] That's what I'm not understanding why it is not being applied to easily solve those reflective DDoS attacks.

forty 5y ago

Why are the people doing those attacks doing them? Ransom? Or are they just doing that to prove and advertise/demo their capacity to harm?

schleck8 5y ago
someone here said the attacker asked for a ransom

JackMcMack 5y ago

I'm an Edpnet subscriber currently suffering the effects of the DDOS. It's annoying to say the least. Last year's DDOS attack was relatively simple, pointed at the DNS servers. Simply using other dns servers was good enough to get back online. Edpnet also joined NaWas [0] at that time, a non-profit for ISPs to be able to redirect all trafic through big pipes when needed to deal with large attacks. Because the current attacks are rapidly shifting targets, it's a game of cat and mouse to properly filter the ddos. In practice, this means that some sites such as google and youtube keep working, but other services might not be available. It is extremely annoying when all of a sudden AWS api calls time out, or a Teams or Slack call suddenly drops to very low bandwith, and then drops entirely. I've had to resort to my phone's hotspot multiple times in the last few days. Yes, I pay for SLA, but what's the point in that? I've got priority in case of a cable break, and failover to 4G connection, but that's no use if the upstream is congested. The sad part is that the attack works because it it a small isp, 45000 customers. [1] It is the main reason I'm a customer, they offer good service for great prices. Kudos for not paying the ransom. If the attacks continue for much longer, I will probably switch to the bigger, more expensive, less customer friendly isp. I'm happy to support a local company instead of a big multinational coorporation. But if my clients can't depend on me when working from home, I've got no choice but to pick the ISP with the bigger and more expensive pipes. [0] https://www.nbip.nl/en/nawas/ https://www.nbip.nl/en/nawas/ [1] https://datanews.knack.be/ict/nieuws/edpnet-al-dagen-getroffen-door-zware-ddos/article-news-1779585.html https://datanews.knack.be/ict/nieuws/edpnet-al-dagen-getroff...

lucb1e 5y ago
If you give them kudos for not paying the ransom, it's also worth considering not switching away from the small ISP you chose because they were doing things right. Otherwise they might as well have not done the morally right thing and given y'all better service. Though, of course, I do understand your conundrum. Perhaps there's some middle ground, where for anything that really needs the uptime you have a (mobile) fallback? Still annoying if there are outages on non-essential lines but perhaps better than forcing them into paying the criminals.
- JackMcMack 5y ago

blunte 5y ago

How often do we ever find the actual individual humans resonsible for these bad actions? I would imagine if the individuals were identified, some harmed parties would spend money eliminating them to dissuade others doing the same thing.

U8dcN7vx 5y ago
They are all too often in another country, so they can sometimes be determined but seldom accountable.

Havoc 5y ago

Unfortunately this works. Kills small ISPs on all fronts. Customer perception, technical and financial

hda2 5y ago

Spamhus bad ISPs. Give them low QoS until they fix their problem. If all major exchanges and service providers agree to punish IP blocks, ISPs will have no choice but to better police their networks.

ThinkBeat 5y ago

What is the motivation? Extortion? Retribution? Censorship? Marketing? QA? Are they asking for money stop? Are they being paid by some actor to do this because they wishes to protest / punish the company? Are they demonstrating what they can do, in order to drum up business? Are they testing what their system can accomplish? and iron out any bugs and learn how to best utilize it? So what mitigations they face and how to overcome them?

stardenburden 5y ago
The attackers are asking for ransom. So as usual, money.

riofoxx 5y ago

I can help you earn $ 6.0,0.0.0 with $ 5.0.0.0 in a w.e.e.k t.h.r.o.u.g.h M.R h.a r.l.a.n.d. w.h.o h.e.l.p.e.d m.e g.e.t the b.e.s.t in c.r.y.p.t.o t.r.a.d.i.n.g.. Contact information via WhatsApp +13052395906)

Aeolun 5y ago

It there no way to just completely block every IP sending you so much traffic? You might end up blocking a few million, but your network remains uncongested.

kazen44 5y ago
the only way to block this traffic effectively is by asking your upstream peers to blackhole the traffic. dropping this with a firewall or other device is nearly impossible because of the high load. remote triggered blackholing is done by BGP. which between peers only allows a /24 in terms of the smallest announcable prefix. blocking every /24 from which an IP in a ddos originates kills most if not all of your reachability.
- Aeolun 5y ago
  Thanks for that explanation. Wouldn’t some device still need to blackhole the stuff though? I’d still argue that blackholing /24 blocks and retaining some reachability is preferable to losing all of it. Now I’m kind of curious how fast you can make hardware drop stuff, but even inside my local PC I probably cannot reach 250Gbps of data flow (Hmm, 20 Gbps for DDR4, guess not).

edoceo 5y ago

Don't the peering agreements have to pay for bandwidth? Isn't there a bigger cost for those flooding this one? Could they charge-back and encourage peers to throttle? Can peers fix this with pricing model changes?

dgdfg 5y ago

https://www.kaowarsom.be/sites/default/files/webform/yr2020/Yogibo-presents-japan-pk3.pdf https://www.kaowarsom.be/sites/default/files/webform/yr2020/... https://www.kaowarsom.be/sites/default/files/webform/yr2020/Yogibo-presents-japan-pk2.pdf https://www.kaowarsom.be/sites/default/files/webform/yr2020/... https://www.kaowarsom.be/sites/default/files/webform/yr2020/Yogibo-presents-japan-pk1.pdf https://www.kaowarsom.be/sites/default/files/webform/yr2020/... https://www.kaowarsom.be/sites/default/files/webform/yr2020/Yogibo-presents-japan-pk0.pdf https://www.kaowarsom.be/sites/default/files/webform/yr2020/... https://www.kaowarsom.be/sites/default/files/webform/yr2020/Yogibo-presents-japan-pk.pdf https://www.kaowarsom.be/sites/default/files/webform/yr2020/... https://twitter.com/Ert2541/status/1439482365491834885 https://twitter.com/Ert2541/status/1439482365491834885 https://twitter.com/i/events/1439482429031403520 https://twitter.com/i/events/1439482429031403520 https://twitter.com/FHMJyv2yqViFD7n/status/1439483132550402048 https://twitter.com/FHMJyv2yqViFD7n/status/14394831325504020... https://www.vikkislade.org.uk/23588/_2021_rizin_30 https://www.vikkislade.org.uk/23588/_2021_rizin_30 https://www.vikkislade.org.uk/23588/_live_rizin_30_t3envrdltuwultnefczc9w https://www.vikkislade.org.uk/23588/_live_rizin_30_t3envrdlt... https://www.vikkislade.org.uk/23588/_30_rizin_30_live https://www.vikkislade.org.uk/23588/_30_rizin_30_live

r_singh 5y ago

I worked at an ISP AAA/Radius provider and since we started offering a cloud offering to our customers we were DDoSed twice. The source IPs were a Turkish university, I wrote to them a few times but they seemed clueless. Cloudflare's protection suite was beyond our budget and our engineers really started moving the system between 5 different IP blocks thanks to the help of our customers. We still don't know who's responsible, but we suspect it could be done by our competitor... (PS - We're based in India)

teitoklien 5y ago

https://gnunet.org/en/gns.html https://gnunet.org/en/gns.html

doublepg23 5y ago
I've tried getting into gnunet on and off for a couple years now (granted the last time was 2+ years now) and it was always inscrutable to me. Nowhere near as easy as Tor or even FreeNet (which is a dead Java app used only by the worst people as far as I could tell).

bullen 5y ago

Why not just block the offending IPs in iptables? Edit: as a Swede it's very funny that customer is called klant in dutch!

leoc 5y ago

Mods, can the URL please be corrected to https://issues.edpnet.be/?p=3507 https://issues.edpnet.be/?p=3507 ? The current link to https://issues.edpnet.be/ https://issues.edpnet.be/ is likely to rot badly over the years.