5 ms·

How many of you all are running bare metal hooked right up to the internet? Is DDoS or any of that actually a super common problem? I know it happens, but also

How many of you all are running bare metal hooked right up to the internet? Is DDoS or any of that actually a super common problem? I know it happens, but also I've run plenty of servers hooked directly to the internet (with standard *nix security precautions and hosting provider DDoS protection) and haven't had it actually be an issue. So why run absolutely everything through Cloudflare?

uniformlyrandom 2y ago
Most exploits target the software, not the hardware. CF is a good reverse proxy.
raffraffraff 2y ago
They make it easy to delegate a DNS zone to them and use their API to create records (eg: install external-dns on kubernetes and key it create records automatically for ingresses)
dosdosdosdos 2y ago
[flagged]
codexon 2y ago
It is common once your website hits a certain threshold in popularity. If you are just a small startup or a blog, you'll probably never see an attack. Even if you don't host anything offensive you can be targeted by competitors, blackmailed for money, or just randomly selected by a hacker to test the power of their botnet.
progmetaldev 2y ago
Web scraping without any kind of sleeping in between requests (usually firing many threads at once), as well as heavy exploit scanning is a near constant for most websites. With AI technology, it's only getting worse, as vendors attempt to bring in content from all over the web without regard for resource usage. Depending on the industry, DDoS can be very common from competitors that aren't afraid to rent out botnets to boost their business and tear down those they compete against.
rpgwaiter 2y ago
It’s free unless you’re rolling in traffic, it’s extremely easy to setup, and CF can handle pretty much all of your infra with tools way better than AWS. Also you can buy a cheaper ipv6 only VPS and run it thru free CF proxy to allow ipv4 traffic to your site
- zelphirkalt 2y ago
  Easy to set up, easy to screw up user experience. Easy-peasy.
grishka 2y ago
> How many of you all are running bare metal hooked right up to the internet? I do. Many people I know do. In my risk model, DDoS is something purely theoretical. Yes it can happen, but you have to seriously upset someone for it to maybe happen.
- maples37 2y ago
  From my experience, if you tick off the wrong person, the threshold for them starting a DDoS is surprisingly low. A while ago, my company was hiring and conducting interviews, and after one candidate was rejected, one of our sites got hit by a DDoS. I wasn't in the room when people were dealing with it, but in the post-incident review, they said "we're 99% sure we know exactly who this came from".
  - Loughla 2y ago
    What the hell is wrong with people? Honestly the lack of substantive human interaction in a lot of folks' lives, except via the Internet, is a real problem. Take that story for instance. Here's how that goes in the physical world, just to show how unbelievably ridiculous it is. So you didn't get the job? What's your next step? I'll stop by their office and keep people from entering the front doors by running around in front of them. That'll show those bastards.
    - deleted 2y ago
      [deleted]
nijave 2y ago
Small/medium SaaS. Had ~8 hours of 100k reqs/sec last year when we usually see 100-150 reqs/sec. Moved everything behind a Cloudflare Enterprise setup and ditched AWS Client Access VPN (OpenVPN) for Cloudflare WARP I've only been here 1.5 years but sounds like we usually see 1 decent sized DDoS a year plus a handful of other "DoS" usually AI crawler extensions or 3rd parties calling too aggressively There are some extensions/products that create a "personal AI knowledge base" and they'll use the customers login credentials and scrape every link once an hour. Some links are really really resource intensive data or report requests that are very rare in real usage
- gamegod 2y ago
  Did you put rate limiting rules on your webserver? Why was that not enough to mitigate the DDoS?
  - danielheath 2y ago
    Not the same poster, but the first "D" in "DDoS" is why rate-limiting doesn't work - attackers these days usually have a _huge_ (tens of thousands) pool of residential ip4 addresses to work with.
    - chillfox 2y ago
      They were talking about logged in accounts, so you would group by accounts for the rate limiting and not by ip addresses.
      - nijave 2y ago
        They were unauthenticated requests making GETs to the login page
    - rixed 2y ago
      Is ten of thousands a big number again?
matt_heimer 2y ago
Yes, [D]DoS is a problem. Its not uncommon for a single person with residential fiber to have more bandwidth than your small site hosted on a 1u box or VPS. Either your bandwidth is rate limited and they can denial of service your site or your bandwidth is greater but they can still cause you to go over your allocation and cause massive charges. In the past you could ban IPs but that's not very useful anymore. The distributed attacks tend to be AI companies that assume every site has infinite bandwidth and their crawlers tend to run out of different regions. Even if you aren't dealing with attacks or outages, Cloudflare's caching features can save you a ton of money. If you haven't used Cloudflare, most sites only need their free tier offering. It's hard to say no to a free service that provides feature you need. Source: I went over a decade hosting a site without a CDN before it became too difficult to deal with. Basically I spent 3 days straight banning ips at the hosting company level, tuning various rate limiting web server modules and even scaling the hardware to double the capacity. None of it could keep the site online 100% of the time. Within 30 mins of trying Cloudflare it was working perfectly.
- radlad 2y ago
  > It's hard to say no to a free service that provides feature you need. Very true! Though you still see people who are surprised to learn that CF DDOS protection acts as a MITM proxy and can read your traffic plaintext. This is of course by design, to inspect the traffic. But admittedly, CF is not very clear about this in the Admin Panel or docs. Places one might expect to learn this, but won't: - https://developers.cloudflare.com/dns/manage-dns-records/reference/proxied-dns-records/ https://developers.cloudflare.com/dns/manage-dns-records/ref... - https://developers.cloudflare.com/fundamentals/concepts/how-cloudflare-works/ https://developers.cloudflare.com/fundamentals/concepts/how-... - https://imgur.com/a/zGegZ00 https://imgur.com/a/zGegZ00
  - sophacles 2y ago
    How would you do DDoS protection without having something in path?
    - 2y ago
motiejus 2y ago
I've been running jakstys.lt (and subdomains like git.jakstys.lt) from my closet, a simple residential connection with a small monthly price for a static IP. The only time I had a problem was when gitea started caching git bundles of my Linux kernel mirror, which bots kept downloading (things like a full targz of every commit since 2005). Server promptly went out of disk space. I fixed gitea settings to not cache those. That was it. Not ever ddos. Or I (and uptimerobot) did not notice it. :)
Puts 2y ago
Most (D)DOS attacks are just either UDP floods or SYN floods that iptables will handle without any problem. Sometimes what people think are DDOS is just their application DDOSing themself because they are doing recursive calls to some back-end micro-service. If it was actually a traffic based DDOS someone still needs to pay for that bandwidth which would be too expansive for most companies anyway - even if it kept your site running. But you can sell a lot of services to incompetent people.
- hombre_fatal 2y ago
  You need an answer to someone buying $10 of booter time and sending a volumetric attack your way. If any of the traffic is even reaching your server, you've already lost, so iptables isn't going to help you because your link is saturated. Cloudflare offers protection for free.
- sophacles 2y ago
  What's the iptables invocation that will let my 10Gbps connection drop a a 100Gbps syn flood while also serving good traffic?
  - truetraveller 2y ago
    xdp
  - Puts 2y ago
    The point with a syn flood is to try to saturate the OS limit for open sockets. From an attackers perspective the whole point of a syn flood is to do a DOS without needing much bandwidth. My experience form 15 years working in the hosting industry is that volumetric attacks are extremely rare but customers that turn to Cloudflare as a solution are more often than not DDOS-ing them self because of bad configured systems, but their junior developers lack any networking troubleshooting skills.
buyucu 2y ago
DDoS is a problem, but for most ordinary problems it's not as bad as people make it out to be. Even something very simple like fail2ban will go a long way.
professorsnep 2y ago
I run a Mediawiki instance for an online community on a fairly cheap box (not a ton of traffic) but had a few instances of AI bots like Amazon's crawling a lot of expensive API pages thousands of times an hour (despite robots.txt preventing those). Turned on Cloudflare's bot blocking and 50% of total traffic instantly went away. Even now, blocked bot requests make up 25% of total requests to the site. Without blocking I would have needed to upgrade quite a bit or play a tiring game of whack a mole blocking any new IP ranges for the dozens of bots.
- CGamesPlay 2y ago
  How do you feel, knowing that some portion of the 25% “detected bot traffic” are actually people in this comment thread?
- mrweasel 2y ago
  AI bots are a huge issue for a lot of sites. Just putting intentional DDoS attacks aside, AI scrapers can frequently tip over a site because many of them don't know how to back off. Google is an exception really, their experience with creating GoogleBot as ensured that they are never a problem. Many of the AI scrapers don't identify themselves, they live on AWS, Azure, Alibaba Cloud, and Tencent Cloud, so you can't really block them and rate limiting also have limited effect as they just jump to new IPs. As a site owner, you can't really contact AWS and ask them to terminate their customers service in order for you to recover.
- account42 2y ago
  You don't need buttflare's mistery juice to rate-limit or block bad users.
blablabla123 2y ago
The biggest problems I see with DDoS is metered traffic and availability. The largest Cloud providers all meter their traffic. The availability part on the other hand is maybe something that's not so business critical for many but for targeted long-term attacks it probably is. So I think for some websites, especially smaller ones it's totally feasible to not use Cloudflare but involves planning the hosting really carefully.
megous 2y ago
I also rely on hosting provider DDoS protection and don't use very intrusive protection like Cloudflare. Only issues I had to deal with are when someone finds some slow endpoint, and manages to overload the server with it, and my go to approach is to optimize it to max <10-20ms response time, while blocking the source of traffic if it keeps being too annoying after optimization. And this happened like 2-3 times over 20 years of hosting the eshop. Much better than exposing users to CF or likes of it.
johnklos 2y ago
I've been hosting web sites on my own bare metal in colo for more than 25 years. In all that time I've dealt with one DDoS that was big enough to bring everything down, and that was because of a specific person being pissed at another specific person. The attacker did jail time for DDoS activities. Every other attempt at DDoS has been ineffective, has been form abuse and credential stuffing, has been generally amateurish enough to not take anything down. I host (web, email, shells) lots of people including kids (young adults) who're learning about the Internet, about security, et cetera, who do dumb things like talk shit on irc. You'd think I'd've had more DDoS attacks than that rather famous one. So when people assert with confidence that the Internet would fall over if companies like Cloudflare weren't there to "protect" them, I have to wonder how Cloudflare marketed so well that these people believe this BS with no experience. Sure, it could be something else, like someone running Wordpress with a default admin URL left open who makes a huge deal about how they're getting "hacked", but that wouldn't explain all the Cloudflare apologists. Cloudflare wants to be a monopoly. They've shown they have no care in the world for marginalized people, whether they're people who don't live in a western country or people who simply prefer to not run mainstream OSes and browsers. They protect scammers because they make money from scammers. So why would people want to use them? That's a very good question.
- mvdtnz 2y ago
  I'm sorry but lumping in people who prefer to use a weird browser with "marginalised people" does not help your credibility.
  - Aachen 2y ago
    What bit do you mean specifically? As a fellow web hoster, who also hosted kids before (from a game making forum), I can fully corroborate what they're saying
    - mvdtnz 2y ago
      Clearly you didn't even read his post (or mine) if you're asking. I'm obviously referring to > Cloudflare wants to be a monopoly. They've shown they have no care in the world for marginalized people, whether they're people who don't live in a western country or people who simply prefer to not run mainstream OSes and browsers.
porty 2y ago
I would feel pretty safe running my own hand-written services against the raw Internet, but if I was to host Wordpress or other large/complicated/legacy codebases I'd start to get worried. Also the CDN aspect is useful - having lived in Australia you like connections that don't have to traverse continents for every request.
itomato 2y ago
Check your logs, you might be surprised.
betaby 2y ago
Other comments say that DDoS are common, not my experience though. I run a couple of API/SAAS sites and DDoSes are rare. Sites are in Canada and Brazil if that matters, although I won't disclose what data-centers. Most strange thing is that no one demanded any ransom during those DDoS attacks ever. Just some flooding for 1-2 days. Most of the times I did't even care - servers are on 10G ports and I pay 95% percentile for the traffic with a cap on final bill. Sites are geo-fenced by nftables rules, only countries of interest are allowed.
tombert 2y ago
I run my "server" [1] straight to my home internet, and maybe I should count my blessings but I haven't had any issues with DDoS in the years I've done this. I have relatively fast internet, so maybe it's fast enough to absorb a lot of the problems, but I've had good enough luck with some basic Nginx settings and fail2ban. [1] a small little mini gaming PC running NixOS.

uniformlyrandom 2y ago

Most exploits target the software, not the hardware. CF is a good reverse proxy.

raffraffraff 2y ago

They make it easy to delegate a DNS zone to them and use their API to create records (eg: install external-dns on kubernetes and key it create records automatically for ingresses)

dosdosdosdos 2y ago

[flagged]

codexon 2y ago

It is common once your website hits a certain threshold in popularity. If you are just a small startup or a blog, you'll probably never see an attack. Even if you don't host anything offensive you can be targeted by competitors, blackmailed for money, or just randomly selected by a hacker to test the power of their botnet.

progmetaldev 2y ago

Web scraping without any kind of sleeping in between requests (usually firing many threads at once), as well as heavy exploit scanning is a near constant for most websites. With AI technology, it's only getting worse, as vendors attempt to bring in content from all over the web without regard for resource usage. Depending on the industry, DDoS can be very common from competitors that aren't afraid to rent out botnets to boost their business and tear down those they compete against.

rpgwaiter 2y ago

It’s free unless you’re rolling in traffic, it’s extremely easy to setup, and CF can handle pretty much all of your infra with tools way better than AWS. Also you can buy a cheaper ipv6 only VPS and run it thru free CF proxy to allow ipv4 traffic to your site

zelphirkalt 2y ago
Easy to set up, easy to screw up user experience. Easy-peasy.

grishka 2y ago

> How many of you all are running bare metal hooked right up to the internet? I do. Many people I know do. In my risk model, DDoS is something purely theoretical. Yes it can happen, but you have to seriously upset someone for it to maybe happen.

maples37 2y ago
From my experience, if you tick off the wrong person, the threshold for them starting a DDoS is surprisingly low. A while ago, my company was hiring and conducting interviews, and after one candidate was rejected, one of our sites got hit by a DDoS. I wasn't in the room when people were dealing with it, but in the post-incident review, they said "we're 99% sure we know exactly who this came from".
- Loughla 2y ago
  What the hell is wrong with people? Honestly the lack of substantive human interaction in a lot of folks' lives, except via the Internet, is a real problem. Take that story for instance. Here's how that goes in the physical world, just to show how unbelievably ridiculous it is. So you didn't get the job? What's your next step? I'll stop by their office and keep people from entering the front doors by running around in front of them. That'll show those bastards.
  - deleted 2y ago
    [deleted]

nijave 2y ago

Small/medium SaaS. Had ~8 hours of 100k reqs/sec last year when we usually see 100-150 reqs/sec. Moved everything behind a Cloudflare Enterprise setup and ditched AWS Client Access VPN (OpenVPN) for Cloudflare WARP I've only been here 1.5 years but sounds like we usually see 1 decent sized DDoS a year plus a handful of other "DoS" usually AI crawler extensions or 3rd parties calling too aggressively There are some extensions/products that create a "personal AI knowledge base" and they'll use the customers login credentials and scrape every link once an hour. Some links are really really resource intensive data or report requests that are very rare in real usage

gamegod 2y ago
Did you put rate limiting rules on your webserver? Why was that not enough to mitigate the DDoS?
- danielheath 2y ago
  Not the same poster, but the first "D" in "DDoS" is why rate-limiting doesn't work - attackers these days usually have a _huge_ (tens of thousands) pool of residential ip4 addresses to work with.
  - chillfox 2y ago
    They were talking about logged in accounts, so you would group by accounts for the rate limiting and not by ip addresses.
    - nijave 2y ago
      They were unauthenticated requests making GETs to the login page
  - rixed 2y ago
    Is ten of thousands a big number again?

matt_heimer 2y ago

Yes, [D]DoS is a problem. Its not uncommon for a single person with residential fiber to have more bandwidth than your small site hosted on a 1u box or VPS. Either your bandwidth is rate limited and they can denial of service your site or your bandwidth is greater but they can still cause you to go over your allocation and cause massive charges. In the past you could ban IPs but that's not very useful anymore. The distributed attacks tend to be AI companies that assume every site has infinite bandwidth and their crawlers tend to run out of different regions. Even if you aren't dealing with attacks or outages, Cloudflare's caching features can save you a ton of money. If you haven't used Cloudflare, most sites only need their free tier offering. It's hard to say no to a free service that provides feature you need. Source: I went over a decade hosting a site without a CDN before it became too difficult to deal with. Basically I spent 3 days straight banning ips at the hosting company level, tuning various rate limiting web server modules and even scaling the hardware to double the capacity. None of it could keep the site online 100% of the time. Within 30 mins of trying Cloudflare it was working perfectly.

radlad 2y ago
> It's hard to say no to a free service that provides feature you need. Very true! Though you still see people who are surprised to learn that CF DDOS protection acts as a MITM proxy and can read your traffic plaintext. This is of course by design, to inspect the traffic. But admittedly, CF is not very clear about this in the Admin Panel or docs. Places one might expect to learn this, but won't: - https://developers.cloudflare.com/dns/manage-dns-records/reference/proxied-dns-records/ https://developers.cloudflare.com/dns/manage-dns-records/ref... - https://developers.cloudflare.com/fundamentals/concepts/how-cloudflare-works/ https://developers.cloudflare.com/fundamentals/concepts/how-... - https://imgur.com/a/zGegZ00 https://imgur.com/a/zGegZ00
- sophacles 2y ago
  How would you do DDoS protection without having something in path?
  - 2y ago

motiejus 2y ago

I've been running jakstys.lt (and subdomains like git.jakstys.lt) from my closet, a simple residential connection with a small monthly price for a static IP. The only time I had a problem was when gitea started caching git bundles of my Linux kernel mirror, which bots kept downloading (things like a full targz of every commit since 2005). Server promptly went out of disk space. I fixed gitea settings to not cache those. That was it. Not ever ddos. Or I (and uptimerobot) did not notice it. :)

Puts 2y ago

Most (D)DOS attacks are just either UDP floods or SYN floods that iptables will handle without any problem. Sometimes what people think are DDOS is just their application DDOSing themself because they are doing recursive calls to some back-end micro-service. If it was actually a traffic based DDOS someone still needs to pay for that bandwidth which would be too expansive for most companies anyway - even if it kept your site running. But you can sell a lot of services to incompetent people.

hombre_fatal 2y ago
You need an answer to someone buying $10 of booter time and sending a volumetric attack your way. If any of the traffic is even reaching your server, you've already lost, so iptables isn't going to help you because your link is saturated. Cloudflare offers protection for free.
sophacles 2y ago
What's the iptables invocation that will let my 10Gbps connection drop a a 100Gbps syn flood while also serving good traffic?
- truetraveller 2y ago
  xdp
- Puts 2y ago
  The point with a syn flood is to try to saturate the OS limit for open sockets. From an attackers perspective the whole point of a syn flood is to do a DOS without needing much bandwidth. My experience form 15 years working in the hosting industry is that volumetric attacks are extremely rare but customers that turn to Cloudflare as a solution are more often than not DDOS-ing them self because of bad configured systems, but their junior developers lack any networking troubleshooting skills.

buyucu 2y ago

DDoS is a problem, but for most ordinary problems it's not as bad as people make it out to be. Even something very simple like fail2ban will go a long way.

professorsnep 2y ago

I run a Mediawiki instance for an online community on a fairly cheap box (not a ton of traffic) but had a few instances of AI bots like Amazon's crawling a lot of expensive API pages thousands of times an hour (despite robots.txt preventing those). Turned on Cloudflare's bot blocking and 50% of total traffic instantly went away. Even now, blocked bot requests make up 25% of total requests to the site. Without blocking I would have needed to upgrade quite a bit or play a tiring game of whack a mole blocking any new IP ranges for the dozens of bots.

CGamesPlay 2y ago
How do you feel, knowing that some portion of the 25% “detected bot traffic” are actually people in this comment thread?
mrweasel 2y ago
AI bots are a huge issue for a lot of sites. Just putting intentional DDoS attacks aside, AI scrapers can frequently tip over a site because many of them don't know how to back off. Google is an exception really, their experience with creating GoogleBot as ensured that they are never a problem. Many of the AI scrapers don't identify themselves, they live on AWS, Azure, Alibaba Cloud, and Tencent Cloud, so you can't really block them and rate limiting also have limited effect as they just jump to new IPs. As a site owner, you can't really contact AWS and ask them to terminate their customers service in order for you to recover.
account42 2y ago
You don't need buttflare's mistery juice to rate-limit or block bad users.

blablabla123 2y ago

The biggest problems I see with DDoS is metered traffic and availability. The largest Cloud providers all meter their traffic. The availability part on the other hand is maybe something that's not so business critical for many but for targeted long-term attacks it probably is. So I think for some websites, especially smaller ones it's totally feasible to not use Cloudflare but involves planning the hosting really carefully.

megous 2y ago

I also rely on hosting provider DDoS protection and don't use very intrusive protection like Cloudflare. Only issues I had to deal with are when someone finds some slow endpoint, and manages to overload the server with it, and my go to approach is to optimize it to max <10-20ms response time, while blocking the source of traffic if it keeps being too annoying after optimization. And this happened like 2-3 times over 20 years of hosting the eshop. Much better than exposing users to CF or likes of it.

johnklos 2y ago

I've been hosting web sites on my own bare metal in colo for more than 25 years. In all that time I've dealt with one DDoS that was big enough to bring everything down, and that was because of a specific person being pissed at another specific person. The attacker did jail time for DDoS activities. Every other attempt at DDoS has been ineffective, has been form abuse and credential stuffing, has been generally amateurish enough to not take anything down. I host (web, email, shells) lots of people including kids (young adults) who're learning about the Internet, about security, et cetera, who do dumb things like talk shit on irc. You'd think I'd've had more DDoS attacks than that rather famous one. So when people assert with confidence that the Internet would fall over if companies like Cloudflare weren't there to "protect" them, I have to wonder how Cloudflare marketed so well that these people believe this BS with no experience. Sure, it could be something else, like someone running Wordpress with a default admin URL left open who makes a huge deal about how they're getting "hacked", but that wouldn't explain all the Cloudflare apologists. Cloudflare wants to be a monopoly. They've shown they have no care in the world for marginalized people, whether they're people who don't live in a western country or people who simply prefer to not run mainstream OSes and browsers. They protect scammers because they make money from scammers. So why would people want to use them? That's a very good question.

mvdtnz 2y ago
I'm sorry but lumping in people who prefer to use a weird browser with "marginalised people" does not help your credibility.
- Aachen 2y ago
  What bit do you mean specifically? As a fellow web hoster, who also hosted kids before (from a game making forum), I can fully corroborate what they're saying
  - mvdtnz 2y ago
    Clearly you didn't even read his post (or mine) if you're asking. I'm obviously referring to > Cloudflare wants to be a monopoly. They've shown they have no care in the world for marginalized people, whether they're people who don't live in a western country or people who simply prefer to not run mainstream OSes and browsers.

porty 2y ago

I would feel pretty safe running my own hand-written services against the raw Internet, but if I was to host Wordpress or other large/complicated/legacy codebases I'd start to get worried. Also the CDN aspect is useful - having lived in Australia you like connections that don't have to traverse continents for every request.

itomato 2y ago

Check your logs, you might be surprised.

betaby 2y ago

Other comments say that DDoS are common, not my experience though. I run a couple of API/SAAS sites and DDoSes are rare. Sites are in Canada and Brazil if that matters, although I won't disclose what data-centers. Most strange thing is that no one demanded any ransom during those DDoS attacks ever. Just some flooding for 1-2 days. Most of the times I did't even care - servers are on 10G ports and I pay 95% percentile for the traffic with a cap on final bill. Sites are geo-fenced by nftables rules, only countries of interest are allowed.

tombert 2y ago

I run my "server" [1] straight to my home internet, and maybe I should count my blessings but I haven't had any issues with DDoS in the years I've done this. I have relatively fast internet, so maybe it's fast enough to absorb a lot of the problems, but I've had good enough luck with some basic Nginx settings and fail2ban. [1] a small little mini gaming PC running NixOS.