5 ms·

I wonder if there is some major propaganda push on HN to discourage actual software engineers from ever running their servers. Judging from the comments in this

by sph 2y ago

I wonder if there is some major propaganda push on HN to discourage actual software engineers from ever running their servers. Judging from the comments in this and similar posts, sounds like it is common belief that it's super expensive and a full time job to run bare metal services. AWS and GCP free credits have created a generation of developers that think running a webserver, a webapp and a database is rocket science, and when stuff breaks the whole company might go bankrupt because no one could ever save a corrupted database. Now a company with thousands of server should probably leverage big cloud offerings, but when we see a post that someone is able to save 10x by going bare metal, and they don't seem to be running a large operation, we should celebrate that, rather than parrot that it is a bad idea, oh god what have you done, no one will ever be able to run servers on their own, it's not worth it. For comparison, I manage half a dozen servers for clients and I literally do 1 hour of maintenance a year.

llm_trw 2y ago
I have a 10kW 4090 training cluster in my garage with water cooling. It paid itself back in 3 months if I'd used AWS services.
- alanfranz 2y ago
  Not something that requires HA, backups, scaling, auditing, logs, security, and such, though. Would you run your prod inference cluster the same way?
  - llm_trw 2y ago
    It's little brother is the prod inference cluster. It's had an uptime of 100% over the last two years. People seriously don't realize just how _big_ computers have gotten. The AWS mentality is still stuck in 2005 when a t2.micro was a decent computer.
    - Dunedan 2y ago
      > The AWS mentality is still stuck in 2005 when a t2.micro was a decent computer. The T2 EC2 instance family got announced in 2014: https://aws.amazon.com/blogs/startups/announcing-amazon-ec2-t2/ https://aws.amazon.com/blogs/startups/announcing-amazon-ec2-...
      - llm_trw 2y ago
        The first instance was the m1.small it's specs are marginally better than those of t2.micro, it's not been generally available since 2022 if memory serves. https://aws.amazon.com/blogs/aws/ec2-instance-history/ https://aws.amazon.com/blogs/aws/ec2-instance-history/ https://instances.vantage.sh/aws/ec2/t2.micro https://instances.vantage.sh/aws/ec2/t2.micro https://instances.vantage.sh/aws/ec2/m1.small https://instances.vantage.sh/aws/ec2/m1.small Please nitpick better. Nothing is more embarrassing then being pedantic and wrong.
      - rcxdude 2y ago
        And t2.micro was not a decent computer at that time.
  - apitman 2y ago
    You don't need most of that until you're making enough money to hire someone to worry about it.
- sulandor 2y ago
  is that enough to heat your house in the winter?
apitman 2y ago
It's weird how certain everyone is that it would take a ton of time to maintain. I mean, maybe. Depends on what you're running.
- ghomem 2y ago
  People want simple answers. "It depends" annoys everyone :-\
bryanrasmussen 2y ago
In my experience running AWS etc. is something that takes at least one dev who is a subject matter expert, so it doesn't seem like it would be at all problematic to have the same thing for bare metal - a dev who is the SME on the bare metal. Also if you have everything ready to push to a cloud provider in the case of the server going down, someone like Vercel or Netlify (easier to maintain than AWS in my experience) then I guess you are pretty setup.
Cpoll 2y ago
Disagreeing with your thesis isn't propaganda. If anything, you could argue the tweet is propaganda, because the labels for the AWS bar chart is missing. (Not that I think that's what's happening). Hetzner nodes aren't actually 10x cheaper than AWS instances, so there's stuff like bandwidth usage going on. My point being there's no silver bullet, sometimes cloud makes sense and sometimes it doesn't.
- earnesti 2y ago
  Yeah, hetzner nodes aren't 10x cheaper, because the products are not comparable. AWS/cloud services innovation is that their offering is incomparable and setup in a way which makes comparison very difficult. It is not only the hardware cost that saves you money, but also the fact that if you rent dedicated servers you tend to develop your infra in a different way which is ultimately way more cost effective. Dedicated servers definitely aren't a silver bullet, nobody is saying that. Managing your own servers requires work - how much, that is arguable and depends on many factors. To me the problem is more that AWS and cloud companies are selling a very expensive silver bullet, which really isn't even a silver bullet. In my experience most of the time it doesn't make sense to buy that.
  - Cpoll 2y ago
    I often find myself in a mid-size situation that's awkward. Hetzner doesn't provide SLAs, so if I have to then I need to run my clusters across multiple providers. If I need to support full DR in 8hrs I need to have something on another data center, I can't just run my AWS terraform in a different region. If my infra were 10x larger I'd go dedicated, but right now I'm happy enough to eat the AWS premium. > develop your infra in a different way Totally agree, but for many people that ship has sailed.
    - dijit 2y ago
      > Hetzner doesn't provide SLAs They do if you enter an agreement with them for professional services; I'm not certain but I think there's an SLA also on their "managed" servers: https://www.hetzner.com/managed-server/ https://www.hetzner.com/managed-server/ However, if you want a "no-human" SLA with bare-metal rented servers, I could recommend Gcore: https://gcore.com/hosting/dedicated https://gcore.com/hosting/dedicated Their managed kubernetes even supports bare-metal nodes, which is actually something I'm using.
cjfd 2y ago
I think so. People create and/or sell new technologies. Then they try to market them. For instance, by writing articles exaggerating the advantages and/or downplaying the disadvantages. Then this get picked up by other folks who want something new and interesting to write about. And suddenly there is a new and hot thing even if there are lots of cases where it does not help.
Dalewyn 2y ago
>sounds like it is common belief that it's super expensive and a full time job to run bare metal services. Remember the age old saying: Time is money. If you run your own servers, you are spending your own time and thus money to manage them. If you buy services from hosting ("cloud") providers, you spend money to have someone else manage them. The question then becomes: Do you value your time or your money more? And then you pick the one that spends the one you value less. Some people value their time, some people value their money. YMMV.
- sph 2y ago
  Your comment doesn't take into account that some things cost more than others, and might be not worth it their price. Running a server by yourself isn't more time consuming than having AWS do it for you. The issues that cause very long downtimes are flooding of the server room, subpar connectivity, low quality hardware, CPU is on fire. Just do not host your production services in your basement. Any VPS saves you from these, at 1/10th the AWS cost. This is obvious to anyone that has the basic skills to run a Linux host. The fact that it needs to be spelled out means that these days few have the skills and/or the others have been convinced that it is "a waste of time" and "makes only sense if your time is worthless" which is total FUD.
  - Dalewyn 2y ago
    I've realized that my time only becomes more valuable as I get older, to the point that if a problem can be solved by just throwing more money at it then that is a very compelling solution. Sure, buying a VPS or whatever from a hosting provider might be cheaper than buying a full blown cloud solution, but at what personal time cost? Time is a finite resource, much more than most of us ever realize. Worse, time is a resource that cannot be replenished. Money, on the other hand, is a replenishable and theoretically infinite resource. Sometimes, paying big bucks for premium services is the cheaper solution to moving on in life.
    - jgb1984 2y ago
      It takes me more time to find what I need in the infernal hell hole of an AWS web interface, try to understand their hopelessly complex jungle of lingo and abstraction layers. Hell, I'll deploy a fresh Hetzner bare metal server with Debian in a matter of 3-4 minutes and roll with it. 10 times more bang for your buck, no vendor lock in, and guess what: I am in control. I know what runs on my server. I understand the platform inside out. I don't trust developers who are afraid of setting up their own Linux server.
fileeditview 2y ago
> I wonder if there is some major propaganda push on HN to discourage actual software engineers from ever running their servers. Judging from the comments in this and similar posts, sounds like it is common belief that it's super expensive and a full time job to run bare metal services. AWS and GCP free credits have created a generation of developers that think running a webserver, a webapp and a database is rocket science, and when stuff breaks the whole company might go bankrupt because no one could ever save a corrupted database. Related to this observation you really should watch DHHs keynote talk at Rails World 2024 from about minute 20 to 30: https://youtu.be/-cEn_83zRFw?t=1296 https://youtu.be/-cEn_83zRFw?t=1296 He addresses exactly this phenomenon. EDIT: changed URL to include timestamp to the start of the relevant content.
ghomem 2y ago
>I wonder if there is some major propaganda push on HN to discourage actual >software engineers from ever running their servers How to scare a computer engineer in 2024? Invite him to configure a server.
deleted 2y ago
[deleted]
hagbard_c 2y ago
I thing this is partly the result of people simply parroting shallow statements concerning 'backup, availability, disaster recovery,...' they've read or heard elsewhere, partly the result of a general push towards a service economy where it is considered smart to outsource to 'the experts'. While this may make sense in some cases - e.g. those where the task at hand is not recurring and requires knowledge or equipment which is not at hand and hard/expensive to come by - it is often more expensive and less flexible than just doing the thing yourself. Of course HN is a site frequented by people who are wont to start those same service companies so it is not surprising for there to be a tendency to play up the 'risks' of doing things yourself instead of 'leaving it to the experts'.

by sph 2y ago

llm_trw 2y ago

I have a 10kW 4090 training cluster in my garage with water cooling. It paid itself back in 3 months if I'd used AWS services.

alanfranz 2y ago
Not something that requires HA, backups, scaling, auditing, logs, security, and such, though. Would you run your prod inference cluster the same way?
- llm_trw 2y ago
  It's little brother is the prod inference cluster. It's had an uptime of 100% over the last two years. People seriously don't realize just how _big_ computers have gotten. The AWS mentality is still stuck in 2005 when a t2.micro was a decent computer.
  - Dunedan 2y ago
    > The AWS mentality is still stuck in 2005 when a t2.micro was a decent computer. The T2 EC2 instance family got announced in 2014: https://aws.amazon.com/blogs/startups/announcing-amazon-ec2-t2/ https://aws.amazon.com/blogs/startups/announcing-amazon-ec2-...
    - llm_trw 2y ago
      The first instance was the m1.small it's specs are marginally better than those of t2.micro, it's not been generally available since 2022 if memory serves. https://aws.amazon.com/blogs/aws/ec2-instance-history/ https://aws.amazon.com/blogs/aws/ec2-instance-history/ https://instances.vantage.sh/aws/ec2/t2.micro https://instances.vantage.sh/aws/ec2/t2.micro https://instances.vantage.sh/aws/ec2/m1.small https://instances.vantage.sh/aws/ec2/m1.small Please nitpick better. Nothing is more embarrassing then being pedantic and wrong.
    - rcxdude 2y ago
      And t2.micro was not a decent computer at that time.
- apitman 2y ago
  You don't need most of that until you're making enough money to hire someone to worry about it.
sulandor 2y ago
is that enough to heat your house in the winter?

apitman 2y ago

It's weird how certain everyone is that it would take a ton of time to maintain. I mean, maybe. Depends on what you're running.

ghomem 2y ago
People want simple answers. "It depends" annoys everyone :-\

bryanrasmussen 2y ago

In my experience running AWS etc. is something that takes at least one dev who is a subject matter expert, so it doesn't seem like it would be at all problematic to have the same thing for bare metal - a dev who is the SME on the bare metal. Also if you have everything ready to push to a cloud provider in the case of the server going down, someone like Vercel or Netlify (easier to maintain than AWS in my experience) then I guess you are pretty setup.

Cpoll 2y ago

Disagreeing with your thesis isn't propaganda. If anything, you could argue the tweet is propaganda, because the labels for the AWS bar chart is missing. (Not that I think that's what's happening). Hetzner nodes aren't actually 10x cheaper than AWS instances, so there's stuff like bandwidth usage going on. My point being there's no silver bullet, sometimes cloud makes sense and sometimes it doesn't.

earnesti 2y ago
Yeah, hetzner nodes aren't 10x cheaper, because the products are not comparable. AWS/cloud services innovation is that their offering is incomparable and setup in a way which makes comparison very difficult. It is not only the hardware cost that saves you money, but also the fact that if you rent dedicated servers you tend to develop your infra in a different way which is ultimately way more cost effective. Dedicated servers definitely aren't a silver bullet, nobody is saying that. Managing your own servers requires work - how much, that is arguable and depends on many factors. To me the problem is more that AWS and cloud companies are selling a very expensive silver bullet, which really isn't even a silver bullet. In my experience most of the time it doesn't make sense to buy that.
- Cpoll 2y ago
  I often find myself in a mid-size situation that's awkward. Hetzner doesn't provide SLAs, so if I have to then I need to run my clusters across multiple providers. If I need to support full DR in 8hrs I need to have something on another data center, I can't just run my AWS terraform in a different region. If my infra were 10x larger I'd go dedicated, but right now I'm happy enough to eat the AWS premium. > develop your infra in a different way Totally agree, but for many people that ship has sailed.
  - dijit 2y ago
    > Hetzner doesn't provide SLAs They do if you enter an agreement with them for professional services; I'm not certain but I think there's an SLA also on their "managed" servers: https://www.hetzner.com/managed-server/ https://www.hetzner.com/managed-server/ However, if you want a "no-human" SLA with bare-metal rented servers, I could recommend Gcore: https://gcore.com/hosting/dedicated https://gcore.com/hosting/dedicated Their managed kubernetes even supports bare-metal nodes, which is actually something I'm using.

cjfd 2y ago

I think so. People create and/or sell new technologies. Then they try to market them. For instance, by writing articles exaggerating the advantages and/or downplaying the disadvantages. Then this get picked up by other folks who want something new and interesting to write about. And suddenly there is a new and hot thing even if there are lots of cases where it does not help.

Dalewyn 2y ago

>sounds like it is common belief that it's super expensive and a full time job to run bare metal services. Remember the age old saying: Time is money. If you run your own servers, you are spending your own time and thus money to manage them. If you buy services from hosting ("cloud") providers, you spend money to have someone else manage them. The question then becomes: Do you value your time or your money more? And then you pick the one that spends the one you value less. Some people value their time, some people value their money. YMMV.

sph 2y ago
Your comment doesn't take into account that some things cost more than others, and might be not worth it their price. Running a server by yourself isn't more time consuming than having AWS do it for you. The issues that cause very long downtimes are flooding of the server room, subpar connectivity, low quality hardware, CPU is on fire. Just do not host your production services in your basement. Any VPS saves you from these, at 1/10th the AWS cost. This is obvious to anyone that has the basic skills to run a Linux host. The fact that it needs to be spelled out means that these days few have the skills and/or the others have been convinced that it is "a waste of time" and "makes only sense if your time is worthless" which is total FUD.
- Dalewyn 2y ago
  I've realized that my time only becomes more valuable as I get older, to the point that if a problem can be solved by just throwing more money at it then that is a very compelling solution. Sure, buying a VPS or whatever from a hosting provider might be cheaper than buying a full blown cloud solution, but at what personal time cost? Time is a finite resource, much more than most of us ever realize. Worse, time is a resource that cannot be replenished. Money, on the other hand, is a replenishable and theoretically infinite resource. Sometimes, paying big bucks for premium services is the cheaper solution to moving on in life.
  - jgb1984 2y ago
    It takes me more time to find what I need in the infernal hell hole of an AWS web interface, try to understand their hopelessly complex jungle of lingo and abstraction layers. Hell, I'll deploy a fresh Hetzner bare metal server with Debian in a matter of 3-4 minutes and roll with it. 10 times more bang for your buck, no vendor lock in, and guess what: I am in control. I know what runs on my server. I understand the platform inside out. I don't trust developers who are afraid of setting up their own Linux server.

fileeditview 2y ago

> I wonder if there is some major propaganda push on HN to discourage actual software engineers from ever running their servers. Judging from the comments in this and similar posts, sounds like it is common belief that it's super expensive and a full time job to run bare metal services. AWS and GCP free credits have created a generation of developers that think running a webserver, a webapp and a database is rocket science, and when stuff breaks the whole company might go bankrupt because no one could ever save a corrupted database. Related to this observation you really should watch DHHs keynote talk at Rails World 2024 from about minute 20 to 30: https://youtu.be/-cEn_83zRFw?t=1296 https://youtu.be/-cEn_83zRFw?t=1296 He addresses exactly this phenomenon. EDIT: changed URL to include timestamp to the start of the relevant content.

ghomem 2y ago

>I wonder if there is some major propaganda push on HN to discourage actual >software engineers from ever running their servers How to scare a computer engineer in 2024? Invite him to configure a server.

deleted 2y ago

[deleted]

hagbard_c 2y ago

I thing this is partly the result of people simply parroting shallow statements concerning 'backup, availability, disaster recovery,...' they've read or heard elsewhere, partly the result of a general push towards a service economy where it is considered smart to outsource to 'the experts'. While this may make sense in some cases - e.g. those where the task at hand is not recurring and requires knowledge or equipment which is not at hand and hard/expensive to come by - it is often more expensive and less flexible than just doing the thing yourself. Of course HN is a site frequented by people who are wont to start those same service companies so it is not surprising for there to be a tendency to play up the 'risks' of doing things yourself instead of 'leaving it to the experts'.