6 ms·

I bet that compared to an equivalent load hosted on AWS, that lovely box pays for itself in full every month, if not every week...

by sho 3y ago

deleted 3y ago
[deleted]
d_silin 3y ago
Absolutely. Never understood why people are so infatuated with "cloud" options. Yes, it is convinient, but you are absolutely paying at least order of magnitude more for the same amount of compute/storage.
- civilitty 3y ago
  It's gotten absolutely bonkers with all the demand for GPUs and the cloud companies are just raking it in hand over fist. On AWS reserved instances with 8x A100s or H100s (if you can even get them) cost more per year than the total upfront retail price of the equivalent pods from Lambda Labs. The on-demand price is even more absurd.
- qwertox 3y ago
  If you don't need to host something big, where you can get away with one good VPS, the cloud has the benefit of offering cheap bandwidth. Once you have the bandwidth at your location and you don't need to be present in multiple locations, it's cheaper to self-host. Next step would be colocation, but for a start, using cloud offerings is a cheap way to be a part of the internet.
  - nacs 3y ago
    Cloud bandwidth is definitely not cheap. If anything, it's where they rip you off the most. You can get "baremetal"/dedicated servers from places like Hetzner and OVH that give unmetered gigabit connections for like $50.
    - nmjohn 3y ago
      You can not get actual unmetered 1gb/s for anywhere close to that. If you start pushing anywhere close to that much bandwidth, you will be throttled / have your account closed. For example, Hetzner caps your bandwidth at 20TB per month. Additionally, if you are actually pushing close to that much traffic, you can negotiate guaranteed commit prices w/ AWS that are competitive (especially when you consider the quality of bandwidth. I can only get ~100 mb/s to my hetzner server because of how bad their peering is. I can easily saturate my 1GB connection to anywhere in AWS.) --- (Having said that, this does only apply to egress via cloudfront. Things like charging for intra A-Z bandwidth within the same region is insane, and for many workloads may be surprisingly expensive.)
      - riku_iki 3y ago
        > For example, Hetzner caps your bandwidth at 20TB per month. looks like your information is obsolete by 5 years: https://www.hetzner.com/news/traffic-limit https://www.hetzner.com/news/traffic-limit
        nmjohn 3y ago
        Your random blog post is the outdated one. Their actual docs: https://docs.hetzner.com/robot/general/traffic/ https://docs.hetzner.com/robot/general/traffic/ And reports of throttling once you actually hit certain limits: https://lowendtalk.com/discussion/180504/hetzner-traffic-use-notice-unlimited-unlimited https://lowendtalk.com/discussion/180504/hetzner-traffic-use...
        riku_iki 3y ago
        your link says exactly what blog post said: root servers with 1gb/s uplink have unlimited traffic. Other offerings (e.g. 20gb/s uplink) and small cloud servers have traffic limitations.
        nmjohn 3y ago
        Yes, I was pointing out that many of their products have explicit limits. But more importantly, the second link I posted shows how that despite being "unlimited" - it's not uncommon for hetzner to throttle / close your account if you go over unstated traffic limits.
- landemva 3y ago
  In my experience, in-house groups racking and configuring and maintaining boxes often become a priesthood of negative value bickering over pet machine names. While cloud overcharges for McD product, I know what I get.
  - WJW 3y ago
    Not to mention that you get it within minutes of asking for it. I've worked at places where the internal bureaucracy managed to take double digit weeks to deliver a new database instance.
    - JambalayaJim 3y ago
      It‘a very depressing hearing this. Anything cloud at my company takes exponentially longer to setup as we have layers and layers of bureaucracy for any data that leaves our network.
      - WJW 3y ago
        The grass is always greener on the other side of the fence I guess. :)
    - narag 3y ago
      I've worked at places where the internal bureaucracy managed to take double digit weeks to deliver a new database instance. That wouldn't happen if the internal bureaucracy for firing the laggards wasn't so excruciatingly slow.
- dharmab 3y ago
  What you pay for is the ability to provision hardware for new R&D and projects in minutes rather than days, weeks or months. Companies are willing to spend millions a month in cloud fees to accelerate hundreds of millions in revenue.
- nunez 3y ago
  Because there's no such thing as a free lunch, running your own datacenter (or managing datacenter managed services) is work, and the clouds are better at providing exactly the compute you need when you need it than buying a ton of excess hardware you might or might not use (or can't scale quickly enough if you _do_ use it all) A datacenter makes sense if your usage profile is steady state and hardly ever changes, or if growth rate is predictable and capacity can be procured in advance. Any other use case is better suited for the cloud, IMO.
  - riku_iki 3y ago
    there are several middle points between full cloud and full own datacenter: rent dedicated server, rent rack in collocation datacenter.
    - djoletina 3y ago
      You still need to manage the underlying database and security which requires a lot of knowledge to do well.
      - riku_iki 3y ago
        correct, and managed solution will have cost too, quality issues: 3p services have bugs and are hacked too and that's something you can't control, and added complexity. To me it looks like if you have small traffic/storage requirements, then cloud may look good. But if you have lots of data and need compute then running beefy bare-metal server can be very cost beneficial.
      - kemotep 3y ago
        I still think that you need to manage some security things even if you are using AWS.
  - alberth 3y ago
    People forget that Colo & Leasing servers exist. And it costs like 1/5th of cost of “cloud”, and the later still gets you “the cloud”.
- YetAnotherNick 3y ago
  Because cloud infatuation is largely a myth. Every middle or large organization does periodic cost analysis and select cloud based on that analysis not on convenience. e.g. See this: https://aws.amazon.com/solutions/case-studies/emirates-case-study/?did=cr_card&trk=cr_card https://aws.amazon.com/solutions/case-studies/emirates-case-....
  - riku_iki 3y ago
    > See this that looks like some marketing material without any specifics..
    - dharmab 3y ago
      I used to lead a widely used and influential platform eng team at Adobe. Parent comment is right. We had our own datacenters and multiple cloud providers, and growing the public cloud infrastructure had a demonstrably higher return on investment than the private cloud.
louwrentius 3y ago
I think there are ton of great use cases for the cloud, but people should try and think for themselves and decide if their circumstances and workload is really a good fit. A ton of people forget that a bunch of servers across a few colocations can pay for itself in months, especially if you go for second-hand gear that is dirt cheap. Again, going (back) to colocating hardware could not be good fit. But with modern management tools and datacenter services like 'remote hands' I think people should not reject it upfront.
pid-1 3y ago
I did some napkin math (could be very, very wrong) but this server costs ~ 225,000 USD according to Dell's webpage. AWS does not have a 100% similar VM, but you could have something close for ~ 20,000 USD monthly. Not that bad. However, storage costs alone would be astronomic. Like > 100,000 USD / month. I have no idea how much outbound traffic Let's Encrypt serves, but that also could be a quite relevant expense. OFC I also don't know how much Let's Encrypt pays for energy, cooling, operations, real estate, etc... but: > I bet that compared to an equivalent load hosted on AWS, that lovely box pays for itself in full every month I would not take the other side on that bet
- ttegloma 3y ago
  > AWS does not have a 100% similar VM, but you could have something close for ~ 20,000 USD monthly. Not that bad. Is that the on-demand cost, or the reserved cost? For comparing to buying a server outright, you should be comparing the reserved cost. I’m not sure exactly which instances you’re looking at to get $20k/mo, but I see some instances with 64-128 cores/1-2 TB memory for <10k/month. For storage, I’m not sure how you’re getting >100k… I plugged in the highest IOPS I could for io2 volumes for 150 TB of storage and got 30k/mo. Also worth considering here that you don’t have to provision all 150 TB up front - you could start with 5 TB and increase in size as you grow, for example. Still gonna be hella expensive but all of this changes the calculus quite a bit from your estimates.
  - Shakahs 3y ago
    They're also using ZFS in Raid1+0, so 38.4TB of usable storage. $14,566/month on AWS io2 with max IOPS.
- jaas 3y ago
  This did not cost us $225k. About half that. Nobody pays the website price, you pay a lot less via a VAR. - ED of ISRG / Let's Encrypt
- alberth 3y ago
  It’s closer to $80k, if you went via a VAR. I spec’d a current gen server with: - 2x more cores - 50% more RAM - 50% more storage For only ~$80k, and that’s not even with a discount and is a more powerful current gen server with more cores/ram/storage. It’s $41k if I matched the specs exact (but that’s a bit unfair to do because the Dell server is 2-3 years old). https://www.siliconmechanics.com/system/rackform-a335.v9 https://www.siliconmechanics.com/system/rackform-a335.v9

I bet that compared to an equivalent load hosted on AWS, that lovely box pays for itself in full every month, if not every week...

by sho 3y ago

I bet that compared to an equivalent load hosted on AWS, that lovely box pays for itself in full every month, if not every week...

deleted 3y ago

[deleted]

d_silin 3y ago

Absolutely. Never understood why people are so infatuated with "cloud" options. Yes, it is convinient, but you are absolutely paying at least order of magnitude more for the same amount of compute/storage.

civilitty 3y ago
It's gotten absolutely bonkers with all the demand for GPUs and the cloud companies are just raking it in hand over fist. On AWS reserved instances with 8x A100s or H100s (if you can even get them) cost more per year than the total upfront retail price of the equivalent pods from Lambda Labs. The on-demand price is even more absurd.
qwertox 3y ago
If you don't need to host something big, where you can get away with one good VPS, the cloud has the benefit of offering cheap bandwidth. Once you have the bandwidth at your location and you don't need to be present in multiple locations, it's cheaper to self-host. Next step would be colocation, but for a start, using cloud offerings is a cheap way to be a part of the internet.
- nacs 3y ago
  Cloud bandwidth is definitely not cheap. If anything, it's where they rip you off the most. You can get "baremetal"/dedicated servers from places like Hetzner and OVH that give unmetered gigabit connections for like $50.
  - nmjohn 3y ago
    You can not get actual unmetered 1gb/s for anywhere close to that. If you start pushing anywhere close to that much bandwidth, you will be throttled / have your account closed. For example, Hetzner caps your bandwidth at 20TB per month. Additionally, if you are actually pushing close to that much traffic, you can negotiate guaranteed commit prices w/ AWS that are competitive (especially when you consider the quality of bandwidth. I can only get ~100 mb/s to my hetzner server because of how bad their peering is. I can easily saturate my 1GB connection to anywhere in AWS.) --- (Having said that, this does only apply to egress via cloudfront. Things like charging for intra A-Z bandwidth within the same region is insane, and for many workloads may be surprisingly expensive.)
    - riku_iki 3y ago
      > For example, Hetzner caps your bandwidth at 20TB per month. looks like your information is obsolete by 5 years: https://www.hetzner.com/news/traffic-limit https://www.hetzner.com/news/traffic-limit
      - nmjohn 3y ago
        Your random blog post is the outdated one. Their actual docs: https://docs.hetzner.com/robot/general/traffic/ https://docs.hetzner.com/robot/general/traffic/ And reports of throttling once you actually hit certain limits: https://lowendtalk.com/discussion/180504/hetzner-traffic-use-notice-unlimited-unlimited https://lowendtalk.com/discussion/180504/hetzner-traffic-use...
        riku_iki 3y ago
        your link says exactly what blog post said: root servers with 1gb/s uplink have unlimited traffic. Other offerings (e.g. 20gb/s uplink) and small cloud servers have traffic limitations.
        nmjohn 3y ago
        Yes, I was pointing out that many of their products have explicit limits. But more importantly, the second link I posted shows how that despite being "unlimited" - it's not uncommon for hetzner to throttle / close your account if you go over unstated traffic limits.
landemva 3y ago
In my experience, in-house groups racking and configuring and maintaining boxes often become a priesthood of negative value bickering over pet machine names. While cloud overcharges for McD product, I know what I get.
- WJW 3y ago
  Not to mention that you get it within minutes of asking for it. I've worked at places where the internal bureaucracy managed to take double digit weeks to deliver a new database instance.
  - JambalayaJim 3y ago
    It‘a very depressing hearing this. Anything cloud at my company takes exponentially longer to setup as we have layers and layers of bureaucracy for any data that leaves our network.
    - WJW 3y ago
      The grass is always greener on the other side of the fence I guess. :)
  - narag 3y ago
    I've worked at places where the internal bureaucracy managed to take double digit weeks to deliver a new database instance. That wouldn't happen if the internal bureaucracy for firing the laggards wasn't so excruciatingly slow.
dharmab 3y ago
What you pay for is the ability to provision hardware for new R&D and projects in minutes rather than days, weeks or months. Companies are willing to spend millions a month in cloud fees to accelerate hundreds of millions in revenue.
nunez 3y ago
Because there's no such thing as a free lunch, running your own datacenter (or managing datacenter managed services) is work, and the clouds are better at providing exactly the compute you need when you need it than buying a ton of excess hardware you might or might not use (or can't scale quickly enough if you _do_ use it all) A datacenter makes sense if your usage profile is steady state and hardly ever changes, or if growth rate is predictable and capacity can be procured in advance. Any other use case is better suited for the cloud, IMO.
- riku_iki 3y ago
  there are several middle points between full cloud and full own datacenter: rent dedicated server, rent rack in collocation datacenter.
  - djoletina 3y ago
    You still need to manage the underlying database and security which requires a lot of knowledge to do well.
    - riku_iki 3y ago
      correct, and managed solution will have cost too, quality issues: 3p services have bugs and are hacked too and that's something you can't control, and added complexity. To me it looks like if you have small traffic/storage requirements, then cloud may look good. But if you have lots of data and need compute then running beefy bare-metal server can be very cost beneficial.
    - kemotep 3y ago
      I still think that you need to manage some security things even if you are using AWS.
- alberth 3y ago
  People forget that Colo & Leasing servers exist. And it costs like 1/5th of cost of “cloud”, and the later still gets you “the cloud”.
YetAnotherNick 3y ago
Because cloud infatuation is largely a myth. Every middle or large organization does periodic cost analysis and select cloud based on that analysis not on convenience. e.g. See this: https://aws.amazon.com/solutions/case-studies/emirates-case-study/?did=cr_card&trk=cr_card https://aws.amazon.com/solutions/case-studies/emirates-case-....
- riku_iki 3y ago
  > See this that looks like some marketing material without any specifics..
  - dharmab 3y ago
    I used to lead a widely used and influential platform eng team at Adobe. Parent comment is right. We had our own datacenters and multiple cloud providers, and growing the public cloud infrastructure had a demonstrably higher return on investment than the private cloud.

louwrentius 3y ago

I think there are ton of great use cases for the cloud, but people should try and think for themselves and decide if their circumstances and workload is really a good fit. A ton of people forget that a bunch of servers across a few colocations can pay for itself in months, especially if you go for second-hand gear that is dirt cheap. Again, going (back) to colocating hardware could not be good fit. But with modern management tools and datacenter services like 'remote hands' I think people should not reject it upfront.

pid-1 3y ago

I did some napkin math (could be very, very wrong) but this server costs ~ 225,000 USD according to Dell's webpage. AWS does not have a 100% similar VM, but you could have something close for ~ 20,000 USD monthly. Not that bad. However, storage costs alone would be astronomic. Like > 100,000 USD / month. I have no idea how much outbound traffic Let's Encrypt serves, but that also could be a quite relevant expense. OFC I also don't know how much Let's Encrypt pays for energy, cooling, operations, real estate, etc... but: > I bet that compared to an equivalent load hosted on AWS, that lovely box pays for itself in full every month I would not take the other side on that bet

ttegloma 3y ago
> AWS does not have a 100% similar VM, but you could have something close for ~ 20,000 USD monthly. Not that bad. Is that the on-demand cost, or the reserved cost? For comparing to buying a server outright, you should be comparing the reserved cost. I’m not sure exactly which instances you’re looking at to get $20k/mo, but I see some instances with 64-128 cores/1-2 TB memory for <10k/month. For storage, I’m not sure how you’re getting >100k… I plugged in the highest IOPS I could for io2 volumes for 150 TB of storage and got 30k/mo. Also worth considering here that you don’t have to provision all 150 TB up front - you could start with 5 TB and increase in size as you grow, for example. Still gonna be hella expensive but all of this changes the calculus quite a bit from your estimates.
- Shakahs 3y ago
  They're also using ZFS in Raid1+0, so 38.4TB of usable storage. $14,566/month on AWS io2 with max IOPS.
jaas 3y ago
This did not cost us $225k. About half that. Nobody pays the website price, you pay a lot less via a VAR. - ED of ISRG / Let's Encrypt
alberth 3y ago
It’s closer to $80k, if you went via a VAR. I spec’d a current gen server with: - 2x more cores - 50% more RAM - 50% more storage For only ~$80k, and that’s not even with a discount and is a more powerful current gen server with more cores/ram/storage. It’s $41k if I matched the specs exact (but that’s a bit unfair to do because the Dell server is 2-3 years old). https://www.siliconmechanics.com/system/rackform-a335.v9 https://www.siliconmechanics.com/system/rackform-a335.v9