5 ms·

Signals says cloud repatriation has already saved $1M

31 points by rcarmo 3y ago

nonrandomstring 3y ago
Had a student do a research thesis on this last year and IIRC whether it's a massive cost saver or an expensive disaster boils down to good old fashioned "system analysis". Know your system, every part. Big cost savings are often made at this stage just discovering a shit-ton of obsolete stuff. Don't just map the network, map the _function_ of the network. Then look at its logical versus physical topology for anomalies (also a great security exercise). What you bring back on-prem must be systematically viable. Factor in on-prem energy costs, they are not trivial. The biggest overall factor is people and wages. Any significant repatriation project ends up needing to hire back the system administrator you fired a couple of years ago.
- blitzar 3y ago
  > boils down to good old fashioned "system analysis". Know your system, every part This type of answer has managed to talk me out of some amazing roles (and dodge some horrible people). Apparently the correct answer is "there is a magic formula whereby you dont need to understand anything or do any analysis and it will deliver you sunshine and rainbows"
  - thelastparadise 3y ago
    Reminds me of a (public) talk by DIRNSA on security a few years back. The talk boiled down to: know your network. Know the traffic on it. Then you have a chance of being secure.
    - blitzar 3y ago
      NSA isnt selling anything, they can tell the truth. $Corp who sells Worlds Best network intrusion detection firewall will be telling a different story.
- vidarh 3y ago
  Actual on prem is not for most people. Managed hosting in a data centre run by someone else is the halfway house (and what 37Signals is doing). You get most of the savings, and very few of the pain points. One of the biggest benefits besides egress should be people and wages. When I consulted in this space, clients on AWS were gold dust, because they tended to spend far more. A significant reason is that you tend to be able to drastically simplify when you can pick hardware that fits your workloads rather than what fits your cloud provider, and can forego all the multi-tenancy complications your cloud provider needs to account for.
- coastermug 3y ago
  Are you able to share that research thesis? Or any key papers on how this should be done? Knowing SOTA on these approaches is always nice.
  - nonrandomstring 3y ago
    Sorry no, I quit teaching at that uni and don't keep contact with the cohort now. But a little more info I remember: It arose under a project suggestion three or four students took me up on, and that was "Long term resilience". The obvious antecedent being "If Google goes down tomorrow....." "De-clouding" was in the title. And the conclusions, perhaps predictably, were that hybrid systems were the sweet spot across a range of business types, with no panaceas, or even decent maxima to be found in fully on-prem or total cloud. The students artifact IIRC was a "matrix" (a graph theoretical model) of the edge-costs for traversing well modular systems into different states - something that could be used to plan a "repatriation" FWIW the student was African and we had a good laugh at the racist undertones of that language and it's subtle implications about "Empires".
    - vidarh 3y ago
      Hybrid is interesting because once you set a system up for hybrid it tends to drive down costs without increasing cloud usage much in my experience. When you can spin up cloud instances, that capability in itself means you can justify loading individual servers closer to 100%. Couple that with managed / leased capacity in data centres rather than "true" on prem, and managed providers often delivering many server models very rapidly, and in most hybrid setups I've worked on, we've ended up using the ability to spin up cloud instances very little, while slowing down the rate of growth of new server capacity overall. So I agree, they're the sweet spot, but in terms of dollars spent my hunch is that most people deploying them will find their spend dominated by "more fixed" servers, ranging from own/leased servers via managed hosting depending on local power and real estate costs (last placed I ran a physical hosting facility, what finally tipped it towards managed hosting for us was that the price of land grew to a point that empty rack space near us vs. at the managed hosting provider we chose was so much more expensive that the difference paid for our servers).
mdhb 3y ago
I spent over ten years in the Rails community and as a result consider myself fairly well versed in the things that tend to come out of the mouth of DHH. This particular one is very much in line with some of his other greatest hits where the more you dig into his claims and assumptions the more things don’t seem to always add up in quite the way he is presenting them. Without going super deep on the topic, he came to cloud with his on-prem architecture choices, made very few adaptions to take advantage of what cloud is good at besides trying to slap K8s around it and then was upset that the benefits never materialised. I can understand that being a shitty and painful experience but he is now acting as an anti-cloud evangelist of sorts while arguing against a series of constraints and choices that he picked and clearly weren’t appropriate for his business. But now for him that’s all that cloud can ever be as it’s all he ever knew.
- willtemperley 3y ago
  So in reality they've done two unnecessary migrations, burned a ton of cash on cloud compute and are selling it as a success?
- sgarland 3y ago
  While I don’t doubt your personal experience, I’ve spent time at a cloud-native company and boy, do they know how to spend money. Just because you’ve bought into every .aaS the cloud has doesn’t mean you’ll save money. RDS, for example, is stupidly expensive. I maintain that it’s not that hard to use Xtrabackup or any of the other free and open tooling to run regular backups for RDBMS. It’s also not that hard to set up HA. If those things seem hard, your Aurora Serverless bill is going to be even harder to swallow when it hits.
  - ckdarby 3y ago
    Sounds like a great business opportunity for you that will make millions and millions.
    - sgarland 3y ago
      Touché. I have no desire to start an IaaS company. However, if I ever launch a product, it will be running on my own hardware (to be clear, I own the hardware now; this isn’t a future CAPEX) until there is a clear financial incentive to not do so.
flanked-evergl 3y ago
I dunno, to me the argument is basically, "We can outcompete our competitors and the cloud provides at once". I doubt it, actually, because it requires 37 Signals to make tradeoffs against their primary business, which is whatever people actually pay them for, in favour of something which is not their primary business, i.e. running a cloud platform. But sure, if all they buy from AWS is VMs, may be better.
- pclmulqdq 3y ago
  The trick is that 99% of the engineering expense at cloud providers is to enable weird multi-tenancy use cases and scale to obscene size. You don't have to do either of those if you run your own hardware - your infrastructure can be quite simple.
sixhobbits 3y ago
"37Signals" , i.e. basecamp, Hey, etc. I know HN strips out numbers automatically to remove clickbait articles like "37 signals you need to watch for when buying a SaaS" or whatever, but pretty annoying when it butchers something like this.
- opan 3y ago
  Thanks, I missed the second S and thought this was about the Signal messenger.
darthrupert 3y ago
AWS is always more expensive than owned hardware + good team -- it's after all what they generate their profits from. However, it's also probable that they were using the cloud wrong. Their software (afaik) is a typical 3-tier application coded on Ruby on Rails, which probably implies that they used just EC2s, with probably an orchestrator on top. Also, if they didn't need any new staff after moving to their own server rooms, this implies that either their staff was having lots of free time or that their staff currently is overworked.
- vidarh 3y ago
  When I've done contract devops, clients on AWS always tended to need more of my time. The level of devops complexity for AWS is often far higher than self hosted. It's really hard not to beat AWS on cost, including person hours to manage the setup.
johnklos 3y ago
Of course it saves money. The main selling line of the cloud is you're too dumb to administer, so just trust them and throw money at them. If you're not too dumb to administer, then running your own hardware is invariably cheaper. What's surprising is that so many otherwise technical people believe the cloud BS without really thinking things through. Anyone who can do basic maths can see that aside from trivial uses renting is always going to cost more in the long run.
- aaronharnly 3y ago
  I don’t think that’s the main selling point of cloud? The main selling points are that (1) you can scale capacity up and down, so you aren’t purchasing, provisioning, and operating hardware for peak demand (2) with infra-as-code tooling, software teams can configure and deploy significant infrastructural capabilities without scaling specialized operations roles. Dumbness does not play into it.
  - sgarland 3y ago
    IME, much of the “need to easily scale” is fixed by understanding your bottlenecks, being able to understand the metrics being captured, and how to address them. You would be shocked (or not) at how few people raised on cloud can explain what IOPS are, or what IOWAIT is, let alone linking the two up. Re: IaC, you definitely should understand what it is you’re instantiating (and why), no matter how you’re doing it. AWS Community TF Modules are all well and good until something breaks, and you realize you don’t really understand what’s going on.
    - sokoloff 3y ago
      > how few people raised on cloud can explain what IOPS are I think you would be shocked at how few people raised in a fully on-prem world can adequately explain even that simple concept and its first-order implications.
      - sgarland 3y ago
andrewstuart 3y ago
The absolute bottom of the market lowest price AWS GPU spot instance costs 15 cents an hour. That’s USD$1,314 per annum for an extremely underpowered machine. The numbers don’t add up. Just buy a physical machine, plop it on a network somewhere, you’ll have dramatically more computing power and you won’t be paying for it again next year, and the year after that etc “The cloud is where Moore’s law goes to die.”
- ckdarby 3y ago
  Except the need for managing it securely. Oh, it is very successful? Now the 50 dev team each want one. Oh, you maxed out the number of ports on the switch. One of those GPUs have customer information that only select individuals should have. One of them needs to write only for off-site backups. Every time this on-prem vs cloud stuff comes up folks just skim over all the other pieces included. Didn't even talk about racking, network and or power. Yes, there is a spot and time where on-prem makes sense and there is also a time for cloud. I barely cover some of them in this 30 min video, https://youtu.be/8i7i1VFSJrw?si=pLyIhwgxe4d3efvb https://youtu.be/8i7i1VFSJrw?si=pLyIhwgxe4d3efvb
  - Spooky23 3y ago
    You’re adopting an almost religious dogma about this. At the end of the day, it’s a dollars and cents decision. During development and growth of services, cloud makes total sense as you abstract the capital costs. If you’re a startup making Uber for cupcake makers, you’d be dumb to do anything else. Depending on your needs, once things settle down, it’s often cheaper and more effective to capitalize your costs yourself. It’s akin to a car. If you go somewhere for a day or two, Uber makes sense. For a week or more, rent a car, for a year, you probably want to buy or lease a vehicle. Again, it depends on you. For me, I have large workloads running in GCP and Amazon. But 80% of the production I’m responsible for is “on-prem”, because it’s about 30% cheaper. YMMV
  - alberth 3y ago
    Lease a Server You get all of above (mgmt, etc) for the same cost. For example, you can get for the same $1200/year: - 16 core (Zen4 current gen) - 128 GB RAM (ECC/DDR5) - 2 x 2TB NVME For just $100/mo. https://www.hetzner.com/dedicated-rootserver/ax102/configurator#/ https://www.hetzner.com/dedicated-rootserver/ax102/configura... I feel like the greatest marketing move AWS has played is getting people to forget that Leasing Servers is a thing.
alberth 3y ago
Source: https://world.hey.com/dhh/our-cloud-exit-has-already-yielded-1m-year-in-savings-db358dea https://world.hey.com/dhh/our-cloud-exit-has-already-yielded...
midwit 3y ago
How practical is it to run a server at your house? Is fiber a must? What if the power goes out? What about exposing your home ip?
- unforeseen9991 3y ago
  Colo is a much better option, but i've hosted at home before. I have fiber, UPS, and a generator. The home IP wasn't a concern, but you could proxy behind cloudflare or something?
datadrivenangel 3y ago
"We’re still storing eight petabytes of data in AWS’s S3 service. We’re leaving that one for next year. " - DHH in July [0] That means 37 signals probably still have a 7 figure cloud bill [0] https://37signals.com/podcast/leaving-the-cloud-the-finale/ https://37signals.com/podcast/leaving-the-cloud-the-finale/

31 points by rcarmo 3y ago

Had a student do a research thesis on this last year and IIRC whether it's a massive cost saver or an expensive disaster boils down to good old fashioned "system analysis". Know your system, every part. Big cost savings are often made at this stage just discovering a shit-ton of obsolete stuff. Don't just map the network, map the _function_ of the network. Then look at its logical versus physical topology for anomalies (also a great security exercise). What you bring back on-prem must be systematically viable. Factor in on-prem energy costs, they are not trivial. The biggest overall factor is people and wages. Any significant repatriation project ends up needing to hire back the system administrator you fired a couple of years ago.

blitzar 3y ago
> boils down to good old fashioned "system analysis". Know your system, every part This type of answer has managed to talk me out of some amazing roles (and dodge some horrible people). Apparently the correct answer is "there is a magic formula whereby you dont need to understand anything or do any analysis and it will deliver you sunshine and rainbows"
- thelastparadise 3y ago
  Reminds me of a (public) talk by DIRNSA on security a few years back. The talk boiled down to: know your network. Know the traffic on it. Then you have a chance of being secure.
  - blitzar 3y ago
    NSA isnt selling anything, they can tell the truth. $Corp who sells Worlds Best network intrusion detection firewall will be telling a different story.
vidarh 3y ago
Actual on prem is not for most people. Managed hosting in a data centre run by someone else is the halfway house (and what 37Signals is doing). You get most of the savings, and very few of the pain points. One of the biggest benefits besides egress should be people and wages. When I consulted in this space, clients on AWS were gold dust, because they tended to spend far more. A significant reason is that you tend to be able to drastically simplify when you can pick hardware that fits your workloads rather than what fits your cloud provider, and can forego all the multi-tenancy complications your cloud provider needs to account for.
coastermug 3y ago
Are you able to share that research thesis? Or any key papers on how this should be done? Knowing SOTA on these approaches is always nice.
- nonrandomstring 3y ago
  Sorry no, I quit teaching at that uni and don't keep contact with the cohort now. But a little more info I remember: It arose under a project suggestion three or four students took me up on, and that was "Long term resilience". The obvious antecedent being "If Google goes down tomorrow....." "De-clouding" was in the title. And the conclusions, perhaps predictably, were that hybrid systems were the sweet spot across a range of business types, with no panaceas, or even decent maxima to be found in fully on-prem or total cloud. The students artifact IIRC was a "matrix" (a graph theoretical model) of the edge-costs for traversing well modular systems into different states - something that could be used to plan a "repatriation" FWIW the student was African and we had a good laugh at the racist undertones of that language and it's subtle implications about "Empires".
  - vidarh 3y ago
    Hybrid is interesting because once you set a system up for hybrid it tends to drive down costs without increasing cloud usage much in my experience. When you can spin up cloud instances, that capability in itself means you can justify loading individual servers closer to 100%. Couple that with managed / leased capacity in data centres rather than "true" on prem, and managed providers often delivering many server models very rapidly, and in most hybrid setups I've worked on, we've ended up using the ability to spin up cloud instances very little, while slowing down the rate of growth of new server capacity overall. So I agree, they're the sweet spot, but in terms of dollars spent my hunch is that most people deploying them will find their spend dominated by "more fixed" servers, ranging from own/leased servers via managed hosting depending on local power and real estate costs (last placed I ran a physical hosting facility, what finally tipped it towards managed hosting for us was that the price of land grew to a point that empty rack space near us vs. at the managed hosting provider we chose was so much more expensive that the difference paid for our servers).

mdhb 3y ago

I spent over ten years in the Rails community and as a result consider myself fairly well versed in the things that tend to come out of the mouth of DHH. This particular one is very much in line with some of his other greatest hits where the more you dig into his claims and assumptions the more things don’t seem to always add up in quite the way he is presenting them. Without going super deep on the topic, he came to cloud with his on-prem architecture choices, made very few adaptions to take advantage of what cloud is good at besides trying to slap K8s around it and then was upset that the benefits never materialised. I can understand that being a shitty and painful experience but he is now acting as an anti-cloud evangelist of sorts while arguing against a series of constraints and choices that he picked and clearly weren’t appropriate for his business. But now for him that’s all that cloud can ever be as it’s all he ever knew.

willtemperley 3y ago
So in reality they've done two unnecessary migrations, burned a ton of cash on cloud compute and are selling it as a success?
sgarland 3y ago
While I don’t doubt your personal experience, I’ve spent time at a cloud-native company and boy, do they know how to spend money. Just because you’ve bought into every .aaS the cloud has doesn’t mean you’ll save money. RDS, for example, is stupidly expensive. I maintain that it’s not that hard to use Xtrabackup or any of the other free and open tooling to run regular backups for RDBMS. It’s also not that hard to set up HA. If those things seem hard, your Aurora Serverless bill is going to be even harder to swallow when it hits.
- ckdarby 3y ago
  Sounds like a great business opportunity for you that will make millions and millions.
  - sgarland 3y ago
    Touché. I have no desire to start an IaaS company. However, if I ever launch a product, it will be running on my own hardware (to be clear, I own the hardware now; this isn’t a future CAPEX) until there is a clear financial incentive to not do so.

flanked-evergl 3y ago

I dunno, to me the argument is basically, "We can outcompete our competitors and the cloud provides at once". I doubt it, actually, because it requires 37 Signals to make tradeoffs against their primary business, which is whatever people actually pay them for, in favour of something which is not their primary business, i.e. running a cloud platform. But sure, if all they buy from AWS is VMs, may be better.

pclmulqdq 3y ago
The trick is that 99% of the engineering expense at cloud providers is to enable weird multi-tenancy use cases and scale to obscene size. You don't have to do either of those if you run your own hardware - your infrastructure can be quite simple.

sixhobbits 3y ago

"37Signals" , i.e. basecamp, Hey, etc. I know HN strips out numbers automatically to remove clickbait articles like "37 signals you need to watch for when buying a SaaS" or whatever, but pretty annoying when it butchers something like this.

opan 3y ago
Thanks, I missed the second S and thought this was about the Signal messenger.

darthrupert 3y ago

AWS is always more expensive than owned hardware + good team -- it's after all what they generate their profits from. However, it's also probable that they were using the cloud wrong. Their software (afaik) is a typical 3-tier application coded on Ruby on Rails, which probably implies that they used just EC2s, with probably an orchestrator on top. Also, if they didn't need any new staff after moving to their own server rooms, this implies that either their staff was having lots of free time or that their staff currently is overworked.

vidarh 3y ago
When I've done contract devops, clients on AWS always tended to need more of my time. The level of devops complexity for AWS is often far higher than self hosted. It's really hard not to beat AWS on cost, including person hours to manage the setup.

johnklos 3y ago

Of course it saves money. The main selling line of the cloud is you're too dumb to administer, so just trust them and throw money at them. If you're not too dumb to administer, then running your own hardware is invariably cheaper. What's surprising is that so many otherwise technical people believe the cloud BS without really thinking things through. Anyone who can do basic maths can see that aside from trivial uses renting is always going to cost more in the long run.

aaronharnly 3y ago
I don’t think that’s the main selling point of cloud? The main selling points are that (1) you can scale capacity up and down, so you aren’t purchasing, provisioning, and operating hardware for peak demand (2) with infra-as-code tooling, software teams can configure and deploy significant infrastructural capabilities without scaling specialized operations roles. Dumbness does not play into it.
- sgarland 3y ago
  IME, much of the “need to easily scale” is fixed by understanding your bottlenecks, being able to understand the metrics being captured, and how to address them. You would be shocked (or not) at how few people raised on cloud can explain what IOPS are, or what IOWAIT is, let alone linking the two up. Re: IaC, you definitely should understand what it is you’re instantiating (and why), no matter how you’re doing it. AWS Community TF Modules are all well and good until something breaks, and you realize you don’t really understand what’s going on.
  - sokoloff 3y ago
    > how few people raised on cloud can explain what IOPS are I think you would be shocked at how few people raised in a fully on-prem world can adequately explain even that simple concept and its first-order implications.
    - sgarland 3y ago

andrewstuart 3y ago

The absolute bottom of the market lowest price AWS GPU spot instance costs 15 cents an hour. That’s USD$1,314 per annum for an extremely underpowered machine. The numbers don’t add up. Just buy a physical machine, plop it on a network somewhere, you’ll have dramatically more computing power and you won’t be paying for it again next year, and the year after that etc “The cloud is where Moore’s law goes to die.”

ckdarby 3y ago
Except the need for managing it securely. Oh, it is very successful? Now the 50 dev team each want one. Oh, you maxed out the number of ports on the switch. One of those GPUs have customer information that only select individuals should have. One of them needs to write only for off-site backups. Every time this on-prem vs cloud stuff comes up folks just skim over all the other pieces included. Didn't even talk about racking, network and or power. Yes, there is a spot and time where on-prem makes sense and there is also a time for cloud. I barely cover some of them in this 30 min video, https://youtu.be/8i7i1VFSJrw?si=pLyIhwgxe4d3efvb https://youtu.be/8i7i1VFSJrw?si=pLyIhwgxe4d3efvb
- Spooky23 3y ago
  You’re adopting an almost religious dogma about this. At the end of the day, it’s a dollars and cents decision. During development and growth of services, cloud makes total sense as you abstract the capital costs. If you’re a startup making Uber for cupcake makers, you’d be dumb to do anything else. Depending on your needs, once things settle down, it’s often cheaper and more effective to capitalize your costs yourself. It’s akin to a car. If you go somewhere for a day or two, Uber makes sense. For a week or more, rent a car, for a year, you probably want to buy or lease a vehicle. Again, it depends on you. For me, I have large workloads running in GCP and Amazon. But 80% of the production I’m responsible for is “on-prem”, because it’s about 30% cheaper. YMMV
- alberth 3y ago
  Lease a Server You get all of above (mgmt, etc) for the same cost. For example, you can get for the same $1200/year: - 16 core (Zen4 current gen) - 128 GB RAM (ECC/DDR5) - 2 x 2TB NVME For just $100/mo. https://www.hetzner.com/dedicated-rootserver/ax102/configurator#/ https://www.hetzner.com/dedicated-rootserver/ax102/configura... I feel like the greatest marketing move AWS has played is getting people to forget that Leasing Servers is a thing.

alberth 3y ago

Source: https://world.hey.com/dhh/our-cloud-exit-has-already-yielded-1m-year-in-savings-db358dea https://world.hey.com/dhh/our-cloud-exit-has-already-yielded...

midwit 3y ago

How practical is it to run a server at your house? Is fiber a must? What if the power goes out? What about exposing your home ip?

unforeseen9991 3y ago
Colo is a much better option, but i've hosted at home before. I have fiber, UPS, and a generator. The home IP wasn't a concern, but you could proxy behind cloudflare or something?

datadrivenangel 3y ago

"We’re still storing eight petabytes of data in AWS’s S3 service. We’re leaving that one for next year. " - DHH in July [0] That means 37 signals probably still have a 7 figure cloud bill [0] https://37signals.com/podcast/leaving-the-cloud-the-finale/ https://37signals.com/podcast/leaving-the-cloud-the-finale/