3 ms·

37signals spends more than $3M a year on cloud. So while it definitely isn't a major enterprise. It is also a lot more than a a few dozen servers. I am not ant

by ksec 2y ago

37signals spends more than $3M a year on cloud. So while it definitely isn't a major enterprise. It is also a lot more than a a few dozen servers. I am not anti-cloud and pro cloud. My major problem with the new trend is that a lot of people are basically rediscovering pre "Cloud" era. which is VPS, Dedicated server and Colocation. And people are suggesting Hetzner or OVH or many other players are equivalent to AWS. While I dont disagree AWS is charging a lot for their offering, putting AWS to other services isn't even a valid comparison. Completely ignoring the basics such as Server / CPU / RAM / SSD quality. Network quality such as interconnect, redundancy, as well as Data Center quality. If you rally want to do simple price and spec comparison you might as well go to Lowendbox to find a low cost VPS which some people have been doing since 2008. I really wish there is a middle ground somewhere before using Hyperscalers. Both DO / Linode couldn't reach a larger scale. Hetzner is expanding their Cloud offering only and no dedicated outside EU.

ceejayoz 2y ago
> 37signals spends more than $3M a year on cloud. Isn’t most of that just S3 storage/bandwidth?
- jgalt212 2y ago
  If so, they should move to R2.
RateMyPE 2y ago
Yep, Hetzner, OVH or even DO aren't even close to offering what AWS offers. Once you start exploring all the things they have to offer you understand why so many large companies use hyperscalers. Although to be fair, most hobbyists only need basic services like cloud servers/VMs, and hyperscalers like AWS are an awful deal compared to other options if you only need compute + storage + bandwidth. You don't need to use S3, Lambdas and Cloudfront to host a personal blog, a simple VPS will be more than enough. It feels like most devs nowadays prefer using services that abstract away the infrastructure, at the cost of not developing SysOps skills, so I don't see a future where the Cloud is going to lose relevance.
- littlestymaar 2y ago
  > you understand why so many large companies use hyperscalers. While there are valid use-case where you get value from the extra services the hyperscaller are providing, most of the time people go for AWS “because everybody does it” or because the choice was made by a consulting company that doesn't pay the final cloud bill and is optimizing their own added value at the expenses of the customer's one. I've been doing freelance work for 7 years now for roughly two dozens if companies of various size, and I can't tell you how many massively underused AWS / Azure VPS I've seen, but it's more than half the cloud bills of the said companies (or division for big companies since I obviously only had the vision on the division I worked for and not the whole company).
- mbreese 2y ago
  > Hetzner, OVH or even DO aren't even close to offering what AWS offers But I think the argument is - do they need to be? How much of the services of AWS (Google/azure/etc) are really needed by the majority of customers? For companies that need hyperscaling services, I get it. There are definite benefits to the cloud when operating at extremes (auto scaling up and down). But for the majority of workloads, I think you could be well served by a more barebones offering.
  - itake 2y ago
    > are really needed by the majority of customers? Some companies want the space to grow into it. At my job, we just started getting into video decoding. AWS has elastic video processing services. Where as DO would cost way more to setup those services on our own.
  - inkyoto 2y ago
    > How much of the services of AWS (Google/azure/etc) are really needed by the majority of customers? Very many. And none of them are EC2 (or its equivalent). Any service that comes with the consumption based charging (i.e. no 24x7 running costs whether it is used or not) and offers a clearly defined functional feature, has plenty of appeal to cloud customers. Another part of the appeal is the mix and match nature of mature cloud platforms: the customers get substantial freedom to choose from services they can instantly start using, or roll (and maintain) their own albeir at a higher cost. I.e. if the customer wants a queue, they get a queue, and nothing else. The cloud platform abstracts away and takes care of the underlying platform that «runs» the queue and eliminates the operational overhead that comes with having to look after the message broker that provides the said queue. There are other many examples.
kredd 2y ago
I totally agree, but I’ve also worked on a project with 0 customers spending about $2M/year on AWS, and there was absolutely zero incentive from the stakeholders to reduce the cost. There’s a certain disconnect between boots on the ground engineers, and decision makers when it comes to infra management.
dumbledoren 2y ago
> Server / CPU / RAM / SSD quality I had no issues with Hetzner's component and especially network quality for a decade now. Before that, yeah, there could be some hiccups and issues. But they stopped being issues a long time ago. And really, what hardware issues do you expect on this hardware: https://www.hetzner.com/dedicated-rootserver/brands/matrix-dell/ https://www.hetzner.com/dedicated-rootserver/brands/matrix-d...
roncesvalles 2y ago
My opinion from analyzing the 37signals cloud offboard case is that it shouldn't have been done. They didn't save a whole lot of money from it (they aren't spending a whole lot on it anyway), and now their business ever so slightly loses focus. Not to mention, as you said, the quality aspects. Cloud gives you many things "for free" (like local disk RAID, network and power redundancy, datacenter compliance, in-transit and on-disk encryption, optimized OS images, overall network security, controls around what their employees can do - that $5/month lowendbox provider from Romania is almost certainly logging into your VM and going through your files) which you lose when going to a "pre-cloud" provider.
- blibble 2y ago
  there's a mile of difference between Romanian lowendbox.com and renting a cage in, say, an equinix datacentre if this approach to DC compliance/security/redundancy is good enough for the world's financial services industry then it's probably good enough for everyone else too (but yes, then only saves about 90% of the cost instead of 95%)
- tommica 2y ago
  Umm, millions saved in the future seems like a decent amount of money? Yeah, they paid for the hardware in year 0 that equals to their AWS bill, but the subsequent years that money is not spent on aws or new servers
  - roncesvalles 2y ago
    Although it's private so we can never be sure, their revenue seems to be in the ballpark of $100m with about 40% margin. So even if they save a million per year, it's not worth it, especially when it's a trade-off.
umur 2y ago
There is indeed a large gap in the market between outsourcing all your infrastructure to Hyperscalers vs. hosting it on DIY-bare-metal and/or VPC providers. An open source alternative to AWS would do much to fill that gap, and we are building just that at Ubicloud (I'm one of the co-founders). So far with Ubicloud, you get virtual machines, load balancers, private networking, managed PostgreSQL, all with encryption at rest and in-transit. The Ubicloud managed service uses Hetzner bare metal as one of its hosting providers, which cuts costs 2x - 10x compared to AWS/Azure. Would love to hear any feedback if you'd like to give it a try, or go through the repo here: https://github.com/ubicloud/ubicloud https://github.com/ubicloud/ubicloud
- Brain_Man 2y ago
  > You can set it up yourself on these providers or you can use our managed service. Are all the bits and pieces necessary for starting one's own managed service open source? In case somebody is interested in starting their own commercial cloud. How easy would that be to deploy?
  - sekh60 2y ago
    There's OpenStack. It's a private IaaS. Had loadbalancers, ipv6 support, support for K8s hosting via the magnum component (and other container orchestrators), HA via Masakari component. The networking is very flexible. It does not currently have functions as a service, I believe that was in the Senlin component, but that's been abandoned, I believe a new incarnation of the idea is in the works though. With something like Kolla-ansible a containerized OpenStack infrastructure is pretty damn easy to manage, upgrades are just making sure you make any needded changes in the global config file (just a vimdiff with the new sample one included in the release) and then literally just a kolla-ansible upgrade -i inventory-file.yml. I'm just a home labber and I've run OpenStack via kolla-ansible for like 7 years now, and Ceph since the jewel release I think almost 8 years ago for storage. Both are pretty easy to manage.

by ksec 2y ago

ceejayoz 2y ago

> 37signals spends more than $3M a year on cloud. Isn’t most of that just S3 storage/bandwidth?

jgalt212 2y ago
If so, they should move to R2.

RateMyPE 2y ago

Yep, Hetzner, OVH or even DO aren't even close to offering what AWS offers. Once you start exploring all the things they have to offer you understand why so many large companies use hyperscalers. Although to be fair, most hobbyists only need basic services like cloud servers/VMs, and hyperscalers like AWS are an awful deal compared to other options if you only need compute + storage + bandwidth. You don't need to use S3, Lambdas and Cloudfront to host a personal blog, a simple VPS will be more than enough. It feels like most devs nowadays prefer using services that abstract away the infrastructure, at the cost of not developing SysOps skills, so I don't see a future where the Cloud is going to lose relevance.

littlestymaar 2y ago
> you understand why so many large companies use hyperscalers. While there are valid use-case where you get value from the extra services the hyperscaller are providing, most of the time people go for AWS “because everybody does it” or because the choice was made by a consulting company that doesn't pay the final cloud bill and is optimizing their own added value at the expenses of the customer's one. I've been doing freelance work for 7 years now for roughly two dozens if companies of various size, and I can't tell you how many massively underused AWS / Azure VPS I've seen, but it's more than half the cloud bills of the said companies (or division for big companies since I obviously only had the vision on the division I worked for and not the whole company).
mbreese 2y ago
> Hetzner, OVH or even DO aren't even close to offering what AWS offers But I think the argument is - do they need to be? How much of the services of AWS (Google/azure/etc) are really needed by the majority of customers? For companies that need hyperscaling services, I get it. There are definite benefits to the cloud when operating at extremes (auto scaling up and down). But for the majority of workloads, I think you could be well served by a more barebones offering.
- itake 2y ago
  > are really needed by the majority of customers? Some companies want the space to grow into it. At my job, we just started getting into video decoding. AWS has elastic video processing services. Where as DO would cost way more to setup those services on our own.
- inkyoto 2y ago
  > How much of the services of AWS (Google/azure/etc) are really needed by the majority of customers? Very many. And none of them are EC2 (or its equivalent). Any service that comes with the consumption based charging (i.e. no 24x7 running costs whether it is used or not) and offers a clearly defined functional feature, has plenty of appeal to cloud customers. Another part of the appeal is the mix and match nature of mature cloud platforms: the customers get substantial freedom to choose from services they can instantly start using, or roll (and maintain) their own albeir at a higher cost. I.e. if the customer wants a queue, they get a queue, and nothing else. The cloud platform abstracts away and takes care of the underlying platform that «runs» the queue and eliminates the operational overhead that comes with having to look after the message broker that provides the said queue. There are other many examples.

kredd 2y ago

I totally agree, but I’ve also worked on a project with 0 customers spending about $2M/year on AWS, and there was absolutely zero incentive from the stakeholders to reduce the cost. There’s a certain disconnect between boots on the ground engineers, and decision makers when it comes to infra management.

dumbledoren 2y ago

> Server / CPU / RAM / SSD quality I had no issues with Hetzner's component and especially network quality for a decade now. Before that, yeah, there could be some hiccups and issues. But they stopped being issues a long time ago. And really, what hardware issues do you expect on this hardware: https://www.hetzner.com/dedicated-rootserver/brands/matrix-dell/ https://www.hetzner.com/dedicated-rootserver/brands/matrix-d...

roncesvalles 2y ago

My opinion from analyzing the 37signals cloud offboard case is that it shouldn't have been done. They didn't save a whole lot of money from it (they aren't spending a whole lot on it anyway), and now their business ever so slightly loses focus. Not to mention, as you said, the quality aspects. Cloud gives you many things "for free" (like local disk RAID, network and power redundancy, datacenter compliance, in-transit and on-disk encryption, optimized OS images, overall network security, controls around what their employees can do - that $5/month lowendbox provider from Romania is almost certainly logging into your VM and going through your files) which you lose when going to a "pre-cloud" provider.

blibble 2y ago
there's a mile of difference between Romanian lowendbox.com and renting a cage in, say, an equinix datacentre if this approach to DC compliance/security/redundancy is good enough for the world's financial services industry then it's probably good enough for everyone else too (but yes, then only saves about 90% of the cost instead of 95%)
tommica 2y ago
Umm, millions saved in the future seems like a decent amount of money? Yeah, they paid for the hardware in year 0 that equals to their AWS bill, but the subsequent years that money is not spent on aws or new servers
- roncesvalles 2y ago
  Although it's private so we can never be sure, their revenue seems to be in the ballpark of $100m with about 40% margin. So even if they save a million per year, it's not worth it, especially when it's a trade-off.

umur 2y ago

There is indeed a large gap in the market between outsourcing all your infrastructure to Hyperscalers vs. hosting it on DIY-bare-metal and/or VPC providers. An open source alternative to AWS would do much to fill that gap, and we are building just that at Ubicloud (I'm one of the co-founders). So far with Ubicloud, you get virtual machines, load balancers, private networking, managed PostgreSQL, all with encryption at rest and in-transit. The Ubicloud managed service uses Hetzner bare metal as one of its hosting providers, which cuts costs 2x - 10x compared to AWS/Azure. Would love to hear any feedback if you'd like to give it a try, or go through the repo here: https://github.com/ubicloud/ubicloud https://github.com/ubicloud/ubicloud

Brain_Man 2y ago
> You can set it up yourself on these providers or you can use our managed service. Are all the bits and pieces necessary for starting one's own managed service open source? In case somebody is interested in starting their own commercial cloud. How easy would that be to deploy?
- sekh60 2y ago
  There's OpenStack. It's a private IaaS. Had loadbalancers, ipv6 support, support for K8s hosting via the magnum component (and other container orchestrators), HA via Masakari component. The networking is very flexible. It does not currently have functions as a service, I believe that was in the Senlin component, but that's been abandoned, I believe a new incarnation of the idea is in the works though. With something like Kolla-ansible a containerized OpenStack infrastructure is pretty damn easy to manage, upgrades are just making sure you make any needded changes in the global config file (just a vimdiff with the new sample one included in the release) and then literally just a kolla-ansible upgrade -i inventory-file.yml. I'm just a home labber and I've run OpenStack via kolla-ansible for like 7 years now, and Ceph since the jewel release I think almost 8 years ago for storage. Both are pretty easy to manage.