13 ms·

Regardless of the merits or drawbacks of "de-clouding" for this particular company, it seems to me that their ops team is just really bored or permanently unsat

by Ameo 4y ago

Regardless of the merits or drawbacks of "de-clouding" for this particular company, it seems to me that their ops team is just really bored or permanently unsatisfied with any solution. They say that they've tried deploying their apps in all of: * Their own Datacenter * ECS * GKE * EKS * and now back to their own Datacenter Even with their new "de-clouded" deployment, it seems like they have created an absolutely immense amount of complexity to deploy what seems to be a variety of generic Ruby CRUD apps (I might be wrong but I didn't see anything different in the post). They have a huge list of tools and integrations that they've tried out with crazy names; Capistrano, Chef, mrsk, Filebeat, traefik... It seems well on par complexity-wise with a full K8s deploy with all the bells and whistles (logging, monitoring, networking, etc.) Google says that this company, 37signals, has 34 employees. This seems like such a monumental amount of orchestration and infra stuff unless they're deploying some crazy complex stuff they're not talking about. Idk what the lesson is here, if there is one, but this seems like a poor example to follow.

butt__hugger 4y ago
37signals and RoR have a habit of flip flopping on their decisions. See CoffeeScript.
- cutler 4y ago
  There's an operator for that.
- tptacek 4y ago
  Each of these "flip flops" probably lasted a good deal longer than the median 20+ person startup, so that seems pretty facile. But the parallel with CoffeeScript seems valid --- people on message boards are really not OK with nonstandard languages, and are never less happy than when a company they've heard something about does actual computer science of any kind. See, for instance, Fog Creek and Wasabi.
- mirekrusin 4y ago
  CoffeeScript impacted ECMAScript more than anything else.
jrumbut 4y ago
This is the company that gave birth to Ruby on Rails. They appear to have a culture of being very opinionated about their tools and unafraid of doing things their own way. Probably not an example most companies that size ought to follow but I'm glad they were crazy enough to do it!
tptacek 4y ago
We're talking about a product that has existed since 2004. They did: * Their own data center, before Docker existed * The non-K8s Docker way of deploying in AWS * The GCP, and then AWS, ways of doing K8s Docker * Docker in their own data center. For 20 years of deployment, that doesn't look crazy to me. The actual components of each environment are pretty standard. They wrote Capistrano, which predates Chef. Filebeat is a tiny part of ELK, which is the de facto standard logging stack. They use a smart reverse proxy, like... everybody else. It's easy to anything sound complicated if you expand the stack to as many acronyms as you can.
- andrewf 4y ago
  > * Their own data center, before Docker existed Also, it might be worth calling out: their product launched in 2004, Linode and Xen were launched in 2003, S3 and EC2 launched in 2007. The cloud as we know it today didn't exist when they started.
  - grogenaut 4y ago
    Pretty sure they knew the linode folks and were on there early iirc my history. This from hanging out with one of the linode owners back then randomly at a bar in stl
- ghaff 4y ago
  Whether DHH is "right" in some philosophical sense, this is a small company with a lot of technical experience in a variety of technologies and with presumably a lot of technical chops, so generalizing their experience to "cloud is good" or "cloud is bad" isn't really possible.
  - tptacek 4y ago
    I mean, I work for a cloud hosting vendor. I'm not saying one side or the other is right, only that people who are dunking on 37signals for this are telling on themselves.
- Rapzid 4y ago
klabb3 4y ago
I think you’re right that they doing it for fun or because they can, primarily. But I am excited to see them pioneer in this area, both because it’s more open and hacker friendly, but also because they’re moving the needle towards healthy competition amongst the providers. Our big-three cloud hegemony has already showed it’s ugly sides, both in terms of price (egress, anyone?) and quality (hello, zero interop and opaqueness). I’d argue we’ve seen significant complexity increases in especially server-side tech in the last 5-10 years, with relatively little to show for it, despite massive economic investments. I expect that trend to continue downwards unless we take back the hacker friendliness of infra & ops. PS. Actually scratch that I’m excited, that’s an understatement. I’m thrilled!
- jb_gericke 4y ago
  Pioneer? Other than rewriting docket swarm it sounds like a stack from the early 2000s…
- Melingo 4y ago
  Kubernetes is a poster child on open source and transparent development. And it pushed the needle tremendously. I'm lost on how you can compare this even.
spiralpolitik 4y ago
From my experiences there are two lessons: There is usually a sweet spot in terms of size where being on the public cloud makes sense, both from a cost and management perspective. Once you go above that size then having to manage IAM starting becoming a pain. Usually around the same point public cloud costs start becoming noticeable to your finance team and so you have to start dealing with questions around that. Usually that's a good point to do a sanity check before things get even bigger and more expensive. Similar k8s works well for certain classes of problem, but doesn't work well for all classes of problem. Any form of k8s has an operational overhead and you really need to make sure that you are going to get the ROI from the effort of maintaining the stack for it to be worthwhile.
- cyberax 4y ago
  > having to manage IAM starting becoming a pain Just use multiple AWS accounts. You won't need any complicated IAM policies.
hsn915 4y ago
This is very common in almost all web companies since around 2015. I've never seen a company with a simple infrastructure, no matter how simple their actual application actually is. If you choose a slow dynamic language (Ruby/Python) your deployment has to be massively complicated; you have no choice about it. For one simple reason: you will need a multitude of separate components to be made to work together. You need many application instances because there's no way one machine can handle all your traffic: Ruby is just too slow. A sharded database cluster as a source of truth: You went through the effort of making several applications instances with a load balancer: you don't want a single database server to be a single point of failure. A distributed redis/memcache index to accelerate queries and lower the pressure on the real database. You might have several index-type engines for different types of quries. Most people use ElasticSearch in addition to Redis. You need some system to manage all this complexity: monitor it, deploy new versions, rollback to a previous version, run migrations and monitor their state, etc etc. This is the bare minimum. Most people have a setup that is way more complicated than this. I don't even know how they come up with these complexties, but they not only come up with frequently: they love it! To them it's a feature, not a bug.
- hw 4y ago
  So, you’re explaining a stack with: - application instances - load balancer - database - cache - search cluster if application search is necessary Sounds like any cookie cutter application to me, even modern ones. How is that complicated?
  - kgeist 4y ago
    Sometimes I think, maybe our complex cluster which runs PHP software (load balancer, app instances, cache etc.) can be replaced with a single performant machine running something like Rust
    - hsn915 4y ago
      It can. You don't even need to go all the way to Rust. I'm doing it with Go, which has a GC and a runtime. A single executable on a single machine can handle millions of users per month.
jillesvangurp 4y ago
When I read stuff like this it strikes me that probably, by far, their largest operational expense is their staffing cost to orchestrate all of this. I come from a background of running small startups on a shoe string budget. I need to make tough choices when it comes to this stuff. I can either develop features or start spending double digit percentages of my development budget on devops. So, I aim to minimize cost and time (same thing) for all of that. At the same time, I appreciate things like observable software, rapid CI/CD cycles, and generally not having a lot of snow flakes as part of my deployment architecture. I actually have worked with a lot of really competent people over the past two decades and I like to think I'm not a complete noob on this front. In other words, I'm not a naive idiot but actually capable of making some informed choices here. That has lead me down a path of making very consistent choices over the years: 1) no kubernetes and no microservices. Microservices are Conways Law mapped to your deployment architecture. You don't need that if you do monoliths. And if you have a monolith, kubernetes is a waste of CPU, Memory, and development time. Complete overkill with zero added value. 2) the optimal size of a monolith deployment is 2 cheap VMs and a load balancer. You can run that for tens of dollars in most popular cloud environments. Good enough for zero down time deployments and having failover across availability zones. And you can scale it easily if needed (add more vms, bigger vms, etc.). 3) those two vms must not be snow flakes and be replaceable without fanfare, ceremony, or any manual intervention. So use docker and docker-compose on a generic linux host, preferably of the managed variety. Most developers can do a simple Dockerfile and wing it with docker-compose. It's not that hard. And it makes CI/CD really straight forward. Put the thing in the container registry, run the thing. Use something like Github actions to automate. Cheap and easy. 4) Use hosted/managed middleware (databases, search clusters, queues, etc). Running that stuff in some DIY setup is rarely worth the development time and operational overhead (devops, monitoring, backups, upgrades, etc). All this overhead rapidly adds up to costing more than years of paying for a managed solution. If you think in hours and market conform rates for people even capable of doing this stuff, that is. Provision the thing, use the thing, and pay tens of dollars per month for it. Absolute no brainer. When you hit thousands per month, you might dedicate some human resources to figuring out something cheaper. 5) Automate things that you do often. Don't automate things that you only do once (like creating a production environment). Congratulations, you just removed the need for having people do anything with teraform, cloudformation, chef, puppet, ansible, etc. Hiring people that can do those things is really expensive. And even though I can do all of those, it's literally not worth my time. Document it, but don't automate it unless you really need to and spend your money on feature development. But when I need to choose between hiring 1 extra developer or paying similarly expensive hosting bills, I'll prefer to have the extra developer on my team. Every time. Hosting bills can be an order of magnitude cheaper than a single developer on a monthly basis if you do it properly. For reference, we pay around 400/month for our production environment. That's in Google cloud and with an Elastic Cloud search cluster included. Other companies make other choices of course for all sorts of valid reasons. But these work fine for me and I feel good about the trade offs.
hw 4y ago
I wonder how much of these movements are them iterating and hunting for ROI in their infrastructure costs. Did GCP and AWS salespeople sell them on the benefits of the cloud, offer discounts, white glove migration help, showed some calculation on how much $$ they will save in the cloud, etc that on paper sounded great, but wasn’t ultimately a good fit? Their market is probably saturated and perhaps declining that they are reaching for optimizations elsewhere.
- Jnr 4y ago
  There is no such thing as "saving money in cloud". It is all about convenience and it always costs more than a smart team could achieve elsewhere. I tend to hear an argument that it is cheaper since you do not have to pay people to maintain those services, but in reality you still need that person to set up and maintain your particular cloud setup. And the services themselves are much much more expensive than maintaining your own servers in a data center. In my opinion cloud hosting and services are more meant for large corporations where no one wants to take responsibility and is scared of doing anything. Cloud is a nice way to shift the blame if/when things go bad - "but cloud is industry standard, everyone does it".
  - awesomegoat_com 4y ago
    Indeed. Hacker news crowd is drinking their own cool aid on this topic and not recognizing how much costs can be avoided if they just drop EKS from their stack. Remember that in SRE all the abstractions are leaky and thus having more abstractions means having more complexity not less.
    - awesomegoat_com 4y ago
      Yes, when that grows you can build a new corporate team babysitting control tower.
- ExoticPearTree 4y ago
  If you have a fairly stable traffic pattern, hosting your own stuff tends to be significantly cheaper than any cloud provider.
kodah 4y ago
> They have a huge list of tools and integrations that they've tried out with crazy names; Capistrano, Chef, mrsk, Filebeat, traefik These tools are pretty stock standard in the Systems Engineering world. I think anyone that's been a Systems Engineer that's over 30 has probably deployed every one of these. One thing I've learned over my mixed SWE and SE career is that infrastructure is expensive and grows regardless of revenue. I didn't truly appreciate this until I launched Kubernetes on Digital Ocean and began running my personal cloud on it. It was costing me over $100/m for very little. That money was gone whether I pushed a ton of VPN traffic over my mesh or not. It didn't care about how much I stored in the disk I reserved, and frankly, that cost was going to grow as time went on. I pulled the plug, setup servers in my house, wired up Traefik and Docker Compose V2 with a little Tailscale sprinkled on. The servers stay up to date with some scripts, I deploy new apps on select servers with Docker Compose and Docker profiles. It's possible for companies to do similar things, but not to the extremes I took it to. A really good infrastructure SWE generally goes for $300k. You can pay people with an expertise in these things who can streamline them and create maintainable products out of your infrastructure or you can pay for Legos and glue from a managed service provider like AWS, GCP, or Azure. At some point the latters costs will not scale, you'll pivot and cost reduce many times - maybe even begin rearchitecting. I think there's a lot of companies that are noe realizing the cheap money is gone, and the cloud has somewhat relied on cheap money.
mpweiher 4y ago
> Idk what the lesson is here I'd say the lesson is that we, as an industry, haven't figured out this "cloud" stuff yet. And it looks to me that what we need, roughly, is some sort of "deployment polymorphism" that separates interface from implementation.
datadeft 4y ago
> Idk what the lesson is here, if there is one, but this seems like a poor example to follow. The lesson is not to focus on tech tooling as much and focus more on product instead. Imagine this energy doing into the product....
shafyy 4y ago
In addition to what other commentators have stated: 37signals has ~ 80 employees, not 34.
dpkirchner 4y ago
> They have a huge list of tools and integrations that they've tried out with crazy names; Capistrano, Chef, mrsk, Filebeat, traefik I use a lot of this or similar (terraform instead of Chef, logstash instead of filebeat) and I'm a one person team. If anything these tools make my job a lot easier and less complex.

by Ameo 4y ago

butt__hugger 4y ago

37signals and RoR have a habit of flip flopping on their decisions. See CoffeeScript.

cutler 4y ago
There's an operator for that.
tptacek 4y ago
Each of these "flip flops" probably lasted a good deal longer than the median 20+ person startup, so that seems pretty facile. But the parallel with CoffeeScript seems valid --- people on message boards are really not OK with nonstandard languages, and are never less happy than when a company they've heard something about does actual computer science of any kind. See, for instance, Fog Creek and Wasabi.
mirekrusin 4y ago
CoffeeScript impacted ECMAScript more than anything else.

jrumbut 4y ago

This is the company that gave birth to Ruby on Rails. They appear to have a culture of being very opinionated about their tools and unafraid of doing things their own way. Probably not an example most companies that size ought to follow but I'm glad they were crazy enough to do it!

tptacek 4y ago

We're talking about a product that has existed since 2004. They did: * Their own data center, before Docker existed * The non-K8s Docker way of deploying in AWS * The GCP, and then AWS, ways of doing K8s Docker * Docker in their own data center. For 20 years of deployment, that doesn't look crazy to me. The actual components of each environment are pretty standard. They wrote Capistrano, which predates Chef. Filebeat is a tiny part of ELK, which is the de facto standard logging stack. They use a smart reverse proxy, like... everybody else. It's easy to anything sound complicated if you expand the stack to as many acronyms as you can.

andrewf 4y ago
> * Their own data center, before Docker existed Also, it might be worth calling out: their product launched in 2004, Linode and Xen were launched in 2003, S3 and EC2 launched in 2007. The cloud as we know it today didn't exist when they started.
- grogenaut 4y ago
  Pretty sure they knew the linode folks and were on there early iirc my history. This from hanging out with one of the linode owners back then randomly at a bar in stl
ghaff 4y ago
Whether DHH is "right" in some philosophical sense, this is a small company with a lot of technical experience in a variety of technologies and with presumably a lot of technical chops, so generalizing their experience to "cloud is good" or "cloud is bad" isn't really possible.
- tptacek 4y ago
  I mean, I work for a cloud hosting vendor. I'm not saying one side or the other is right, only that people who are dunking on 37signals for this are telling on themselves.
Rapzid 4y ago

klabb3 4y ago

I think you’re right that they doing it for fun or because they can, primarily. But I am excited to see them pioneer in this area, both because it’s more open and hacker friendly, but also because they’re moving the needle towards healthy competition amongst the providers. Our big-three cloud hegemony has already showed it’s ugly sides, both in terms of price (egress, anyone?) and quality (hello, zero interop and opaqueness). I’d argue we’ve seen significant complexity increases in especially server-side tech in the last 5-10 years, with relatively little to show for it, despite massive economic investments. I expect that trend to continue downwards unless we take back the hacker friendliness of infra & ops. PS. Actually scratch that I’m excited, that’s an understatement. I’m thrilled!

jb_gericke 4y ago
Pioneer? Other than rewriting docket swarm it sounds like a stack from the early 2000s…
Melingo 4y ago
Kubernetes is a poster child on open source and transparent development. And it pushed the needle tremendously. I'm lost on how you can compare this even.

spiralpolitik 4y ago

From my experiences there are two lessons: There is usually a sweet spot in terms of size where being on the public cloud makes sense, both from a cost and management perspective. Once you go above that size then having to manage IAM starting becoming a pain. Usually around the same point public cloud costs start becoming noticeable to your finance team and so you have to start dealing with questions around that. Usually that's a good point to do a sanity check before things get even bigger and more expensive. Similar k8s works well for certain classes of problem, but doesn't work well for all classes of problem. Any form of k8s has an operational overhead and you really need to make sure that you are going to get the ROI from the effort of maintaining the stack for it to be worthwhile.

cyberax 4y ago
> having to manage IAM starting becoming a pain Just use multiple AWS accounts. You won't need any complicated IAM policies.

hsn915 4y ago

This is very common in almost all web companies since around 2015. I've never seen a company with a simple infrastructure, no matter how simple their actual application actually is. If you choose a slow dynamic language (Ruby/Python) your deployment has to be massively complicated; you have no choice about it. For one simple reason: you will need a multitude of separate components to be made to work together. You need many application instances because there's no way one machine can handle all your traffic: Ruby is just too slow. A sharded database cluster as a source of truth: You went through the effort of making several applications instances with a load balancer: you don't want a single database server to be a single point of failure. A distributed redis/memcache index to accelerate queries and lower the pressure on the real database. You might have several index-type engines for different types of quries. Most people use ElasticSearch in addition to Redis. You need some system to manage all this complexity: monitor it, deploy new versions, rollback to a previous version, run migrations and monitor their state, etc etc. This is the bare minimum. Most people have a setup that is way more complicated than this. I don't even know how they come up with these complexties, but they not only come up with frequently: they love it! To them it's a feature, not a bug.

hw 4y ago
So, you’re explaining a stack with: - application instances - load balancer - database - cache - search cluster if application search is necessary Sounds like any cookie cutter application to me, even modern ones. How is that complicated?
- kgeist 4y ago
  Sometimes I think, maybe our complex cluster which runs PHP software (load balancer, app instances, cache etc.) can be replaced with a single performant machine running something like Rust
  - hsn915 4y ago
    It can. You don't even need to go all the way to Rust. I'm doing it with Go, which has a GC and a runtime. A single executable on a single machine can handle millions of users per month.

jillesvangurp 4y ago

When I read stuff like this it strikes me that probably, by far, their largest operational expense is their staffing cost to orchestrate all of this. I come from a background of running small startups on a shoe string budget. I need to make tough choices when it comes to this stuff. I can either develop features or start spending double digit percentages of my development budget on devops. So, I aim to minimize cost and time (same thing) for all of that. At the same time, I appreciate things like observable software, rapid CI/CD cycles, and generally not having a lot of snow flakes as part of my deployment architecture. I actually have worked with a lot of really competent people over the past two decades and I like to think I'm not a complete noob on this front. In other words, I'm not a naive idiot but actually capable of making some informed choices here. That has lead me down a path of making very consistent choices over the years: 1) no kubernetes and no microservices. Microservices are Conways Law mapped to your deployment architecture. You don't need that if you do monoliths. And if you have a monolith, kubernetes is a waste of CPU, Memory, and development time. Complete overkill with zero added value. 2) the optimal size of a monolith deployment is 2 cheap VMs and a load balancer. You can run that for tens of dollars in most popular cloud environments. Good enough for zero down time deployments and having failover across availability zones. And you can scale it easily if needed (add more vms, bigger vms, etc.). 3) those two vms must not be snow flakes and be replaceable without fanfare, ceremony, or any manual intervention. So use docker and docker-compose on a generic linux host, preferably of the managed variety. Most developers can do a simple Dockerfile and wing it with docker-compose. It's not that hard. And it makes CI/CD really straight forward. Put the thing in the container registry, run the thing. Use something like Github actions to automate. Cheap and easy. 4) Use hosted/managed middleware (databases, search clusters, queues, etc). Running that stuff in some DIY setup is rarely worth the development time and operational overhead (devops, monitoring, backups, upgrades, etc). All this overhead rapidly adds up to costing more than years of paying for a managed solution. If you think in hours and market conform rates for people even capable of doing this stuff, that is. Provision the thing, use the thing, and pay tens of dollars per month for it. Absolute no brainer. When you hit thousands per month, you might dedicate some human resources to figuring out something cheaper. 5) Automate things that you do often. Don't automate things that you only do once (like creating a production environment). Congratulations, you just removed the need for having people do anything with teraform, cloudformation, chef, puppet, ansible, etc. Hiring people that can do those things is really expensive. And even though I can do all of those, it's literally not worth my time. Document it, but don't automate it unless you really need to and spend your money on feature development. But when I need to choose between hiring 1 extra developer or paying similarly expensive hosting bills, I'll prefer to have the extra developer on my team. Every time. Hosting bills can be an order of magnitude cheaper than a single developer on a monthly basis if you do it properly. For reference, we pay around 400/month for our production environment. That's in Google cloud and with an Elastic Cloud search cluster included. Other companies make other choices of course for all sorts of valid reasons. But these work fine for me and I feel good about the trade offs.

hw 4y ago

I wonder how much of these movements are them iterating and hunting for ROI in their infrastructure costs. Did GCP and AWS salespeople sell them on the benefits of the cloud, offer discounts, white glove migration help, showed some calculation on how much $$ they will save in the cloud, etc that on paper sounded great, but wasn’t ultimately a good fit? Their market is probably saturated and perhaps declining that they are reaching for optimizations elsewhere.

Jnr 4y ago
There is no such thing as "saving money in cloud". It is all about convenience and it always costs more than a smart team could achieve elsewhere. I tend to hear an argument that it is cheaper since you do not have to pay people to maintain those services, but in reality you still need that person to set up and maintain your particular cloud setup. And the services themselves are much much more expensive than maintaining your own servers in a data center. In my opinion cloud hosting and services are more meant for large corporations where no one wants to take responsibility and is scared of doing anything. Cloud is a nice way to shift the blame if/when things go bad - "but cloud is industry standard, everyone does it".
- awesomegoat_com 4y ago
  Indeed. Hacker news crowd is drinking their own cool aid on this topic and not recognizing how much costs can be avoided if they just drop EKS from their stack. Remember that in SRE all the abstractions are leaky and thus having more abstractions means having more complexity not less.
  - awesomegoat_com 4y ago
    Yes, when that grows you can build a new corporate team babysitting control tower.
ExoticPearTree 4y ago
If you have a fairly stable traffic pattern, hosting your own stuff tends to be significantly cheaper than any cloud provider.

kodah 4y ago

> They have a huge list of tools and integrations that they've tried out with crazy names; Capistrano, Chef, mrsk, Filebeat, traefik These tools are pretty stock standard in the Systems Engineering world. I think anyone that's been a Systems Engineer that's over 30 has probably deployed every one of these. One thing I've learned over my mixed SWE and SE career is that infrastructure is expensive and grows regardless of revenue. I didn't truly appreciate this until I launched Kubernetes on Digital Ocean and began running my personal cloud on it. It was costing me over $100/m for very little. That money was gone whether I pushed a ton of VPN traffic over my mesh or not. It didn't care about how much I stored in the disk I reserved, and frankly, that cost was going to grow as time went on. I pulled the plug, setup servers in my house, wired up Traefik and Docker Compose V2 with a little Tailscale sprinkled on. The servers stay up to date with some scripts, I deploy new apps on select servers with Docker Compose and Docker profiles. It's possible for companies to do similar things, but not to the extremes I took it to. A really good infrastructure SWE generally goes for $300k. You can pay people with an expertise in these things who can streamline them and create maintainable products out of your infrastructure or you can pay for Legos and glue from a managed service provider like AWS, GCP, or Azure. At some point the latters costs will not scale, you'll pivot and cost reduce many times - maybe even begin rearchitecting. I think there's a lot of companies that are noe realizing the cheap money is gone, and the cloud has somewhat relied on cheap money.

mpweiher 4y ago

> Idk what the lesson is here I'd say the lesson is that we, as an industry, haven't figured out this "cloud" stuff yet. And it looks to me that what we need, roughly, is some sort of "deployment polymorphism" that separates interface from implementation.

datadeft 4y ago

> Idk what the lesson is here, if there is one, but this seems like a poor example to follow. The lesson is not to focus on tech tooling as much and focus more on product instead. Imagine this energy doing into the product....

shafyy 4y ago

In addition to what other commentators have stated: 37signals has ~ 80 employees, not 34.

dpkirchner 4y ago

> They have a huge list of tools and integrations that they've tried out with crazy names; Capistrano, Chef, mrsk, Filebeat, traefik I use a lot of this or similar (terraform instead of Chef, logstash instead of filebeat) and I'm a one person team. If anything these tools make my job a lot easier and less complex.