12 ms·

37signals Leaves the Cloud

285 points by klelatti 4y ago

vikeri 4y ago
Curious to know how to which degree they are self hosting, seems they'll buy their own servers but I doubt they'll go all the way to buying land and building their own data centers?
- Nextgrid 4y ago
  It would make little sense when there are plenty of bare-metal providers who are happy to provide this as a service at a reasonable price.
- wlll 4y ago
  When I was there, they rented space in a DC outside Chicago (near the airport), and later near NY for a backup location. The DC in Chicago was fantastic. We had a whole load of racks (about a row in the DC), and we bought our own servers. Running our own DC would have crossed the line, it's a whole huge thing you need to do and it only makes sense at scale if you want to do it well, Basecamp used a tiny fraction of the DC capacity.
  - ksec 4y ago
    >We had a whole load of racks A whole load of Racks for running Basecamp? We are talking about 42U per Rack, and total of 420U of Servers? The scale seems quite massive. At least to the idea / perception of what I had about Basecamp. Would be nice to see those specification and see how much of an improvement it is 10 years later and if we could fit those into 2-3 Racks.
    - wlll 4y ago
      *whoops, this got long* Basecamp started on one single Rackspace server, before I was there. I started at 20 people, left at ~55. When I left there were I don't remember how many racks exactly, but more than 10, less than 30 in the primary location. 42U of servers in each. There was a mix of a whole load of (Dell, never pay list price!) blade servers, DB appliances (~12 in total across ~6 apps ISTR), Isilon storage[0][1], F5 kit, juniper routers etc. etc. We had some epically fast storage in some of the servers for the time, way faster than SSDs. Later we added two more sites. One in I think Virginia, one in NY. The one in Virginia was a replica of what we needed to run Basecamp, the one in NY was a half-rack data replication location (I think I got that the right way round). We had 10G fibre (we rented wavelengths not the whole fibre) between each location. We could lose one DC and remain RW for our block data, 2 DCs and we'd have to drop down to RO. Block data was things like uploads, so DBs, search etc. wouldn't have been affected. We could lose one of the /main/ DCs and still be RW for everything. With all this kit we were able to run both main DCs hot. With our Geo DNS you could hit either of our DCs and you'd get served pages, you could even write to both locations. One DC was always the "RO" DC, it always replicated the databases. If you tried to write to that DC we proxied your request to the RW DC over our 10G links and proxied any more requests you made for n seconds to the RW DC too, at which point we reverted you back to the RO DC. Now, NY to Virginia isn't that far, so why bother with the hot/hot config? Because it played into the rest of our plan, which was DC failover. With some pretty epic voodoo (Juniper/F5/OpenResty etc.) we could fail over the datacentres, swapping the RO and RW locations. We could also do this if one of the locations was unavailable. We could do this in 4 seconds /without losing a single in-flight request/ (we tested it). This ended up a bit longer than I was intending, but it illustrates a few things: - I think people underestimate Basecamp. It's /huge/ (money and users, not employees). Not so much in the tech world (anymore), but even with this kit, even with the (at the time 6) sysadmins that maintained it, it still made a shit load of cash. I guestimated the net-worth of the two owners as in the hundreds of millions of dollars each, entirely because of basecamp. Cash that as a privately owned company all went to the owners (who gave some of it to us, they treated us fairly well). I think it was Patio11 who said that people under estimated the market for software, it's easy to do, these aren't human-scale numbers. - The owners are right about keeping it private. You might not make "larger yacht than Larry Ellison" money, but you sure as hell have a lot higher chance of making "pretty damn big yacht and not having to work again" sort of money. If I was rolling the dice I know which I'd gamble on. - I don't think I ever actually calculated it, but the amount of money our servers were worth while running was /immense/. When you went to the DC and looked at the row of racks, you could kinda see just how dense the $ value there was. The efficiencies that the cloud brings gives money to AWS, not to the clients. - All of this was still /way/ cheaper than running cloud infra. I'm not against the cloud (I use it), but we made more because we had this infra than if we were on the cloud. We were dense when it came to # customers per $ spent on infra. - The flexibility we had because we controlled everything was also extraordinary. It came at a cost, we had to do it ourselves, but we had a /lot/ of power to make things work the way we wanted to. - We had access to everything, from the network (hell, even the light :) ) to the JS. This gave us optimisation options not available to a lot of people. Network topologies, buying the right CPUs (we benchmarked them), configuring the CPUs, etc. etc. [0] An Isilon storage engineer once dropped one of these on the floor, while it was powered on and the drives were running. It took out a metal floor tile. [1] They were a PITA, everyone hated them. There was a running joke among the sysadmins that when we decommissioned them we would take one on an Ops meet and use it as target practice.
      - lamontcg 4y ago
        That is about the amazon.com architecture as late as 2006 (only slightly higher scale of three entire datacenters scattered around virginia and I think dual 40G links between datacenters, but other than that exactly the same principles). Lot of people think that DC failover needs to be east coast / west coast but you can achieve most of your DC redundancy goals separated by 100 miles or less and have a lot lower latency, higher bandwidth and lower costs. Might want to think about different geographic flood plains and different power companies / grids. Could still nerd fight about EMPs from nuclear war and a sufficiently massive hurricane or an earthquake out here on the west coast, but at some point you need to accept some risks.
      - ksec 4y ago
        The reason was that I remember DHH stating they were doing 2K RPS with 30 App Server in 2015 ~ 2016. And they were on one Primary DB. ( At least that was what I jot down in my notes ). I was assuming they could fit multiple "App Server" Node inside a single 1U Blade. But even if it was 1U per App Server, that would be 30U + likely a Powerful 4U DB Monster. Along with probably some Cache instances. Even if the number above does not include redundancy, that still only makes it 2 Racks of Servers with spare. 30 Racks is a lot. What am I missing here? Apart from Storage.
        flakeoil 4y ago
        Do you guys define racks the same way?
        wlll 4y ago
        I think so, 42U of full-depth server capacity.
        wlll 4y ago
        The place I work (CTO for now, looking for opportunities pretty soon) can sustain nearly 400 RPS through a Rails app on a single Performance-L Heroku dyno without a sweat (though we run two min), but it's the tip of the iceberg. The infra to support those web servers is way more than that. Aurora Postgres x 3 for now, a large ES cluster, 2x redis clusters, memcached etc. Bear in mind that we had I think 6 apps. Basecamp 1, 2 and 3 (all separate infra), Highrise, Campfire, some other internal stuff. Our ES cluster was pretty damn big, redis and memcached too. Juniper, F5, network switches (rack infra) etc. Storage was pretty big, quite a few 4u servers with spinning rust. The blade servers were I think 6 blades in 2u. Everything was redundant, everything had 2x or more. I honestly can't remember how many racks we had. "More than 10" is my hand-wavey guess. I don't think DHH was being disingenuous with his "2k on 30 servers" message, it was likely more about the scalability of Rails rather than the infra required to run the app.
- Abe9000 4y ago
  Who told you they need to build a massive data center? They are NOT RENTING SERVERS but buying space for the ones they use. With the budget they can afford to buy super beefy rack servers that occupy no space. With a single rack they can have a ton of late tech servers.
klelatti 4y ago
Could anyone expand on why it makes sense for Netflix to use AWS so extensively?
- Nextgrid 4y ago
  The bulk of the media is served from appliances they give out to ISPs, so their AWS costs might be reasonable. It's also a huge company and a massive distributed system, so moving that off from AWS is an insane amount of effort and there might be legitimate benefits from AWS' management and autoscaling layer that aren't as valuable for 37Signals which runs simpler, monolithic systems.
  - klelatti 4y ago
    Thanks - I'm guessing that they get a great deal from AWS too.
  - mch82 4y ago
    Additionally, Netflix falls into one category of website DHH writes about: unpredictable, significant bursts of traffic. When they publish Tiger King during the pandemic there’s a spike in streaming no one could have predicted. When people finish the show it goes away. The arrival time of the next breakout show is likely uncertain, even if it can be modeled using a statistical distribution.
    - wlll 4y ago
      I worked at Basecamp/37signals up until 2015. We had peaks and troughs, with as you might predict traffic ramping at ET morning, and tailing off PT afternoon/evening. We had some European traffic, but it was substantially less than the US. It still cost /far/ less to buy/rack/power etc. our servers 100% of the time than it would have done to use the cloud and scale up and down with demand. There is definitely a situation when bursty requirements will make cloud the more obvious answer, but given the near order of magnitude cost increase for cloud it's not as clear cut as just seeing a wave on a graph :)
- bigiain 4y ago
commitpizza 4y ago
Very interesting post! I think there will be a movement away from the cloud in the coming time. People will realize that the cloud comes with it's problems when regulations come down, governments require companies to spy on them etc. The only way of being really sure of what is going on is to own it yourself. Unfortunately that has become more and more important.
- Breza 4y ago
  I think there will be more competition for specific types of cloud needs. AWS, GCP, and Azure offer every service you could ever need in one place. At the other extreme, you can pay a colo facility to rack the servers you own. I think the in-between market has the potential to grow. Cheaper than AWS but without all the drawbacks of true self-hosting.
senttoschool 4y ago
So their AWS bill is $500k/year. Hire one person to buy, configure, manage hardware is what? $250k/year? And is this self-hosting more reliable than AWS? For an email service, you'd think that you want the most reliable host. Weird.
- brenogazzola 4y ago
  Their RDS and ES are 500k. They still need to pay for EC2, S3 and any other services they might be using, like ELB and Elasticache. So probably higher than that. He also did not mention if they are going to self-host, colocate, or rent bare metal in a more traditional host (like softlayer). Probably colocate as I think that’s how they used to run? Both your points are still valid though.
  - senttoschool 4y ago
    So they'd need to hire a person to setup and manage hardware. And another person to setup and manage RDS, ES, etc. If it's $250k each, that's already $500k. Unless they can find someone who can do both for... $300k? I don't know. He doesn't provide enough details to make the math work.
    - 988747 4y ago
      $250k is Sillicon Valley rate. In Europe similarly skilled people earn less than that, typically $80k.
      - pmontra 4y ago
        Even $25k if you look at small companies in southern Europe.
        danwee 4y ago
        That's just for juniors. Seniors are more around $60K-$70K (although, it's still a low number of US standards, yeah)
mattbrewsbytes 4y ago
Very valid points. Its hard to describe the complexities and weird error or latency or other conditions that happen in cloud environments. OPs people kinda shrug and ask that you essentially turn it off and back on again and that usually resolves the issue at that time. There are so many abstraction layers when all the infrastructure is software run and I bet a lot of those introduce performance degradation. Another element of this is I would think its more fun for a systems person to be closer to the hardware. Specifying, ordering and unboxing servers, disks, switches and setting things up sounds satisfying to me.
- senttoschool 4y ago
  >Its hard to describe the complexities and weird error or latency or other conditions that happen in cloud environments. As oppose to complexities of self-host using self purchased hardware?
  - Nextgrid 4y ago
    They are much less frequent, and at the very least you have full access to the hardware to debug if needed. Can you get me access to the AWS control plane's logs so I can understand why my RDS instance is stuck in "modifying" for 12+ hours?
    - senttoschool 4y ago
      AWS has over a decade of experience and a ton of full-time dev ops to make complicated cloud usage easy. You're not going to beat the reliability by going inhouse.
      - BackBlast 4y ago
        If something goes wrong at AWS, and it does. You have no power over recovery. The more of the stack you own, the more such ability you do have. If you run a competent organization, this can be a boon. If you have serious deficiencies, it may well be the opposite.
      - 4y ago
thrillgore 4y ago
Interesting. I helped put together a cost estimates for Amazon Workspaces for our development infrastructure and we quickly found it costing us up to a quarter million compared to moving it to a colocated server. AWS is not looking that affordable anymore.
- kristianpaul 4y ago
  AWS is never affordable, but offers interesting commodities as a service that these days many companies could not afford to build or maintain themselves. So it's a matter of choice.
frankfrank13 4y ago
When we got containers and cloud providers at the same time, we somehow collectively threw the baby out with the bathwater. I suspect this will be a trend -- https://pulse2.com/how-bank-of-america-is-saving-2-billion-every-year-by-building-its-own-cloud/ https://pulse2.com/how-bank-of-america-is-saving-2-billion-e...
rr808 4y ago
> If one of the primary AWS regions go down, seemingly half the internet is offline along with it. This is not what DARPA designed! This is a good point. I really dont like the centralization on AWS.
- karmakaze 4y ago
  That's really only us-east-1. It's a special pet region, don't use it.
  - dijit 4y ago
    Some backplane stuff is only in us-east-1, which gets brought up every time there's a us-east-1 outage.
karmakaze 4y ago
I worked at a hybrid shop that had main MySQL and MongoDB databases on bare metal rented servers and all the new extra features/services built in AWS. It was a pretty good split. We did have some human cost in handling hardware failures and scaling of the hardware but it was way less than cloud hosting for that much CPU and RAM would have been as well as access to higher performance machines. I'm glad that most of the infrastructure handling scripts in Ansible and Chef were made before I got there and only had to make a few new specialized ones.
- lastofus 4y ago
  I’d think the latency between the app servers and the DBs would be a problem, given they didn’t live in the same data center. How was that mitigated?
  - karmakaze 4y ago
    Oh right, we did have many app servers located with the DBs as well. They were stateless and numerous so could be anywhere. The datacentre with the bare metal did have dedicated links to the nearby AWS region we were using. At one point I think one of their redundant links got severed on purpose by accident, so things were slow for a while. The microservices in AWS were feature-services that had their own datastores so only paid a single overall request latency than the many app-server to db queries per user request.
homarp 4y ago
good to read along https://news.ycombinator.com/item?id=33257204 https://news.ycombinator.com/item?id=33257204 "81% of IT teams directed to reduce or halt cloud spending by C-suite"
adrianmsmith 4y ago
> The cloud excels at two ends of the spectrum ... The first end is when your application is so simple and low traffic that you really do save on complexity by starting with fully managed services .. The second is when your load is highly irregular I think there's a third, which is when the company is so rich it just doesn't matter. Then saving even a little bit of employee time (which are more likely to be the bottleneck in such a company) is worth basically infinite spend. I worked at a place where they spent $6k/month on cloud, just for a Java app not doing much plus a PostgreSQL database. I came up with a way to save $2k/month (better indexes), when I did it nobody cared or even noticed I think. I know that $2k/month might not sound like much, but I think the same principle would have been true even if it'd been much more. They spent about $2m/year on an external software development agency to create a React project, full of inefficient scrum meetings and billable hours from project managers and so on, and they didn't really care about that either.
- flakeoil 4y ago
  Sounds like pure stupidity and incompetence. Why not buy a bunch of Ferrari's and beat them up with sledge hammers while they are at it?
  - adrianmsmith 4y ago
    I think if you're working at a company that makes a huge amount of money, it just isn't really rational to care that much about spend. I think of it like this: If a person had $100m in the bank, I reckon they're not going to care whether their beer costs $7.00 or $7.10. There are just more important things to worry about.
    - flakeoil 4y ago
      Sure between $7.10 and $7.00, who cares, but now we are talking more like $7 or $70. If I had $100m I would prefer to pay $7 for the beer and spend the remaining $63 on something more useful.
- pfhayes 4y ago
uvdn7 4y ago
Managing machine life cycles, data intensive and stateful systems is very hard. I was hoping to see more concrete examples/data points instead of just saying we can also run the same software on our own machines.
Abe9000 4y ago
DEATH TO THE CLOUD. For far too long they have profited out of the ignorance of those who do not want to learn how to virtualize, or host their own site. Have you seen the specs of the server you rent? A 2 GHZ core? Really? Time to spend real money in a real hardware. Fiber optic has dropped latency to really low levels. You don't need a super close geolocation server anymore. With the money they spend on the cloud, they can rent small offices all over the world and have a small rack server with beefy servers in a secure server room they can build.
WhyNotHugo 4y ago
The premise of the cloud is actually a good idea: if you only use the server for 1cpu-hours per day, you should only pay that, and let others use it for the other 23cpu-hours in the day. The executions we've had so far are terrible. There's huge markup, piles of complex abstractions, and all the other issues mentioned in this article. When I was a teen, I used to pay this guy in town like $1.5 a month for a user on a *nix machine where I could run scripts and run my own IRC bouncer and stuff. A bunch of people shared this machine (mostly running IRC bouncers, AFAIK). I've no idea now many, it was stable, and nobody was eating up all the CPU. It worked. I feel that this nostalgic memory of the past is closer to a utopic "cloud" than anything any modern provider is offering.
- tomaskafka 4y ago
  You just described a low budget shared VPS. I’m running everything on these for a few EUR per month :).
  - aliqot 4y ago
    Vultr has a good 2 dollar tier for this, my only gripe being that they're like Hetzner and will shut your shit down at random in the first 6 months and ask you for your passport or ID or a pic of you holding it. This has happened to me personally on Vultr and Hetzner.
    - tomaskafka 4y ago
      I understand why they need to do it, and am fine with that.
thinkpad13 4y ago
I also think the same, I want metal self-hosted, anyone have tips how to do the security too?
tomaskafka 4y ago
Why Weathergraph (my hourly forecast app for iOS) was never in the cloud: Because I’d be dumb to spend hundreds of dollars and engineering complexity for a stable load service, when a 20 EUR VPS (10 EUR until recently - see, scaling!) runs everything I need (forecast proxy, geocoding, website, payment verification) just fine :). (Link: https://apps.apple.com/cz/app/weathergraph-weather-widget/id1501958576 https://apps.apple.com/cz/app/weathergraph-weather-widget/id... )
papito 4y ago
The problem with the cloud is that it makes things easy. Just throw it in, eh? Developers keep adding toys that cost money, and then at some point, someone at the top notices the bill, and it's all hands on deck. The cost due diligence should happened before anything, not when your ass is on fire and you need to save money yesterday. "Do we need that beefy RDS instance?" ¯\_(ツ)_/¯ "Why does our ElasticCache data keep growing at such a high rate and nothing is expiring?" ¯\_(ツ)_/¯ "Could X be done much cheaper with Y?" ¯\_(ツ)_/¯ "Why did we spin up DynamoDB for this if we already have Postgres?". ¯\_(ツ)_/¯
ericraio 4y ago
It almost seems like their is an opportunity to build a cloud hosting service, where mom and pop data centers can plug in their infrastructure to a cloud service and let the platform manage the renting and latency based routing. If the software is developed enough, this enables an even more global distribution. Software services don't want to manage machines and data centers don't want to manage customers.
khebbie 4y ago
I didn’t listen to the accompanying podcast so I might be missing some details. It strikes me that it seems like 37signals are just using AWS as a glorified virtual machine with maybe some db hosting. To me the big benefit of the cloud is using all the little services they provide, like s3, sns, sqs, CloudSearch ECS, cloudwatch and the list goes on and on. If we were to host all those service ourselves and keep them up to date the operating cost would be huge. Furthermore I work for a site that is very public and has a lot of for instance ddos threats hanging over head. AWS handles lots of ddos threats all the time and they a much better suited for handling that than our own ops team.
tritiy 4y ago
I've always considered that you pay extra for the cloud for scalability and security. First one you may not need but the second one is a must.

285 points by klelatti 4y ago

Curious to know how to which degree they are self hosting, seems they'll buy their own servers but I doubt they'll go all the way to buying land and building their own data centers?

Nextgrid 4y ago
It would make little sense when there are plenty of bare-metal providers who are happy to provide this as a service at a reasonable price.
wlll 4y ago
When I was there, they rented space in a DC outside Chicago (near the airport), and later near NY for a backup location. The DC in Chicago was fantastic. We had a whole load of racks (about a row in the DC), and we bought our own servers. Running our own DC would have crossed the line, it's a whole huge thing you need to do and it only makes sense at scale if you want to do it well, Basecamp used a tiny fraction of the DC capacity.
- ksec 4y ago
  >We had a whole load of racks A whole load of Racks for running Basecamp? We are talking about 42U per Rack, and total of 420U of Servers? The scale seems quite massive. At least to the idea / perception of what I had about Basecamp. Would be nice to see those specification and see how much of an improvement it is 10 years later and if we could fit those into 2-3 Racks.
  - wlll 4y ago
    *whoops, this got long* Basecamp started on one single Rackspace server, before I was there. I started at 20 people, left at ~55. When I left there were I don't remember how many racks exactly, but more than 10, less than 30 in the primary location. 42U of servers in each. There was a mix of a whole load of (Dell, never pay list price!) blade servers, DB appliances (~12 in total across ~6 apps ISTR), Isilon storage[0][1], F5 kit, juniper routers etc. etc. We had some epically fast storage in some of the servers for the time, way faster than SSDs. Later we added two more sites. One in I think Virginia, one in NY. The one in Virginia was a replica of what we needed to run Basecamp, the one in NY was a half-rack data replication location (I think I got that the right way round). We had 10G fibre (we rented wavelengths not the whole fibre) between each location. We could lose one DC and remain RW for our block data, 2 DCs and we'd have to drop down to RO. Block data was things like uploads, so DBs, search etc. wouldn't have been affected. We could lose one of the /main/ DCs and still be RW for everything. With all this kit we were able to run both main DCs hot. With our Geo DNS you could hit either of our DCs and you'd get served pages, you could even write to both locations. One DC was always the "RO" DC, it always replicated the databases. If you tried to write to that DC we proxied your request to the RW DC over our 10G links and proxied any more requests you made for n seconds to the RW DC too, at which point we reverted you back to the RO DC. Now, NY to Virginia isn't that far, so why bother with the hot/hot config? Because it played into the rest of our plan, which was DC failover. With some pretty epic voodoo (Juniper/F5/OpenResty etc.) we could fail over the datacentres, swapping the RO and RW locations. We could also do this if one of the locations was unavailable. We could do this in 4 seconds /without losing a single in-flight request/ (we tested it). This ended up a bit longer than I was intending, but it illustrates a few things: - I think people underestimate Basecamp. It's /huge/ (money and users, not employees). Not so much in the tech world (anymore), but even with this kit, even with the (at the time 6) sysadmins that maintained it, it still made a shit load of cash. I guestimated the net-worth of the two owners as in the hundreds of millions of dollars each, entirely because of basecamp. Cash that as a privately owned company all went to the owners (who gave some of it to us, they treated us fairly well). I think it was Patio11 who said that people under estimated the market for software, it's easy to do, these aren't human-scale numbers. - The owners are right about keeping it private. You might not make "larger yacht than Larry Ellison" money, but you sure as hell have a lot higher chance of making "pretty damn big yacht and not having to work again" sort of money. If I was rolling the dice I know which I'd gamble on. - I don't think I ever actually calculated it, but the amount of money our servers were worth while running was /immense/. When you went to the DC and looked at the row of racks, you could kinda see just how dense the $ value there was. The efficiencies that the cloud brings gives money to AWS, not to the clients. - All of this was still /way/ cheaper than running cloud infra. I'm not against the cloud (I use it), but we made more because we had this infra than if we were on the cloud. We were dense when it came to # customers per $ spent on infra. - The flexibility we had because we controlled everything was also extraordinary. It came at a cost, we had to do it ourselves, but we had a /lot/ of power to make things work the way we wanted to. - We had access to everything, from the network (hell, even the light :) ) to the JS. This gave us optimisation options not available to a lot of people. Network topologies, buying the right CPUs (we benchmarked them), configuring the CPUs, etc. etc. [0] An Isilon storage engineer once dropped one of these on the floor, while it was powered on and the drives were running. It took out a metal floor tile. [1] They were a PITA, everyone hated them. There was a running joke among the sysadmins that when we decommissioned them we would take one on an Ops meet and use it as target practice.
    - lamontcg 4y ago
      That is about the amazon.com architecture as late as 2006 (only slightly higher scale of three entire datacenters scattered around virginia and I think dual 40G links between datacenters, but other than that exactly the same principles). Lot of people think that DC failover needs to be east coast / west coast but you can achieve most of your DC redundancy goals separated by 100 miles or less and have a lot lower latency, higher bandwidth and lower costs. Might want to think about different geographic flood plains and different power companies / grids. Could still nerd fight about EMPs from nuclear war and a sufficiently massive hurricane or an earthquake out here on the west coast, but at some point you need to accept some risks.
    - ksec 4y ago
      The reason was that I remember DHH stating they were doing 2K RPS with 30 App Server in 2015 ~ 2016. And they were on one Primary DB. ( At least that was what I jot down in my notes ). I was assuming they could fit multiple "App Server" Node inside a single 1U Blade. But even if it was 1U per App Server, that would be 30U + likely a Powerful 4U DB Monster. Along with probably some Cache instances. Even if the number above does not include redundancy, that still only makes it 2 Racks of Servers with spare. 30 Racks is a lot. What am I missing here? Apart from Storage.
      - flakeoil 4y ago
        Do you guys define racks the same way?
        wlll 4y ago
        I think so, 42U of full-depth server capacity.
      - wlll 4y ago
        The place I work (CTO for now, looking for opportunities pretty soon) can sustain nearly 400 RPS through a Rails app on a single Performance-L Heroku dyno without a sweat (though we run two min), but it's the tip of the iceberg. The infra to support those web servers is way more than that. Aurora Postgres x 3 for now, a large ES cluster, 2x redis clusters, memcached etc. Bear in mind that we had I think 6 apps. Basecamp 1, 2 and 3 (all separate infra), Highrise, Campfire, some other internal stuff. Our ES cluster was pretty damn big, redis and memcached too. Juniper, F5, network switches (rack infra) etc. Storage was pretty big, quite a few 4u servers with spinning rust. The blade servers were I think 6 blades in 2u. Everything was redundant, everything had 2x or more. I honestly can't remember how many racks we had. "More than 10" is my hand-wavey guess. I don't think DHH was being disingenuous with his "2k on 30 servers" message, it was likely more about the scalability of Rails rather than the infra required to run the app.
Abe9000 4y ago
Who told you they need to build a massive data center? They are NOT RENTING SERVERS but buying space for the ones they use. With the budget they can afford to buy super beefy rack servers that occupy no space. With a single rack they can have a ton of late tech servers.

klelatti 4y ago

Could anyone expand on why it makes sense for Netflix to use AWS so extensively?

Nextgrid 4y ago
The bulk of the media is served from appliances they give out to ISPs, so their AWS costs might be reasonable. It's also a huge company and a massive distributed system, so moving that off from AWS is an insane amount of effort and there might be legitimate benefits from AWS' management and autoscaling layer that aren't as valuable for 37Signals which runs simpler, monolithic systems.
- klelatti 4y ago
  Thanks - I'm guessing that they get a great deal from AWS too.
- mch82 4y ago
  Additionally, Netflix falls into one category of website DHH writes about: unpredictable, significant bursts of traffic. When they publish Tiger King during the pandemic there’s a spike in streaming no one could have predicted. When people finish the show it goes away. The arrival time of the next breakout show is likely uncertain, even if it can be modeled using a statistical distribution.
  - wlll 4y ago
    I worked at Basecamp/37signals up until 2015. We had peaks and troughs, with as you might predict traffic ramping at ET morning, and tailing off PT afternoon/evening. We had some European traffic, but it was substantially less than the US. It still cost /far/ less to buy/rack/power etc. our servers 100% of the time than it would have done to use the cloud and scale up and down with demand. There is definitely a situation when bursty requirements will make cloud the more obvious answer, but given the near order of magnitude cost increase for cloud it's not as clear cut as just seeing a wave on a graph :)
bigiain 4y ago

commitpizza 4y ago

Very interesting post! I think there will be a movement away from the cloud in the coming time. People will realize that the cloud comes with it's problems when regulations come down, governments require companies to spy on them etc. The only way of being really sure of what is going on is to own it yourself. Unfortunately that has become more and more important.

Breza 4y ago
I think there will be more competition for specific types of cloud needs. AWS, GCP, and Azure offer every service you could ever need in one place. At the other extreme, you can pay a colo facility to rack the servers you own. I think the in-between market has the potential to grow. Cheaper than AWS but without all the drawbacks of true self-hosting.

senttoschool 4y ago

So their AWS bill is $500k/year. Hire one person to buy, configure, manage hardware is what? $250k/year? And is this self-hosting more reliable than AWS? For an email service, you'd think that you want the most reliable host. Weird.

brenogazzola 4y ago
Their RDS and ES are 500k. They still need to pay for EC2, S3 and any other services they might be using, like ELB and Elasticache. So probably higher than that. He also did not mention if they are going to self-host, colocate, or rent bare metal in a more traditional host (like softlayer). Probably colocate as I think that’s how they used to run? Both your points are still valid though.
- senttoschool 4y ago
  So they'd need to hire a person to setup and manage hardware. And another person to setup and manage RDS, ES, etc. If it's $250k each, that's already $500k. Unless they can find someone who can do both for... $300k? I don't know. He doesn't provide enough details to make the math work.
  - 988747 4y ago
    $250k is Sillicon Valley rate. In Europe similarly skilled people earn less than that, typically $80k.
    - pmontra 4y ago
      Even $25k if you look at small companies in southern Europe.
      - danwee 4y ago
        That's just for juniors. Seniors are more around $60K-$70K (although, it's still a low number of US standards, yeah)

mattbrewsbytes 4y ago

Very valid points. Its hard to describe the complexities and weird error or latency or other conditions that happen in cloud environments. OPs people kinda shrug and ask that you essentially turn it off and back on again and that usually resolves the issue at that time. There are so many abstraction layers when all the infrastructure is software run and I bet a lot of those introduce performance degradation. Another element of this is I would think its more fun for a systems person to be closer to the hardware. Specifying, ordering and unboxing servers, disks, switches and setting things up sounds satisfying to me.

senttoschool 4y ago
>Its hard to describe the complexities and weird error or latency or other conditions that happen in cloud environments. As oppose to complexities of self-host using self purchased hardware?
- Nextgrid 4y ago
  They are much less frequent, and at the very least you have full access to the hardware to debug if needed. Can you get me access to the AWS control plane's logs so I can understand why my RDS instance is stuck in "modifying" for 12+ hours?
  - senttoschool 4y ago
    AWS has over a decade of experience and a ton of full-time dev ops to make complicated cloud usage easy. You're not going to beat the reliability by going inhouse.
    - BackBlast 4y ago
      If something goes wrong at AWS, and it does. You have no power over recovery. The more of the stack you own, the more such ability you do have. If you run a competent organization, this can be a boon. If you have serious deficiencies, it may well be the opposite.
    - 4y ago

thrillgore 4y ago

Interesting. I helped put together a cost estimates for Amazon Workspaces for our development infrastructure and we quickly found it costing us up to a quarter million compared to moving it to a colocated server. AWS is not looking that affordable anymore.

kristianpaul 4y ago
AWS is never affordable, but offers interesting commodities as a service that these days many companies could not afford to build or maintain themselves. So it's a matter of choice.

frankfrank13 4y ago

When we got containers and cloud providers at the same time, we somehow collectively threw the baby out with the bathwater. I suspect this will be a trend -- https://pulse2.com/how-bank-of-america-is-saving-2-billion-every-year-by-building-its-own-cloud/ https://pulse2.com/how-bank-of-america-is-saving-2-billion-e...

rr808 4y ago

> If one of the primary AWS regions go down, seemingly half the internet is offline along with it. This is not what DARPA designed! This is a good point. I really dont like the centralization on AWS.

karmakaze 4y ago
That's really only us-east-1. It's a special pet region, don't use it.
- dijit 4y ago
  Some backplane stuff is only in us-east-1, which gets brought up every time there's a us-east-1 outage.

karmakaze 4y ago

I worked at a hybrid shop that had main MySQL and MongoDB databases on bare metal rented servers and all the new extra features/services built in AWS. It was a pretty good split. We did have some human cost in handling hardware failures and scaling of the hardware but it was way less than cloud hosting for that much CPU and RAM would have been as well as access to higher performance machines. I'm glad that most of the infrastructure handling scripts in Ansible and Chef were made before I got there and only had to make a few new specialized ones.

lastofus 4y ago
I’d think the latency between the app servers and the DBs would be a problem, given they didn’t live in the same data center. How was that mitigated?
- karmakaze 4y ago
  Oh right, we did have many app servers located with the DBs as well. They were stateless and numerous so could be anywhere. The datacentre with the bare metal did have dedicated links to the nearby AWS region we were using. At one point I think one of their redundant links got severed on purpose by accident, so things were slow for a while. The microservices in AWS were feature-services that had their own datastores so only paid a single overall request latency than the many app-server to db queries per user request.

homarp 4y ago

good to read along https://news.ycombinator.com/item?id=33257204 https://news.ycombinator.com/item?id=33257204 "81% of IT teams directed to reduce or halt cloud spending by C-suite"

adrianmsmith 4y ago

> The cloud excels at two ends of the spectrum ... The first end is when your application is so simple and low traffic that you really do save on complexity by starting with fully managed services .. The second is when your load is highly irregular I think there's a third, which is when the company is so rich it just doesn't matter. Then saving even a little bit of employee time (which are more likely to be the bottleneck in such a company) is worth basically infinite spend. I worked at a place where they spent $6k/month on cloud, just for a Java app not doing much plus a PostgreSQL database. I came up with a way to save $2k/month (better indexes), when I did it nobody cared or even noticed I think. I know that $2k/month might not sound like much, but I think the same principle would have been true even if it'd been much more. They spent about $2m/year on an external software development agency to create a React project, full of inefficient scrum meetings and billable hours from project managers and so on, and they didn't really care about that either.

flakeoil 4y ago
Sounds like pure stupidity and incompetence. Why not buy a bunch of Ferrari's and beat them up with sledge hammers while they are at it?
- adrianmsmith 4y ago
  I think if you're working at a company that makes a huge amount of money, it just isn't really rational to care that much about spend. I think of it like this: If a person had $100m in the bank, I reckon they're not going to care whether their beer costs $7.00 or $7.10. There are just more important things to worry about.
  - flakeoil 4y ago
    Sure between $7.10 and $7.00, who cares, but now we are talking more like $7 or $70. If I had $100m I would prefer to pay $7 for the beer and spend the remaining $63 on something more useful.
pfhayes 4y ago

uvdn7 4y ago

Managing machine life cycles, data intensive and stateful systems is very hard. I was hoping to see more concrete examples/data points instead of just saying we can also run the same software on our own machines.

Abe9000 4y ago

DEATH TO THE CLOUD. For far too long they have profited out of the ignorance of those who do not want to learn how to virtualize, or host their own site. Have you seen the specs of the server you rent? A 2 GHZ core? Really? Time to spend real money in a real hardware. Fiber optic has dropped latency to really low levels. You don't need a super close geolocation server anymore. With the money they spend on the cloud, they can rent small offices all over the world and have a small rack server with beefy servers in a secure server room they can build.

WhyNotHugo 4y ago

The premise of the cloud is actually a good idea: if you only use the server for 1cpu-hours per day, you should only pay that, and let others use it for the other 23cpu-hours in the day. The executions we've had so far are terrible. There's huge markup, piles of complex abstractions, and all the other issues mentioned in this article. When I was a teen, I used to pay this guy in town like $1.5 a month for a user on a *nix machine where I could run scripts and run my own IRC bouncer and stuff. A bunch of people shared this machine (mostly running IRC bouncers, AFAIK). I've no idea now many, it was stable, and nobody was eating up all the CPU. It worked. I feel that this nostalgic memory of the past is closer to a utopic "cloud" than anything any modern provider is offering.

tomaskafka 4y ago
You just described a low budget shared VPS. I’m running everything on these for a few EUR per month :).
- aliqot 4y ago
  Vultr has a good 2 dollar tier for this, my only gripe being that they're like Hetzner and will shut your shit down at random in the first 6 months and ask you for your passport or ID or a pic of you holding it. This has happened to me personally on Vultr and Hetzner.
  - tomaskafka 4y ago
    I understand why they need to do it, and am fine with that.

thinkpad13 4y ago

I also think the same, I want metal self-hosted, anyone have tips how to do the security too?

tomaskafka 4y ago

Why Weathergraph (my hourly forecast app for iOS) was never in the cloud: Because I’d be dumb to spend hundreds of dollars and engineering complexity for a stable load service, when a 20 EUR VPS (10 EUR until recently - see, scaling!) runs everything I need (forecast proxy, geocoding, website, payment verification) just fine :). (Link: https://apps.apple.com/cz/app/weathergraph-weather-widget/id1501958576 https://apps.apple.com/cz/app/weathergraph-weather-widget/id... )

papito 4y ago

The problem with the cloud is that it makes things easy. Just throw it in, eh? Developers keep adding toys that cost money, and then at some point, someone at the top notices the bill, and it's all hands on deck. The cost due diligence should happened before anything, not when your ass is on fire and you need to save money yesterday. "Do we need that beefy RDS instance?" ¯\_(ツ)_/¯ "Why does our ElasticCache data keep growing at such a high rate and nothing is expiring?" ¯\_(ツ)_/¯ "Could X be done much cheaper with Y?" ¯\_(ツ)_/¯ "Why did we spin up DynamoDB for this if we already have Postgres?". ¯\_(ツ)_/¯

ericraio 4y ago

It almost seems like their is an opportunity to build a cloud hosting service, where mom and pop data centers can plug in their infrastructure to a cloud service and let the platform manage the renting and latency based routing. If the software is developed enough, this enables an even more global distribution. Software services don't want to manage machines and data centers don't want to manage customers.

khebbie 4y ago

I didn’t listen to the accompanying podcast so I might be missing some details. It strikes me that it seems like 37signals are just using AWS as a glorified virtual machine with maybe some db hosting. To me the big benefit of the cloud is using all the little services they provide, like s3, sns, sqs, CloudSearch ECS, cloudwatch and the list goes on and on. If we were to host all those service ourselves and keep them up to date the operating cost would be huge. Furthermore I work for a site that is very public and has a lot of for instance ddos threats hanging over head. AWS handles lots of ddos threats all the time and they a much better suited for handling that than our own ops team.

tritiy 4y ago

I've always considered that you pay extra for the cloud for scalability and security. First one you may not need but the second one is a must.