6 ms·

It's five grand a day to miss our S3 exit

187 points by ksec 2y ago

ksec 2y ago
I wonder how many Racks would there be after the total Cloud Exit from Basecamp. It is a nice way to think about scale. The whole business that used to be 1/5 of the size 10 years ago were running on 100s of racks. And today it could be less than 10.
- mindcrash 2y ago
  David wrote about that earlier. They are running on just about 8 racks in each DC (and they use 2). That might exclude the new Pure Storage setup, though. https://world.hey.com/dhh/we-stand-to-save-7m-over-five-years-from-our-cloud-exit-53996caa https://world.hey.com/dhh/we-stand-to-save-7m-over-five-year...
  - ksec 2y ago
    I think the Pure Storage is at best half a rack. I dont think it is 8 rack per DC, it is 4 rack per DC. >We currently spend about $60,000/month on eight dedicated racks between our two data centers And they are only doing 64 Core. We will have 256 Zen 6 core next year. And it seems by next year, if they were willing to pay for density they could have fitted everything inside one rack per DC. Exciting times. Edit: Actually if Intel were to push 18A on server it would make performance / density even better.
    - nijave 2y ago
      >Actually if Intel were to push 18A on server it would make performance / density even better Don't you end up needing pretty insane electric capacity not to mention cooling for that kind of density?
rvz 2y ago
> $1.5 million/year That is excessive and it's already $4K a day. Lots of teams really underestimate cloud costs since there is an assumption that the hundreds of millions they are raising will give them enough runway to survive a few years despite losing money for years. Even scaling would be somewhat of an issue depending on the tech stack. Imagine the cost of running standard Java micro-services and the "solution" was to "spin up hundreds of more nodes". The worst that I have seen was a bank proudly having up to 8,000 - 10,000 separate micro-services. Just imagine the daily cost of that. Unjustifiable. But of course the AWS cloud consultants would be happy to shill you their offerings at "cheap" prices, but in reality the pricing is designed for you to accumulate millions in costs as you scale on the tiniest amount of usage, even for testing. So before you build the software, one must think about the costs of scaling if it becomes widely used rather than taking the easy approach in just spinning up nodes and increasing more costs and act as if you don't have the capital to solve the problem. You can only do that for so long until you don't.
- whstl 2y ago
  I remember that at a previous company somehow it leaked that the AWS cost was 50% of all the developers staff salary. There was nowhere near the same volume of data as Basecamp/Hey, not there was much processing power needed. It was purely bad engineering accumulated over 10 years.
  - lpapez 2y ago
    I was once contracted to work on a project where the monthly GCP bill for Postgres was $60k per month - this was basically my YEARLY rate at that time, just for managed Postgres. After some time I was quite familiar with their stack and had gathered considerable domain experience. This led to an idea how to halve the database load (and the cost would presumably fall by a similar percentage), which I wanted to use as leverage during contract renegotiation. I boldly offered to work for free to halve their database load, in exchange for being paid half the money this optimization would save over the course of one year. This would basically triple my pay, and they would still save money. They declined, and I moved to a better opportunity. Last I heard they had to pay a team of 4 new consultants for a year to implement the same idea I had. Without the domain knowledge, consultants couldn't progress as fast as I suspect I could have done (my estimated was 2 months of work). I know it's very petty, but I regret revealing too many implementation details of the idea during the pitch and allowing the company to contract other consultants to see it done.
nodesocket 2y ago
Any idea what software they are using for the s3 data migration? rclone?
- roxolotl 2y ago
  Yea I’d love to know the answer to this. Of course you can just download petabytes of data but I imagine there’s gotta be more to it than just rclone. Or maybe it is that simple.
  - Symbiote 2y ago
    I used rclone to migrate about 2PB of data between datacentres last year. There were some ways to make it go a bit faster, found by reading the rclone manual, but otherwise no surprises. I wasn't sure what the maximum transfer speed could have been, but as one side was still the production system I didn't care to reach the limit anyway. Over 10Gb/s anyway.
    - nodesocket 2y ago
      Were the files lots of smaller files or lots of large files? I imagine if lots of small files requires a lot of concurrent threads to utilize a 10Gbps connection.
      - Symbiote 2y ago
        Lots of large files, I think most would have been over 50MB. There was a folder of smaller files to move, around a million files each less than 100kB. That took longer, but not long enough that I bothered to try and speed it up.
comrade1234 2y ago
Years ago (10+) when I was deciding to use aws or hosting our own hardware in a colo for my company I found a spreadsheet where you plug in your hardware and hosting costs and the current Amazon pricing for their various products and it would give you how long to break-even on the initial investment. It was always just above three years to break even. After we decided to self-host I still kept tracking the prices in the spreadsheet and as hardware costs fluctuated Amazon adjusted their prices to match. I figured someone at Amazon must have had the same spreadsheet I was working with.
- rco8786 2y ago
  I'm sure Amazon has mountains of data to arrive at that target number. But subjectively, it also feels about right in terms of juicing just the most profit margin from customers without being obviously overpriced. It's a nice "sweet spot" where it's just too far out on the horizon for companies, especially SMBs, to really account for or to think the up front cost/effort is currently worth it vs putting that effort into revenue generating activities. Hardware cycle is probably about 3 years also.
  - Gigachad 2y ago
    Business needs change often enough that it’s hard to justify locking something in for three years vs paying a small amount extra for flexibility.
- spockz 2y ago
  Is that including or excluding write off? If you have to replace the hardware every 3 years then it would be equivalent. IIRC hardware is replaced every 3-5 years because otherwise it is out of support.
  - j45 2y ago
    Write off is am accounting term, not operational. Hardware like cars and laptop can continue to perform after they are written off, or even after the warranty. The grade of hardware used is critical in servers. Hyperscaling might mean commodity based servers. Hosting a large app does not mean using commodity component servers. Hardware, when self hosting, does not need to be replaced every 3-5 years because it does not fail every 3-5 years. Depends on load and a bunch of factors. Why? We wouldn’t buy the cheap and disposable components a massive cloud or social media network might use to scale faster because they have a massive budget. Besides, do providers really replace all their servers every 3-5 years? Hosting companies don’t seem to. The cloud is many multiples more expensive than self hosting especially at scale. Hosting and cloud tools have brought down labour costs tremendously. For the hardware, with the extremely clean environments servers run in, plus much cleaner electricity hardware runs much longer. Purchase actual enterprise grade servers (HP Proliant, etc) that a company would buy for themselves for maximum reliability (compared to the commodity based ones of clouds) and those have so much reliability built into them that they sometimes never die. You can still buy used proliant servers many, many, many generations old and they hum along just fine. It is bizarre but not. Support is a few things. Warranty on parts and software. Extended support options which amounts to (hardware warranty) are always available for a fee, and achievable on your own. If your software is a hypervisor you will be mirrored. If a server has an issue the affected machine moves the load elsewhere. The server has hot swap equipment. Takes a few moments to swap components if needed. If you are self hosting, you can buy a used server or two or theee to have a backup and mirror and spare parts. It’s like buying a few NUCs. Hosting corporately can be done not just with buying, but leasing too (meaning hardware swapping can happen). Add to this moving older equipment to less demanding tasks (if they ever do stay at load)z
andrewaylett 2y ago
I'd like to note that they're not swapping like-for-like here. The differences might not be material for the use-case, and AWS probably don't even offer a product that does exactly what the Pure Storage system does, but S3 has a lot more resiliency as well as other features. That doesn't mean they're wrong to move, it means you need to be careful to make sure that you pay for what you need, and try to avoid paying extra if all it gives you is stuff you don't want. I value the extra functionality, so I'm not moving my data off S3.
- diggan 2y ago
  I wish people were as careful when thinking about adopting S3 as they seemingly are careful when others move away from S3. Somewhere along the line, people starting defaulting to "at least 3 nodes for backend" and "cloud services for all infrastructure" even if the product they're building haven't even found product market fit. Sure, if you know for a fact that your traffic will go and down more than 50% during a normal day, go for something that scales up and down quickly. But for most other use cases, the extra cost of cloud doesn't really make much financial sense, unless you're a fat VC-funded startup cat.
  - gavinray 2y ago
    That reminds me of a friend who ran a SaaS business off a Heztner VPC. It was a Node.js app he deployed via SSH and ran under a systemd job. Used directories of JSON files as a database and the business logic was handled by a single endpoint that took JSON RPC payloads with different action types and metadata. The app scaled to ~10,000 daily users like this.
    - stavros 2y ago
      I run multiple SaaSes, some popular, some not so much, off a single $20/mo Hetzner VPS with Dokku on it. It works great, and I've never needed to worry about anything. Meanwhile, I see friends working on MVPs with 1-2 non-paying customers who already have costs in the thousands of dollars a month, but "it's fine because we got free money for a year". Yes, but that means that your company now has an expiration date of a year.
deleted 2y ago
[deleted]
Hnrobert42 2y ago
I wonder if the savings includes the cost of labor to maintain the physical servers, cabling, performance and security security monitoring, etc. Not saying it doesn't, I just wonder.
- znpy 2y ago
  > I wonder if the savings includes the cost of labor to maintain the physical servers, cabling, This appears to be a valid point, but it really isn't. In a case you're paying sysadmins, in the other you're paying cloud engineers. > performance and security security monitoring, etc. Not saying it doesn't, I just wonder. You'd be paying those anyways. AWS's mantra is that amazon takes care OF the security of the cloud but YOU (the client company) take care of the security IN the cloud. Same goes for performance etc.
  - exe34 2y ago
    from listening to people who work with aws talking about their work, I don't get the impression they save any time - it's just different stuff. they're constantly doing things with permissions, setting up something or other, trying to work out why something can't find something else - it doesn't just magically solve all your problems. presumably once you figure out everything for you 2000 customers you can scale to a billion with just your credit card info, but most people won't work on stuff that scales to billions of users.
    - vidarh 2y ago
      Indeed, I offer consultancy for both cloud-based clients and ones using managed hosting like Hetzner, and colo'ed own servers, and the clients who host in the cloud routinely need more of my time on a per-server/instance basis. Most of the time when people bring up these costs they have no experience with modern server hardware, and hosting. You can have the server shipped directly to a colo, you can file tickets with the provider to have it plugged into your power bar and your network, and you can connect with your IPMI client and set it to boot from your PXE/bootp/tftp server to get an install image. With a well-configured setup, you have a one-off cost to set up your firewalls, wire-up the switches and power bars, and set up a server for the rest to network boot off, and the rest of bringing up your servers is near automatic and most management can be done remotely via IPMI or similar. It's not the 90's any more.
christina97 2y ago
I very much appreciate the concrete numbers in this post!
RadiozRadioz 2y ago
I thought you could pay Amazon to send you a van full of hard disks if you need to transfer that much data.
- jarito 2y ago
  Pretty sure those services are only for transferring data into S3, not out.
  - byefruit 2y ago
    https://aws.amazon.com/snowball/pricing/ https://aws.amazon.com/snowball/pricing/ snowball seems to support getting data out of S3 though you still end up paying extortionate egress charges.
    - dilyevsky 2y ago
      They wave egress if you’re leaving for 60 days, at least when you migrate via standard api
ifightcrime 2y ago
Cloud has always been more expensive. I remember being quoted 250k/month for bandwidth when I was paying 15k with rackspace 10+ years ago. You’re paying for convenience and speed. The Math stops working when you grow to a certain point. You can mitigate this to some extent by making some key architecture + vendor decisions upfront when first building… or just consider that some day you’ll need to do things like this. It’s not a novel problem.
- vidarh 2y ago
  It's horrifyingly hard to convince people of this, though, even you can present them with actual numbers. A lot of people have convinced themselves that cloud is cheap, to the point that they don't even do a cursory investigation. A lot of those even don't do the bare minimum to reduce hosting costs within the cloud they choose, or choose one of the cheaper clouds (AWS is absolutely extortionate for anything that requires significant amount of outbound bandwith), or put caching/CDN's in front (you can trivially slash your AWS egress costs dramatically). Most of my consultancy work is on driving cost efficiencies for cloud, and I can usually safely guarantee the fee will pay for itself within months because people don't fix even the most low hanging fruit.
  - mbreese 2y ago
    I think one business argument for cloud is capital expenses vs operational expenses. If you’re (over) paying for cloud resources vs an in house option (or colo), those are numbers that are a straight expense. When you own hardware, those are on your books until they depreciate off. For some businesses, that can make sense. Now, a good accountant probably wouldn’t care one way or the other. Debits and credits balance either way. And spending more still means less profit in the long term, no matter how it looks on the books. But, in addition to the flexibility, that was what I always thought of as the main cloud benefit. It’s the same with leasing vs buying cars/computers/etc…
    - tialaramex 2y ago
      Cloud made sense for the startup I worked for previously. If you are a startup then a $1M per year expense makes much more sense than a $5M up front purchase with 5-10 years of life - in five years you might be billionaires or you might be bankrupt and until then the Cloud was better.
tgtweak 2y ago
The frustrating part is they'll give you $1500/day back again if you let them know you're leaving... Very annoying tbh.
calderwoodra 2y ago
Their current rate is about ~20% cheaper (4k/day), so missing the deadline isn't that big of a deal.
DeathArrow 2y ago
Using SSD seems like a waste of money. I think having a raid like structure with good old spinning hard disks is good enough. Does Amazon use SSD only for S3? I know backblaze uses HDDs and some raid like addressing.
- nijave 2y ago
  Not sure Amazon posts much about S3 architecture but some years ago I think they had an article about using spinners. They had so many HDDs, though, that 100-250 IOPs * <the insane number of total drives> meant they still had insane throughput. Not sure they're at the scale in the article to get good performance with HDDs but someone with a storage eng background can correct me. NVMe seems like a waste, though. I imagine you can still get solid performance with SATA/SAS (250 to 50-80k IOPs is still a pretty huge leap)
Axsuul 2y ago
Cloud is expensive but hardware failures are at least handled gracefully. With coloc you'd have some serious downtime. That means you'd need to get to a certain level of redundancy in order to have coloc make sense. I'd love to move to coloc for my SaaS but it doesn't feel as resilient. Please correct me if I'm wrong as I'd love to move off the cloud.
- nijave 2y ago
  Not sure cloud is necessarily more resilient--imo it's less resilient. On the other hand, it's fully automated with robust APIs so there's easy tools to mitigate failures like node/machine sets (scale sets, scaling groups, auto scaling groups, whatever the provider calls them) You could use an orchestration solution to help handle automatic failover. There's a handful of container-based options from heavy duty Kubernetes to Docker Swarm and Nomad. Containers are nice since you can bypass most of the host management where you only need basic security patching and installation of your container runtime. There's also k8s distros like OpenShift to make k8s setup easier if you go that route.
  - Axsuul 2y ago
    Yep I use orchestration (Nomad) but still you would need hardware redundancy. For example, the database server is currently a single point of failure. In the cloud, if there's a hardware failure, it will simply go down and come back up with a new instance. In coloc, you'd need to have the data center debug and replace hardware which means extended downtime.
    - NorwegianDude 2y ago
      When using colocation, nothing is stopping people from storing the database data externally from the server running the database like some cloud services do. But doing so, either in cloud or not, does have a serious downside: greatly increased latency.
      - Axsuul 2y ago
albert_e 2y ago
Even if they move only part of the data before the deadline ..they stop paying for that the moment they delete it from S3. (Doesn't work that way in reverse.) They can start saving thousands of dollars even before the deadline if they are able to start moving as soon as their own infra is up and incrementally move and delete data from S3. If their data consumers can work with that, that is. If part of the data does not need the full S3 guarantees of durability and availability ..they could probably save more by using cheaper tiers while on S3.
siscia 2y ago
What are the stats of the new system? Cost per giga? Latency curve on read? Latency curve on write? Support for concurrency? Max throughput? I think the article is somehow shallow, but S3 is a very deep and flexible system. Of course, someone will just need it's basic features, but someone else may be interested in advanced features as well.
hbogert 2y ago
if vendors like Pure weren't so incredibly irritating with the documentation behind a paywall i'd actually recommend them to clients. What happens now is, a client already has pure, and as a contractor I can't get access to documentation. Seriously, what year is it.
- otterley 2y ago
  If someone is hiding something, it means they have something to hide.
theptip 2y ago
Would love to see their growth rates, but Basecamp seems like a canonical example of where renting makes the least sense. They are bootstrapped and not hyperscaling, so they don’t need flexibility for rapid and unpredictable growth (VC backed startups do). And they have a strong engineering org, so they don’t benefit as much from buying reliability (large non-tech companies often do).
FireBeyond 2y ago
> But those big numbers always seem a bit abstract to me. The idea of paying $5,000/day, if we miss our departure date, is awfully concrete in comparison. Says the guy who owns multiple hypercars, some of which are $3M+ (Aston Martin Valkyrie)...
otterley 2y ago
Is Hey not able to take advantage of S3 intelligent tiering? It’s way cheaper than S3 standard for infrequently accessed objects. I wouldn’t be surprised if Hey’s access patterns lend themselves well to it. (Yes, I’m aware that Hey’s decision to evacuate the cloud is a fait accompli, but I also can’t help but wonder if there are loads of potential savings that are being left out of the discussion.)
npalli 2y ago
DHH's focus getting massive saving on storage costs is a head-scratcher. Usually, egress costs are punitive. Given LLM's are pretty good now, I modeled the break-even point of such a move from S3 (6PB) to a colo. Breakeven time 18 years(!!), even with 8 year depreciation cycle still looking at 8 years to breakeven. This is not including falling S3 costs in Amazon. Truth be told he is probably spending 10's of millions on servers running Rails (SSR), probably can cut that in half by moving to golang or java. This is peanuts in comparison. 1. Initial costs of $2.5M to provision the hardware (disks, servers, enclosures, networking equipment, redundancy, software solutions etc.) 2. Facility OPEX: $50,000/month (Power, connective, monitoring etc..) 3. Staffing and Operation tools: $10,000 / month 4. Replacement cycle of 5 years so assume $500,000/year ~ $41K/month
- bauruine 2y ago
  >2. Facility OPEX: $50,000/month (Power, connective, monitoring etc..) Are you sure? It looks like they only have 1 rack (+1 in another facility for redundancy) and seem to have 40Gbit/s connectivity. A full rack is in the range of 1k, connectivity around 600$ per 10Gbit/s. I have no idea how much power they consume but I doubt it's 40k$+ per month for a storage workload. I would guess they are in the 10k$ range. Those are only list prices I've seen in the wild so take it with a grain of salt but 50k seem VERY high.
  - omnimus 2y ago
    The power must be way less especially without spinning drives.
  - npalli 2y ago
    1 rack is a recipe for failure, you need to split, even without that consider how many SSD's you need. power usage for all those SSD's inter-site connectivity (need to keep transferring data between the sites otherwise customers are going to be very surprised. maintenance and software costs (at the colo level) All these add up, closer to $50K than some $2K (LOL). The way you guys (below) are talking, this is not some home server that serves personal videos. It runs (some) business operations for thousands of small/medium companies.
killer32 2y ago
We’ve been running Rook-Ceph in production across multiple client environments. In one example, we built a setup with 8 refurbished Dell servers (128GB RAM, 8–14 JBOD disks each) over 10G networking. It supports geo-replication between sites and has been stable for over 2 years. Total hardware cost was under $100k. Rook simplifies the operational overhead of Ceph quite a lot, especially in Kubernetes-native stacks. For teams with large data and HA requirements, it's been a solid on-prem alternative. SLA-backed managed services are also becoming more common, which helps reduce the operational burden even further.

187 points by ksec 2y ago

ksec 2y ago

I wonder how many Racks would there be after the total Cloud Exit from Basecamp. It is a nice way to think about scale. The whole business that used to be 1/5 of the size 10 years ago were running on 100s of racks. And today it could be less than 10.

mindcrash 2y ago
David wrote about that earlier. They are running on just about 8 racks in each DC (and they use 2). That might exclude the new Pure Storage setup, though. https://world.hey.com/dhh/we-stand-to-save-7m-over-five-years-from-our-cloud-exit-53996caa https://world.hey.com/dhh/we-stand-to-save-7m-over-five-year...
- ksec 2y ago
  I think the Pure Storage is at best half a rack. I dont think it is 8 rack per DC, it is 4 rack per DC. >We currently spend about $60,000/month on eight dedicated racks between our two data centers And they are only doing 64 Core. We will have 256 Zen 6 core next year. And it seems by next year, if they were willing to pay for density they could have fitted everything inside one rack per DC. Exciting times. Edit: Actually if Intel were to push 18A on server it would make performance / density even better.
  - nijave 2y ago
    >Actually if Intel were to push 18A on server it would make performance / density even better Don't you end up needing pretty insane electric capacity not to mention cooling for that kind of density?

rvz 2y ago

> $1.5 million/year That is excessive and it's already $4K a day. Lots of teams really underestimate cloud costs since there is an assumption that the hundreds of millions they are raising will give them enough runway to survive a few years despite losing money for years. Even scaling would be somewhat of an issue depending on the tech stack. Imagine the cost of running standard Java micro-services and the "solution" was to "spin up hundreds of more nodes". The worst that I have seen was a bank proudly having up to 8,000 - 10,000 separate micro-services. Just imagine the daily cost of that. Unjustifiable. But of course the AWS cloud consultants would be happy to shill you their offerings at "cheap" prices, but in reality the pricing is designed for you to accumulate millions in costs as you scale on the tiniest amount of usage, even for testing. So before you build the software, one must think about the costs of scaling if it becomes widely used rather than taking the easy approach in just spinning up nodes and increasing more costs and act as if you don't have the capital to solve the problem. You can only do that for so long until you don't.

whstl 2y ago
I remember that at a previous company somehow it leaked that the AWS cost was 50% of all the developers staff salary. There was nowhere near the same volume of data as Basecamp/Hey, not there was much processing power needed. It was purely bad engineering accumulated over 10 years.
- lpapez 2y ago
  I was once contracted to work on a project where the monthly GCP bill for Postgres was $60k per month - this was basically my YEARLY rate at that time, just for managed Postgres. After some time I was quite familiar with their stack and had gathered considerable domain experience. This led to an idea how to halve the database load (and the cost would presumably fall by a similar percentage), which I wanted to use as leverage during contract renegotiation. I boldly offered to work for free to halve their database load, in exchange for being paid half the money this optimization would save over the course of one year. This would basically triple my pay, and they would still save money. They declined, and I moved to a better opportunity. Last I heard they had to pay a team of 4 new consultants for a year to implement the same idea I had. Without the domain knowledge, consultants couldn't progress as fast as I suspect I could have done (my estimated was 2 months of work). I know it's very petty, but I regret revealing too many implementation details of the idea during the pitch and allowing the company to contract other consultants to see it done.

nodesocket 2y ago

Any idea what software they are using for the s3 data migration? rclone?

roxolotl 2y ago
Yea I’d love to know the answer to this. Of course you can just download petabytes of data but I imagine there’s gotta be more to it than just rclone. Or maybe it is that simple.
- Symbiote 2y ago
  I used rclone to migrate about 2PB of data between datacentres last year. There were some ways to make it go a bit faster, found by reading the rclone manual, but otherwise no surprises. I wasn't sure what the maximum transfer speed could have been, but as one side was still the production system I didn't care to reach the limit anyway. Over 10Gb/s anyway.
  - nodesocket 2y ago
    Were the files lots of smaller files or lots of large files? I imagine if lots of small files requires a lot of concurrent threads to utilize a 10Gbps connection.
    - Symbiote 2y ago
      Lots of large files, I think most would have been over 50MB. There was a folder of smaller files to move, around a million files each less than 100kB. That took longer, but not long enough that I bothered to try and speed it up.

comrade1234 2y ago

Years ago (10+) when I was deciding to use aws or hosting our own hardware in a colo for my company I found a spreadsheet where you plug in your hardware and hosting costs and the current Amazon pricing for their various products and it would give you how long to break-even on the initial investment. It was always just above three years to break even. After we decided to self-host I still kept tracking the prices in the spreadsheet and as hardware costs fluctuated Amazon adjusted their prices to match. I figured someone at Amazon must have had the same spreadsheet I was working with.

rco8786 2y ago
I'm sure Amazon has mountains of data to arrive at that target number. But subjectively, it also feels about right in terms of juicing just the most profit margin from customers without being obviously overpriced. It's a nice "sweet spot" where it's just too far out on the horizon for companies, especially SMBs, to really account for or to think the up front cost/effort is currently worth it vs putting that effort into revenue generating activities. Hardware cycle is probably about 3 years also.
- Gigachad 2y ago
  Business needs change often enough that it’s hard to justify locking something in for three years vs paying a small amount extra for flexibility.
spockz 2y ago
Is that including or excluding write off? If you have to replace the hardware every 3 years then it would be equivalent. IIRC hardware is replaced every 3-5 years because otherwise it is out of support.
- j45 2y ago
  Write off is am accounting term, not operational. Hardware like cars and laptop can continue to perform after they are written off, or even after the warranty. The grade of hardware used is critical in servers. Hyperscaling might mean commodity based servers. Hosting a large app does not mean using commodity component servers. Hardware, when self hosting, does not need to be replaced every 3-5 years because it does not fail every 3-5 years. Depends on load and a bunch of factors. Why? We wouldn’t buy the cheap and disposable components a massive cloud or social media network might use to scale faster because they have a massive budget. Besides, do providers really replace all their servers every 3-5 years? Hosting companies don’t seem to. The cloud is many multiples more expensive than self hosting especially at scale. Hosting and cloud tools have brought down labour costs tremendously. For the hardware, with the extremely clean environments servers run in, plus much cleaner electricity hardware runs much longer. Purchase actual enterprise grade servers (HP Proliant, etc) that a company would buy for themselves for maximum reliability (compared to the commodity based ones of clouds) and those have so much reliability built into them that they sometimes never die. You can still buy used proliant servers many, many, many generations old and they hum along just fine. It is bizarre but not. Support is a few things. Warranty on parts and software. Extended support options which amounts to (hardware warranty) are always available for a fee, and achievable on your own. If your software is a hypervisor you will be mirrored. If a server has an issue the affected machine moves the load elsewhere. The server has hot swap equipment. Takes a few moments to swap components if needed. If you are self hosting, you can buy a used server or two or theee to have a backup and mirror and spare parts. It’s like buying a few NUCs. Hosting corporately can be done not just with buying, but leasing too (meaning hardware swapping can happen). Add to this moving older equipment to less demanding tasks (if they ever do stay at load)z

andrewaylett 2y ago

I'd like to note that they're not swapping like-for-like here. The differences might not be material for the use-case, and AWS probably don't even offer a product that does exactly what the Pure Storage system does, but S3 has a lot more resiliency as well as other features. That doesn't mean they're wrong to move, it means you need to be careful to make sure that you pay for what you need, and try to avoid paying extra if all it gives you is stuff you don't want. I value the extra functionality, so I'm not moving my data off S3.

diggan 2y ago
I wish people were as careful when thinking about adopting S3 as they seemingly are careful when others move away from S3. Somewhere along the line, people starting defaulting to "at least 3 nodes for backend" and "cloud services for all infrastructure" even if the product they're building haven't even found product market fit. Sure, if you know for a fact that your traffic will go and down more than 50% during a normal day, go for something that scales up and down quickly. But for most other use cases, the extra cost of cloud doesn't really make much financial sense, unless you're a fat VC-funded startup cat.
- gavinray 2y ago
  That reminds me of a friend who ran a SaaS business off a Heztner VPC. It was a Node.js app he deployed via SSH and ran under a systemd job. Used directories of JSON files as a database and the business logic was handled by a single endpoint that took JSON RPC payloads with different action types and metadata. The app scaled to ~10,000 daily users like this.
  - stavros 2y ago
    I run multiple SaaSes, some popular, some not so much, off a single $20/mo Hetzner VPS with Dokku on it. It works great, and I've never needed to worry about anything. Meanwhile, I see friends working on MVPs with 1-2 non-paying customers who already have costs in the thousands of dollars a month, but "it's fine because we got free money for a year". Yes, but that means that your company now has an expiration date of a year.

deleted 2y ago

[deleted]

Hnrobert42 2y ago

I wonder if the savings includes the cost of labor to maintain the physical servers, cabling, performance and security security monitoring, etc. Not saying it doesn't, I just wonder.

znpy 2y ago
> I wonder if the savings includes the cost of labor to maintain the physical servers, cabling, This appears to be a valid point, but it really isn't. In a case you're paying sysadmins, in the other you're paying cloud engineers. > performance and security security monitoring, etc. Not saying it doesn't, I just wonder. You'd be paying those anyways. AWS's mantra is that amazon takes care OF the security of the cloud but YOU (the client company) take care of the security IN the cloud. Same goes for performance etc.
- exe34 2y ago
  from listening to people who work with aws talking about their work, I don't get the impression they save any time - it's just different stuff. they're constantly doing things with permissions, setting up something or other, trying to work out why something can't find something else - it doesn't just magically solve all your problems. presumably once you figure out everything for you 2000 customers you can scale to a billion with just your credit card info, but most people won't work on stuff that scales to billions of users.
  - vidarh 2y ago
    Indeed, I offer consultancy for both cloud-based clients and ones using managed hosting like Hetzner, and colo'ed own servers, and the clients who host in the cloud routinely need more of my time on a per-server/instance basis. Most of the time when people bring up these costs they have no experience with modern server hardware, and hosting. You can have the server shipped directly to a colo, you can file tickets with the provider to have it plugged into your power bar and your network, and you can connect with your IPMI client and set it to boot from your PXE/bootp/tftp server to get an install image. With a well-configured setup, you have a one-off cost to set up your firewalls, wire-up the switches and power bars, and set up a server for the rest to network boot off, and the rest of bringing up your servers is near automatic and most management can be done remotely via IPMI or similar. It's not the 90's any more.

christina97 2y ago

I very much appreciate the concrete numbers in this post!

RadiozRadioz 2y ago

I thought you could pay Amazon to send you a van full of hard disks if you need to transfer that much data.

jarito 2y ago
Pretty sure those services are only for transferring data into S3, not out.
- byefruit 2y ago
  https://aws.amazon.com/snowball/pricing/ https://aws.amazon.com/snowball/pricing/ snowball seems to support getting data out of S3 though you still end up paying extortionate egress charges.
  - dilyevsky 2y ago
    They wave egress if you’re leaving for 60 days, at least when you migrate via standard api

ifightcrime 2y ago

Cloud has always been more expensive. I remember being quoted 250k/month for bandwidth when I was paying 15k with rackspace 10+ years ago. You’re paying for convenience and speed. The Math stops working when you grow to a certain point. You can mitigate this to some extent by making some key architecture + vendor decisions upfront when first building… or just consider that some day you’ll need to do things like this. It’s not a novel problem.

vidarh 2y ago
It's horrifyingly hard to convince people of this, though, even you can present them with actual numbers. A lot of people have convinced themselves that cloud is cheap, to the point that they don't even do a cursory investigation. A lot of those even don't do the bare minimum to reduce hosting costs within the cloud they choose, or choose one of the cheaper clouds (AWS is absolutely extortionate for anything that requires significant amount of outbound bandwith), or put caching/CDN's in front (you can trivially slash your AWS egress costs dramatically). Most of my consultancy work is on driving cost efficiencies for cloud, and I can usually safely guarantee the fee will pay for itself within months because people don't fix even the most low hanging fruit.
- mbreese 2y ago
  I think one business argument for cloud is capital expenses vs operational expenses. If you’re (over) paying for cloud resources vs an in house option (or colo), those are numbers that are a straight expense. When you own hardware, those are on your books until they depreciate off. For some businesses, that can make sense. Now, a good accountant probably wouldn’t care one way or the other. Debits and credits balance either way. And spending more still means less profit in the long term, no matter how it looks on the books. But, in addition to the flexibility, that was what I always thought of as the main cloud benefit. It’s the same with leasing vs buying cars/computers/etc…
  - tialaramex 2y ago
    Cloud made sense for the startup I worked for previously. If you are a startup then a $1M per year expense makes much more sense than a $5M up front purchase with 5-10 years of life - in five years you might be billionaires or you might be bankrupt and until then the Cloud was better.

tgtweak 2y ago

The frustrating part is they'll give you $1500/day back again if you let them know you're leaving... Very annoying tbh.

calderwoodra 2y ago

Their current rate is about ~20% cheaper (4k/day), so missing the deadline isn't that big of a deal.

DeathArrow 2y ago

Using SSD seems like a waste of money. I think having a raid like structure with good old spinning hard disks is good enough. Does Amazon use SSD only for S3? I know backblaze uses HDDs and some raid like addressing.

nijave 2y ago
Not sure Amazon posts much about S3 architecture but some years ago I think they had an article about using spinners. They had so many HDDs, though, that 100-250 IOPs * <the insane number of total drives> meant they still had insane throughput. Not sure they're at the scale in the article to get good performance with HDDs but someone with a storage eng background can correct me. NVMe seems like a waste, though. I imagine you can still get solid performance with SATA/SAS (250 to 50-80k IOPs is still a pretty huge leap)

Axsuul 2y ago

Cloud is expensive but hardware failures are at least handled gracefully. With coloc you'd have some serious downtime. That means you'd need to get to a certain level of redundancy in order to have coloc make sense. I'd love to move to coloc for my SaaS but it doesn't feel as resilient. Please correct me if I'm wrong as I'd love to move off the cloud.

nijave 2y ago
Not sure cloud is necessarily more resilient--imo it's less resilient. On the other hand, it's fully automated with robust APIs so there's easy tools to mitigate failures like node/machine sets (scale sets, scaling groups, auto scaling groups, whatever the provider calls them) You could use an orchestration solution to help handle automatic failover. There's a handful of container-based options from heavy duty Kubernetes to Docker Swarm and Nomad. Containers are nice since you can bypass most of the host management where you only need basic security patching and installation of your container runtime. There's also k8s distros like OpenShift to make k8s setup easier if you go that route.
- Axsuul 2y ago
  Yep I use orchestration (Nomad) but still you would need hardware redundancy. For example, the database server is currently a single point of failure. In the cloud, if there's a hardware failure, it will simply go down and come back up with a new instance. In coloc, you'd need to have the data center debug and replace hardware which means extended downtime.
  - NorwegianDude 2y ago
    When using colocation, nothing is stopping people from storing the database data externally from the server running the database like some cloud services do. But doing so, either in cloud or not, does have a serious downside: greatly increased latency.
    - Axsuul 2y ago

albert_e 2y ago

Even if they move only part of the data before the deadline ..they stop paying for that the moment they delete it from S3. (Doesn't work that way in reverse.) They can start saving thousands of dollars even before the deadline if they are able to start moving as soon as their own infra is up and incrementally move and delete data from S3. If their data consumers can work with that, that is. If part of the data does not need the full S3 guarantees of durability and availability ..they could probably save more by using cheaper tiers while on S3.

siscia 2y ago

What are the stats of the new system? Cost per giga? Latency curve on read? Latency curve on write? Support for concurrency? Max throughput? I think the article is somehow shallow, but S3 is a very deep and flexible system. Of course, someone will just need it's basic features, but someone else may be interested in advanced features as well.

hbogert 2y ago

if vendors like Pure weren't so incredibly irritating with the documentation behind a paywall i'd actually recommend them to clients. What happens now is, a client already has pure, and as a contractor I can't get access to documentation. Seriously, what year is it.

otterley 2y ago
If someone is hiding something, it means they have something to hide.

theptip 2y ago

Would love to see their growth rates, but Basecamp seems like a canonical example of where renting makes the least sense. They are bootstrapped and not hyperscaling, so they don’t need flexibility for rapid and unpredictable growth (VC backed startups do). And they have a strong engineering org, so they don’t benefit as much from buying reliability (large non-tech companies often do).

FireBeyond 2y ago

> But those big numbers always seem a bit abstract to me. The idea of paying $5,000/day, if we miss our departure date, is awfully concrete in comparison. Says the guy who owns multiple hypercars, some of which are $3M+ (Aston Martin Valkyrie)...

otterley 2y ago

Is Hey not able to take advantage of S3 intelligent tiering? It’s way cheaper than S3 standard for infrequently accessed objects. I wouldn’t be surprised if Hey’s access patterns lend themselves well to it. (Yes, I’m aware that Hey’s decision to evacuate the cloud is a fait accompli, but I also can’t help but wonder if there are loads of potential savings that are being left out of the discussion.)

npalli 2y ago

DHH's focus getting massive saving on storage costs is a head-scratcher. Usually, egress costs are punitive. Given LLM's are pretty good now, I modeled the break-even point of such a move from S3 (6PB) to a colo. Breakeven time 18 years(!!), even with 8 year depreciation cycle still looking at 8 years to breakeven. This is not including falling S3 costs in Amazon. Truth be told he is probably spending 10's of millions on servers running Rails (SSR), probably can cut that in half by moving to golang or java. This is peanuts in comparison. 1. Initial costs of $2.5M to provision the hardware (disks, servers, enclosures, networking equipment, redundancy, software solutions etc.) 2. Facility OPEX: $50,000/month (Power, connective, monitoring etc..) 3. Staffing and Operation tools: $10,000 / month 4. Replacement cycle of 5 years so assume $500,000/year ~ $41K/month

bauruine 2y ago
>2. Facility OPEX: $50,000/month (Power, connective, monitoring etc..) Are you sure? It looks like they only have 1 rack (+1 in another facility for redundancy) and seem to have 40Gbit/s connectivity. A full rack is in the range of 1k, connectivity around 600$ per 10Gbit/s. I have no idea how much power they consume but I doubt it's 40k$+ per month for a storage workload. I would guess they are in the 10k$ range. Those are only list prices I've seen in the wild so take it with a grain of salt but 50k seem VERY high.
- omnimus 2y ago
  The power must be way less especially without spinning drives.
- npalli 2y ago
  1 rack is a recipe for failure, you need to split, even without that consider how many SSD's you need. power usage for all those SSD's inter-site connectivity (need to keep transferring data between the sites otherwise customers are going to be very surprised. maintenance and software costs (at the colo level) All these add up, closer to $50K than some $2K (LOL). The way you guys (below) are talking, this is not some home server that serves personal videos. It runs (some) business operations for thousands of small/medium companies.

killer32 2y ago

We’ve been running Rook-Ceph in production across multiple client environments. In one example, we built a setup with 8 refurbished Dell servers (128GB RAM, 8–14 JBOD disks each) over 10G networking. It supports geo-replication between sites and has been stable for over 2 years. Total hardware cost was under $100k. Rook simplifies the operational overhead of Ceph quite a lot, especially in Kubernetes-native stacks. For teams with large data and HA requirements, it's been a solid on-prem alternative. SLA-backed managed services are also becoming more common, which helps reduce the operational burden even further.