6 ms·

We stand to save $7M over five years from our cloud exit

103 points by klelatti 4y ago

senttoschool 4y ago
I think it'll be interesting to see. It seems like they won't move their 8PB of data off the cloud. This seems smart because you can always spin up servers when one goes down, but if your storage system loses important user emails, that's the end of your service.
- gabrielgio 4y ago
  > It seems like they won't move their 8PB of data off the cloud. For now. He wrote they will do that next year.
  - senttoschool 4y ago
    Good luck to them. I'm sure they can do it but I wouldn't want to be working on that project.
- bluedino 4y ago
  What do you use when you decide to host 8PB of data yourself? 60x18TB - 1PB raw $10,000 for a 60 drive chassis $250/drive = $15,0000 $25,000 total You're going to need 12-15 of those, right? $300-400,000 Now what do you run on top of that, CEPH? NFS shares? Present it as iSCSI?
  - wmf 4y ago
    Ceph, Minio, or some other S3-compatible system.
    - hintymad 4y ago
      Minio is dual licensed: AGPS or their enterprise license. Both can be a big no-no for companies. I was wondering what could be the alternative to MinIO, in addition to Ceph.
      - wmf 4y ago
        IMO "never pay for software no matter what" is a red flag to me. Buying a license can be the right choice if it saves you money on ops or hardware.
        hintymad 4y ago
        I see two issues with licensed systems for large enough workload. One is scale of cost. Enterprise software charges by some normalized server configuration (CPU, memory, and etc. MinIO seems an exception, though). As a result, the cost of the license can grow to be prohibitive for large workload. Another is that enterprise licenses usually do not allow users to modify the software easily, while for any significant workload, we almost always need to add code for smooth operations.
rkwasny 4y ago
Good idea, however remember to calculate power requirements in the DC and get at least 100% more space than you currently need! I've seen cases where you want to provision the new rack but unfortunately DC is full and you have to end up running infrastructure spanning across different buildings or even different DCs. With Zen4 I would go with single CPU option, it's hard to saturate 256 threads. Main cost in the cloud is always the database server, because it needs to be on 100% of the time, cannot be on spot instance, and needs lots of memory and CPU just in case of a spike of traffic.
dijit 4y ago
I think it's very important that we have someone willing to put their money where their mouth is. I've been saying for years that cloud is more cost but you end up needing the same number of people if you have a large enough scope. Scalability isn't actually all that hard, you solve it in cloud or you solve it in hardware. People are always very quick to say that running on bare metal means you have to start working in nuclear power plants or CPU lithography from sand. It's intellectually dishonest of course, but we need new players in the game that can't be accused of being old and actually weighing the benefits and drawbacks publicly. So, Bravo.
locusm 4y ago
When companies migrate back to colo/on-prem from cloud are they disabling CPU vuln mitigations for non public facing servers? On some server generations the performance uplift is huge.
- senttoschool 4y ago
  They're using Zen4. Zen4 should have those issues fixed in the hardware.
all_usernames 4y ago
> Just under a million of that was on storing 8 petabytes of files in S3, fully replicated across several regions Hol' up. They're replicating data across regions -- across /several/ regions -- on top of a service (S3) which already has 99.999999999% data durability in a single region? ...Why?
- shrimpx 4y ago
  Latency?
ravagat 4y ago
Very happy to see this. I have been a silent advocate for bringing back/investing into in-house "cloud" infrastructure for the many SME I've done some work for. I have already seen corporate (non-tech/startup/saas) do this but have yet to see true adoption in-house for more-tech oriented companies
dustedcodes 4y ago
The real question is, how much cloud computing cost would DHH save if he'd stopped using Ruby
- gabrielgio 4y ago
  That is not the real question. Choosing another language won't fix any of the issues he is addressing with the moving to on-prem servers. It is irrelevant for the topic.
- wmf 4y ago
  He wrote a blog post about that topic. IIRC around 10% of their cost comes from Ruby. Meanwhile they're aiming to save ~50% by moving off AWS.

103 points by klelatti 4y ago

I think it'll be interesting to see. It seems like they won't move their 8PB of data off the cloud. This seems smart because you can always spin up servers when one goes down, but if your storage system loses important user emails, that's the end of your service.

gabrielgio 4y ago
> It seems like they won't move their 8PB of data off the cloud. For now. He wrote they will do that next year.
- senttoschool 4y ago
  Good luck to them. I'm sure they can do it but I wouldn't want to be working on that project.
bluedino 4y ago
What do you use when you decide to host 8PB of data yourself? 60x18TB - 1PB raw $10,000 for a 60 drive chassis $250/drive = $15,0000 $25,000 total You're going to need 12-15 of those, right? $300-400,000 Now what do you run on top of that, CEPH? NFS shares? Present it as iSCSI?
- wmf 4y ago
  Ceph, Minio, or some other S3-compatible system.
  - hintymad 4y ago
    Minio is dual licensed: AGPS or their enterprise license. Both can be a big no-no for companies. I was wondering what could be the alternative to MinIO, in addition to Ceph.
    - wmf 4y ago
      IMO "never pay for software no matter what" is a red flag to me. Buying a license can be the right choice if it saves you money on ops or hardware.
      - hintymad 4y ago
        I see two issues with licensed systems for large enough workload. One is scale of cost. Enterprise software charges by some normalized server configuration (CPU, memory, and etc. MinIO seems an exception, though). As a result, the cost of the license can grow to be prohibitive for large workload. Another is that enterprise licenses usually do not allow users to modify the software easily, while for any significant workload, we almost always need to add code for smooth operations.

rkwasny 4y ago

Good idea, however remember to calculate power requirements in the DC and get at least 100% more space than you currently need! I've seen cases where you want to provision the new rack but unfortunately DC is full and you have to end up running infrastructure spanning across different buildings or even different DCs. With Zen4 I would go with single CPU option, it's hard to saturate 256 threads. Main cost in the cloud is always the database server, because it needs to be on 100% of the time, cannot be on spot instance, and needs lots of memory and CPU just in case of a spike of traffic.

dijit 4y ago

I think it's very important that we have someone willing to put their money where their mouth is. I've been saying for years that cloud is more cost but you end up needing the same number of people if you have a large enough scope. Scalability isn't actually all that hard, you solve it in cloud or you solve it in hardware. People are always very quick to say that running on bare metal means you have to start working in nuclear power plants or CPU lithography from sand. It's intellectually dishonest of course, but we need new players in the game that can't be accused of being old and actually weighing the benefits and drawbacks publicly. So, Bravo.

locusm 4y ago

When companies migrate back to colo/on-prem from cloud are they disabling CPU vuln mitigations for non public facing servers? On some server generations the performance uplift is huge.

senttoschool 4y ago
They're using Zen4. Zen4 should have those issues fixed in the hardware.

all_usernames 4y ago

> Just under a million of that was on storing 8 petabytes of files in S3, fully replicated across several regions Hol' up. They're replicating data across regions -- across /several/ regions -- on top of a service (S3) which already has 99.999999999% data durability in a single region? ...Why?

shrimpx 4y ago
Latency?

ravagat 4y ago

Very happy to see this. I have been a silent advocate for bringing back/investing into in-house "cloud" infrastructure for the many SME I've done some work for. I have already seen corporate (non-tech/startup/saas) do this but have yet to see true adoption in-house for more-tech oriented companies

dustedcodes 4y ago

The real question is, how much cloud computing cost would DHH save if he'd stopped using Ruby

gabrielgio 4y ago
That is not the real question. Choosing another language won't fix any of the issues he is addressing with the moving to on-prem servers. It is irrelevant for the topic.
wmf 4y ago
He wrote a blog post about that topic. IIRC around 10% of their cost comes from Ruby. Meanwhile they're aiming to save ~50% by moving off AWS.