5 ms·

Once you've had an outage on AWS, Cloudflare, Google Cloud, Akismet. What are you going to do? Host in house? None of them seem to be immune from some outage at

Once you've had an outage on AWS, Cloudflare, Google Cloud, Akismet. What are you going to do? Host in house? None of them seem to be immune from some outage at some point. Get your refund and carry on. It's less work for the same outcome.

CobrastanJorji 1y ago
Multi-cloud. It's fairly unlikely that AWS and Google Cloud are going to fail at the same time.
- radium3d 1y ago
  Yeah, just double++ the cost to have a clone of all your systems. Worth it if you need to guarantee uptime. Although, it also doubles your exposure to potential data breaches as well.
  - jancsika 1y ago
    > double++ I'd suggest to ++double the cost. Compare: ++double: spoken as "triple" -> team says that double++ was a joke, we can obviously only double the cost -> embarrassingly you quickly agree -> team laughs -> team approves doubling -> you double the cost -> team goes out for beers -> everyone is happy double++: spoken as "double" -> team quickly agrees and signs off -> you consequently triple the cost per c precedence rules -> manager goes ballistic -> you blithely recount the history of c precedence in a long monotone style -> job returns EINVAL -> beers = 0
    - radium3d 1y ago
      Lol :)
  - unethical_ban 1y ago
    Shouldn't be double in the long term. Think of the second cloud as a cold standby. Depends on the system. Periodic replication of data layer (object storage/database) and CICD configured to be able to build services and VMs on multiple clouds. Have automatic tests weekly/monthly that represent end-to-end functionality, have scaled tests semi-annually. This is all very, very hand-wavey. And if one says "golly gee, all our config is too cloud specific to do multi-cloud" then you've figured out why cloud blows and that there is no inherent reason not to have API standards for certain mature cloud services like serverless functions, VMs and networks. Edit to add- ink ow how grossly simplified this is, and that most places have massively complex systems.
  - dexterdog 1y ago
    And likely far more than double the cost since you have to use the criminally-priced outbound bandwidth to keep everything in sync.
  - jimbob45 1y ago
    And data egress fees just to get the clone set up, right? This doesn’t seem feasible as a macrostrategy. Maybe for a small number of critical services.
- yeswecatan 1y ago
  How do you handle replication lag for databases?
  - zacmps 1y ago
    If you use something like cockroachdb you can have a multi-master cluster and use regional-by-row tables to locate data close to users. It'll fail over fine to other regions if needed.
cakeday 1y ago
> Host in house? Yes, mostly.
grogers 1y ago
Certainly if you aren't even multi-region, then multi-cloud is a pipe dream
bean469 1y ago
> What are you going to do? Host in house? Yep. Although it's just anecdata, it's what we do where I work - haven't had a slightest issue in years.
- nxpnsv 1y ago
  Cheaper, faster, in house people understands what’s going on. It should be a given for many services but somehow it’s not.
  - Breza 1y ago
    I totally agree with you. Where I work, we self-host almost everything. Exceptions are we use a CDN for one area where we want lower latency, and we use BigQuery when we need to parse a few billion datapoints into something usable. It's amazing how few problems we have. Honestly, I don't think we have to worry about configuration issues as often as people who rely on the cloud.
- erikpukinskis 1y ago
  On premise? Or do you build servers in a data center? Or do you lease dedicated servers?
  - Breza 1y ago
    Not GP, but my company also self-hosts. We rent rackspace in a colo. We used to keep my team's research server in the back closet before we went full-remote.
  - bean469 1y ago
    We have our own data center with servers. The upfront costs are high, but it was worth it in our use-case
Breza 1y ago
Why not host in house? If you have an application with stable resource needs, it can often be the cheaper and more stable option. At a certain scale, you can buy the servers, hire a sysadmin, and still spend less money than relying on AWS. If you have an app that experiences 1000x demand spikes at unpredictable times then sure, go with the cloud. But there are a lot of companies that would be better off if they seriously considered their options before choosing the cloud for everything.

CobrastanJorji 1y ago

Multi-cloud. It's fairly unlikely that AWS and Google Cloud are going to fail at the same time.

radium3d 1y ago
Yeah, just double++ the cost to have a clone of all your systems. Worth it if you need to guarantee uptime. Although, it also doubles your exposure to potential data breaches as well.
- jancsika 1y ago
  > double++ I'd suggest to ++double the cost. Compare: ++double: spoken as "triple" -> team says that double++ was a joke, we can obviously only double the cost -> embarrassingly you quickly agree -> team laughs -> team approves doubling -> you double the cost -> team goes out for beers -> everyone is happy double++: spoken as "double" -> team quickly agrees and signs off -> you consequently triple the cost per c precedence rules -> manager goes ballistic -> you blithely recount the history of c precedence in a long monotone style -> job returns EINVAL -> beers = 0
  - radium3d 1y ago
    Lol :)
- unethical_ban 1y ago
  Shouldn't be double in the long term. Think of the second cloud as a cold standby. Depends on the system. Periodic replication of data layer (object storage/database) and CICD configured to be able to build services and VMs on multiple clouds. Have automatic tests weekly/monthly that represent end-to-end functionality, have scaled tests semi-annually. This is all very, very hand-wavey. And if one says "golly gee, all our config is too cloud specific to do multi-cloud" then you've figured out why cloud blows and that there is no inherent reason not to have API standards for certain mature cloud services like serverless functions, VMs and networks. Edit to add- ink ow how grossly simplified this is, and that most places have massively complex systems.
- dexterdog 1y ago
  And likely far more than double the cost since you have to use the criminally-priced outbound bandwidth to keep everything in sync.
- jimbob45 1y ago
  And data egress fees just to get the clone set up, right? This doesn’t seem feasible as a macrostrategy. Maybe for a small number of critical services.
yeswecatan 1y ago
How do you handle replication lag for databases?
- zacmps 1y ago
  If you use something like cockroachdb you can have a multi-master cluster and use regional-by-row tables to locate data close to users. It'll fail over fine to other regions if needed.

cakeday 1y ago

> Host in house? Yes, mostly.

grogers 1y ago

Certainly if you aren't even multi-region, then multi-cloud is a pipe dream

bean469 1y ago

> What are you going to do? Host in house? Yep. Although it's just anecdata, it's what we do where I work - haven't had a slightest issue in years.

nxpnsv 1y ago
Cheaper, faster, in house people understands what’s going on. It should be a given for many services but somehow it’s not.
- Breza 1y ago
  I totally agree with you. Where I work, we self-host almost everything. Exceptions are we use a CDN for one area where we want lower latency, and we use BigQuery when we need to parse a few billion datapoints into something usable. It's amazing how few problems we have. Honestly, I don't think we have to worry about configuration issues as often as people who rely on the cloud.
erikpukinskis 1y ago
On premise? Or do you build servers in a data center? Or do you lease dedicated servers?
- Breza 1y ago
  Not GP, but my company also self-hosts. We rent rackspace in a colo. We used to keep my team's research server in the back closet before we went full-remote.
- bean469 1y ago
  We have our own data center with servers. The upfront costs are high, but it was worth it in our use-case

Breza 1y ago

Why not host in house? If you have an application with stable resource needs, it can often be the cheaper and more stable option. At a certain scale, you can buy the servers, hire a sysadmin, and still spend less money than relying on AWS. If you have an app that experiences 1000x demand spikes at unpredictable times then sure, go with the cloud. But there are a lot of companies that would be better off if they seriously considered their options before choosing the cloud for everything.