3 ms·

I struggle to find one correct remark in your comment. This take clearly suggests you have never worked on any similar infrastructure / distributed systems befo

I struggle to find one correct remark in your comment. This take clearly suggests you have never worked on any similar infrastructure / distributed systems before at scale. So you're telling me there's no backend or compute workers? And you're telling me that there are no opportunities for performance optimization? That they are already 100% native C++ services at full parallel utilization? That they've already tuned the size and cost of their ML models? That their model evaluation infra is fully saturated? Storage is cheap. I believe they are on AWS, so hardware costs are already taken out of the equation. If so, even just moving to on-prem would save them the $1B easily.

koyanisqatsi 4y ago
You should sit down and perform the storage calculation instead of struggling to find something correct in what I said because that will be a better use of your limited time.
trowawee 4y ago
> If so, even just moving to on-prem would save them the $1B easily Except you can't wave a magic wand and shift the entire distributed infrastructure for one of the five most visited sites on earth on-prem. Twitter has two DCs. Running their infra on-prem somewhere would require reworking the entire company from the ground up in a way that would take years and cost substantially more than $1b.
- deleted 4y ago
  [deleted]
- indigodaddy 4y ago
  Having their own DCs means they are on-prem already.
devoutsalsa 4y ago
> Storage is cheap Seriously. They could back up all of Twitter on Backblaze for $7/month.
- indigodaddy 4y ago
  Hah, I chuckled bigly, nice one.
drewbug01 4y ago
> I believe they are on AWS, so hardware costs are already taken out of the equation. If so, even just moving to on-prem would save them the $1B easily. They already run their own datacenters (not uncommon for companies started in that era). They have moved partially into the cloud for data science stuff - GCP, not AWS, according to their engineering blog. So… odds are that there isn’t nearly as much waste as everyone wants to believe.
- unboxingelf 4y ago
  DC is far more cost effective than public cloud for data intensive workloads.
  - drewbug01 4y ago
    Right, which is why they only executed a partial migration for certain workloads that make sense: https://blog.twitter.com/engineering/en_us/topics/infrastructure/2019/partly-cloudy-architecture https://blog.twitter.com/engineering/en_us/topics/infrastruc... I think what really irks me is this meme that “obviously they have massive infrastructure waste.” Does anyone have credible evidence that there is enormous waste in Twitter’s infrastructure? Just because Musk says “cut $1b in infra spending” doesn’t mean that there is $1b of things to cut, just lying about being wasted. Some of the smartest minds in our industry work/worked at Twitter. Are they that bad at their jobs? What is actually going to happen is this: they’ll turn off all of their data warehousing stuff, blinding the business. They’ll cut their redundancies and backups, reducing the probability that they can do effective DR. They’ll reduce spending on observability so far that they won’t even know what’s going wrong (it’s surprisingly expensive). And that’ll get them to the $1b in cuts - by flying blind (both in a business sense, and a technical sense), and just hoping they don’t need to recover from a disaster.

koyanisqatsi 4y ago

You should sit down and perform the storage calculation instead of struggling to find something correct in what I said because that will be a better use of your limited time.

trowawee 4y ago

> If so, even just moving to on-prem would save them the $1B easily Except you can't wave a magic wand and shift the entire distributed infrastructure for one of the five most visited sites on earth on-prem. Twitter has two DCs. Running their infra on-prem somewhere would require reworking the entire company from the ground up in a way that would take years and cost substantially more than $1b.

deleted 4y ago
[deleted]
indigodaddy 4y ago
Having their own DCs means they are on-prem already.

devoutsalsa 4y ago

> Storage is cheap Seriously. They could back up all of Twitter on Backblaze for $7/month.

indigodaddy 4y ago
Hah, I chuckled bigly, nice one.

drewbug01 4y ago

> I believe they are on AWS, so hardware costs are already taken out of the equation. If so, even just moving to on-prem would save them the $1B easily. They already run their own datacenters (not uncommon for companies started in that era). They have moved partially into the cloud for data science stuff - GCP, not AWS, according to their engineering blog. So… odds are that there isn’t nearly as much waste as everyone wants to believe.

unboxingelf 4y ago
DC is far more cost effective than public cloud for data intensive workloads.
- drewbug01 4y ago
  Right, which is why they only executed a partial migration for certain workloads that make sense: https://blog.twitter.com/engineering/en_us/topics/infrastructure/2019/partly-cloudy-architecture https://blog.twitter.com/engineering/en_us/topics/infrastruc... I think what really irks me is this meme that “obviously they have massive infrastructure waste.” Does anyone have credible evidence that there is enormous waste in Twitter’s infrastructure? Just because Musk says “cut $1b in infra spending” doesn’t mean that there is $1b of things to cut, just lying about being wasted. Some of the smartest minds in our industry work/worked at Twitter. Are they that bad at their jobs? What is actually going to happen is this: they’ll turn off all of their data warehousing stuff, blinding the business. They’ll cut their redundancies and backups, reducing the probability that they can do effective DR. They’ll reduce spending on observability so far that they won’t even know what’s going wrong (it’s surprisingly expensive). And that’ll get them to the $1b in cuts - by flying blind (both in a business sense, and a technical sense), and just hoping they don’t need to recover from a disaster.