9 ms·

To be fair, there are going to be plenty of critical leaks coming out of Twitter with a 50% cutting of staff. That's so many whistleblowers, my backyard is comp

To be fair, there are going to be plenty of critical leaks coming out of Twitter with a 50% cutting of staff. That's so many whistleblowers, my backyard is completely leaf free now. Frankly, if Musk cuts $1B from the run rate and there isn't a bad outage, a lot of people will look pretty stupid. To be fair, I don't know if Twitter is overspending on infrastructure, but I would guess there are always places to streamline. You could frankly save a ton of money by re-negotiating commits with Google. Also, $1B to me shows how insane running a large business in the cloud can be. This is sort of where Facebook/Meta got it right, IMHO - run infrastructure on-prem, at scale, in a super-efficient "everything fails" sort of way.

bushbaba 4y ago
I don’t think the 1B/year shows issue on cloud pricing. It’s instead bloat of twitter’s setup. That’s way in excess of what should reasonably be spent. I suspect there’s a lot of bad coding practice going on. Of which the bulk is data analytics.
- bagels 4y ago
  AWS profit margin is something like 35%. I think some of their infra is there, so that might be an upper bound?
  - djmips 4y ago
    Do they use AWS though? https://blog.twitter.com/engineering/en_us/topics/infrastructure/2017/the-infrastructure-behind-twitter-scale https://blog.twitter.com/engineering/en_us/topics/infrastruc...
colechristensen 4y ago
Everyone is “overspending” on infrastructure. There is always money to save the question comes down to where you want to expend your engineering effort and what risk levels you want to take. “Make fewer features, focus on spending less” is a valid executive decision, it’s just a matter of priority. Twitter seemingly being feature complete and losing lots of money most years makes doing this now a decent decision. Spending too much like that is generally a symptom of not paying attention and noticing the waste, it happens from time to time. Gets to feel normal, then you notice and correct. I’ve been mostly responsible for cloud spend a number of times and had several events of cutting spending by 20% or more over the years.
- bagels 4y ago
  I've had the same experience at startups. Usually it's time to make features, but eventually the spend gets high enough to justify spending a couple weeks of engineer time on it to cut out waste. Same thing happened when I worked a large internet company, just on a different scale.
rippercushions 4y ago
AFAIK Twitter's serving infra is mostly on-prem and they use cloud mostly for analytics. https://blog.twitter.com/engineering/en_us/topics/infrastructure/2017/the-infrastructure-behind-twitter-scale https://blog.twitter.com/engineering/en_us/topics/infrastruc... (2017 tho)
partiallypro 4y ago
I had a friend that worked for a massive company in the Valley which I'll leave unnamed, that told me about them just leaving servers running hot with no throughput for months at a time. So, I guess it's possible. I doubt his estimate was pulled out of a hat (but I could be wrong, or the article could just be pulling that out of a hat/using an unreliable source.) I've taken over AWS and Azure accounts before that were paying tens of thousands for unattached disks, load balancers, big servers that were doing very little that could have just been on a burstable VM. Massive messes that in some cases took months to clean up. So, I can believe there could be a massive bit of cost savings available if it were poorly managed. I've never seen the scale of a company like Twitter, but I really doubt there isn't some dead weight. I know storage is a massive cost, and he has stated he was going to purge accounts that were unused over a certain time frame.
- SanjayMehta 4y ago
  One of our smallest customers had a VM for each application server process. CPU occupancy alone was rarely over 20%. Easily cut 70% of cloud costs.
- tluyben2 4y ago
  Yep, same here; as a (well paid) hobby, I help smaller companies lower their software bills. When they use 'cloud hosting' instead of a vps or something, that's usually the first point where I can cut 80%+. It is crazy how many things are running there that are not used or needed in almost every company I see. Usually there are software issues (most commonly, no, not enough or plainly wrong db indexes being the #1 top scorer) which drive rds bills to scary height for no reason at all, but the low hanging fruit is unused cloud 'stuff'.
  - tomas_nort 4y ago
    Could you explain how db indexes are costs having no reason at all?
    - Thorrez 4y ago
kirso 4y ago
I am quite puzzled, at such scale with low seasonality what are the reasons they haven't moved to their own infra? Managing and hiring people to do so would result in big decrease in cost compared to the cloud.
- gpapilion 4y ago
  Twitter is mostly on their own infra. 1 billion is 25% of total revenue. I think they are close to that spend with cloud but it’s more like 750m if I were to venture a guess.
throwoutway 4y ago
They barely run anything in the cloud, they’ve been operating their own data centers & infrastructure for over a decade. So they’re doing the same thing as Meta.
reaperducer 4y ago
if Musk cuts $1B from the run rate and there isn't a bad outage, a lot of people will look pretty stupid. A then-current Twitter employee told one of the New York Times podcasts that Twitter's infra looks overbuilt because the teams focus primarily on reliability, with new features presumably coming second. Feels like Twitter is now joining the "move fast and break things" club.
UncleMeat 4y ago
> Frankly, if Musk cuts $1B from the run rate and there isn't a bad outage, a lot of people will look pretty stupid. Is this a good way of reasoning? If somebody said "frankly, if Bob stops wearing his seatbelt and doesn't die, a lot of people will look pretty stupid" we'd push back. Now the opposite is also not true, where if things go fine Musk is still an idiot. Only the people who know the internals of the system can really say with confidence. One thing I am sure of is that Musk is not one of those people.
- tenpies 4y ago
  The analogy you're using is binary (seat belt on or off), whereas an infrastructure cut is more like reducing your thermostat by a couple of degrees, then concluding that no one froze to death, therefore it was an acceptable cut. Many parts of Europe will be running this experiment, in fact.
  - fragmede 4y ago
    more like, cutting off heating to the building, telling everyone to wear sweaters, and just wait and see if anyone has a serious medical emergency.
- kevinventullo 4y ago
  Except Twitter going down is not as serious as death. The answer to “How much $ is it worth to get an extra 9 of reliability?” is never “Infinite”. So, I agree with OP, if they can save a big chunk of their opex with a negligible decrease in reliability (measured by something like, how much user churn are we seeing as a result of the downtime), then it’s a good call. I don’t know enough about Twitter’s internals to guess whether this is a good call. A bit off-topic, but having worked at Meta I think it would probably not be a good call there. Infra engineers are already directly incentivized to find fractional-percentage wins, because the savings to the company are often greater than their total comp, and that’s a pretty straightforward way to justify a high performance rating.
  - metadat 4y ago
spoils19 4y ago
> This is sort of where Facebook/Meta got it right, IMHO - run infrastructure on-prem, at scale, in a super-efficient "everything fails" sort of way. It's fairly ignorant to make this claim without realizing that Twitter also runs the majority of their stuff on-prem as well.

bushbaba 4y ago

I don’t think the 1B/year shows issue on cloud pricing. It’s instead bloat of twitter’s setup. That’s way in excess of what should reasonably be spent. I suspect there’s a lot of bad coding practice going on. Of which the bulk is data analytics.

bagels 4y ago
AWS profit margin is something like 35%. I think some of their infra is there, so that might be an upper bound?
- djmips 4y ago
  Do they use AWS though? https://blog.twitter.com/engineering/en_us/topics/infrastructure/2017/the-infrastructure-behind-twitter-scale https://blog.twitter.com/engineering/en_us/topics/infrastruc...

colechristensen 4y ago

Everyone is “overspending” on infrastructure. There is always money to save the question comes down to where you want to expend your engineering effort and what risk levels you want to take. “Make fewer features, focus on spending less” is a valid executive decision, it’s just a matter of priority. Twitter seemingly being feature complete and losing lots of money most years makes doing this now a decent decision. Spending too much like that is generally a symptom of not paying attention and noticing the waste, it happens from time to time. Gets to feel normal, then you notice and correct. I’ve been mostly responsible for cloud spend a number of times and had several events of cutting spending by 20% or more over the years.

bagels 4y ago
I've had the same experience at startups. Usually it's time to make features, but eventually the spend gets high enough to justify spending a couple weeks of engineer time on it to cut out waste. Same thing happened when I worked a large internet company, just on a different scale.

rippercushions 4y ago

AFAIK Twitter's serving infra is mostly on-prem and they use cloud mostly for analytics. https://blog.twitter.com/engineering/en_us/topics/infrastructure/2017/the-infrastructure-behind-twitter-scale https://blog.twitter.com/engineering/en_us/topics/infrastruc... (2017 tho)

partiallypro 4y ago

I had a friend that worked for a massive company in the Valley which I'll leave unnamed, that told me about them just leaving servers running hot with no throughput for months at a time. So, I guess it's possible. I doubt his estimate was pulled out of a hat (but I could be wrong, or the article could just be pulling that out of a hat/using an unreliable source.) I've taken over AWS and Azure accounts before that were paying tens of thousands for unattached disks, load balancers, big servers that were doing very little that could have just been on a burstable VM. Massive messes that in some cases took months to clean up. So, I can believe there could be a massive bit of cost savings available if it were poorly managed. I've never seen the scale of a company like Twitter, but I really doubt there isn't some dead weight. I know storage is a massive cost, and he has stated he was going to purge accounts that were unused over a certain time frame.

SanjayMehta 4y ago
One of our smallest customers had a VM for each application server process. CPU occupancy alone was rarely over 20%. Easily cut 70% of cloud costs.
tluyben2 4y ago
Yep, same here; as a (well paid) hobby, I help smaller companies lower their software bills. When they use 'cloud hosting' instead of a vps or something, that's usually the first point where I can cut 80%+. It is crazy how many things are running there that are not used or needed in almost every company I see. Usually there are software issues (most commonly, no, not enough or plainly wrong db indexes being the #1 top scorer) which drive rds bills to scary height for no reason at all, but the low hanging fruit is unused cloud 'stuff'.
- tomas_nort 4y ago
  Could you explain how db indexes are costs having no reason at all?
  - Thorrez 4y ago

kirso 4y ago

I am quite puzzled, at such scale with low seasonality what are the reasons they haven't moved to their own infra? Managing and hiring people to do so would result in big decrease in cost compared to the cloud.

gpapilion 4y ago
Twitter is mostly on their own infra. 1 billion is 25% of total revenue. I think they are close to that spend with cloud but it’s more like 750m if I were to venture a guess.

throwoutway 4y ago

They barely run anything in the cloud, they’ve been operating their own data centers & infrastructure for over a decade. So they’re doing the same thing as Meta.

reaperducer 4y ago

if Musk cuts $1B from the run rate and there isn't a bad outage, a lot of people will look pretty stupid. A then-current Twitter employee told one of the New York Times podcasts that Twitter's infra looks overbuilt because the teams focus primarily on reliability, with new features presumably coming second. Feels like Twitter is now joining the "move fast and break things" club.

UncleMeat 4y ago

> Frankly, if Musk cuts $1B from the run rate and there isn't a bad outage, a lot of people will look pretty stupid. Is this a good way of reasoning? If somebody said "frankly, if Bob stops wearing his seatbelt and doesn't die, a lot of people will look pretty stupid" we'd push back. Now the opposite is also not true, where if things go fine Musk is still an idiot. Only the people who know the internals of the system can really say with confidence. One thing I am sure of is that Musk is not one of those people.

tenpies 4y ago
The analogy you're using is binary (seat belt on or off), whereas an infrastructure cut is more like reducing your thermostat by a couple of degrees, then concluding that no one froze to death, therefore it was an acceptable cut. Many parts of Europe will be running this experiment, in fact.
- fragmede 4y ago
  more like, cutting off heating to the building, telling everyone to wear sweaters, and just wait and see if anyone has a serious medical emergency.
kevinventullo 4y ago
Except Twitter going down is not as serious as death. The answer to “How much $ is it worth to get an extra 9 of reliability?” is never “Infinite”. So, I agree with OP, if they can save a big chunk of their opex with a negligible decrease in reliability (measured by something like, how much user churn are we seeing as a result of the downtime), then it’s a good call. I don’t know enough about Twitter’s internals to guess whether this is a good call. A bit off-topic, but having worked at Meta I think it would probably not be a good call there. Infra engineers are already directly incentivized to find fractional-percentage wins, because the savings to the company are often greater than their total comp, and that’s a pretty straightforward way to justify a high performance rating.
- metadat 4y ago

spoils19 4y ago

> This is sort of where Facebook/Meta got it right, IMHO - run infrastructure on-prem, at scale, in a super-efficient "everything fails" sort of way. It's fairly ignorant to make this claim without realizing that Twitter also runs the majority of their stuff on-prem as well.