9 ms·
Musk orders Twitter to cut infrastructure costs by $1B – sources
- whateveracct 4y agohm how much does a 9 cost?
- derwiki 4y agothe fail whale was a nice "maybe you should go do something else" reminder
- bravetraveler 4y agoSurprisingly little, 89.999% maintains four :D
- afavour 4y ago$1bn, the exact amount of debt interest Twitter now has to pay off after Musk saddled it with so much debt. So much of this feels like such a waste. I don’t have any problem with the idea of someone taking Twitter private in order to fix it. But with this kind of debt financing and with a CEO that doesn’t seem to understand the company (and tried to get out of buying it)…
- Aphataeros 4y agoYou've captured my sentiments exactly...
- refurb 4y agoWhy a waste? Sounds like a smart move? Buy a company, drastically reduce unnecessary expenses.
- tfsh 4y agoAll those unnecessary expenses like content moderation and robust fault tolerant infrastructure.
- valine 4y agoTwitter going down occasionally wouldn’t be the worst thing in the world.
- PubliusMI 4y ago
- factsarelolz 4y agoOr gigantic offices in expensive down town areas, all the coworkering space rent, food served at said office buildings, and personnel.
- listless 4y agoThis was my thought. Or at least move HQ somewhere else. Although he’ll probably end up doing that anyway.
- 8f2ab37a-ed6c 4y agoIt's an app for billionaires and journos to crank out viral memes, not a pacemaker. The world will be ok.
- afavour 4y agoNot really the point here. If you spent tens of billions purchasing Twitter you presumably don’t want the world to be without it. You have a very active interest in making it a success.
- darthrupert 4y agoUnless you're so beyond caring and vindictive that you're willing to do that just to screw over the shareholders who forced you to buy the company.
- nemothekid 4y ago>Buy a company, drastically reduce unnecessary expenses. This famously works out very well for PE firms. It seems very naive to assume Twitter was casually spending 1B/yr in "unnecessary expenses". Maybe the previous CEO/CTO was a baffling moron, but I tend to have more faith in people absent of evidence otherwise. I'm going to be surprised if Twitter is around in year - I'm just they have the capital to keep the lights on but it will be a very different story if advertisers start to abandon the platform because the adops infrastructure was gutted as an "unnecessary expense".
- nothatscool 4y agoWeren’t they spending almost half a billion a year on r&d?
- GoOnThenDoTell 4y agoThat usually just means regular engineering staff
- refurb 4y agoTwitter has expenses of $5B per year. https://finance.yahoo.com/quote/TWTR/financials/ https://finance.yahoo.com/quote/TWTR/financials/ So just view this as "A 20% reduction in expenses". Doesn't seem quite to scary huh?
- deleted 4y ago[deleted]
- EricDeb 4y agowould have been smart if you bought it for a good deal.
- bagels 4y agoIgnoring all the other things that are wrong with this whole situation... if they can make it $1B more efficient, how is it a waste even given the loans? Presumably the plan is to get rid of the debt eventually, and then you'll have a much more financially healthy company, assuming it can actually get $1B more efficient.
- dkjaudyeqooe 4y agoIf they can, that's great, if not then they're setting up Twitter for bankruptcy, or asset sales, if they have any. Generally speaking, loading up a company with debt is corrosive to good management. If you have to cover your interest payments each month, and that's hard, it focuses the business on short term survival rather than investing in growth in the longer term.
- bagels 4y agoDefinitely, the debt is a burden. I don't see how it's a waste to save money though. Maybe the cuts are so dramatic it damages the business, but reducing wasted money is generally good management.
- dkjaudyeqooe 4y agoIt's like surgery: making cuts is part of the job, but the more drastic and indiscriminate the more likely the patient dies. Given that Musk has just arrived, how can these cuts be carefully considered? He burst through the door and started slashing from what anyone can tell. Even if Twitter is heavily bloated, the cuts may be fatal anyway on that basis.
- GoOnThenDoTell 4y agoCutting costs is good, quickly slashing 50% doesn’t seem carefully considered and may cause actual problems.
- afavour 4y agoSeems like a pretty massive “if” there. An extra one billion in efficiency without detriment to the service?
- colechristensen 4y agoWait, what debt has Musk made Twitter take on? (being at the helm mere days, I really doubt there's any, debt like that takes longer to raise) Musk himself took on ~$12 billion to finance the deal, that debt isn't on Twitter's books.
- deleted 4y ago[deleted]
- hither_shores 4y agoIt was a leveraged buyout
- credit_guy 4y agoOk, so what if it was a leveraged buyout? All buyouts are. How did Twitter get "saddled" with debt?
- fakedang 4y agoNo, all-cash buyouts are also a thing.
- credit_guy 4y agoThis was an all-cash buyout. It does not mean it was not leveraged. All-cash buyouts are those buyouts where the acquirer pays with cash and not their own stock. In this case, Musk used only cash. But some of that cash (about $12.7 BN) was borrowed from banks. So, it was leveraged. In principle some buyer like Musk could buy without any help from banks, but why would they do that? You don't get to be a multi-billionaire without getting familiar with financing. In any case, leveraged or not, Musk did not saddle Twitter with any debt. Maybe he will do it in the future, but for the time being he didn't, because he did not have time to do it. He could try to issue a huge amount of debt later, and pay himself an extremely fat dividend, so he can turn a quick profit. But in the current market environment, that does not seem even remotely likely. Who is going to buy newly issued Twitter bonds? At what yields? A one year US Treasury yields 4.8%, what type of yield would one require for a bond issued by Twitter? 10%? 15%? If investors get a hint that Musk is just trying to run Twitter into the ground and get himself a nice exit, they won't invest even at 30%. After the whole saga with Musk repeatedly changing his mind on the deal (and before that the $420-for-sure Tesla buyout), how many investors do you think would fall for a get-rich-quick scheme perpetrated by a guy who can't actually climb the richest man ladder anymore?
- rhaway84773 4y agoThe risk for Twitter isn’t that it might go down for some periods of time. The risk is that if it starts going down on a regular basis and/or performance is degraded, a competitor could swoop in and clean their clocks. This might be a good time for Google to start Google+ again as an independent product one needs to sign in and create a new account for.
- jjordan 4y agoThe network effect is too strong for some occasional downtime to give even a well-funded competitor a window in.
- bagels 4y agoIt seems like a good time for Meta & Google just to make Twitter clones and see if they can get it to stick.
- jterrys 4y agoSocial media has always been a ghetto and it'll be interesting to see how many guys try and fail to build the next twitter in a few years before they realize why it was doomed from the start
- 0xy 4y agoTwitter was famously unreliable for years with the "fail whale", and no such competitor swooped in despite years of problems. So this argument doesn't seem to hold water.
- moistly 4y agoI suppose if everyone is fleeing the service, less infrastructure is required.
- fsdjkflsjfsoij 4y agoThat most likely won't happen though. In a month most people won't be talking about this and Twitter will be running fine.
- nemo 4y agoThis one looks meaningfully different. I won't be logging on in a month since I've already deleted my account (had it since 2010).
- fsdjkflsjfsoij 4y ago> This one looks meaningfully different. You might be right but plenty of people say this every time some major event at Twitter takes place. The vast majority of people don't care that much about layoffs at Twitter and certainly aren't going to migrate to another service over it.
- kQq9oHeAz6wLLS 4y agoThis is another "I'm moving to Canada if Trump wins", and nobody of note moved to Canada.
- rlt 4y agoAnd you think you are representative of the average Twitter user, why?
- nemo 4y agolol
- nemo 4y agoCare to revisit this one in a month?
- lawgimenez 4y agoI guess they should bring back the Twitter whale.
- alex_young 4y agoThe fail whale was amazing.
- lawgimenez 4y agoIt is amazing and an indicator if something went viral. I remember Kendrick Lamar dropped his first LPs and Twitter just couldn’t recover for hours. There were no “trends” list before.
- koyanisqatsi 4y agoElon used to be a software engineer so it's kinda weird how he thinks a legacy codebase like Twitter can be fixed up to use $1B less every year. Twitter is limited mainly by storage, their compute costs are negligible in comparison. So unless there is so much bloat in their hardware procurement process that they're over-provisioning storage by more than $1B a year there is no way to reduce costs. The hardware costs what it does and there is no way to rewrite the software at this point to reduce hardware costs and still maintain backward compatibility with the existing Twitter functionality (no matter how minimal the feature set seems to be for external observers). So either this article is a lie or Elon is just making nonsensical statement to focus attention on what it costs to run Twitter at such a scale. If people don't pay then the functionality will continue to degrade and it's looking like there aren't enough willing buyers for $8/month. The only way Twitter can continue to operate is if the people that actively use the service start paying for it, every other financing option is now closed off and if Elon can't turn it around then I doubt anyone else can.
- deleted 4y ago[deleted]
- Hydraulix989 4y agoI struggle to find one correct remark in your comment. This take clearly suggests you have never worked on any similar infrastructure / distributed systems before at scale. So you're telling me there's no backend or compute workers? And you're telling me that there are no opportunities for performance optimization? That they are already 100% native C++ services at full parallel utilization? That they've already tuned the size and cost of their ML models? That their model evaluation infra is fully saturated? Storage is cheap. I believe they are on AWS, so hardware costs are already taken out of the equation. If so, even just moving to on-prem would save them the $1B easily.
- koyanisqatsi 4y agoYou should sit down and perform the storage calculation instead of struggling to find something correct in what I said because that will be a better use of your limited time.
- frellus 4y agoTo be fair, there are going to be plenty of critical leaks coming out of Twitter with a 50% cutting of staff. That's so many whistleblowers, my backyard is completely leaf free now. Frankly, if Musk cuts $1B from the run rate and there isn't a bad outage, a lot of people will look pretty stupid. To be fair, I don't know if Twitter is overspending on infrastructure, but I would guess there are always places to streamline. You could frankly save a ton of money by re-negotiating commits with Google. Also, $1B to me shows how insane running a large business in the cloud can be. This is sort of where Facebook/Meta got it right, IMHO - run infrastructure on-prem, at scale, in a super-efficient "everything fails" sort of way.
- bushbaba 4y agoI don’t think the 1B/year shows issue on cloud pricing. It’s instead bloat of twitter’s setup. That’s way in excess of what should reasonably be spent. I suspect there’s a lot of bad coding practice going on. Of which the bulk is data analytics.
- bagels 4y agoAWS profit margin is something like 35%. I think some of their infra is there, so that might be an upper bound?
- djmips 4y agoDo they use AWS though? https://blog.twitter.com/engineering/en_us/topics/infrastructure/2017/the-infrastructure-behind-twitter-scale https://blog.twitter.com/engineering/en_us/topics/infrastruc...
- colechristensen 4y agoEveryone is “overspending” on infrastructure. There is always money to save the question comes down to where you want to expend your engineering effort and what risk levels you want to take. “Make fewer features, focus on spending less” is a valid executive decision, it’s just a matter of priority. Twitter seemingly being feature complete and losing lots of money most years makes doing this now a decent decision. Spending too much like that is generally a symptom of not paying attention and noticing the waste, it happens from time to time. Gets to feel normal, then you notice and correct. I’ve been mostly responsible for cloud spend a number of times and had several events of cutting spending by 20% or more over the years.
- seibelj 4y agoEveryone on the internet knows how to run Twitter better than the richest man on earth who revolutionized both the electric car and private space industries. I wish I could invest in Twitter now but I’ll wait for the cryptocurrency launch.
- bagels 4y agoIs there some kind of scam coin associated with the twitter deal?
- valeg 4y agoDoge?
- bagels 4y agoIs there any connection beyond Musk hyped it a year or two ago?
- sigmar 4y agohe's been hyping it non stop, including taking doge for payments with the perfume he launched last month https://twitter.com/search?q=from%3Aelonmusk%20doge&src=typed_query&f=live https://twitter.com/search?q=from%3Aelonmusk%20doge&src=type...
- paleotrope 4y agoPeople are treating Musk pronouncements like they did Trumps. Taking extremely literally even when it's obvious he's pulling your leg.
- seibelj 4y agoI am assuming twitcoin is coming and I will buy in
- SicSemperUranus 4y ago
- edge17 4y agoWhy does this article claim to know that Twitter might go down due to cost cutting? Isn't there a possibility that their infrastructure is just hugely wasteful? Having never worked at giant web app company, if it were run efficiently then what should we expect their server and infra costs to be? Surely that is the type of Fermi calculation someone on this site can make.
- colechristensen 4y agoRight? The real cost cutting is usually “hey there’s this huge X sitting around that nobody uses 99% of the time” or “This process is 10000x faster after someone did some performance engineering” or “we’re storing this PB dataset a dozen times for no reason” There are often lots of ways to cut the fat that have nothing to do with reducing (reliability related) redundancy.
- bagels 4y agoThere's also reducing replication from 4x to 3x which can save a lot but also reduce redundancy.
- volume 4y ago> Why does this article claim to know that Twitter might go down due to cost cutting? Isn't there a possibility that their infrastructure is just hugely wasteful? I see the authors (Sheila Dang, Paresh Dave and Katie Paul) quote "sources".
- watwut 4y agoIt is possible to simultaneously have unnecessary spending on infrastructure and then cut it in a way that makes whole thing unreliable. Given large layoffs and general chaos, mistakes and bad cuts are much more likely. Especially when you git deadline and goal saving made by clueless executive.
- shrubble 4y agoThere is plenty of opportunity to renegotiate contracts, since capacity is exploding with 400Gbit going over a single wavelength. It used to be amazing to hit 1 Tb/s with a piece of core network equipment and now it is routine... whether tjat and many other changes is enough to save 1B per year, I don't know.
- brian-armstrong 4y agoWhat does that have to do with anything? The cost of bandwidth is just one part of infra costs, and it comes from peering agreements. Those are expensive because you're dealing with monopolies and they don't really have any reason to bargain with you.
- pcurve 4y ago$1B out of?
- avrionov 4y ago$3B. Twitter operational costs are $3B or 60%.
- bagels 4y agoThis is an incredible opportunity for the engineers remain which Musk surely has converted in to saboteurs with his antics. Sabotage disguised as cost savings.
- partiallypro 4y agoThey'd be easily found out by a smart engineer and promptly fired and probably banned by every competitor once word got out.
- GoOnThenDoTell 4y agoIncompetence and mistakes are usually hard to prove is malice
- partiallypro 4y agoYou can find a pattern or motive pretty quick if it happens often. I don't think many would be stupid enough to risk their high paying job for that. Any sabotage would probably be from a low-level employee with git access or is more customer facing. An engineer making in the high 6 figures isn't going to risk it all just to "own Elon" or something stupid.
- watwut 4y agoone does not want to employ either
- hackeraccount 4y agoThere's proof and then there's the recommendation that you ask for when trying to find your next job.
- tjr225 4y agoI dislike Elon Musk as much as the next guy but why are people upset about Twitter burning to the ground? It’s just another paint brush used to paint the downfall of our society. Good riddance. I hope Elon goes under with them. This is a platform built on driving clicks and driving clicks on fear and anger. More fear = more clicks = more ad revenue = negative feedback loop. Just like the New York Times. Just like NPR. Just like Fox News. Just like Reddit. Just another piece of shit we should all enjoy watching burn into the ground. Spend some time with your family. We will all be incredibly lucky if this ad-tech insanity falls apart.
- koyanisqatsi 4y agoI don't think ad-tech is going away anytime soon. It would be nice but it's not gonna happen because the fundamental problem is not the ads but the fact that most social networks are not sustainable without advertising so the platform participants end up being treated like cash cows that are milked for all they're worth. If Twitter was paid for by its participants then they would not have to put up with advertisers. Right now the advertisers subsidize the operational costs so they get to dictate what features are developed and what kind of invasive access they have to behavioral data for optimizing their marketing campaigns.
- tcmart14 4y agoAnd the worst part. I think there are a lot of us who would be willing to pay for being on a good social network without ads, but I don't think there is enough of us. Too many people have been trained via myspace, facebook and twitter that social media is just "free." It it is going to be hard to change the masses on that.
- hnfong 4y agoAs long as somebody has friends who can't afford say $10 per month for social media, the network effect takes over and the result will inevitably be ad-ridden crap. I might be willing to pay to turn off ads and tracking on such a platform, but if all the middle-class+ people pay to turn off ads, which advertisers would want to spend money targeting "poor" people and people too cheap to pay a couple dollars for using social media?
- wyldfire 4y agoIt could be the case that Musk is an a-hole who happens to be right. Maybe Twitter's OPEX expanded to fill whatever seemed right for the massive ad revenue (or "potential" revenue). They probably did not need an enormous engineering team to keep the site running. But beyond "just running" I am curious how difficult it is for them to combat spam/bots/griefers/hatemongers. Spending money on moderators and some kind of engineering to make the moderation team productive is a good investment. As much as some folks hate it, I think paying for the service makes a great deal of sense. Just imagine how much better these things might be if they weren't all driven by adtech. But yeah it could also easily become Twitter's death knell. That network effect will work in your favor right up until it starts working in your biggest competitor's favor. Market leaders can't hold on forever. re: "paying for it" - I think it would be a potentially awesome experiment to provide a service like [Gmail+Twitter+LinkedIn] for something like $5/mo or maybe even $10/yr. And what if you could get some incremental funding from government grants or NGOs & philanthropists who saw this as a public good (the same way some endowments fund public radio, TV etc).
- bumblebritches5 4y ago
- shawnb576 4y agoI worked at a large valley based company that had their own data centers and badly wanted to decrease infra spend. It's WAY harder than it sounds. Remember Twitter is not a profitable company which means the likelihood of a bunch of low-hanging fruit for savings is very low. That's been done already. There are not likely "oh look we found a whole DC sitting idle, oops" sort of savings. Any vendors they use are ALREADY giving them huge discounts. I'm not saying Twitter is efficient or that this is physically impossible (tho it might be), I'm saying its not easy and takes skilled people a long time to iterate to lower-cost systems or designs that don't have massive stability or performance tradeoffs.
- djmips 4y agoAll of what you say is speculation. In reality there might be a huge amount of waste where it's just become an accustomed part of the culture <- also speculation.
- nullc 4y ago> Remember Twitter is not a profitable company which means the likelihood of a bunch of low-hanging fruit for savings is very low. That's circular. Their revenue is astronomical. They're not profitable because they're spending it all. That doesn't mean there aren't plenty of low hanging opportunities (or that there are). Hidden recordings of twitter staff had the describing the places as more of a lifestyle business where profit isn't a major goal and many people worked very limited amounts. It could have been useless bluster but if there was any basis to it, it's a good indication that massive savings should be possible.
- kornhole 4y agoAre they selling any servers? I need them to setup Mastodon servers.
- encryptluks2 4y agoSure you can find unused cloud environments to spin down and try to switch away from expensive vendor agreements, but any significant costs savings is going to require a lot of optimization and talented employees and I get the impression that the talented employees that wanted to work at Twitter already do.
- BrandoElFollito 4y agoI work and worked for huge companies in the direct CIO team. Many times I thought "that is wasteful, we should do X". I self-host a lot at home and am a standard nerd and what I learned over 30 years is that my knowledge does not scale at all. In such companies even if the DC is idle, knowing what happens when you switch it off does not exist. Even worse, when cutting/moving a service, three blocks down something crumbles and their custodian has no idea why (and worse: they would not even know that something was switched off if there is no strict change management). It is easier if you are a modern company that uses some modern best practices, there is at least hope that the configuration gives you a clue.
- 7e 4y agoMusk failed to deliver the $35K Model 3 and Starlink prices are rising. I doubt there is a billion dollars left lying on the table, at least not accessible to a man of Musk’s nature.
- mlindner 4y agoRight now Tesla has waiting lists so long they've head to shut off ordering of new vehicles. They hardly need a vehicle that's any cheaper at this point. Additionally, Starlink has so many people wanting to use the service that the entire eastern half of the US is basically at capacity because demand is too high. Increasing prices for both products seems like a very good idea.
- hackeraccount 4y agoI'm not convinced this is bad for Twitter - if they reduce their scope with it. They should open the API back up and let third party developers step in and start doing all the stuff - filtering for example - that they don't do a good job at. Just be a login and a platform.