5 ms·

Reminds me of: "Nobody ever got fired for buying cloud". Spotify was a stand-out, almost no large tech company bought into cloud like they did, and everyone sa

• ttymck 3y ago
  This is a wild leap of twisted logic. You posit that they might not have to lay off employees if they instead.... Paid a team to manage their infrastructure instead of outsourcing that responsibility to the cloud.
• kranke155 3y ago
  How big is the difference in your estimate, between the bills they have now vs cost of cloud + staff to support it?
  • dijit 3y ago
    Largely depends. For ML workloads anything between 6-8x cost increase for using cloud. For CDN costs somewhere like 3x (they're pretty smart on discounts here, without those it could be 100's of x) For compute somewhere like 3-4x (would be 6x before the discounts) Managed DB is a hard one because the cost is really high but I'm not 100% sure how much time is saved with the solution they chose because while I saw the bill I don't work there. It gets dicey with humans because it's apples to oranges, expensive SRE's who understand the inner-workings of cloud vs cheap(er) hardware folks who set things up from first principles with a higher up-front cost. -- FinOps with more traditional solutions is easier but you still need people for that. Spotify's backend load is not entirely elastic. There are some elastic bits but a large chunk is pretty static.
• stbenjam 3y ago
  Finding people who can run infra at Spotify scale isn’t easy. Even if they could find those people, it tough to leave aws (by design). They could’ve been more careful about avoiding lock-in by building more on something like kube but most startups don’t have that foresight, and the expense of moving to on-prem is compounded even more when they have so much wrapped up in the aws ecosystem.
  • dijit 3y ago
    They've already managed a switch from AWS to Google Cloud once in their past.
  • mrweasel 3y ago
    > Finding people who can run infra at Spotify scale isn’t easy. You won't get any argument from me on that one, but it's worth remembering that it's also hard to find people who are able to run systems in the cloud cost effectively.
  • shiftpgdn 3y ago
    TikTok found all of those people very quickly, they famously don't use a public cloud provider. Infra isn't magic.
    • deleted 3y ago
      [deleted]
    • hiddencost 3y ago
      TikTok has 100k employees and shoes shits money.
      • dijit 3y ago
        Not sure what "shoes shits money" means, but contextually they had $9.9B in revenue for the year 2022 and an operational expenditure of $9.4B. I believe this means they generated $500m for bytedance, but I can't find a good citation, bytedance itself generated $6B in profits though for the fiscal year 2022 https://www.statista.com/statistics/1342785/bytedance-key-quarterly-financial-figures/ https://www.statista.com/statistics/1342785/bytedance-key-qu...
• ctvo 3y ago
  Where in the article did you see cloud costs being the reason for this layoff? > Spotify is cutting almost 1,600 jobs as the music streaming service blamed a slowing economy and higher borrowing costs in the latest round of redundancies at big tech companies. It clearly states later that in a zero interest environment Spotify borrowed heavily and over hired, in this environment it’s no longer sustainable and they had to let employees go. Did you just assume “costs” meant infrastructure costs like cloud?
  • dijit 3y ago
    Cost's don't exist in a vacuum. it's easier to lay people off than to save millions in cloud spending when you've already committed to $x-billion over 5 years.
    • ctvo 3y ago
      A single engineer is 400k in total costs, and they hired thousands for initiatives that didn’t pan out. Thinking cloud costs come anywhere close to being a factor here is nonsense and not supported by anything in the article. You extrapolated that costs meant cloud costs vs. them overhiring during the pandemic and investing in things like podcasts, which haven’t had the expected returns. You did this in an attempt to tie this back to some injustice that happened to you in a comment section far far away, which you’ve edited the post to remove now, thankfully.
      • dijit 3y ago
        Five things wrong 1. I didn't edit my comment to remove anything like that, so I'm not sure what you're suggesting. 2. I didn't perceive injustice, I thought it was a bit brainless to not associate company costs with long term survivability. 3. An engineer being 400k TC is an anomaly, Spotify does not pay any of it's Swedish engineering force nearly that much, and since we don't know the demographics of the layoffs it's hard to argue engineers anyway. (Citation here says the avg was 125.000[0] which is still very high if these were europeans) 4. Their cloud bill would still be roughly 500 people based on that TC based on committed use alone. 5. Costs = Costs. Overspending in many areas = no more money. I'm merely suggesting that they overspent in one area that is now affecting another area indirectly. Yes, I drew a line. [0]: https://news.ycombinator.com/item?id=38516794 https://news.ycombinator.com/item?id=38516794
• cedws 3y ago
  This is why I'm bullish about Oxide (who are building hardware to make it easy to essentially run your own cloud.) Nobody wants to deal with the nasty parts of running servers but if a company can take some of that complexity away, you could save a tonne of money over AWS.
  • notRobot 3y ago
    How is running your own cloud different than traditional bare metal?
    • cedws 3y ago
      For one, you can use your compute resources much more elasticly. You can create and delete VMs on demand and don't need to drive to the colo to reimage.
    • dboreham 3y ago
      More automation.
    • throwboatyface 3y ago
      Traditional bare metal involved complicated packaging and release processes. You would literally have a diagram of "these are the physical servers running this service (LB, web server, etc.)" and be able to point to them. You could overprovision workloads to tolerate failure but not just shuffle a service from one machine to another. When I used to work in a similar environment we would develop code, then we would give it to a QA team. They would test it and give it to an Ops team. The Ops team would schedule a maintenance window and roll out the new code on each server. This happened maybe once a quarter because testing and releasing was a week-long process. Racking new servers and provisioning them also required some manual labour. We had a process to use PXE to provision the machines but it was still toil. Virtualization was a big benefit because you could at least create and blow away VMs without having to re-image a whole server from scratch. "Running your own cloud" implies that developers can treat instances like cattle and interact via an API. But it also means there's a standard set of tooling for fleet management. None of this stuff is entirely new but in small or mid-sized orgs it was out of reach 10 years ago.
• throwaw12 3y ago
  > Spotify was a stand-out, almost no large tech company bought into cloud like they did Isn't this what made Spotify success? Yes, they could have build their own datacenters, but someone else could outcompete them in the meantime. They have traded off something for another thing.
  • philistine 3y ago
    I don't think one of you has to be wrong for another one to be right. Should Spotify have focused on its core competency early in its life? Sure. But at some point in the long years since, Spotify should have 100% looked at their cloud bills and built a way out of that lock-in.
    • throwaw12 3y ago
      > Spotify should have 100% looked at their cloud bills and built a way out of that lock-in. Absolutely, but everything has ROI, ROI of fixing cloud bills were probably smaller than "possible growth" Now they know which features failed, they can start deprecating or stop growing them and refocus on cloud bills
  • dijit 3y ago
    I think that's a good take, it's easy to look in hindsight and say that they made wrong choices but we wouldn't even be having the conversation had they not been at least somewhat successful. The question becomes then: 9,500 employee's and none among them know how (or wanted) to regain enough costs to save 17% of the company from losing their job just before Christmas? Certain features (spotify wrapped for example) would definitely be a lot harder with more traditional hosting. But the bulk of Spotify's workload are not significantly hampered by moving slower on infrastructure and recuperating significant costs. It's just very easy to keep piling infra costs on top of infra costs and not thinking about it, especially if it feels disconnected from company financials for most developers and they do not see a line between spending and jobs.
    • marcinzm 3y ago
• marcinzm 3y ago
  It's actually non-trivial to run an efficient cost center department that actually supports other departments properly. The incentive structures are just horribly difficult to align. Cloud's selling point is that you won't have to spend 6 months filling out forms to get a single out of date VM (yes, that happened to me).
  • nopassrecover 3y ago
    Until you get the worst of both worlds and have to align an enterprise architecture approval committee, get cyber sign-off, put the request through your service desk, have it land with some sort of enterprise task prioritisation team, and finally land with your internal cloud team to action the request your team isn’t empowered to do, only to then have a simple service provisioning turn into a massively overcomplicated set of extra cloud bits and pieces getting provisioned and retemplated and secured and configured over the course of months, as that overallocated team find 20 minutes here or there only to hit some new wall as the cloud settings look different to the last time they did it or some unforeseen setting requirement was overlooked in the original request, leading to a two week lag on email responses and ticket handover before it gets picked up again.
    • marcinzm 3y ago
      That is why companies such as Spotify don't focus on cost savings as much as people here seem to think they should. A focus on cost savings inevitably means bureaucracy which means time spent on that instead of on product features. That means not just less growth for the stock markets but also more attrition due to good engineers hating the process. Of course at some point it's unsustainable but right until that moment it makes perfect sense. edit: And that moment depends not just on costs but the growth rate of the company so it's almost impossible to predict ahead of time (otherwise wall street investing would be a lot more boring).
    • pjmorris 3y ago
      You forgot to include the meetings and paperwork involved in reporting on the SOC2 compliance.
  • 3y ago
• spaceywilly 3y ago
  I would caution against drawing grand conclusions from these recent layoff rounds. Companies overhired during the pandemic, and now they are trimming that fat. It’s not necessarily a sign of any big strategic mistakes beyond “we hired too many people”
  • jklinger410 3y ago
    > Companies overhired during the pandemic, and now they are trimming that fat Citation needed
    • rcbdev 3y ago
      Anyone who had a LinkedIn profile in tech with open DMs during that time will probably confirm this.
    • spaceywilly 3y ago
      The original article >ValueAct’s chief executive, Mason Morfit, said Spotify’s costs had “exploded” and that it was “built for the bubble”
      • jklinger410 3y ago
        COVID != "the bubble"
    • dlkf 3y ago
      I think this is actually pretty well-established, eg https://news.crunchbase.com/layoffs/analysis-big-tech-pandemic-amzn-meta/ https://news.crunchbase.com/layoffs/analysis-big-tech-pandem...
• soderfoo 3y ago
  Generally I agree with what your saying, but > I've seen their cloud bill and it's significantly higher than 1,600 peoples jobs An apples-to-apples comparison requires looking at all associated costs of switching to on prem (salaries, hardware, etc...). I've only been a part of this analysis at a company dealing with a cloud bill in the low tens of millions, nowhere near the scale of Spotify (and I'm still relatively new to all of this). > And cloud has a lot more lock-in than most employees. It's a shame there are not better abstractions to facilitate moving to and fro.
  • falserum 3y ago
    Cloud providers would benefit from actively fighting such abstractions.
  • therealdrag0 3y ago
    Kubernetes is a pretty decent abstraction no? If all your apps and databases are containerized you can deploy in azure or aws just as well. And many cloud DBs support open source interfaces like Cassandra or JMS or Kafka etc.
    • nunez 3y ago
      It's a great abstraction that is _still_ a huge leap to make for companies who aren't bought in yet.
• tootie 3y ago
  Is their cloud bill more or less than they pay Joe Rogan?
  • dijit 3y ago
    more, actually.
• bogomipz 3y ago
  >"Spotify was a stand-out, almost no large tech company bought into cloud like they did, and everyone said the same thing: "It's not our core competence, it would require more people". I'm not sure what you mean by this as Spotify was actually quite late buying into cloud. They ran on bare metal through at least 2015. From their engineering blog: ">Thus, in early 2015 we started exploring what a cloud strategy would look like for Spotify."[1] So they would have been running on bare metal through some of the biggest spikes in growth i.e launching in the US market. I don't think they bought in any more or any less than anyone else who transitioned from bare metal to cloud and for probably the same reasons. [1] https://engineering.atspotify.com/2019/12/views-from-the-cloud-a-history-of-spotifys-journey-to-the-cloud-part-1-2/ https://engineering.atspotify.com/2019/12/views-from-the-clo...
• apwell23 3y ago
  >I've seen their cloud bill how did you see this. How do you know what all the discounts and backdoor deals that were applied to the bill. How do you know the final amount that was paid out. i am skeptical that some rando is privy to this information.