6 ms·

Despite all of the snark here, in my experience Salesforce SRE team is quite competent. The engineering challenges of running a large PaaS - not just with own a

by stmw 11d ago

Despite all of the snark here, in my experience Salesforce SRE team is quite competent. The engineering challenges of running a large PaaS - not just with own apps, but with millions of customer-written apps running on it - are quite interesting, and sadly things happen. The status page makes sense to actual customers, it's the particular "pods" where a given service runs.

mulmen 11d ago
I honestly don’t get the snark. The status page has: Seemingly meaningful IDs Search Region filter Email update signup Predictable URLs for instance status so they can be deep linked in runbooks What appears to be the actual live instance status. What appears to be the actual live service status in each instance. An update log with frequent detailed updates.
- dominotw 11d ago
  [flagged]
- troyvit 11d ago
  I despise Salesforce, but when I landed on this page I was like, huh. wow. honesty. Looks at GitHub So yeah you're exactly right, the snark is not deserved if you ask me, and I'm 82% snark.
shimman 11d ago
This is the case for every single B2B saas product. This is like the "bar is rolling on the floor" level of competence required. Please have higher standards for paid products.
- eastbound 11d ago
  What do you think of Atlassian?
  - shimman 11d ago
    Terrible, I'd argue the vast majority of modern big tech offerings are extremely poor quality where the need for surveillance in the form of constant monitoring/advertising metrics deliberately makes these types of services more costly to maintain and repair over time. Sure there are like 3 or 5 decent services out there (like S3) but the vast majority are over engineered to be user hostile while extracting out whatever resources they can from their customers.
Anon1096 11d ago
Hacker News is much easier to read when you realize that 95% of people have never worked on a "high" (maybe we could say >1B requests per day as a starting point) scale distributed service and think it's trivial to run one with more than 2 nines. You see comments all the time here mentioning that their own desktop at home is achieving more than that which belies deep misunderstanding of how systems are measured. Or that unofficial github status page repeatedly posted here that counts all github services together into one number.
- ocdtrekkie 11d ago
  > which belies deep misunderstanding I think you are missing the point. When I state my Exchange server is more reliable than Exchange Online, I don't think I'm a better engineer. I recognize Microsoft has harder problems to solve than I do. I think building overengineered, oversized SaaS environments is introducing extreme risk. It's an inherent flaw of the current approach. Smaller is, in fact, better, because it's easier to operate reliably.
  - toomuchtodo 11d ago
    Indeed, the scale Anon1096 refers to wrt distributed systems is anti pattern. It is designed to vacuum up revenue and create enterprise value with scale, not to create resiliency for customers (although resiliency might be a byproduct of a well architected and operated distributed system at scale). "Simplicity is the ultimate sophistication." -- Da Vinci
    - a_conservative 11d ago
      Hidden in this discussion around self-hosting reliability are other options as well. Depending on your time and appetite for tinkering with all of this, it's not hard to imagine a home setup that fails over to a cheap Hetzner or DO VM. A manual failover at the DNS level isn't overly complex, and could be scripted. Keeping a database in sync between home and the instance might be simple or more complex depending on needs, but would it really be that hard to have Claude help you setup a replicating Postgres server? If your database (or data files) are 1 gigabyte and don't update that often... maybe just rsync it every night or something There's a thread you and others are pulling on here, and we need to pull it. Hosting doesn't have to be the domain of the big vendors anymore.
gibsonf1 11d ago
Hmm, could the use of genAI have anything to do with this failure and the inability to quickly fix it?
- Jach 11d ago
  It's not impossible, but Salesforce has had big outages before LLMs. For a disruption that began at 1am pacific, the response time isn't that bad. 3 hours total to give up on restarts, 4 hours total to validate a quick fix and begin rollout, and the rest of the time since has been waiting for the rollout + addressing subsets of instances that had some issues with restarting+the quick fix. It's nearly 9am pacific now, so Dreamforce is saved~ (It's Dreamforce week this week. Most devs are either focused on that or on soft-vacation / working on lower priority non-feature-work items, it's surprising anything would be updated to production this week that could do this.) The architecture and approval process of everything there has long been setup so that things can't be changed quickly.
- karagenit 11d ago
  The bug was from 2009, so probably not :)
trebligdivad 11d ago
What I'm curious about is why it is a single-PaaS; I'd have expected Salesforce to have the customers quite isolated so the chance of bringing down multiple customers at once was much smaller.
- prpl 11d ago
  a few things are global (login service) to some extent, just as AWS places several such things in us-east-1
- kakwa_ 11d ago
  You still have fleet wide management which can cause issues. Plus there are always a few core services like queues, authz, presentation layer. Also, mono-tenant architectures is no golden bullet either. Such architecture (often coming from a formerly on-premise product that was SaaS-ified) can easily become hell to operate as it multiplies the integration points (DB parameters, URLs, allowlists, etc). It's also quite wasteful in terms of resource utilization and hosting costs.
- stmw 11d ago
  The customers are quite isolated, but it doesn't mean that some services or errors do not propagate. In public cloud terms, think back on some AWS or Azure or even Gmail outages - you probably wouldn't even hear about them if it didn't affect millions of users at once, across security and availability boundaries.
  - Cthulhu_ 11d ago
    I think it's easy to underestimate how many smaller outages there are in any period of time but which do not affect anyone or only a small number of people due to all the mitigations, due diligence that developers and SREs do, and the self-repairing nature of modern systems.
nightski 11d ago
For me personally the snark isn't because their SRE team is incompotent. It's because software that tries to be everything to everyone is inherently terrible. It's not fun to use for the users, and so many compromises need to be made on the technology side to make that happen that it ends up just being crap all around. This includes Salesforce, SAP, Dynamics, any platforms like that which scale many industries. Flexibility and abstraction come at a high cost. It doesn't really matter though, world domination at all costs is the name of the game.
- elzbardico 11d ago
  AI changes this equation fundamentally, in a way that VC and SaaS founders still haven't realized. Nobody ever liked having to change their business, their workflows, or ducting taping a customization in a SaaS, they did because as we moved from centralized mainframe apps, to PC client-server and then Web based distributed apps, it became increasingly more cost-effective to suffer with a generic, one-size-fits-all SaaS than building at home. AI coding changes this a lot.
  - Cthulhu_ 11d ago
    I'm going to give this a "maybe"; the challenge with large scale software isn't in authoring new code or whatever, it's in managing complexity. I think AIs / agents (and more importantly how we are learning to use them effectively) may help in that regard, but only if they are able to manage that complexity. This'll depend on context window sizes, their ability to explore a codebase, and how well their operators can provide relevant information. But that's only what they can consume (so codebase, documentation, etc), on top of that are the people that work with / for these systems for decades and who know a lot about things outside of what's written down.
- Cthulhu_ 11d ago
  Thing is, if one piece or suite of software doesn't try to be everything (in a fairly consistent way), businesses that need certain functionality will end up with various different tools from different suppliers, which is at the very least just as complex and expensive to manage, and in practice more expensive.
kennethops 11d ago
As a previous SRE at Cloudflare, I'll never shit-talk fellow SREs at big companies. The level of scale and complexity a big tech SRE has to deal with on a constant day-to-day is a very imbalanced proposition. A lot of people, in my experience, are not fully comprehending. You have to be a jack-of-all-trades and a master of all.
codeulike 11d ago
Yes, the thing Salesforce are good at, and is little understood here, is that they've kept their platform online for 27 years so far. Its constantly evolving, three upgrades per year, but changes that require customers to change their customisations are rare, and when they happen they are communicated at least a year in advance. Approx 150,000 tenants, all with different configurations and some so heavily customized that they are effectively unique apps. Salesforce keeps them all online and evolving. In those 27 years there hasnt been a 'lets trash this and rewrite from scratch' and there hasn't been a 'you must migrate your data to our new platform, we're closing the old one'. They've just evolved it while running. They must have got some things very right in the original architecture to be able to do that. One thing that I find interesting is that they launched their platform language Apex (a sortof subset of Java) in 2007 when TDD was the hot new thing, so TDD is baked into the platform - your Apex code must have at least 75% test coverage, and the tests must pass, before you are allowed to deploy to prod. They leverage that test coverage when they are upgrading the platform - they have an internal process called The Hammer where they run all customer-created tests against customers own unique configs on the current platform and then again on the next version of the platform to see if any customer tests are being broken. Look it up, its really interesting.
- Cthulhu_ 11d ago
  > They must have got some things very right in the original architecture to be able to do that. I'd argue they (and many larger, older, established etc systems) may not have, but it's part of how it works so while it may not be the best it's the one that is working right now and earning them money - working (and earning) software always trumps correctness etc, in practice.
  - deleted 11d ago
    [deleted]

by stmw 11d ago

mulmen 11d ago

I honestly don’t get the snark. The status page has: Seemingly meaningful IDs Search Region filter Email update signup Predictable URLs for instance status so they can be deep linked in runbooks What appears to be the actual live instance status. What appears to be the actual live service status in each instance. An update log with frequent detailed updates.

dominotw 11d ago
[flagged]
troyvit 11d ago
I despise Salesforce, but when I landed on this page I was like, huh. wow. honesty. Looks at GitHub So yeah you're exactly right, the snark is not deserved if you ask me, and I'm 82% snark.

shimman 11d ago

This is the case for every single B2B saas product. This is like the "bar is rolling on the floor" level of competence required. Please have higher standards for paid products.

eastbound 11d ago
What do you think of Atlassian?
- shimman 11d ago
  Terrible, I'd argue the vast majority of modern big tech offerings are extremely poor quality where the need for surveillance in the form of constant monitoring/advertising metrics deliberately makes these types of services more costly to maintain and repair over time. Sure there are like 3 or 5 decent services out there (like S3) but the vast majority are over engineered to be user hostile while extracting out whatever resources they can from their customers.

Anon1096 11d ago

Hacker News is much easier to read when you realize that 95% of people have never worked on a "high" (maybe we could say >1B requests per day as a starting point) scale distributed service and think it's trivial to run one with more than 2 nines. You see comments all the time here mentioning that their own desktop at home is achieving more than that which belies deep misunderstanding of how systems are measured. Or that unofficial github status page repeatedly posted here that counts all github services together into one number.

ocdtrekkie 11d ago
> which belies deep misunderstanding I think you are missing the point. When I state my Exchange server is more reliable than Exchange Online, I don't think I'm a better engineer. I recognize Microsoft has harder problems to solve than I do. I think building overengineered, oversized SaaS environments is introducing extreme risk. It's an inherent flaw of the current approach. Smaller is, in fact, better, because it's easier to operate reliably.
- toomuchtodo 11d ago
  Indeed, the scale Anon1096 refers to wrt distributed systems is anti pattern. It is designed to vacuum up revenue and create enterprise value with scale, not to create resiliency for customers (although resiliency might be a byproduct of a well architected and operated distributed system at scale). "Simplicity is the ultimate sophistication." -- Da Vinci
  - a_conservative 11d ago
    Hidden in this discussion around self-hosting reliability are other options as well. Depending on your time and appetite for tinkering with all of this, it's not hard to imagine a home setup that fails over to a cheap Hetzner or DO VM. A manual failover at the DNS level isn't overly complex, and could be scripted. Keeping a database in sync between home and the instance might be simple or more complex depending on needs, but would it really be that hard to have Claude help you setup a replicating Postgres server? If your database (or data files) are 1 gigabyte and don't update that often... maybe just rsync it every night or something There's a thread you and others are pulling on here, and we need to pull it. Hosting doesn't have to be the domain of the big vendors anymore.

gibsonf1 11d ago

Hmm, could the use of genAI have anything to do with this failure and the inability to quickly fix it?

Jach 11d ago
It's not impossible, but Salesforce has had big outages before LLMs. For a disruption that began at 1am pacific, the response time isn't that bad. 3 hours total to give up on restarts, 4 hours total to validate a quick fix and begin rollout, and the rest of the time since has been waiting for the rollout + addressing subsets of instances that had some issues with restarting+the quick fix. It's nearly 9am pacific now, so Dreamforce is saved~ (It's Dreamforce week this week. Most devs are either focused on that or on soft-vacation / working on lower priority non-feature-work items, it's surprising anything would be updated to production this week that could do this.) The architecture and approval process of everything there has long been setup so that things can't be changed quickly.
karagenit 11d ago
The bug was from 2009, so probably not :)

trebligdivad 11d ago

What I'm curious about is why it is a single-PaaS; I'd have expected Salesforce to have the customers quite isolated so the chance of bringing down multiple customers at once was much smaller.

prpl 11d ago
a few things are global (login service) to some extent, just as AWS places several such things in us-east-1
kakwa_ 11d ago
You still have fleet wide management which can cause issues. Plus there are always a few core services like queues, authz, presentation layer. Also, mono-tenant architectures is no golden bullet either. Such architecture (often coming from a formerly on-premise product that was SaaS-ified) can easily become hell to operate as it multiplies the integration points (DB parameters, URLs, allowlists, etc). It's also quite wasteful in terms of resource utilization and hosting costs.
stmw 11d ago
The customers are quite isolated, but it doesn't mean that some services or errors do not propagate. In public cloud terms, think back on some AWS or Azure or even Gmail outages - you probably wouldn't even hear about them if it didn't affect millions of users at once, across security and availability boundaries.
- Cthulhu_ 11d ago
  I think it's easy to underestimate how many smaller outages there are in any period of time but which do not affect anyone or only a small number of people due to all the mitigations, due diligence that developers and SREs do, and the self-repairing nature of modern systems.

nightski 11d ago

For me personally the snark isn't because their SRE team is incompotent. It's because software that tries to be everything to everyone is inherently terrible. It's not fun to use for the users, and so many compromises need to be made on the technology side to make that happen that it ends up just being crap all around. This includes Salesforce, SAP, Dynamics, any platforms like that which scale many industries. Flexibility and abstraction come at a high cost. It doesn't really matter though, world domination at all costs is the name of the game.

elzbardico 11d ago
AI changes this equation fundamentally, in a way that VC and SaaS founders still haven't realized. Nobody ever liked having to change their business, their workflows, or ducting taping a customization in a SaaS, they did because as we moved from centralized mainframe apps, to PC client-server and then Web based distributed apps, it became increasingly more cost-effective to suffer with a generic, one-size-fits-all SaaS than building at home. AI coding changes this a lot.
- Cthulhu_ 11d ago
  I'm going to give this a "maybe"; the challenge with large scale software isn't in authoring new code or whatever, it's in managing complexity. I think AIs / agents (and more importantly how we are learning to use them effectively) may help in that regard, but only if they are able to manage that complexity. This'll depend on context window sizes, their ability to explore a codebase, and how well their operators can provide relevant information. But that's only what they can consume (so codebase, documentation, etc), on top of that are the people that work with / for these systems for decades and who know a lot about things outside of what's written down.
Cthulhu_ 11d ago
Thing is, if one piece or suite of software doesn't try to be everything (in a fairly consistent way), businesses that need certain functionality will end up with various different tools from different suppliers, which is at the very least just as complex and expensive to manage, and in practice more expensive.

kennethops 11d ago

As a previous SRE at Cloudflare, I'll never shit-talk fellow SREs at big companies. The level of scale and complexity a big tech SRE has to deal with on a constant day-to-day is a very imbalanced proposition. A lot of people, in my experience, are not fully comprehending. You have to be a jack-of-all-trades and a master of all.

codeulike 11d ago

Yes, the thing Salesforce are good at, and is little understood here, is that they've kept their platform online for 27 years so far. Its constantly evolving, three upgrades per year, but changes that require customers to change their customisations are rare, and when they happen they are communicated at least a year in advance. Approx 150,000 tenants, all with different configurations and some so heavily customized that they are effectively unique apps. Salesforce keeps them all online and evolving. In those 27 years there hasnt been a 'lets trash this and rewrite from scratch' and there hasn't been a 'you must migrate your data to our new platform, we're closing the old one'. They've just evolved it while running. They must have got some things very right in the original architecture to be able to do that. One thing that I find interesting is that they launched their platform language Apex (a sortof subset of Java) in 2007 when TDD was the hot new thing, so TDD is baked into the platform - your Apex code must have at least 75% test coverage, and the tests must pass, before you are allowed to deploy to prod. They leverage that test coverage when they are upgrading the platform - they have an internal process called The Hammer where they run all customer-created tests against customers own unique configs on the current platform and then again on the next version of the platform to see if any customer tests are being broken. Look it up, its really interesting.

Cthulhu_ 11d ago
> They must have got some things very right in the original architecture to be able to do that. I'd argue they (and many larger, older, established etc systems) may not have, but it's part of how it works so while it may not be the best it's the one that is working right now and earning them money - working (and earning) software always trumps correctness etc, in practice.
- deleted 11d ago
  [deleted]