13 ms·
Salesforce Global Outage
- WhenItRuns 12d ago[flagged]
- cmiles8 12d agoDreamforce, wake up. You’ve overslept!
- cyberpunk 12d agoThat status page is the most salesforce thing ever. Scroll down. >_<
- santoshalper 12d agoAt least they're consistent about UX. My only complaint is that it needs more tabs.
- dgellow 12d agoI see tabs with a spinner loading infinitely, which is indeed very salesforce like
- sunrunner 12d agoWow, it's almost as long as the Every UUID V4 or Every Floating Point Number pages.
- chrisjj 12d agoYup. No mention of outage. Even drilling down gets nothing more than "Service Disruption".
- troupo 12d agoIt lists all instances, you can drill down into each one of them, see which services are affected, and each affected service pops up the incident timeline? Isn't it actually amazing, and not "the most salesforce thing ever"?
- gjvc 12d ago[flagged]
- arionmiles 12d agoThis is certainly a unique status page.
- dboreham 12d agoAt least now we can figure out what Salesforce does.
- sunrunner 12d ago"And in tonight's news, the worldwide CRM solution Salesforce had a global outage affecting one hundred percent of its customer base. We interviewed users of the service to find out the scope of the impact. Everyone agreed that they were impacted, but strangely, nobody could describe _in what way_ they were affected."
- adamw2k 12d agoPerfect timing with Dreamforce this week.
- sparkling 12d agoRemind me please, what are folks currently paying per seat for this glorified CRUD app?
- warmedcookie 12d agoIt's this thing called golf course driven development
- electroweak 12d agoPermanent cache for that one.
- raverbashing 12d agoIf I had been just out of Uni I would think this is edgy Now I'm just glad I'm not responsible for this fire
- abeyer 11d agoThere is a published sticker price, but I don't think anyone outside of tiny installs with just a few users pays that... it's very much the enterprise sales model of "let's schedule a call and talk about it." They also for many years would let previously negotiated prices stand during renewals even when sticker price went up, so older orgs often have the "same" license being charged at many different prices based on when it was first purchased and how the negotiations went at the time.
- fr2029 12d ago[dead]
- theshrike79 12d agoWas it DNS? Any guesses? :)
- dude250711 12d agoVibe-slop, push to Prod?
- geerlingguy 12d agoThey let their agentic AI take over system maintenance at Dreamforce yesterday like they were pushing in the talks /s (partly)
- mariopt 12d agoCould it be people doing Claude/GPT automations and they just can't handle it?
- mulmen 12d agoCould be. With a big conference going on and the updates mentioning resource exhaustion it could be a bunch of people doing demos, AI driven or not. Basically slashdotting themselves.
- jackdecker 12d agoAh yes. Exactly what a status page should look like: an endless list of random ID’s that don’t mean anything and no information whatsoever At least salesforce is consistent with their design language
- chasd00 12d ago> random ID’s that don’t mean anything and no information whatsoever If you use salesforce you know what all of that stuff means. Just click on one, it’s not rocket surgery.
- jackdecker 12d agoWas really just poking fun at them - AWS’s status page isn’t much better
- deleted 12d ago[deleted]
- BoorishBears 12d agoLooks like a region list to me, maybe just with a lot of regions
- deleted 12d ago[deleted]
- deleted 12d ago[deleted]
- dd8601fn 12d agoRandom Ids? If you mean the “USA324” ones, those are pods. If you’re a customer you know which one(s) you care about.
- xboxnolifes 11d agoThis might actually be one of the most useful status pages ive seen. Its not just random green tick marks representing the entire service that only change yellow when someone gives and admits that 5 hours of bad service is an outage.
- jtrn 12d agoAnd nothing of value was lost. God, I hate everything about Salesforce. Sometimes I have to integrate against their services, and it is always a pain, not to mention what the core project actually is: optimization of marketing and spam.
- chasd00 12d agonot really defending salesforce but OAuth+REST is a pain? Pretty plain vanilla in terms of integration requirements.
- dd8601fn 11d agoSure, if you’re struggling with weird custom endpoint behavior, or custom triggers and validation, that’s on your admins and devs. But if you’re having trouble with the standard rest or bulk apis, that’s 100% on you.
- jtrn 11d agoIf you think integration with Salesforce = REST API then you have no idea what you are talking about.
- felineflock 11d agoWhat is painful about a plain REST API?
- jtrn 11d agoA normal sensible REST api ? Nothing. Salesforce API? Sandboxes drift from production in both metadata and data. Refreshes change record ids. API access is gated by edition. Professional edition has none by default. Salesforce sells MuleSoft as the fix to their own shit architecture. Custom objects and fields mean that connecting successfully doesn’t establish what represents a customer, subscription or completed sale. Even standard objects can be used differently between organizations. Metadata deployment is SOAP-based and slow. Dependencies between components break deploys and is massive pain to debug. And notice that I said integration, not API integration. There’s a shit storm of terrible way to integrate beyond just simple API. Have fun with Bulk 1 or 2, Composite, Streaming, Platform Events, Change Data Capture, Tooling, GraphQL. An my most hated item, anything web based. I seriously want to talk to someone that have a good time integrating against Salesforce. Maybe I will have my mind blown as to how easy it could be, but I suspect I will have the same experience I have every time I critical scrutinize and such situation: it’s just as terrible as it seems and the person claiming it’s easy is hands down producing close to nothing of value.
- AmazingTurtle 12d agoYou better bet someone started their agents with a prompt "Make a salesforce clone but with 100% uptime"
- Bluestein 12d ago"Make a salesforce clone but with 100% uptime" ⎿ You've hit your session limit · resets 2:53am (48°52.6′S, 123°23.6′W Etc/GMT+8) /upgrade to increase your usage limit.
- danjc 12d agoIt's dns isn't it
- afr0ck 12d agoClassic
- ghusto 12d agoNo, IPv6
- Bluestein 12d agoIt's always DNS.-
- bearjaws 12d agoFeel like it has to be for all of this to go down at the same time.
- dogas 12d agoI was scrolling to see this comment, haha
- whatthesmack 11d agoThe last time it was DNS at Salesforce, the poor guy got unceremoniously canned. So there's some incentive for it to... not be DNS.
- themgt 12d agoNever before in the history of global compute outages was so little lost by so many down servers, whose purpose was known to so few.
- Havoc 12d agoSo I guess today everyone gets actual work done
- voidUpdate 12d agoWow, the intern must have tripped over a very big power cable this time
- monkeydust 12d agoSam speaks at Salesforce. Salesforce goes down. No causation here...move on.
- hgoel 12d agoWasn't it Dario
- felineflock 11d agoHuang, Sam and Dario speaking at Dreamforce.
- danvesma 12d ago"Trust just got personal!"
- ssk42 12d agoMy gut instinct is that this is about when all of their on prem servers were EOL and their /public cloud solution was required. This must have had something to do with that
- BoorishBears 12d agoMy gut instinct is that this is about Dreamforce with the rickshaws and whatnot
- chasd00 12d agoYou can click on any of the instances and then the service that is down to read the updates. It’s not 100% clear but some sort of issue with a “legacy login service”. The latest updates say a fix is rolling out.
- ownerr 12d agoHere is link to incident details: https://status.salesforce.com/incidents/20004433 https://status.salesforce.com/incidents/20004433
- mannycalavera42 12d ago[flagged]
- alienbaby 12d agoStatus page is still showing all red tho
- aeneas_ory 12d agoIt’s akways DNS or login
- sick_of_slop 12d ago[dead]
- alienbaby 12d agoHaven't they got some kind of new fancy ai interface they can use to fix it?
- raffraffraff 12d agoHave you tried turning it off and then on again? > We're no longer pursuing restarts as a path to remediation. Oh you have
- cube00 12d agoKind of surprised they admit they're going to try restarting and see what happens. I'm sure it happens everywhere but nobody admits it. > We've attempted a rolling restart on one of the impacted instances to see if that resolves the issue. At least it didn't fix the problem so they can actually start finding the real cause. > We're no longer pursuing restarts as a path to remediation. Why isn't the AI they sell telling them what's wrong? Why do they need to take shots in the dark to "see if that resolves the issue"?
- deleted 12d ago[deleted]
- andrewinardeer 12d ago"Yeah, I'm with Rob. Just let's reboot and see what happens"
- jakevoytko 12d agoIn my experience it’s a safe way to do something useful while everyone is getting their bearings. It immediately partitions the situation space between being persisted or systemic vs local or caused by long-running processes. Plus everyone’s going to ask if you’ve tried that already, so you might as well get it out of the way if it makes any amount of sense
- mergy 12d agoUnplanned outage timing is never good but this is really not good. https://www.salesforce.com/dreamforce/ https://www.salesforce.com/dreamforce/ Sept 15-17
- ramesh31 12d agoProbably not a coincidence
- karmakaze 11d agoYup. Could be that everyone was rushing to get all their products and demos ready leading up to it.
- gregw2 11d agoAWS has a history of this before re:Invents (its annual big conference) in my experience. They've gotten a little better in recent years.
- dehrmann 11d agoMost places at this scale have code freezes in place well before conferences. The most likely issues are some launch couldn't handle the scale or periodic deployments have been saving them from some sort of long-standing leak bug, and pausing going into Dreamforce meant some service hasn't been restarted in a week. Historically, Salesforce sharded by customer, so that goes against both of these, unless it's in a routing layer.
- SaucyWrong 11d agoI'm an ex-Salesforce, and yes, at the time I left, there was a huge change freeze surrounding Dreamforce. Unless a demo of an announced feature was coming in really at the buzzer, change velocity would have been low since a few weeks ago. I worked in a sub-cloud, so I can't even speculate as to the reason for the failure. Something I wonder about is whether SRE responses were delayed due to having to be emergency-change-approved because Dreamforce was on. I don't recall a global outage ever occurring during a change freeze when I worked there, so /shrug.
- fhub 12d agoCause: Legacy Salesforce login service got into a resource-exhaustion cascade. Fix: Rolling some unspecified fix they proved in testing out over the fleet seemingly very slowly (After their earlier attempts to roll something out faster failed). Details at https://status.salesforce.com/incidents/20004433 https://status.salesforce.com/incidents/20004433
- dd8601fn 12d agoI wonder if “legacy login” is the shared login gateway. It’s optional but everyone uses it. And it was flaky for an hour or so, like two months ago.
- kgermino 11d agoAnd while I haven’t actually done my homework, the push to email-based login seems like it’s really pushing people to start using their own custom domains for logging in
- ubertaco 11d agoCoincidentally, Salesforce just laid off the senior engineering lead on their shared login service team who 2 months ago was describing the practices his team follows to keep 99.95% uptime on that shared login service (this guy: https://www.linkedin.com/posts/patricktprice_engineeringmanagement-reliability-activity-7475257204628099075-LrN7 https://www.linkedin.com/posts/patricktprice_engineeringmana...) Layoffs are usually short-term money-savers at long-term cost, but hot damn this is the shortest short term I've ever seen.
- lrvick 12d agoSeems it is back up now. Damn.
- irregularbowels 12d ago[dead]
- electroweak 12d agoAt this point OpenAI really ought to let us know when they're testing again.
- DoesntMatter22 11d agoMy thoughts exactly.
- bgro 12d agoLeetcode developers win again
- stmw 12d agoDespite all of the snark here, in my experience Salesforce SRE team is quite competent. The engineering challenges of running a large PaaS - not just with own apps, but with millions of customer-written apps running on it - are quite interesting, and sadly things happen. The status page makes sense to actual customers, it's the particular "pods" where a given service runs.
- mulmen 12d agoI honestly don’t get the snark. The status page has: Seemingly meaningful IDs Search Region filter Email update signup Predictable URLs for instance status so they can be deep linked in runbooks What appears to be the actual live instance status. What appears to be the actual live service status in each instance. An update log with frequent detailed updates.
- shimman 12d agoThis is the case for every single B2B saas product. This is like the "bar is rolling on the floor" level of competence required. Please have higher standards for paid products.
- sidcool 12d agoClaudeForce in action!
- justplay 12d ago[dead]
- elzbardico 12d agoKind of ironic. Salesforce is basically one of the major spiritual grandfathers of Slop. It is not uncommon in production systems to find that objects like Contact and Account have hundreds of custom fields. Sometimes, you find out that several of them have the same meaning and semantics, but were used at different times. Digging out you discover that some Marketing guy that used to work at the company did some task in a certain way that was lost when he was gone, and then a few months later his substitute had the same need and went ahead and created the same field with a slightly different name. Doing data engineering work with Salesforce data is an exercise on archeology, psychology and organizational politics. Slop is basically the ontological and teleological philosophy behind Salesforce very existence. Despite the official discourse that the "No Software" meant no infrastructure, no toil with updates and configuration, the subtext as intended for executives was very clear: "No need for you to be blocked by those pricks from engineering and their stupid, bureaucratic and gatekeeping rules". "No software" was a call-to-arms to a certain subset of managers that were radicalized by Nicholas Carr's 2023 HBR article "IT Doesn't matter". It doesn't matter that Carr was a journalist and a writer with a masters in English that has never ever run even a small bodega, or has never managed an IT department. Anti-intellectualism and the abundance of capital brought in by the petrodollar that allowed the US government to run deficits year by year while exporting the ensuing inflationary effects to rest of world, would ensure that this message would ressonate and then even be amplified during the years of ZIRP and the Baillouts. Play fast and loose, first come, first served, a rising tide rises all boats and all that jazz. Wall Street favors bold, and the heck with the long term! This quarter will only live once! Frankly, this is just poetic justice: Kill by slop, be killed by slop.
- schnevets 12d agoSomething tells me Troy the Salesforce Admin/BD Analyst did not cause the SAAS infrastructure to go down. And I think you're confusing crud with slop.
- isityettime 11d agoI've never worked somewhere that had a Salesforce integration which wasn't an eternal disaster. Have you? Why is every company's Salesforce team absolute bottom of the barrel developers with super high churn, no responsibility, and little competency? Something about the product and its positioning attracts catastrophe. That's what GP is talking about.
- dogas 12d agoObligatory https://xkcd.com/2347/ https://xkcd.com/2347/
- reddalo 11d agoI can't understand how such a huge company can have such a lousy UX.
- ncr100 11d agoIs like Microsoft Windows in that way?
- reddalo 11d agoYes, maybe you're right. Windows used to be good in the 9x era. Then it all went downhill.
- WinstonSmith84 11d agoPeak Tech Salesforce was 2010 +/- 2 years - i.e. after Visualforce and before Aura era. It used to be a developer oriented platform and it became shiny/flashy garbage eventually. But all these shiny things allowed them to get a large market cap with very brilliant sales people, it's hard to deny.
- dd8601fn 11d agoVF and Aura overlapped. Aura was just a bad start and janky. We sometimes just did React instead, for a while. LWC is worlds better. And the local tooling with the cli and VSCode extensions is miles better than the old Eclipse/Sublime FMT days.
- WinstonSmith84 11d agoAura existed simply because Salesforce thought to be smarter than open source, well. Can't deny it though: Salesforce engineers were great on the backend, but frontend dev has never been their thing. Back then there was Angular 1 which was miles ahead. React was released shortly after Aura itself, so to say. LWC is what Aura shall have been 10+ years ago. And that ties back to what OP wrote: awful UX extremely slow bloated with JS. Now, don't talk me about VSCode Extensions. This is the perfect example of an awful dev experience. apex-jorje-lsp.jar with a JVM to parse Apex taking GB of memories, extensions taking dozens of seconds to load (when they load) ... In fact, the only decent LSP is aer, a simple decently working Go binary rather than the monster Salesforce shipped. The one good tooling Salesforce built in the last 15 years is, to some extent, the SF CLI - which came after the `force` CLI from the same guys who built `aer`, anyway. And nowadays, people can use that with their preferred editor from Zed to Vim with shortcuts from built upon the SF CLI. So no, Salesforce didn't do great with tooling, they just did the bare minimum waiting on the (small) community to give them the right ideas.
- zzzeek 11d agoI'm sure the cause of this outage will not be connected to vibe coding in any way
- cmiles8 11d agoSalesforce has turned into the monolithic messy bloatware it set out to replace. Please VCs stop with the AI FOMO and find a few good startups to just go destroy Salesforce and give folks a simple inexpensive replacement.
- sschueller 11d agoAs someone who has never used Salesforce nor hubspot. What are the core features that has people using these services? Is it just the integration between all the different areas where customer data lives?
- cmiles8 11d agoIt’s like a lot of these big SaaS platforms that lock people in to big contracts. They promise the world with all sorts of features and integrations but most customers don’t use that and get locked into buying a very expensive solution to a very simple problem (in this case tracking sales pipelines). Where customers do use the features and integrations it’s often a giant mess that needs a whole separate ecosystem of consultants and “partners” to get the thing working and maintaining it.
- yfontana 10d ago> in this case tracking sales pipelines Salesforce does a lot more than that
- IneffablePigeon 11d agoThey're an incredible marketing machine. But mostly it's a mix of the integration network effects you mention, cost of reimplementation if you want to leave, and the good old "nobody gets fired for buying IBM" dynamic.
- kakwa_ 11d agoIntegration and reconciliation between arbitrary data sources (from SFTP+CSV files to connecting to data warehouse like snowflakes, potentially through vpn tunnels). Data querying for creating marketing campaign audiences or generating some reporting reporting. Little development for landing pages, forms, or custom integrations. Multiple channels (SMS, push, email) & message templating. And many marketing stuff like AB testing/fatigue, etc
- wronex 11d agoDoes it mean I won’t get any AI generated slop spam for a few hours?
- imagetic 11d agoMore of this!
- jason_zig 11d agoI was just working on a zigpoll integration and thought it was me...
- sergiotapia 11d agoRPC errors in my Dunkin app and my wife's Poshmark app. Is it an infrastructure problem?
- haemdahl 11d ago[dead]
- treebeard901 11d agoAll the prompt engineers clogging up San Francisco for dreamforce need to get back to the office.