10 ms·
Salesforce enables ‘modify all’ in user profiles
- BonesJustice 7y agoOopsie. Some poor bastard is having the worst day of their career. I find it concerning that it was even possible for this to happen, regardless of whether it was intentional.
- gambler 7y agoStuff like this is always possible wherever you have fully centralized architecture. I don't know how many massive cloud failures it will take for IT community at large to realize this.
- ralph84 7y agoSalesforce launched in 1999 before before cloud was a thing. It's basically just a big Oracle database.
- mpeg 7y agoThat's not very fair to a company that was one of the pioneers of a lot of the things we consider normal on today's software as a service. Also incorrect, they have great cloud and devops practices. If anything it's likely this bug's impact would be limited due to how decentralised SFDC operates. Still a massive fuck-up, I'm interested in seeing if they'll release any more detail on why it happened.
- the_duke 7y agoWhile a little unfair, and Salesforce is a decent product with nice dev tooling (apart from the weird ancient Java ish custom language), but under the hood, it really is just a Oracle database per org.
- moocowtruck 7y ago> decent product with nice dev tooling whaaaaaaaaaaaaaaaaat hahahaha i can't take that seriously; I've used it and it felt like a giant pit of despair
- vips7L 7y agoWeird ancient proprietary language. Impossible to run locally. No debuggers. Virtually impossible to put an entire org in source control. No package manager. More undefined behavior than a C compiler.
- no_wizard 7y agoWell, they do own heroku https://www.heroku.com/ https://www.heroku.com/
- sbr464 7y agoThey purchased Heroku.
- sb8244 7y agoIn fairness they did do long enough ago that they would have massively messed it up if they at least didn't understand something about running software. SFDC hate is pretty common, maybe because of how big they are. I think that their tech is actually pretty impressive.
- deleted 7y ago[deleted]
- deleted 7y ago[deleted]
- CPLX 7y agoThe hate is likely propelled in part because their sales process is the work of Satan. Like seriously I would rather lick alcohol soaked razor blades than do a standard annual renewal of a Salesforce contract.
- tastroder 7y agoOther than being pretty public at this point, this incident could have easily happened in an upgrade/rollout in on-premise settings as well (at least without a good staging environment and test process).
- meddlepal 7y agoYou get all kinds of new and exciting failure and fuck up modes in a decentralized architecture.
- rightbyte 7y agoIf I add a service listening on port 1234 that pipes text input to a root terminal most likely no one will ever know. That's the advantage of decentralized architecture. It's a disadvantage too though ...
- rc_kas 7y agoRight. Like add a unit test or something. Geez.
- finaliteration 7y agoI’m more curious how a script that does this even made it through review. And if there wasn’t a review... why not?
- finaliteration 7y agoThis is just a reminder that no matter how big or successful you are shit like this can always happen. And it’s usually not the fault of a single person, but rather some lack of process/review/control that made it possible in the first place. I feel terrible for whoever initiated this. I’ve been in that boat and it -really- sucks.
- llamataboot 7y agoIt's hard to make things totally impossible, but hopefully a good post-mortem will identify the systemic issues that led to it happening, fix them, and not throw anyone under the bus.
- londons_explore 7y ago"I'll just ask engineering to fix your wonky account" "Ah yeah - it got in a bad state somehow, let me fix it manually" UPDATE permissions SET allow = 1 WHERE user=671156 AND permission=16 AND org=101 OR 102; Classic SQL blunder...
- londons_explore 7y agoFor anyone reading this who has login access to a production SQL database...: * Change your account to readonly. Make a new admin account, and put its credentials somewhere hard to get (and audited!). * Make a directory in git for 'one off sql statements'. Make them all go through code review and have an automated system run them on merge/deploy. * Enforce style rules with a linter/test, like "UPDATE must have a LIMIT" * Anything the above process is too burdensome to do should have an API or admin interface built for the purpose. * Aim to eventually get rid of your readonly account. A leaked customer data dump could kill the company and shouldn't be available to any malware on your machine. You aren't as secure as you think you are.
- megous 7y agoYou can also do changes in the transaction and check that it did what you expected before committing.
- PowerfulWizard 7y agoThat is smart, I would add "limit 2" to a single row update, so if it returns 2 rows updated I know it was wrong.
- SOLAR_FIELDS 7y ago
- idlewords 7y agoThe worst day of their career so far.
- KallDrexx 7y agoI can imagine it happening, but it's hard to imagine how they applied it to their disaster recovery backups before noticing the issue.
- GuiA 7y agoAs a manager, if you see a poor bastard having the worst day of their career because of something similar, it means you have inadequate safeguards in place and you’re not doing your job correctly.
- jeromegv 7y agoLooks like they shut down a big part of it, our instances are all down.
- bradleybuda 7y agoSalesforce incident: https://status.salesforce.com/incidents/3815 https://status.salesforce.com/incidents/3815 Our product syncs data to Salesforce - we're seeing hit and miss connectivity across our customers' instances. Some API calls are still working, I'm unable to sign in to a developer instance in NA49.
- tetrep 7y agoit'd be nice if we could get the hn link pointing to that instead, since it's actually got a bit more details than the reddit post and the reddit post doesn't seem to be getting updated, so it's only going to get more "stale"
- aaomidi 7y agoFWIW that link doesn't load for me. I think they're getting hammered.
- deleted 7y ago[deleted]
- duxup 7y agoI get a lot of just loading at: https://status.salesforce.com/ https://status.salesforce.com/ Not really what you want with something like this... but the folks holding the keys to a site like that are often never around / fast enough to make those sites helpful.
- trollied 7y agoReading between the lines, I think someone forgot a WHERE clause in their UPDATE statement....
- joshcain 7y agoI mean, you would hope it's not possible for it to be that simple but... you could see it, couldn't you?
- finaliteration 7y agoConsidering that just yesterday I had to quickly cancel an operation that had the wrong criteria specified, I can absolutely see it...
- JanneVee 7y agoHow about turning autocommit off?
- llamataboot 7y ago"The Salesforce Technology team is investigating an issue impacting Salesforce customers who use Pardot, or have used Pardot in the past. The deployment of a database script resulted in granting users broader data access than intended. To protect our customers, we have blocked access to all instances that contain impacted customers until we can complete the removal of the inadvertent permissions in the impacted customer orgs. As a result, customers who were not impacted may experience service disruption. In parallel, we are working to restore the original permissions as quickly as possible. Customers should continue to check Trust for updates."
- deleted 7y ago[deleted]
- aaomidi 7y agoFor people not familiar with salesforce, what does this mean?
- finaliteration 7y agoFor a time, users on many instances were able to read/modify data that they shouldn't have been. They got full CRUD access to -all data-. This includes some external users of things like Customer and Partner portals (where functionality and data are made available to external users via Salesforce). When they decided to try to mitigate the issue, they locked down all access and took away CRUD permissions from all users/profiles in Salesforce on those affected instances. We woke up to a bunch of users unable to do their jobs because they suddenly started receiving "NO ACCESS" errors, effectively. We also haven't been able to modify the profiles and fix the access effectively.
- scottmcf 7y agoThis is absolutely disastrous for healthcare companies.
- aaomidi 7y agoWonder how many lives you could attribute to being lost because of this bug.
- deleted 7y ago[deleted]
- slenk 7y agoWhen computer networks go down, hospitals switch to good old pen and paper. It'll be 0 - its more about the possible breach of PHI/HIPAA violations
- osrec 7y agoAs someone with a big healthcare client on Salesforce, you're not wrong. This was a massive issue for Salesforce Healthcloud users.
- sergers 7y agocan confirm my org is impacted. our SF instance is accessible, but no permissions on login. just got an update from our admin, no eta
- londons_explore 7y agoPresumably this should cause most companies in the EU to announce, within 72 hours, a data breach, since it allows any salesforce user to gain any permissions and view/leak/copy/steal any data on any customer in the org. That will be a lot of companies considering pretty much every big company uses salesforce.
- gruez 7y agoDoes it count as a data breach if unauthorized users were allowed to access data, but they didn't actually? ie. server logs showed that nobody actually read anything they weren't supposed to.
- reilly3000 7y agoI’m not aware of access logs exist, perhaps history tracking if enabled on that object(limited to 10 fields per object). It would absolutely be considered a breach if customer data was accessible by non authorized staff, moreso if by partners.
- samanator 7y agoSalesforce has a feature called Event Monitoring. Event monitoring does show lots of information, like pages access, reports exported, api endpoints used, etc. but it needs to be purchased. However, Salesforce does track and retain that data for some time so you can decide to purchase it retroactively if you ever want.
- reilly3000 7y agoTIL, thanks. Maybe they ought to give that away to those who have an affected org.
- tssva 7y agoEU definition of a personal data breach. > "‘personal data breach’ means a breach of security leading to the accidental or unlawful destruction, loss, alteration, unauthorised disclosure of, or access to, personal data transmitted, stored or otherwise processed; Based upon this definition just allowing the access is a breach.
- Friedduck 7y agoIt didn’t affect any of the 3 orgs within our firm. It was limited to firms who’ve deployed Pardot so a much smaller audience. Still, epic screw-up.
- mediocrejoker 7y agoDoes this allow a salesman from company A to access customer info from company B (where A and B are both customers of salesforce) or is it just intra-organization? Still bad if the latter, but catastrophic if the former
- thrav 7y agoNah, definitely not that. Just ups their permissions to that of a super user / admin in their company.
- jplayer01 7y agoWhy would anybody trust Salesforce after this?
- brogrammernot 7y agoDoesn’t really matter. Moving off Salesforce is a many month project for a small company and possibly years for a larger company depending on the add-ons and everything. Doubt they lose any real customers over this but they’re definitely going to be cutting some checks/credits to a ton of people for the next few months but it’ll fade.