7 ms·

Oopsie. Some poor bastard is having the worst day of their career. I find it concerning that it was even possible for this to happen, regardless of whether it

Oopsie. Some poor bastard is having the worst day of their career. I find it concerning that it was even possible for this to happen, regardless of whether it was intentional.

gambler 7y ago
Stuff like this is always possible wherever you have fully centralized architecture. I don't know how many massive cloud failures it will take for IT community at large to realize this.
- ralph84 7y ago
  Salesforce launched in 1999 before before cloud was a thing. It's basically just a big Oracle database.
  - mpeg 7y ago
    That's not very fair to a company that was one of the pioneers of a lot of the things we consider normal on today's software as a service. Also incorrect, they have great cloud and devops practices. If anything it's likely this bug's impact would be limited due to how decentralised SFDC operates. Still a massive fuck-up, I'm interested in seeing if they'll release any more detail on why it happened.
    - the_duke 7y ago
      While a little unfair, and Salesforce is a decent product with nice dev tooling (apart from the weird ancient Java ish custom language), but under the hood, it really is just a Oracle database per org.
      - moocowtruck 7y ago
        > decent product with nice dev tooling whaaaaaaaaaaaaaaaaat hahahaha i can't take that seriously; I've used it and it felt like a giant pit of despair
      - vips7L 7y ago
        Weird ancient proprietary language. Impossible to run locally. No debuggers. Virtually impossible to put an entire org in source control. No package manager. More undefined behavior than a C compiler.
  - no_wizard 7y ago
    Well, they do own heroku https://www.heroku.com/ https://www.heroku.com/
    - sbr464 7y ago
      They purchased Heroku.
      - sb8244 7y ago
        In fairness they did do long enough ago that they would have massively messed it up if they at least didn't understand something about running software. SFDC hate is pretty common, maybe because of how big they are. I think that their tech is actually pretty impressive.
        deleted 7y ago
        [deleted]
        deleted 7y ago
        [deleted]
        CPLX 7y ago
        The hate is likely propelled in part because their sales process is the work of Satan. Like seriously I would rather lick alcohol soaked razor blades than do a standard annual renewal of a Salesforce contract.
- tastroder 7y ago
  Other than being pretty public at this point, this incident could have easily happened in an upgrade/rollout in on-premise settings as well (at least without a good staging environment and test process).
- meddlepal 7y ago
  You get all kinds of new and exciting failure and fuck up modes in a decentralized architecture.
  - rightbyte 7y ago
    If I add a service listening on port 1234 that pipes text input to a root terminal most likely no one will ever know. That's the advantage of decentralized architecture. It's a disadvantage too though ...
rc_kas 7y ago
Right. Like add a unit test or something. Geez.
- finaliteration 7y ago
  I’m more curious how a script that does this even made it through review. And if there wasn’t a review... why not?
finaliteration 7y ago
This is just a reminder that no matter how big or successful you are shit like this can always happen. And it’s usually not the fault of a single person, but rather some lack of process/review/control that made it possible in the first place. I feel terrible for whoever initiated this. I’ve been in that boat and it -really- sucks.
llamataboot 7y ago
It's hard to make things totally impossible, but hopefully a good post-mortem will identify the systemic issues that led to it happening, fix them, and not throw anyone under the bus.
- londons_explore 7y ago
  "I'll just ask engineering to fix your wonky account" "Ah yeah - it got in a bad state somehow, let me fix it manually" UPDATE permissions SET allow = 1 WHERE user=671156 AND permission=16 AND org=101 OR 102; Classic SQL blunder...
  - londons_explore 7y ago
    For anyone reading this who has login access to a production SQL database...: * Change your account to readonly. Make a new admin account, and put its credentials somewhere hard to get (and audited!). * Make a directory in git for 'one off sql statements'. Make them all go through code review and have an automated system run them on merge/deploy. * Enforce style rules with a linter/test, like "UPDATE must have a LIMIT" * Anything the above process is too burdensome to do should have an API or admin interface built for the purpose. * Aim to eventually get rid of your readonly account. A leaked customer data dump could kill the company and shouldn't be available to any malware on your machine. You aren't as secure as you think you are.
    - megous 7y ago
      You can also do changes in the transaction and check that it did what you expected before committing.
      - PowerfulWizard 7y ago
        That is smart, I would add "limit 2" to a single row update, so if it returns 2 rows updated I know it was wrong.
      - SOLAR_FIELDS 7y ago
idlewords 7y ago
The worst day of their career so far.
KallDrexx 7y ago
I can imagine it happening, but it's hard to imagine how they applied it to their disaster recovery backups before noticing the issue.
GuiA 7y ago
As a manager, if you see a poor bastard having the worst day of their career because of something similar, it means you have inadequate safeguards in place and you’re not doing your job correctly.

Oopsie. Some poor bastard is having the worst day of their career. I find it concerning that it was even possible for this to happen, regardless of whether it was intentional.

gambler 7y ago

Stuff like this is always possible wherever you have fully centralized architecture. I don't know how many massive cloud failures it will take for IT community at large to realize this.

ralph84 7y ago
Salesforce launched in 1999 before before cloud was a thing. It's basically just a big Oracle database.
- mpeg 7y ago
  That's not very fair to a company that was one of the pioneers of a lot of the things we consider normal on today's software as a service. Also incorrect, they have great cloud and devops practices. If anything it's likely this bug's impact would be limited due to how decentralised SFDC operates. Still a massive fuck-up, I'm interested in seeing if they'll release any more detail on why it happened.
  - the_duke 7y ago
    While a little unfair, and Salesforce is a decent product with nice dev tooling (apart from the weird ancient Java ish custom language), but under the hood, it really is just a Oracle database per org.
    - moocowtruck 7y ago
      > decent product with nice dev tooling whaaaaaaaaaaaaaaaaat hahahaha i can't take that seriously; I've used it and it felt like a giant pit of despair
    - vips7L 7y ago
      Weird ancient proprietary language. Impossible to run locally. No debuggers. Virtually impossible to put an entire org in source control. No package manager. More undefined behavior than a C compiler.
- no_wizard 7y ago
  Well, they do own heroku https://www.heroku.com/ https://www.heroku.com/
  - sbr464 7y ago
    They purchased Heroku.
    - sb8244 7y ago
      In fairness they did do long enough ago that they would have massively messed it up if they at least didn't understand something about running software. SFDC hate is pretty common, maybe because of how big they are. I think that their tech is actually pretty impressive.
      - deleted 7y ago
        [deleted]
      - deleted 7y ago
        [deleted]
      - CPLX 7y ago
        The hate is likely propelled in part because their sales process is the work of Satan. Like seriously I would rather lick alcohol soaked razor blades than do a standard annual renewal of a Salesforce contract.
tastroder 7y ago
Other than being pretty public at this point, this incident could have easily happened in an upgrade/rollout in on-premise settings as well (at least without a good staging environment and test process).
meddlepal 7y ago
You get all kinds of new and exciting failure and fuck up modes in a decentralized architecture.
- rightbyte 7y ago
  If I add a service listening on port 1234 that pipes text input to a root terminal most likely no one will ever know. That's the advantage of decentralized architecture. It's a disadvantage too though ...

rc_kas 7y ago

Right. Like add a unit test or something. Geez.

finaliteration 7y ago
I’m more curious how a script that does this even made it through review. And if there wasn’t a review... why not?

finaliteration 7y ago

This is just a reminder that no matter how big or successful you are shit like this can always happen. And it’s usually not the fault of a single person, but rather some lack of process/review/control that made it possible in the first place. I feel terrible for whoever initiated this. I’ve been in that boat and it -really- sucks.

llamataboot 7y ago

It's hard to make things totally impossible, but hopefully a good post-mortem will identify the systemic issues that led to it happening, fix them, and not throw anyone under the bus.

londons_explore 7y ago
"I'll just ask engineering to fix your wonky account" "Ah yeah - it got in a bad state somehow, let me fix it manually" UPDATE permissions SET allow = 1 WHERE user=671156 AND permission=16 AND org=101 OR 102; Classic SQL blunder...
- londons_explore 7y ago
  For anyone reading this who has login access to a production SQL database...: * Change your account to readonly. Make a new admin account, and put its credentials somewhere hard to get (and audited!). * Make a directory in git for 'one off sql statements'. Make them all go through code review and have an automated system run them on merge/deploy. * Enforce style rules with a linter/test, like "UPDATE must have a LIMIT" * Anything the above process is too burdensome to do should have an API or admin interface built for the purpose. * Aim to eventually get rid of your readonly account. A leaked customer data dump could kill the company and shouldn't be available to any malware on your machine. You aren't as secure as you think you are.
  - megous 7y ago
    You can also do changes in the transaction and check that it did what you expected before committing.
    - PowerfulWizard 7y ago
      That is smart, I would add "limit 2" to a single row update, so if it returns 2 rows updated I know it was wrong.
    - SOLAR_FIELDS 7y ago

idlewords 7y ago

The worst day of their career so far.

KallDrexx 7y ago

I can imagine it happening, but it's hard to imagine how they applied it to their disaster recovery backups before noticing the issue.

GuiA 7y ago

As a manager, if you see a poor bastard having the worst day of their career because of something similar, it means you have inadequate safeguards in place and you’re not doing your job correctly.