10 ms·

Google suffers data breach in ongoing Salesforce data theft attacks

207 points by mikece 1y ago

shadowgovt 1y ago
I'm modestly surprised to learn Google was using Salesforce internally at all; the NIH runs deep with that company (they even have their own bugtracker because every other option just wouldn't cut it). On the other hand, the past decade-ish has seen them grow very rapidly via acquisition, so perhaps this DB was grandfathered in via an acquired company and hadn't yet been replaced by anything internal. (For Salesforce in particular though, I'd be willing to believe Google doesn't have an in-house alternative... People asked for a Salesforce-like in Google Workspace for years and the company had no interest. I have a hunch that most Googlers find the idea of creating a new CRM to be a profoundly boring intellectual exercise).
- mc32 1y ago
  Google uses lots of non-Google solutions for many things —just imagine all the facilities stuff. But so does any software company, including Microsoft and Amazon. That said, you can hire people for any purpose (specific roles) and you can build what you want. It’s more a question of whether it’s worth it to build such solutions, after all you have a main line of business to tend to. That’s to say even Google and Apple have so called “boring “ roles and there are lots of people who don’t see it that way and want to work doing those things.
  - shadowgovt 1y ago
    Given the low expected profit margin, a CRM solution at Google would likely come from a 20% project (or rather, the equivalent thing these days since last I checked 20% is basically dead as a formal concept). Nobody expected GMail to blow up the way it did, for example; it happened because some Googlers decided they could probably do a web-client-fronted mail client with a Google search engine attached to it and if they did it'd be really cool. But even with their, what, 180,000 people these days, I think it's entirely possible nobody is as excited about CRM as Paul Buchheit was about email services.
  - progbits 1y ago
    Actually lot of the facilities stuff is inhouse too - floor plans (not just the seat map but actual floor drawings that include physical infrastructure); the ticketing system for maintenance; work hour tracking for contractors; probably lot more that I'm forgetting. But yes your point stands, sometimes it just makes more sense to use an existing product.
    - eitally 1y ago
      The floor plan tool isn't really in house. It's just an extension of the industry standard real estate management platform they use Tririga (https://www.ibm.com/products/tririga https://www.ibm.com/products/tririga) ... in the same way that go/teams in just an custom visualization of a standard employee directory. You might be surprised how much of what runs Google (Anaplan, for example, for XWS) is fairly industry standard.
      - scottyah 1y ago
        They did acquire (then sell) SketchUp which is what I use for floorplans.
- bpodgursky 1y ago
  Salespeople are VERY familiar with Salesforce and are not very technical. Probably significantly increases onboarding and training time to have a weird new tool. Easy to hire experienced salespeople and have them hit the ground fast if they use standard Salesforce conversion flows.
  - bombcar 1y ago
    It still amazes me that Salesforce, which is good, mind you, is still basically just Microsoft Access as a Service, and yet here we are.
- dilyevsky 1y ago
  iirc google cloud’s entire support ticket system is built on top of sf - it went down when saleforce had an outage a few years back
- eitally 1y ago
  Fwiw, I was hired by Google in 2015 to help answer questions like "if Google were to add a CRM to the GSuite portfolio, should they build one, buy one or partner with key players". My team's charter was to create business cases with various options and run them up to chain (at the time, Prabhakar was running product for "Google for Work"). On more than one occasion we presented cases with 3 year ROIs in the $xxxM range and were shot down every time with a "too small" comment. A couple years later, Google had partnered with Copper CRM and supported extension builds into Workspace/GSuite, but had also begun a major enterprise rationalization project to consolidate a multitude of Salesforce instances into a single one, at the same time as adopting standard enterprise features & processes of Anaplan. This led to consolidation of a number of back office IT teams that ultimately ended up with far more enforcement clout than they'd historically had. By the time Ruth changed roles, most of the "normal" business processes had been fairly standardized. Fwiw, the Cloud instance of SFDC, which is by far the most complex & customized, has been in full use for almost five years now and is the canonical source of truth for sales data.
  - coredog64 1y ago
    I'm surprised Google could get away with only a single SFDC instance. AWS has multiple SFDC installations and is forever having to deal with "Oh, yeah, that data is in this other SFDC installation"
    - ssk42 1y ago
      Yeah, they have the world class Salesforce engineers there. One of Google's Salesforce's last tech leads wound up becoming the Director of the proprietary Salesforce language Apex.
  - shadowgovt 1y ago
    I wonder if the Cloud SFDC is the one that was compromised. It's a little telling Google didn't go into details about which arm of the octopus got attacked (or if they did, I didn't see that reporting yet... Unless Cloud is the implied victim because the description of the attack showed up on the Cloud blog). I feel you about the ROI. In hindsight, it's a little funny to me that Salesforce is doing revenue numbers a little under half of Google Cloud; you'd think that would be large enough value to get Google interested in biting into that pie.
- loeg 1y ago
  > they even have their own bugtracker because every other option just wouldn't cut it Of all the things to NIH, this is one of the most defensible -- lots of bugtracker options just aren't very good.
  - deleted 1y ago
    [deleted]
  - cjpearson 1y ago
    I've generally not had an interest in working for one of the big tech companies, but the opportunity to escape JIRA is tempting.
  - 8n4vidtmkvmk 1y ago
    I found this to be true too, but I don't really get it. Doesn't seem like that complicated of a software. Maybe I'm only thinking like a SWE, and not PM and other laypersons that also need access.
    - IshKebab 1y ago
      It's definitely not that complicated. It's just one of those bits of software that is paid for and managed by people that aren't actually using it, so you get Jira shit. I used to work for Dyson and they moved to Teamcenter for CAD management (basically a shitty VCS for CAD). Similarly to Jira it had all the features you'd ever want in its white paper, but it was abysmal to use - even worse than Jira. Anyway the nicest bug tracking software I've used so far is Phabricator. Quite a lot nicer than anything else, but it is tightly integrated, and I wouldn't really recommend Phabricator these days because a) no integrated CI system, b) it's semi abandoned, and c) PHP. And yes that does matter. (Though TBF it beats Ruby.)
  - kevincox 1y ago
    Yeah, Google's Buganizer was the best bug tracker that I ever used. ...and it still wasn't great.
    - 1y ago
Bluescreenbuddy 1y ago
Surprised Google didn't have some internally developed alternative.
- progbits 1y ago
  From my experience with sales/PM people at google, they refuse to use internal tools and try to get Jira and other shit installed. Regardless of the tool quality, just because that's what they learned already. This mostly didn't work out for them back in the day but in more recent times as more and more low quality middle level managers and execs get hired they manage to get approvals. In my org a new VP demanded Jira instance within a month of joining the company and that it be used for technical project reporting. Of course all the developers said fuck no to that so for a while some managers were trying to do two way sync between Jira and Buganizer. When I left it was mostly abandoned and full of tumbleweed...
  - lenerdenator 1y ago
    > From my experience with sales/PM people at google, they refuse to use internal tools and try to get Jira and other shit installed. Regardless of the tool quality, just because that's what they learned already. That's when you're supposed to pull the smooth-talking people that are usually in those roles and ask them a very simple question: "Do you want this tool more than you want to be employed?"
    - closewith 1y ago
      Good software salespeople are much rarer than good developers, so it's likely that conversion would be had with the other parties.
      - datadrivenangel 1y ago
        And they're better at selling!
      - sigmoid10 1y ago
deleted 1y ago
[deleted]
GHanku 1y ago
The linked article explains how they do it: https://www.bleepingcomputer.com/news/security/google-hackers-target-salesforce-accounts-in-data-extortion-attacks/ https://www.bleepingcomputer.com/news/security/google-hacker... >The attackers impersonate IT support personnel, requesting the target employee accept a connection to Salesforce Data Loader, a client application... "The application supports OAuth and allows for direct "app" integration via the "connected apps" functionality in Salesforce," explains the researchers. "Threat actors abuse this by persuading a victim over the phone to open the Salesforce connect setup page and enter a "connection code," thereby linking the actor-controlled Data Loader to the victim's environment. ... app is used to export data stored in Salesforce instances and then used the access to move laterally through connected platforms such as Okta, Microsoft 365, and Workplace. Accessing these additional cloud platforms allows the threat actors to access more sensitive information stored on those platforms, including sensitive communications, authorization tokens, documents, and more.
wferrell 1y ago
They had an internal CRM. It was buggy, missing key features and engineers didn’t really want to work on it.
- hnthrow90348765 1y ago
  If I had jumped through Google's hiring hoops, I wouldn't either. Of course, this could be solved with money.
  - wferrell 1y ago
    I think the real reason was there was no path to promotion for working on this. For better or worse the incentives were not aligned for great work to happen.
kyrra 1y ago
From the source: https://cloud.google.com/blog/topics/threat-intelligence/voice-phishing-data-extortion https://cloud.google.com/blog/topics/threat-intelligence/voi... > The instance was used to store contact information and related notes for small and medium businesses. Analysis revealed that data was retrieved by the threat actor during a small window of time before the access was cut off. The data retrieved by the threat actor was confined to basic and largely publicly available business information, such as business names and contact details.
- deleted 1y ago
  [deleted]
- jedc 1y ago
  "store contact information and related notes for small and medium businesses" Most likely translation: it affected the Google SMB sales team's Salesforce instance
  - angmarsbane 1y ago
    My understanding is that the Cloud org uses Salesforce, the rest of Google uses a self-developed solution.
- lesuorac 1y ago
  > The data retrieved by the threat actor was confined to basic and largely publicly available business information Which is to say, they took public _and_ private data and the private data is something we don't wish to publicly admit so probably not good.
- sugarpimpdorsey 1y ago
  > Analysis revealed that data was retrieved by the threat actor during a small window of time before the access was cut off. That's a pretty nonchalant way to say "they totally stole stuff before we knew what was going on or could stop them".
  - trhway 1y ago
mrweasel 1y ago
Oh, so I wonder if that's also how KLM lost my data.
- steffanA 1y ago
  Same campaign
ok123456 1y ago
Wonder if it's related to https://venturebeat.com/ai/this-ai-already-writes-20-of-salesforces-code-heres-why-developers-arent-worried/ https://venturebeat.com/ai/this-ai-already-writes-20-of-sale...
- deleted 1y ago
  [deleted]
- superfrank 1y ago
  > In June, Google warned that a threat actor they classify as 'UNC6040' is targeting companies' employees in voice phishing (vishing) social engineering attacks to breach Salesforce instances and download customer data > [...] > In June, one of Google's corporate Salesforce instances was impacted by similar UNC6040 activity described in this post Nope. Good old fashion social engineering.
grumple 1y ago
I'm surprised, mostly because Google seems to have basically no salespeople, account reps, or customer management.
- geodel 1y ago
  Google has ~40K people in sales and marketing. Likely they didn't assign any to your account.
  - wferrell 1y ago
    I have 3 for our google cloud account and we spend like 5k a month.
01HNNWZ0MV43FF 1y ago
> Google suffers Uh, it's the users that suffer. You Suffer https://www.youtube.com/watch?v=_-ywSPWu3K8 https://www.youtube.com/watch?v=_-ywSPWu3K8
1970-01-01 1y ago
Google: Nobody beats the $32,000,000,000 Wiz! Bet! UNC6040: lool.
cjonas 1y ago
My guess is leaked from a misconfigured force.com site often used as a support portal or kb. Up until recently they came misconfigured by default to allow public access to the basic info of accounts, contact, opportunity through list view endpoints. Back in 2019 I had a client affected by this (luckily caught by a white hat). Curious, I searched *.site.force.com and found thousands of potentially impacted sites (vulnerability could be tested without exfil of any data). In recent years SF has had many security patches to try and close these holes, but my understanding is most required action by the admin to take effect. I was always confused how SF managed to keep this out of the news.

207 points by mikece 1y ago

I'm modestly surprised to learn Google was using Salesforce internally at all; the NIH runs deep with that company (they even have their own bugtracker because every other option just wouldn't cut it). On the other hand, the past decade-ish has seen them grow very rapidly via acquisition, so perhaps this DB was grandfathered in via an acquired company and hadn't yet been replaced by anything internal. (For Salesforce in particular though, I'd be willing to believe Google doesn't have an in-house alternative... People asked for a Salesforce-like in Google Workspace for years and the company had no interest. I have a hunch that most Googlers find the idea of creating a new CRM to be a profoundly boring intellectual exercise).

mc32 1y ago
Google uses lots of non-Google solutions for many things —just imagine all the facilities stuff. But so does any software company, including Microsoft and Amazon. That said, you can hire people for any purpose (specific roles) and you can build what you want. It’s more a question of whether it’s worth it to build such solutions, after all you have a main line of business to tend to. That’s to say even Google and Apple have so called “boring “ roles and there are lots of people who don’t see it that way and want to work doing those things.
- shadowgovt 1y ago
  Given the low expected profit margin, a CRM solution at Google would likely come from a 20% project (or rather, the equivalent thing these days since last I checked 20% is basically dead as a formal concept). Nobody expected GMail to blow up the way it did, for example; it happened because some Googlers decided they could probably do a web-client-fronted mail client with a Google search engine attached to it and if they did it'd be really cool. But even with their, what, 180,000 people these days, I think it's entirely possible nobody is as excited about CRM as Paul Buchheit was about email services.
- progbits 1y ago
  Actually lot of the facilities stuff is inhouse too - floor plans (not just the seat map but actual floor drawings that include physical infrastructure); the ticketing system for maintenance; work hour tracking for contractors; probably lot more that I'm forgetting. But yes your point stands, sometimes it just makes more sense to use an existing product.
  - eitally 1y ago
    The floor plan tool isn't really in house. It's just an extension of the industry standard real estate management platform they use Tririga (https://www.ibm.com/products/tririga https://www.ibm.com/products/tririga) ... in the same way that go/teams in just an custom visualization of a standard employee directory. You might be surprised how much of what runs Google (Anaplan, for example, for XWS) is fairly industry standard.
    - scottyah 1y ago
      They did acquire (then sell) SketchUp which is what I use for floorplans.
bpodgursky 1y ago
Salespeople are VERY familiar with Salesforce and are not very technical. Probably significantly increases onboarding and training time to have a weird new tool. Easy to hire experienced salespeople and have them hit the ground fast if they use standard Salesforce conversion flows.
- bombcar 1y ago
  It still amazes me that Salesforce, which is good, mind you, is still basically just Microsoft Access as a Service, and yet here we are.
dilyevsky 1y ago
iirc google cloud’s entire support ticket system is built on top of sf - it went down when saleforce had an outage a few years back
eitally 1y ago
Fwiw, I was hired by Google in 2015 to help answer questions like "if Google were to add a CRM to the GSuite portfolio, should they build one, buy one or partner with key players". My team's charter was to create business cases with various options and run them up to chain (at the time, Prabhakar was running product for "Google for Work"). On more than one occasion we presented cases with 3 year ROIs in the $xxxM range and were shot down every time with a "too small" comment. A couple years later, Google had partnered with Copper CRM and supported extension builds into Workspace/GSuite, but had also begun a major enterprise rationalization project to consolidate a multitude of Salesforce instances into a single one, at the same time as adopting standard enterprise features & processes of Anaplan. This led to consolidation of a number of back office IT teams that ultimately ended up with far more enforcement clout than they'd historically had. By the time Ruth changed roles, most of the "normal" business processes had been fairly standardized. Fwiw, the Cloud instance of SFDC, which is by far the most complex & customized, has been in full use for almost five years now and is the canonical source of truth for sales data.
- coredog64 1y ago
  I'm surprised Google could get away with only a single SFDC instance. AWS has multiple SFDC installations and is forever having to deal with "Oh, yeah, that data is in this other SFDC installation"
  - ssk42 1y ago
    Yeah, they have the world class Salesforce engineers there. One of Google's Salesforce's last tech leads wound up becoming the Director of the proprietary Salesforce language Apex.
- shadowgovt 1y ago
  I wonder if the Cloud SFDC is the one that was compromised. It's a little telling Google didn't go into details about which arm of the octopus got attacked (or if they did, I didn't see that reporting yet... Unless Cloud is the implied victim because the description of the attack showed up on the Cloud blog). I feel you about the ROI. In hindsight, it's a little funny to me that Salesforce is doing revenue numbers a little under half of Google Cloud; you'd think that would be large enough value to get Google interested in biting into that pie.
loeg 1y ago
> they even have their own bugtracker because every other option just wouldn't cut it Of all the things to NIH, this is one of the most defensible -- lots of bugtracker options just aren't very good.
- deleted 1y ago
  [deleted]
- cjpearson 1y ago
  I've generally not had an interest in working for one of the big tech companies, but the opportunity to escape JIRA is tempting.
- 8n4vidtmkvmk 1y ago
  I found this to be true too, but I don't really get it. Doesn't seem like that complicated of a software. Maybe I'm only thinking like a SWE, and not PM and other laypersons that also need access.
  - IshKebab 1y ago
    It's definitely not that complicated. It's just one of those bits of software that is paid for and managed by people that aren't actually using it, so you get Jira shit. I used to work for Dyson and they moved to Teamcenter for CAD management (basically a shitty VCS for CAD). Similarly to Jira it had all the features you'd ever want in its white paper, but it was abysmal to use - even worse than Jira. Anyway the nicest bug tracking software I've used so far is Phabricator. Quite a lot nicer than anything else, but it is tightly integrated, and I wouldn't really recommend Phabricator these days because a) no integrated CI system, b) it's semi abandoned, and c) PHP. And yes that does matter. (Though TBF it beats Ruby.)
- kevincox 1y ago
  Yeah, Google's Buganizer was the best bug tracker that I ever used. ...and it still wasn't great.
  - 1y ago

Bluescreenbuddy 1y ago

Surprised Google didn't have some internally developed alternative.

progbits 1y ago
From my experience with sales/PM people at google, they refuse to use internal tools and try to get Jira and other shit installed. Regardless of the tool quality, just because that's what they learned already. This mostly didn't work out for them back in the day but in more recent times as more and more low quality middle level managers and execs get hired they manage to get approvals. In my org a new VP demanded Jira instance within a month of joining the company and that it be used for technical project reporting. Of course all the developers said fuck no to that so for a while some managers were trying to do two way sync between Jira and Buganizer. When I left it was mostly abandoned and full of tumbleweed...
- lenerdenator 1y ago
  > From my experience with sales/PM people at google, they refuse to use internal tools and try to get Jira and other shit installed. Regardless of the tool quality, just because that's what they learned already. That's when you're supposed to pull the smooth-talking people that are usually in those roles and ask them a very simple question: "Do you want this tool more than you want to be employed?"
  - closewith 1y ago
    Good software salespeople are much rarer than good developers, so it's likely that conversion would be had with the other parties.
    - datadrivenangel 1y ago
      And they're better at selling!
    - sigmoid10 1y ago

deleted 1y ago

[deleted]

GHanku 1y ago

The linked article explains how they do it: https://www.bleepingcomputer.com/news/security/google-hackers-target-salesforce-accounts-in-data-extortion-attacks/ https://www.bleepingcomputer.com/news/security/google-hacker... >The attackers impersonate IT support personnel, requesting the target employee accept a connection to Salesforce Data Loader, a client application... "The application supports OAuth and allows for direct "app" integration via the "connected apps" functionality in Salesforce," explains the researchers. "Threat actors abuse this by persuading a victim over the phone to open the Salesforce connect setup page and enter a "connection code," thereby linking the actor-controlled Data Loader to the victim's environment. ... app is used to export data stored in Salesforce instances and then used the access to move laterally through connected platforms such as Okta, Microsoft 365, and Workplace. Accessing these additional cloud platforms allows the threat actors to access more sensitive information stored on those platforms, including sensitive communications, authorization tokens, documents, and more.

wferrell 1y ago

They had an internal CRM. It was buggy, missing key features and engineers didn’t really want to work on it.

hnthrow90348765 1y ago
If I had jumped through Google's hiring hoops, I wouldn't either. Of course, this could be solved with money.
- wferrell 1y ago
  I think the real reason was there was no path to promotion for working on this. For better or worse the incentives were not aligned for great work to happen.

kyrra 1y ago

From the source: https://cloud.google.com/blog/topics/threat-intelligence/voice-phishing-data-extortion https://cloud.google.com/blog/topics/threat-intelligence/voi... > The instance was used to store contact information and related notes for small and medium businesses. Analysis revealed that data was retrieved by the threat actor during a small window of time before the access was cut off. The data retrieved by the threat actor was confined to basic and largely publicly available business information, such as business names and contact details.

deleted 1y ago
[deleted]
jedc 1y ago
"store contact information and related notes for small and medium businesses" Most likely translation: it affected the Google SMB sales team's Salesforce instance
- angmarsbane 1y ago
  My understanding is that the Cloud org uses Salesforce, the rest of Google uses a self-developed solution.
lesuorac 1y ago
> The data retrieved by the threat actor was confined to basic and largely publicly available business information Which is to say, they took public _and_ private data and the private data is something we don't wish to publicly admit so probably not good.
sugarpimpdorsey 1y ago
> Analysis revealed that data was retrieved by the threat actor during a small window of time before the access was cut off. That's a pretty nonchalant way to say "they totally stole stuff before we knew what was going on or could stop them".
- trhway 1y ago

mrweasel 1y ago

Oh, so I wonder if that's also how KLM lost my data.

steffanA 1y ago
Same campaign

ok123456 1y ago

Wonder if it's related to https://venturebeat.com/ai/this-ai-already-writes-20-of-salesforces-code-heres-why-developers-arent-worried/ https://venturebeat.com/ai/this-ai-already-writes-20-of-sale...

deleted 1y ago
[deleted]
superfrank 1y ago
> In June, Google warned that a threat actor they classify as 'UNC6040' is targeting companies' employees in voice phishing (vishing) social engineering attacks to breach Salesforce instances and download customer data > [...] > In June, one of Google's corporate Salesforce instances was impacted by similar UNC6040 activity described in this post Nope. Good old fashion social engineering.

grumple 1y ago

I'm surprised, mostly because Google seems to have basically no salespeople, account reps, or customer management.

geodel 1y ago
Google has ~40K people in sales and marketing. Likely they didn't assign any to your account.
- wferrell 1y ago
  I have 3 for our google cloud account and we spend like 5k a month.

01HNNWZ0MV43FF 1y ago

> Google suffers Uh, it's the users that suffer. You Suffer https://www.youtube.com/watch?v=_-ywSPWu3K8 https://www.youtube.com/watch?v=_-ywSPWu3K8

1970-01-01 1y ago

Google: Nobody beats the $32,000,000,000 Wiz! Bet! UNC6040: lool.

cjonas 1y ago

My guess is leaked from a misconfigured force.com site often used as a support portal or kb. Up until recently they came misconfigured by default to allow public access to the basic info of accounts, contact, opportunity through list view endpoints. Back in 2019 I had a client affected by this (luckily caught by a white hat). Curious, I searched *.site.force.com and found thousands of potentially impacted sites (vulnerability could be tested without exfil of any data). In recent years SF has had many security patches to try and close these holes, but my understanding is most required action by the admin to take effect. I was always confused how SF managed to keep this out of the news.