7 ms·

Just because there isn’t a “novel exploit” doesn’t mean this isn’t a big deal. Snowflake is susceptible to their SE’s having credentials stolen. These credenti

Just because there isn’t a “novel exploit” doesn’t mean this isn’t a big deal. Snowflake is susceptible to their SE’s having credentials stolen. These credentials can bypass MFA. And per the article, they have no expiry. That’s strikes one, two, and three. Snowflake’s security practices lead to a situation where their customers are either required, or at minimum encouraged, to share access to broad datasets with Snowflake employees. That’s strike four. Yes, there is also issue here that the customers are responsible themselves for not granting too broad access, and that’s on them. But it’s also on Snowflake for not having a better system that doesn’t require this access, or at minimum not having better oversight and control over this transitive access. Once these accounts are granted access to a customer’s data, they aren’t “demo accounts” anymore. They’re real accounts, with real, very valuable data, and they should be treated as such. Edit to add: it is worth noting that Snowflake claims the demo account did not have access to customer data and wasn’t the source of the leak, which is in contradiction with what the attackers claim.

bawolff 2y ago
Indeed, the less novel the exploit the more embarassing it is. Data stolen because of some crazy multi-exploit zero day chain. Well that is understandable, i don't blame the company. Data stolen because no 2FA support? In 2024 that is just embarassing.
- p0seidon 2y ago
  Yes, that's also what I think must have happened—missing 2FA. But it seems it's also not mandatory within Snowflake accounts also, from what I understand from the general message. I've never used Snowflake and assumed that because you push all your data into it, it probably has 2FA enabled by default. Is it optional?
  - Galanwe 2y ago
    I don't quite understand how Snowflake works. My understanding was that you had to grant storage access (e.g. S3) and compute access (e.g. EC2) from your account to Snowflake, which would then use said resources to perform queries that you issue from their hosted web UI. In that case it would mean stealing the Snowflake demo account of a SE should not expose your data unless you forgot to revoke their access to your underlying resources. Can someone explain if that is how it works?
    - adovenmuehle 2y ago
      No, Snowflake runs it's own storage and compute (on either AWS, GCP, or Azure depending on what you pick).
      - Galanwe 2y ago
        So the customer data is actually stored on Snowflakes AWS accounts? What difference does it make what underlying storage / provider it uses then? Also does that mean every data query to snowflake goes out/in to/from internet at egress/Ingress costs?
        p_l 2y ago
        At the snowflake size you get custom price lists from cloud operators. But I think there was also support for peering with client VPCs (or equivalents) which is why they support AWS, Azure, and GCP - you choose the location that is most fitting for linking with your cloud/physical workloads.
        wokwokwok 2y ago
        > So the customer data is actually stored on Snowflakes AWS accounts? Yes. > Also does that mean every data query to snowflake goes out/in to/from internet at egress/Ingress costs? Yes. It's covered comprehensively in their docs, along with the caveats. > What difference does it make what underlying storage / provider it uses then? "Snowflake does not charge data ingress fees. However, a cloud storage provider might charge a data egress fee for transferring data from the provider to your Snowflake account." unsaid: "...and you have to pay for that". Note that when they say 'your Snowflake account' they mean our cloud account which we own, and which we run our workloads in, which we refer to as 'your' snowflake account. Tangibly speaking, what means is that if you want to check up your billing; you go through snowflake; you can't login to a cloud console and see the actual charges the cloud vendor is charging. > What difference does it make what underlying storage / provider it uses then? They pass the specific underlying cloud vendor costs on to you (with, I guess, some markup, though you have no way of know what that is :)
  - deleted 2y ago
    [deleted]
hangtime79 2y ago
Having spent a lot of time in the Snowflake ecosystem, a few things to understand. Snowflake is both a platform and a database. Customers can implement any level of security within the system. For example, when Snowflake introduced OAuth functionality, they put a lot of pressure for us to implement it in our tool. We're small enough and they are a significant partner so we priortised and got it into our platform so that a customer CAN implement it if they want too. The keyword here is CAN. As a platform with database functionality, Snowflake allows customers to implement any level of security for access. So each instance of Snowflake https://XX123456.us-east-1.snowflakecomputing.com/ https://XX123456.us-east-1.snowflakecomputing.com/ security is independent. So let's answer some questions * "Was this a production breach", No, this was not a production breach. This is not when hackers went into the Azure and were able to elevate credentials and see all accounts. * "But doesn't Snowflake use MFA", yes and on INTERNAL systems and the ability to connect to the PROD environment of SF it would absolutely have that. However, every customer has the ability to configure a user's credentials for access in their instance https://XX123456.us-east-1.snowflakecomputing.com/ https://XX123456.us-east-1.snowflakecomputing.com/. Likely, the SE was granted access to the client's instance through the use of a user name and password. * "But why not use MFA then Snowflake", this comes back to Okta. You have to create a trust relationship between providers. That's easy when you are only talking about your own employees. They all sit in your same directory. However, to set up a separate trust relationship for another domain, well now we have to get the security team of the client involved. All this so that someone can take a little bit of data and move it into another environment in order to do a demonstration. Not likely. * "Well that just sounds stupid, my application has SSO enabled." True, but as also a database - Snowflake has to account for entire ecosystem that don't even support SSO-OAuth. For example, two major products that do not support SSO connections to Snowflake: Azure Machine Learning and Zapier. Both only support user name / password connections to Snowflake. So if you were an SE trying to show the value of Snowflake and its integration with AzureML; the ONLY way you could do that would be through user name / password functionality. The issue in all of this is we are so used to thinking of SaaS as just an application. Databases have to support all sorts of application weirdness. Here is my gut thought on how this went down. * The SE was working with multiple clients. These clients in an effort to save time and not wanting to have their security team involved and try and create a trust relationship with Snowflake went ahead and created a user name and password for the SE into the client environment vs setting up SSO. Again, this is ONLY controlled by the client. * The SE at some point had active access to multiple instances of Snowflake to their various clients; with various levels of access depending upon the skill and fastidiousness of the client when they created the account for the Snowflake SE into their account. Again, how the SE is connecting to Snowflake is the same way any other user and application are doing so. I know this for a fact as SEs have to create their own throwaway demo instances https://XX123456.us-east-1.snowflakecomputing.com/ https://XX123456.us-east-1.snowflakecomputing.com/. SEs don't have access into the backend of Snowflake. * At some point this SE had malware installed on to their computer. At this point, this person was cooked and any customer who had given them access to the environment. As for Hudson Rocks, they can go f themselves for doxing the SE. It was quite trivial to find who this person was and based on the Snowflake announcement they no longer have a job. That person was having an awful week already, now there name is out there. So again, F Hudson for doxing. It would have been easy to blind them out of the exchange (they did the hacker), but did not do so for the victim.

bawolff 2y ago

Indeed, the less novel the exploit the more embarassing it is. Data stolen because of some crazy multi-exploit zero day chain. Well that is understandable, i don't blame the company. Data stolen because no 2FA support? In 2024 that is just embarassing.

p0seidon 2y ago
Yes, that's also what I think must have happened—missing 2FA. But it seems it's also not mandatory within Snowflake accounts also, from what I understand from the general message. I've never used Snowflake and assumed that because you push all your data into it, it probably has 2FA enabled by default. Is it optional?
- Galanwe 2y ago
  I don't quite understand how Snowflake works. My understanding was that you had to grant storage access (e.g. S3) and compute access (e.g. EC2) from your account to Snowflake, which would then use said resources to perform queries that you issue from their hosted web UI. In that case it would mean stealing the Snowflake demo account of a SE should not expose your data unless you forgot to revoke their access to your underlying resources. Can someone explain if that is how it works?
  - adovenmuehle 2y ago
    No, Snowflake runs it's own storage and compute (on either AWS, GCP, or Azure depending on what you pick).
    - Galanwe 2y ago
      So the customer data is actually stored on Snowflakes AWS accounts? What difference does it make what underlying storage / provider it uses then? Also does that mean every data query to snowflake goes out/in to/from internet at egress/Ingress costs?
      - p_l 2y ago
        At the snowflake size you get custom price lists from cloud operators. But I think there was also support for peering with client VPCs (or equivalents) which is why they support AWS, Azure, and GCP - you choose the location that is most fitting for linking with your cloud/physical workloads.
      - wokwokwok 2y ago
        > So the customer data is actually stored on Snowflakes AWS accounts? Yes. > Also does that mean every data query to snowflake goes out/in to/from internet at egress/Ingress costs? Yes. It's covered comprehensively in their docs, along with the caveats. > What difference does it make what underlying storage / provider it uses then? "Snowflake does not charge data ingress fees. However, a cloud storage provider might charge a data egress fee for transferring data from the provider to your Snowflake account." unsaid: "...and you have to pay for that". Note that when they say 'your Snowflake account' they mean our cloud account which we own, and which we run our workloads in, which we refer to as 'your' snowflake account. Tangibly speaking, what means is that if you want to check up your billing; you go through snowflake; you can't login to a cloud console and see the actual charges the cloud vendor is charging. > What difference does it make what underlying storage / provider it uses then? They pass the specific underlying cloud vendor costs on to you (with, I guess, some markup, though you have no way of know what that is :)
- deleted 2y ago
  [deleted]

hangtime79 2y ago

Having spent a lot of time in the Snowflake ecosystem, a few things to understand. Snowflake is both a platform and a database. Customers can implement any level of security within the system. For example, when Snowflake introduced OAuth functionality, they put a lot of pressure for us to implement it in our tool. We're small enough and they are a significant partner so we priortised and got it into our platform so that a customer CAN implement it if they want too. The keyword here is CAN. As a platform with database functionality, Snowflake allows customers to implement any level of security for access. So each instance of Snowflake https://XX123456.us-east-1.snowflakecomputing.com/ https://XX123456.us-east-1.snowflakecomputing.com/ security is independent. So let's answer some questions * "Was this a production breach", No, this was not a production breach. This is not when hackers went into the Azure and were able to elevate credentials and see all accounts. * "But doesn't Snowflake use MFA", yes and on INTERNAL systems and the ability to connect to the PROD environment of SF it would absolutely have that. However, every customer has the ability to configure a user's credentials for access in their instance https://XX123456.us-east-1.snowflakecomputing.com/ https://XX123456.us-east-1.snowflakecomputing.com/. Likely, the SE was granted access to the client's instance through the use of a user name and password. * "But why not use MFA then Snowflake", this comes back to Okta. You have to create a trust relationship between providers. That's easy when you are only talking about your own employees. They all sit in your same directory. However, to set up a separate trust relationship for another domain, well now we have to get the security team of the client involved. All this so that someone can take a little bit of data and move it into another environment in order to do a demonstration. Not likely. * "Well that just sounds stupid, my application has SSO enabled." True, but as also a database - Snowflake has to account for entire ecosystem that don't even support SSO-OAuth. For example, two major products that do not support SSO connections to Snowflake: Azure Machine Learning and Zapier. Both only support user name / password connections to Snowflake. So if you were an SE trying to show the value of Snowflake and its integration with AzureML; the ONLY way you could do that would be through user name / password functionality. The issue in all of this is we are so used to thinking of SaaS as just an application. Databases have to support all sorts of application weirdness. Here is my gut thought on how this went down. * The SE was working with multiple clients. These clients in an effort to save time and not wanting to have their security team involved and try and create a trust relationship with Snowflake went ahead and created a user name and password for the SE into the client environment vs setting up SSO. Again, this is ONLY controlled by the client. * The SE at some point had active access to multiple instances of Snowflake to their various clients; with various levels of access depending upon the skill and fastidiousness of the client when they created the account for the Snowflake SE into their account. Again, how the SE is connecting to Snowflake is the same way any other user and application are doing so. I know this for a fact as SEs have to create their own throwaway demo instances https://XX123456.us-east-1.snowflakecomputing.com/ https://XX123456.us-east-1.snowflakecomputing.com/. SEs don't have access into the backend of Snowflake. * At some point this SE had malware installed on to their computer. At this point, this person was cooked and any customer who had given them access to the environment. As for Hudson Rocks, they can go f themselves for doxing the SE. It was quite trivial to find who this person was and based on the Snowflake announcement they no longer have a job. That person was having an awful week already, now there name is out there. So again, F Hudson for doxing. It would have been easy to blind them out of the exchange (they did the hacker), but did not do so for the victim.