5 ms·
SAPwned: SAP AI vulnerabilities expose customers' cloud environments and privat
- jgalt212 2y ago[flagged]
- mvandermeulen 2y agoExcellent write up. This wasn’t a sophisticated attack. Seems like there is very little discipline at Salesforce when it comes to deploying production systems.
- phoe18 2y agoHow is this related to Salesforce?
- bloqs 2y agoSalesforce, SAP, it's all the same sort of
- btown 2y agoThis is like saying Apple and Alphabet are the same. SAP is a 52 year old company and is the largest non-American software company in revenue, and has never been part of Salesforce: https://en.wikipedia.org/wiki/SAP https://en.wikipedia.org/wiki/SAP
- mofoteam 2y ago[flagged]
- cosmotic 2y agoAs security researchers, you think they might have known that pixelating text to redact it is a poor choice. https://www.bleepingcomputer.com/news/security/researcher-reverses-redaction-extracts-words-from-pixelated-image/ https://www.bleepingcomputer.com/news/security/researcher-re...
- NotMichaelBay 2y agoIt looks to me like it was blurred, not pixelated. Edit: Nvm, I guess they blurred text in some places, pixelated in others
- FrostKiwi 2y agoIn both cases, yes and there are multiple projects able to reverse it for both the pixelation case [1] Blurring is like a hashing algorithm. If you know the font, size and placement that was used, you can try out reblurring and Thus brute forcing characters [1] https://github.com/spipm/Depix https://github.com/spipm/Depix
- chatmasta 2y agoThe reported bugs have all been patched, and presumably the compromised secrets have been rotated. The blurring/pixelation is arguably unnecessary, regardless of its effectiveness. The censored data looks like local host names and some image hashes.
- deleted 2y ago
- dotty- 2y agoI hope SAP does a hard retrospective on why Wiz's research was not disrupted before they got full cluster admin. Like, I want to know from SAP's side whether they received any alerts for any of this activity and whether they investigated them properly. I wonder if there is any regulation SAP has to follow that requires them to have adequate alerting for suspicious network activity and whether this research can be used to show that they do not.
- SoftTalker 2y agoIndeed. And if they did not detect it, how can they know that customer data have not been compromised?
- uaas 2y agoUsually security researchers are required to reach out to the target before escalating further into the systems, asking for permissions to proceed. This is also something bug bounty programs require as per their rules for their targets in scope. I’d expect this to be the case here as well, given the researcher is employed by a security company. Researchers also usually mention which points they asked for additional permissions at in writeups, but now always.
- Propelloni 2y agoOh, they have rules and regulations, for sure. Take a look at their certification page: https://www.sap.com/about/trust-center/certification-compliance.html https://www.sap.com/about/trust-center/certification-complia... Question is, do they live it or is it just some binder sitting on a shelf.
- BodyCulture 2y agoThe problem is that people who do decisions don’t understand the technology. Most IT managers in Germany do not even know how programming works. There are exceptions, but the biggest players are people flying in blindsight.
- Propelloni 2y ago
- mac-chaffee 2y agoShocked that there was a tiller instance running. That's been deprecated since 2020: https://helm.sh/blog/helm-v2-deprecation-timeline/ https://helm.sh/blog/helm-v2-deprecation-timeline/
- uaas 2y agoYou would be horrified if you would know how much pre-‘20s or even pre-‘10s software is still running in production out there. Here we are talking about a huge enterprise and a somewhat complex migration (from tiller) but you can easily find outdated software without these aggravating circumstances as well.
- hunter2_ 2y agoSoftware from 2019 is horrifyingly outdated? If updates with security patches exist but haven't been applied, sure, but that's not really a default scenario depending on the stack.
- uaas 2y agoI’ve only used 2020 because of the example in question. Security patches might or might not have been applied in both my imaginary example and in real world.
- ketzu 2y agoIn my experience, "deprecated" is often taken as "we can still use that, it is not removed yet", which I find somewhat disheartening sometimes.
- c0balt 2y agoThat's easy though, the removed part can also be ingored by mirroring package repositories for RHEL/ Debian-based systems.
- sexy_seedbox 2y agoYou should see how much outdated / unsafe / unpatched legacy code is on Microsoft Dynamics.
- 1oooqooq 2y agothe sad part is that all this is going to accomplish is promote that sap has ai product their clients can purchase. it's not like anyone using sap know or care about security other than signing with a company that has all the ISO and whatnot, which is the reason they went with sap to begin with
- j45 2y agoI would assume for the prices SAP charges, it mah start as some kind of bulletin of how to properly secure the AI, and failing that a feature update to tighten some defaults. But a security fist to a leaky side door? I’d bet that upsets some customers. Many of these accounting systems are starting to sell AI to automate transactions, which may explain the read+write nature of the access described in the comments.
- betaby 2y agoAm I reading it correctly, customer's account data is exposed to the same customer? The exception is some logs as I see.
- waterproof 2y agoSome logs… and some other customers’ training data and code… and SAP’s internal docker image repository (with read-write access!)
- betaby 2y agoYou are right! I missed that all NFS folders on the screenshot have rwxr-xr-x permissions.
- jaaron 2y agoWhile I get that it's the AI product, the vulnerability here is the k8s configuration. It really has nothing to do with the AI product itself or AI training or anything related to machine learning or generative AI, it's more about poor cloud computing platform security.
- bilekas 2y agoThe article doesn't say that it is an issue with the product itself though. It explains very well than infact it's the isolation of the AI training models. > The root cause of these issues was the ability for attackers to run malicious AI models and training procedures, which are essentially code It's being researched and investigated, to my understanding, due to the prevalence of AI products and the need to be mindful of the infrastructure.
- j45 2y agoThe brand that sells is the brand at fault. Securing it or knowing to secure it or testing it or never releasing it until it was secure is all things that are with the brand making the sale.
- cchance 2y agoWhich is possibly worse lol, the fact SAP a company as big as they are with as much critical information as they have, fucking up basic cloud security, they didn't even fuck up something new they fucked up common shit from the sound of it.
- sunaookami 2y agoThe bigger the company the less they care since no one will hold them accountable anyways.
- dogbraid 2y agoWhy do you say that? A bigger company has also more to loose with such a problem and therefore they care much more about such things. Smaller companies can hide behind obscurity, not big company. If the same thing happened to a company much smaller then SAP, it wouldn't have made hacker news.
- ec109685 2y agoThis is really bad. They are running a single K8s cluster and expecting hard multi-tenancy guarantees? All the major clouds use vm boundaries and separate K8s clusters between customers. Microsoft was similarly bitten a few years ago with one of their function products that expected K8s to be the primary security boundary.
- bilekas 2y ago> They are running a single K8s cluster and expecting hard multi-tenancy guarantees? Maybe I missed something in the article but where are they expecting any hard guarantees. If there is a model being trained for example (running arbitrary code) where does a multi K8 tenency play? The main issue I see is all internal network communication was trusted once behind the proxy/firewall (Istio) but I probably just don't understand k8 clusters too well.
- robertlagrant 2y agoIstio is point to point between services. It's not a boundary in the sense you're thinking.
- bilekas 2y agoI will admit I don't know a lot about Kubernets at all, but as I see the Istio is supposed to be the proxy layer between services ? https://istio.io/latest/docs/ops/deployment/architecture/ https://istio.io/latest/docs/ops/deployment/architecture/ Being able to run as the Istio user (1337) renders the proxy itself moot right ?
- gbrayut 2y agoThere are a lot of other ways to bypass the Istio sidecar proxy, which is not designed to be a general egress boundary/firewall. See https://blog.howardjohn.info/posts/bypass-egress/ https://blog.howardjohn.info/posts/bypass-egress/
- Arbortheus 2y ago
- tiffanyh 2y agoHas anyone used Wiz? It's possibly the fastest rocket for an enterprise software company ever. $100M in just 1.5 years time $350M at end of 3-year https://www.wiz.io/blog/100m-arr-in-18-months-wiz-becomes-the-fastest-growing-software-company-ever https://www.wiz.io/blog/100m-arr-in-18-months-wiz-becomes-th...
- rtev 2y agoAbout to be acquired by Google for $23 billion as well!
- nicce 2y agoSo... the road for maximizing the profits will begin?
- kchr 2y agoUsing it and loving it. Security aspects aside, it's also the best tool I've tried for proper asset management in multi-cloud scenarios. With the graph feature you can write queries for basically anything, across all accounts if you wish.
- deleted 2y ago[deleted]
- tetha 2y agoThis makes me glad I finally talked people at work into running our annual pentests of our products on production, and putting the entire production infrastructure in scope. Focus may be on a specific product or system, but everything is in scope. And the first test is running, and no one is screaming yet, so fingers crossed.
- DyslexicAtheist 2y agowhen you say yearly I assume you're not conducting regular internal pentests? any pentesting companies that you could recommend which do more than just drive-by shooting with metasploit?
- tetha 2y agoWe pentest all of our developed applications annually and on top, a few customers have internal regulations to pentest applications they use, so some of our applications run through 3-4 pentests per year. This is pretty useful to stay up to date on our TLS configs, the latest security headers, they have found some issues in authorization structures and such. However, what I'd really like is budget and time for a dedicated infrastructure pentest. I'd like to give the pentesters the same access as our office has, to see if that's fine. And since I like pain, I'd also like to simulate compromise of an application server: Deploy some reverse shell / kali container as an unprivileged container with some middleware service access, and later on deploy a privileged containers as well. Ideally the first simulation should only lead to loss of data the service needs to function, but as the article shows: Who knows. Maybe we also lose everything. Regarding companies, at my current job we're having good experiences with Secuvera[1] from germany. They are doing the usual ZAP/Metasploit drive-bys, but they are also poking and prodding at various security boundaries, the services behind the application. We're getting good results from them. At my previous test, we also had a few encounters with Redteam Pentesting[2]. Those guys used an incorrectly used cipher-mode to exploit the links allowing users to "single-sign-on" (only in spirit, not in current tech) from the game-client to the forum in order to hijack arbitrary forum accounts by modifying the encrypted forum-account-id inside the link. And other fun hijinks. 1: https://www.secuvera.de/ https://www.secuvera.de/ (I can't find an english version of that site) 2: https://www.redteam-pentesting.de/en/ https://www.redteam-pentesting.de/en/
- darefalcon 2y agoCompanies that penetrate networks uninvited looking for vulnerabilities to create blog content should be prosecuted IMHO. This piece in particular sounds like a hit piece thinly vailed as a vulnerability disclosure. “We thanked them for their co-operation”. Sounds kinda like extortion.
- ppbjj 2y agoAre you serious?
- _dain_ 2y agoway to shoot the messenger.
- dumpsterdiver 2y ago> Companies that penetrate networks uninvited looking for vulnerabilities to create blog content should be prosecuted IMHO. Your comment could be rephrased as, "Companies who carelessly collect and store sensitive user data insecurely should not be closely scrutinized, and should be left alone to continue exposing innocent user data to malicious cyber criminals." Looks a lot different when you look at it from that angle, right?
- rvnx 2y agoIt's a crime, anything who tries to hack a large company without being invited would get seriously prosecuted, but as the law practice says "If you have billions of USD, laws don't apply to you anymore".