8 ms·
In Digital Ocean, S3-like space keys can access all your buckets
- sergioisidoro 3y agoS3 (and similar storages) have caused plenty of security issues in several occasions (usually because of misconfigured buckets, making all contents available to the public). Given this, it would be expected that companies would pay a bit more attention to the security of these data storage methods. However, in Digital Ocean, by design, you can't restrict keys to certain buckets. Once you issue a key, it can access all buckets within your project, with all operations (list, read, delete) files. The issue has been reported and is known for a long time. It's even the top voted "idea" in the company portal. Posting this here in hopes to bring awareness of this issue.
- karambir 3y agoThings like these are why I stopped using digitalocean. They had issues with DNS PTR records too for years. Same for initial auth tokens being global. I had hopes for them and used them exclusively at 1 time.
- k8sToGo 3y agoIsn't this usually solved through some form of IAM policies? Does DO not have this?
- Operyl 3y agoNo such luck there, but this is kind of the MO for DigitalOcean, sadly.
- dcow 3y agoI think DO is doing fine addressing their target market. I also wish they had IAM but you have to admit IAM is huge and messy and is often more about managing your employees than your infrastructure. I’m okay waiting for them to get it right rather than rushing out a quick half baked idea that ruins their entire product stack err platform.
- Operyl 3y agoI don't care for the full on flexibility of IAM, but I'd kill for a simple "this key is for this bucket" feature. That doesn't seem so far fetched, I don't need per-object permissions within a single bucket!
- deleted 3y ago[deleted]
- dcow 3y agoSince the bucket namespace is global you can always just use different accounts. But I agree it doesn't seem that hard to add this small feature.
- londons_explore 3y agoAs a user, you can fix this by having a proxy that fits in front and manages ACL's. Obviously you have to pay for resources for that proxy, and it's probably going to want a large bandwidth allocation. Lucky because DO doesn't charge for bandwidth.
- rmbyrro 3y agoIf I have to implement and manage an ACL myself (such a basic feature!!), it's better to go ahead and implement something like min.io and ditch DO entirely.
- undyingtrillion 3y agoThis is mostly true, but there is a case where you use your transfer pool. From their product docs: Droplets have their own transfer allowance, independent of Spaces. Traffic from Droplets to Spaces does not count against your Spaces transfer allowance (because inbound bandwidth to Spaces is free), but does currently count against your Droplets’ outbound transfer allowance. https://docs.digitalocean.com/products/spaces/details/pricing/#bandwidth https://docs.digitalocean.com/products/spaces/details/pricin...
- dang 3y agoI'm afraid you broke the site guidelines by using the submission title to editorialize. Please see https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html: "Please use the original title, unless it is misleading or linkbait; don't editorialize." If you want to say what you think is important about a page, that's fine, but do it by adding a comment to the thread. Then your view will be on a level playing field with everyone else's: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&sort=byDate&type=comment&query=%22level%20playing%20field%22%20by:dang https://hn.algolia.com/?dateRange=all&page=0&prefix=false&so.... Alternatively, you could have made a text post to communicate what you think is important about the issue, and then your title would have been fine. What's not ok on HN is to use the title field as a way to comment on an issue, because then the submitter gets a privileged position relative to other commenters. Since the title is by far the largest influence on a thread, we want to avoid that. Being the submitter of a page shouldn't confer any extra special power over the discussion.
- jamiedumont 3y agoI haven't used DO Spaces for a good while as I've shifted to B2 as cheaper S3-compatible storage that suits my archival workload. However I absolutely would be using Spaces as storage alongside any apps running on DO, and would have assumed that all the usual per-bucket permissions I'm used to elsewhere were present, so thanks for the heads up!
- snowstormsun 3y agoAnd DO consistently ignores requests to add this. It prevents me from seriously using Spaces and I think many others, too.
- sneak 3y agoDO has had a terrible attitude to security for as long as they've been around. I reported a major data leak vulnerability to them and they told me it was operating as intended, so I published it, and then they accused me of irresponsible disclosure, while simultaneously claiming that there was no security issue. This was a dozen years ago and the founders were involved in the response. They are at the top of the list of companies with which I will never do business.
- mdekkers 3y agoCould you provide more details?
- sneak 3y agoOn delete of VMs, the next user to be allocated disk would be able to read your data. There was a "scrub" feature that would bill you longer/extra to zeroize your data and not give it to the next customer. Apparently they hadn't heard of TRIM or thin provisioning. https://github.com/fog/fog/issues/2525 https://github.com/fog/fog/issues/2525 https://news.ycombinator.com/item?id=6983097 https://news.ycombinator.com/item?id=6983097 https://news.ycombinator.com/item?id=6983520 https://news.ycombinator.com/item?id=6983520
- belter 3y agoSome of the comments by DO on that GitHub thread, are really surprising, and disappointing to say the least...
- neom 3y agoI wrote those comments, fog wasn't using the API as he would have liked, and then this guy starts a bunch of trouble on HN stating that there was a vun on DigitalOcean, and conflated someone using the API incorrectly with us having a security vun. Two things I was trying to (admittedly not very well) say was: if DO did have a security issue, posting it on HN ("DigitalOcean leaks customer data between VMs" huh?) instead of emailing us about it isn't a very responsible way to disclose it and, there isn't actually a security issue, the fog app wasn't passing the scrub flag. In retrospect, I should have been less of a jerk, but I still think sneak was acting in bad faith in how he went about the whole thing. 10 years later and I still remember how pissed off I was that day, hah.
- jacooper 3y agoSpaces is annoying. Tailscale uses it as their main host for the repos, snd its down right annoying with no CDN and slowdowns occuring often.
- whateveracct 3y agoSpaces has a CDN
- jacooper 3y agoWell a crappy one then. Because tailscales repo is clearly in the US.
- patrakov 3y agoIs this post somehow related to the recent Ceph CVE-2023-43040 ("Improperly verified POST keys"), where Digital Ocean was also credited? https://www.openwall.com/lists/oss-security/2023/09/26/10 https://www.openwall.com/lists/oss-security/2023/09/26/10 --- A flaw was found in Ceph RGW. An unprivileged user can write to any bucket(s) accessible by a given key if a POST's form-data contains a key called 'bucket' with a value matching the name of the bucket used to sign the request. The result of this is that a user could actually upload to any bucket accessible by the specified access key as long as the bucket in the POST policy matches the bucket in said POST form part. We have assigned it a CVE of CVE-2023-43040 and the patch is attached. Credits to Lucas Henry of Digital Ocean.
- Operyl 3y agoNope, this post is from 2021 pretty much begging that DO provide per-bucket level access keys for Spaces. Otherwise, right now, all keys created for an account can access all buckets on that account.
- nik736 3y agoThis is the same for Linode, OVH, etc. One key can see ALL buckets, but access can be limited.
- slig 3y agoWhy the smaller players can't figure this out? Are they inventing their own S3-compatible storage or using a open source project underneath if?
- nik736 3y agoThey are all using Ceph, which is the limitation.
- personomas 3y agoIs this true? That's horrifying
- kbumsik 3y agoDo you mean Ceph Object gateway? I am not familiar with Ceph but does it have serious limitations?
- supriyo-biswas 3y agoThis is correct, Ceph's RadosGW doesn't support this feature. For people trying to figure out if their provider is using Ceph, make a request for https://<s3-endpoint>/test https://<s3-endpoint>/test and see if the x-amz-request-id header begins with "tx000", if it does, then they're using Ceph. $ curl -sSi https://nyc3.digitaloceanspaces.com/test/ https://nyc3.digitaloceanspaces.com/test/ | grep x-amz-request-id x-amz-request-id: tx00000947bc21a401c0f02-00651981d1-4b6a0-nyc3d
- pizzafeelsright 3y agoPainful, slow, costly. Access request needs to be created, validated, logged, processed, expired or revoked. Or just validated, processed, and logged.
- supriyo-biswas 3y agoMinio can do this, so I don't see why RadosGW can't.
- angra_mainyu 3y agoHmm, the only issue I've had with spaces is not being able to trivially serve a static site.
- whateveracct 3y agoWhat part isn't trivial? It has a CDN + public access. What am I missing here?
- roelvanhintum 3y agoThat is what Apps Platform is meant for. You can pick static as an option which, if i understand correctly, should do the same thing.
- fulafel 3y agoClick saver: this is about access keys, not bucket keys. (Which would have been in the original title...)
- sergioisidoro 3y agoThe point is that there are no bucket keys. Only Access keys for all buckets. There is no way to issue bucket specific keys. I'm not sure I follow how the title could be misleading.
- fulafel 3y agoWe're still talking about different things: In S3, for an object known as s3://foo/bar, "bar" is the key. See eg https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/object... So the title is ambiguous, it could mean that keys have to be unique across buckets since they share the same namespace.
- kbumsik 3y agoWell, it used to be same for Cloudflare R2 , which was just fixed less than two months ago [1]. This was an unexpected showstopper for me. I honestly find it hard to understand why this kind of basic security feature wasn't in GA. [1]: https://community.cloudflare.com/t/r2-token-per-bucket/389067/12 https://community.cloudflare.com/t/r2-token-per-bucket/38906...
- turtles3 3y agoThe lack of ACLs or comparable permissions is by far the biggest thing that prevents me from recommending DO for production workloads. This kind of thing is absolutely essential. You can't even separate dev resources from prod resources, every API key has godmode on your whole account. This is a security disaster. For a simple example, I'm running externaldns on a kubernetes cluster. For production use, I'd want to at least have an API key that can only access the DNS domains and nothing else. As is, the key can create compute resources or delete anything (including object storage buckets!). Again with no dev/prod isolation, so a leaked dev key that looks innocuous means your prod resources are now compromised. Some second tier providers do better at this, OVH and scaleway both have some concept of ACLs, but DO just don't even try. Apologies for the slight rant but it's frustrating to see something so basic overlooked by a company the size of DO.
- csnweb 3y agoI agree, probably the biggest point Digitalocean is lacking. I think the only workaround is creating several projects? But that probably becomes annoying and hard to manage quite easily if you want a lot of separation.
- deleted 3y ago[deleted]
- turtles3 3y agoIIRC projects only group resources under the same account. API keys are still account-wide, so projects provide no isolation there. There is a other mechanism DO has to allow multiple accounts under one billing context (I think it might be called 'teams'?) but sadly this is still extremely coarse grained, and doesn't allow you to lock a key down to a particular resource, or even a class of resources.
- unilynx 3y agoTeams have separate billing per team. Other platforms would call them ‘organization’
- drogus 3y agoWait till you hear that when you generate a DO container registry key it includes a key that gives you access to all of the resources, not only the registry.
- sergioisidoro 3y agoAre you saying you have to issue a "god" level API access token to access a single docker (private) image in the Digital Ocean container registry?
- dijksterhuis 3y agoThe permissions options on the API key are read or write for the whole project. There is basically no granularity.
- drogus 3y agoIt's not about the API key. It's about the container registry credentials. Which you would expect to be able to only interact with the container registry. This is not the case. If you go to the container registry and click to download credentials you will get JSON like: {"auths":{"registry.digitalocean.com":{"auth":"<base64-encoded-credentials>"}}} If you decode the base64 encoded credentials it will be a string like "<token>:<token>" with either a read only or read/write token to all of the resources on DO.
- dijksterhuis 3y ago> It's not about the API key. It's about the container registry credentials. Which you would expect to be able to only interact with the container registry. think my reply shortness maybe has you misunderstanding me? The fact that an API token can only have read/write project wide basically results in everything you said. No RBAC on services. No RBAC on specific API actions. Anyone with read+write can do/nuke everything. That’s why I’m migrating $COMPANY off to AWS as fast as I humanly can by myself.
- leros 3y agoI don't know if they've improved it, but 5 years ago I basically couldn't delete a DO Space because they had no way of doing a mass deletion. I even got technical support who basically told me I was doing it the best way possible. My delete operation took several months to run.
- supriyo-biswas 3y agoThat is a limitation that comes along with having to be S3 compatible; the best you can do is to combine ListObjectsV2[1] + DeleteObjects[2] (1000 objects in a single API call). [1] https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObje... [2] https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteOb...
- vlovich123 3y agoWhile S3 doesn’t have a REST API to do it, you can trigger it through the console and it applies pretty quickly if you don’t have a lot of objects (hundreds of thousands if I recall correctly) and will apply in the background otherwise. The way to do it via the s3 API is to install a lifecycle policy to delete everything.
- leros 3y agoAt the time, the lifecycle policy didn't seem to be an option on DO unfortunately.
- supriyo-biswas 3y agoLifecycle requests are a good call, but being an asynchronous process, it may take some time to be processed, even when using S3. On a particularly large bucket, I've observed it taking over a week in one instance. The console isn't magic however, it makes the same ListObjectsV2 + DeleteObjects calls in parallel.
- octagons 3y agohttps://buckets.grayhatwarfare.com/ https://buckets.grayhatwarfare.com/
- deleted 3y ago[deleted]
- ph4te 3y agoI used to use DO Spaces and Backblaze, but then I found Wasabi and have switched everything over since. It pretty much has full S3 compatibility, and doesn’t use Minio or Ceph. Mostly just keep adding data, very rarely delete data.
- jddj 3y agoSame here, very happy with wasabi so far.
- mootline 3y agoI unfortunately had to ditch DigitalOcean because of a number of small problems like this (not taking permissions seriously is a symptom of a larger problem). Other issues included how you can't assign a floating IP to a load balancer, the managed kubernetes is embarassing (to resize nodes, you must destroy and recreate the whole cluster), paltry volumes IOPS, minimal support for DNS stuff, and more.
- slig 3y agoAFAIK, you don't need to destroy the cluster and recreate it, just add the nodes you want and destroy the old ones.
- kyledrake 3y agoAside from this, their Spaces service has had multiple outages, the service seems to be pretty unreliable in general (definitely compared to S3). A quick glance of their NYC3 status suggests they may have finally improved this recently, but I don't know what use case I would have for a version of S3 that goes down multiple times a year.
- tigeroil 3y agoIndeed - I used to use Spaces for a project but found not just the uptime but teh overall reliability and performance was, at the time at least, markedly worse than AWS.
- j45 3y agoDigitalOcean should not be used for production use, or the sole network. There is a critical flaw in their operations management that will delete all of your long standing droplets with a valid credit card on file and zero issue with the conduct of the droplets or with the credit card.
- j45 3y agoAnother reason DigitalOcean should not be used for production use, or the sole network. I am looking into blackblaze and other providers. As seriously, Digital Ocean contains a critical flaw in their operations management that will delete all of your long standing droplets with a valid credit card on file and zero issue with the conduct of the droplets or with the credit card.