5 ms·
There are a couple of things I wanted to say, and I can speak with some authority on the subject as I speak on behalf of DigitalOcean. This was mentioned to me
by neom 13y ago
There are a couple of things I wanted to say, and I can speak with some authority on the subject as I speak on behalf of DigitalOcean.
This was mentioned to me on twitter hours ago, prior to this post. The first thing I said is that most people these days understand the importance of a responsible disclosure, and that we take all security issues very seriously. Not following responsible disclosure with a company such as DigitalOcean is extremely irresponsible and I would be amiss to point that if anyone did ever find a software vulnerability filing it and waiting 24 hours for the appropriate response is preferred. - https://www.digitalocean.com/security https://www.digitalocean.com/security
As far as I can tell here, there is no unexpected behavior that isn't documented or stressed. In both our API documentation, and our control panel, we note that you must either pass a flag or check the box to security delete the data. As far as I can tell, the flag is currently functionally correctly. so..
Is the complaint that customer data is being leaked from VMs? That the flag being passed via our API/Dash isn't actually working? Or, that our policy on not doing a secure delete by default isn't something you agree with?
j.
- innoying 13y agoIt doesn't seem like a security issue by DO in any way. It seems that the fog api (the project this 'issue' is filled for) doesn't allow a user to access the required flag to scrub the drive. Not a DO problem in any way I can tell assuming the scrub parameter is working correctly.
- nixgeek 13y agoThis is a well-documented situation that almost every provider of 'consumable infrastructure' before DigitalOcean came along has faced and solved. It is disheartening to see the same mistakes being made. Whilst I absolutely see their USP was always solid-state storage, and that has pitfalls in terms of how you can scrub data to avoid it being leaked, the platform should take every precaution to protect customers data. There shouldn't be an option to 'scrub data' and it shouldn't be defaulted to off so they can save some hassle, and avoid spending a few dollars. It shouldn't be an option because it should be on all the time, anything else is surprising the customer mightily. "What do you mean, my data leaked? Oh that's fine" -Nobody "Why'd you pick a provider who doesn't take our companies information security seriously?" -Every boss anywhere
- mikeash 13y agoIt is absolutely a DO problem in two ways: 1) they default to bad behavior and 2) their UI does not make it at all clear what the consequences of that bad default actually are.
- lawnchair_larry 13y agoYour attitude and response here is exactly why disclosing this was in fact "responsible".
- neom 13y agoI asked three questions so I could address the title of the tread, that our users data is being leaked.
- nixgeek 13y agoI presume the audience felt a Chief Technology Evangelist had more control over the tone and intent of his messages on the Internet, and therefore actually meant to couch his three questions within pointy remarks about responsible disclosure of a problem which was announced as fixed by DigitalOcean and reported as such in the press over 6 months ago.
- nknighthb 13y agoYou either know or should know the answers to the questions you've asked in plainly bad faith, as the problem has been clearly described both here and in the linked github issue, which isn't even filed with DO because DO has already made it crystal clear you don't care about having secure defaults in this matter. The issue was instead filed against fog, so that users of that library may be protected to the extent possible under the circumstances. In other words: This isn't your issue anymore. You've already publicly dismissed it. Worse, you've gone back on an earlier promise about it. It is now in the hands of the community to try and protect your customers, since you have refused to.
- neom 13y agoTo be perfectly frank, I think the title of this post is totally disingenuous and started the wrong conversation, and that makes me sad because I know we do care about our customers, and their security. Had the title said "DigitalOcean doesn't care about it's customers security" I'd have been happy, because that would start a conversation we should probably be having about how data is deleted. As it seems like this is actually an issue with people not liking how our product works, I've begun the internal conversation going about two things: First, communicating again to our customers by way of a blog post that this is how the production functions, as well as highlighting any relevant tutorials. Second, working with the product team and engineers to either reverse this functionality at best, at minimum draw greater attention to it. j.
- thatthatis 13y agoAs someone who isn't a DO customer, the thing that most dissuades me from this about becoming a customer is that this is a case of insecure by default. How many more of those are lurking around?
- yrro 13y agoHere's another compound fuckup for you: 1. DigitalOcean users are unable to install their own kernel updates! 2. DigitalOcean have to bother making a new kernel image available via their admin interface; they haven't done this for over six months of Debian kernel updated in my experience. 3. Even if DigitalOcean did make a new kernel available, there's no notification to inform the customer that they have to log in to the admin interface and pick the new kernel from the list, then reboot their VM. 4. The list of kernels in the admin interface is sorted... bizarrely. I check it every so often and there is no sensible overall naming scheme; you are presented with a popup menu listing every single kernel for every single distribution; the latest kernel for Debian is in the middle of the list. 5. My attempt to resolve these issues with DigitalOcean support covinced me that the person I was corresponding with has no idea what a kernel even is, much less that DigitalOcean's list of available kernels is... lacking. This situation, plus a couple the longstanding lack of progress towards IPv6 support; lack of ability to control kernel parameters; lack of a way to snapshot the filesystem for backups; makes me an unhappy DigitalOcean user who is going to jump ship for Bytemark at the earliest opportunity.
- warrenm 13y agoFunny, I've been able to update off the standard package repos
- yrro 13y agoYou may have updated the package, but did you really boot from it? Run 'uname -v' to check: $ uname -v #1 SMP Debian 3.2.46-1 DigitalOcean systems do not boot from the kernel image installed within your VM; they are externally provided. This reminds me of something I omitted from my original rant. I've actually had to pin the kernel image package that I've got installed on my VM to the version that DigitalOcean provide: linux-image-3.2.0-4-686-pae: Installed: 3.2.46-1 Candidate: 3.2.51-1 Version table: 3.2.51-1 0 550 http://http.debian.net/debian/ wheezy/main i386 Packages 3.2.46-1+deb7u1 0 550 http://security.debian.org/ wheezy/updates/main i386 Packages *** 3.2.46-1 0 100 /var/lib/dpkg/status Because an unforseen ABI break in some netfilter module means that if I install the newest package, then reboot, one of the modules used by my iptables setup fails to load. ferm notices this and rolls back my firewall configuration--to the default state which allows all traffic. I noticed this, but I wonder how many other customers with similar setups did not, and hence have not noticed that their iptables rules are incorrect or absent.
- coolj 13y ago> Or, that our policy on not doing a secure delete by default isn't something you agree with? This one. Choosing insecure defaults for a virtualization API is a Bad Idea. As a rule of thumb (to put it bluntly), people are dumb. If you give them a loaded gun, they will shoot themselves with it. And they will blame you for it. At least put the safety on and make them take a conscious step before blowing their face off. Don't mean to tell you your business, but seriously, insecure defaults are a Bad Idea for a virt API.
- mikeash 13y agoWhile you're totally right, this doesn't even come down to "people are dumb". The documentation is simply lacking here. At no point that I can find do they discuss the ramifications of not using the scrub option, and even a smart person could reasonably expect that not using that option still doesn't leak your data, just in some other way (one less safe against certain attackers, presumably).
- rythie 13y agoAs a DO customer I find this completely unexpected and unacceptable. Secure delete should be always done before transferring to another customer, if this takes time it should scrubbed as queued job, before the next customer gets it.
- codygman 13y agoAgreed (also as a DO customer), DO please follow the instructions of Aaron Friel: https://news.ycombinator.com/item?id=6983270 https://news.ycombinator.com/item?id=6983270 It's a win-win :)
- integraton 13y agoAny company that has ever stored data on DigitalOcean now needs to operate under the assumption that other DigitalOcean customers have accessed it. Even if every staff member believes they checked the "Scrub Data" checkbox or used the API flag when destroying droplets, human memory is unreliable and people make mistakes. This is a very serious security issue and it's appalling that anyone is making excuses for it, and it's even more appalling that the company responds by blaming customers. Customers should not be able to access other customers' data under any circumstances. It shouldn't even need to be stated that providing access to other customers' data should not be the default.
- icelancer 13y ago>The first thing I said is that most people these days understand the importance of a responsible disclosure, and that we take all security issues very seriously. Not following responsible disclosure with a company such as DigitalOcean is extremely irresponsible Oh bullshit. Don't deflect the issue here by complaining that you don't like full disclosure policies that many security experts agree with. (Such as, I don't know, Bruce Schneier?) If you want to get into secondary levels of annoyance, how about the fact there have been multiple instances in the past with DO that were only resolved by open/full disclosure on forums?
- codygman 13y agoDO please follow the instructions of Aaron Friel: https://news.ycombinator.com/item?id=6983270 https://news.ycombinator.com/item?id=6983270 It's a win-win :)
- csdreamer7 13y agoI have to agree with the other posters, this response is disappointing. The initial concern the other posters have is obviously the lack of a secure default. Now i've known about this for quite some time and I mostly use DO for development and testing Chef recipes so I don't have an issue with it being off by default. I love the transparent $5 pricing. But I also don't store sensitive information on DO, yet. I also assumed when I was a newbie any sensitive info would be deleted. The easiest option would be to include a notice next to that checkbox that if there was any sensitive information you should select this option. I understand that SSD wiping would drive up your costs. Another option would be to switch to mechanical hard drives. I don't how easy that option is with your setup.
- pedrocr 13y agoSo you're saying, in the same response that 1) there was no "responsible disclosure" and 2) that this wasn't actually a security issue. You can't have it both ways.
- mikeash 13y ago"I never borrowed your saw, and anyway it was broken when you lent it to me."
- ceejayoz 13y agoOh FFS. You can't whine about "responsible disclosure" while saying "As far as I can tell here, there is no unexpected behavior that isn't documented or stressed." If it's documented, you've already disclosed it yourselves.
- mikeash 13y ago"Or, that our policy on not doing a secure delete by default isn't something you agree with?" This one. You have chosen a default that fails deadly. It's like designing a car that explodes when you turn it off. Oh, there's a button over here you can push to disable the explosion feature. That doesn't really make it better. You've created an option that can and will deeply screw many of your users. The mere existence of the option is not wrong by itself. But the fact that it can and will so easily screw so many people means that the option needs to have lots of flashing warning lights around it and it needs to be on by default. I just checked out the "Destroy" tab for my droplet. There is absolutely no, none, zero indication that failing to check this box will allow the next person to occupy my spot to read all of my data. Here is the exact text: "This is irreversible. We will destroy your droplet and all associated backups." "Scrub Data - This will strictly write 0s to your prior partition to ensure that all data is completely erased. Estimated Destroy Time: 11 minutes 22 seconds" I would expect "destroy your droplet" to mean that the data gets destroyed. I would expect the "scrub" option to be for paranoid people worried about the FBI seizing your equipment and using electron microscopes to extract residual data. At no point does anything in here give me any expectations that the default is "hand over all of the data currently on the VM to the next random stranger who walks in the door". Do you really speak on behalf of DigitalOcean? If so, you need to get your head straight fast, because this is not even remotely acceptable. You cannot defend the current practice, because it is not defensible. If you don't understand why that is, you need to sit down and think about it until you do. Right now, as a customer of yours, my thought is this: if you think this isn't important and doesn't need to be called out, what else have I missed? What other crazy data leaks do you allow by default with the defense that I could turn them off if I cared? I hope and assume the answer is "none", but now I'm rather worried. I can kind of sort of understand how one might end up building a system like this, thinking that it was a good idea at the time. But I cannot understand at all how someone could possibly defend it once it's pointed out that it's terrible.
- neom 13y agoHey Mike, I'm going to give you a call later this afternoon but I wanted to clarify. First, Moisey and I worked on this this morning, so worth a read: https://digitalocean.com/blog/ https://digitalocean.com/blog/ Second, the way this was approached was super super confusing, originally as it was in 140 characters, on the twitters. Maybe my short comings for not totally understanding the situation before I spoke, but information was fairly fragmented. First: Yes, I do speak on behalf of DigitalOcean. My original understanding was that this issue was with the secure delete flag not working when being passed. This promoted me to request, and continue to request, that if it was the case, security@ was notified with an outline of what is going on per http://digitalocean.com/security/ http://digitalocean.com/security/ - If it's an expected behaviour, while still not good at all, it isn't my call nor was I prepared to call the company into the office at midnight on a Sunday knowing we would issue a software update in the morning. Had the flag not been respected, I would have immediately called the senior engineering team as well as Ben and Moisey, so fully understanding the situation was very important. As a customer, I'd like you to know we do take security very very seriously, it's something we discuss going into everything, as we appreciate healthy conversations about the way our product works. I spend 4 hours last night trying to figure out exactly what was going on, it felt very difficult to get a straight answer of "your policy fucking blows and you better change it tomorrow" - That's something I can take to the bank, but I'm sorry if I wasn't clear. j.