15 ms·

Taking Over DigitalOcean Domains via a Lax Domain Import System

385 points by infosecau 10y ago

arkadiyt 10y ago
I will never stop being infuriated by responses like this from companies - how many more megaleaks have to happen before they realize that they need to embrace white hats, not ban their accounts, not sue them, not swat them / have them arrested, not silence them. Great find / writeup.
- IanCal 10y ago
  Why, when the author realised this was likely to be possible, didn't they get in touch with DO? Or try with a domain they knew was OK to use? Or at least just try with one domain. They took almost twenty thousand, sent all the requests to their own server and logged them. That's surely not your first step.
  - mandatory 10y ago
    See my response below and the logs were secure removed shortly after (as stated in the blog post).
    - xuki 10y ago
      Not that I don't believe you, but you already lost white hat status in this case the moment you log traffic on 20k domains. Just do an attack on another domain you own is white hat, 20k domains is not.
      - mandatory 10y ago
        You're probably right about the logging being a bit too far, it was mainly my curiosity getting the best of me. One of my big assumptions was that all of these domains were just owned by one domain broker and this wasn't actually a systemic problem with the implemented importation methodology. I also thought it would be mild because if they had been deleted from an account they were likely no longer used (or so I had wrongfully assumed).
        IanCal 10y ago
        I certainly don't attribute malice on your part, and I'm sorry if my comment came across this way. My point was (at least intended to be) that I'm not really surprised at DO banning you. I would be careful about doing something like this on a large scale, I would actually be surprised if this doesn't technically violate some laws. Please be careful, you don't want to end up trying to explain nameservers to a judge. [edit - and the rest of us don't want that either, even just selfishly I would like as many security researchers practicing their craft as possible, but also I don't want people being punished for trying to do the right thing]
  - arkadiyt 10y ago
    Domains can only be added if they are not in another account, so he added only domains which were previously deleted but still pointed to DO nameservers (hence almost certainly not being used). And if DO was really concerned about the damage then they would have removed the added A records, but they didn't, they banned the author and continued to let all traffic go to his server.
    - IanCal 10y ago
      > (hence almost certainly not being used) Well they were still being accessed by real people. > And if DO was really concerned about the damage then they would have removed the added A records, but they didn't, they banned the author and continued to let all traffic go to his server. Both seem reasonable. And it does sound like the security team are actually doing something about it. > I reached out to DigitalOcean’s team to see if they can assist in deleting the domains from my account (sadly leaving them vulnerable again) or sinkholing the DNS to 127.0.0.1. I received a very helpful response from someone on the security team and it appears they will look into it. I assume these things happened from different departments. Banning an account because you saw it make 20k requests to your API adding domains seems pretty reasonable, and it was a few hours before they reached out. If you saw that activity, would you ban the account or leave it open hoping that they'd be doing something nice and reach out?
      - wongarsu 10y ago
        > Banning an account because you saw it make 20k requests to your API adding domains seems pretty reasonable, and it was a few hours before they reached out. If you saw that activity, would you ban the account or leave it open hoping that they'd be doing something nice and reach out? If I was a domain reseller, adding my 20k domains to digital ocean just to get banned without warning, explaination or option for reconsideration I would be rightfully upset. If they didn't want people adding large numbers of domains they could just have limited the feature instead of banning people who reach some arbitrary threshold. If on the other hand the department that executed the ban knew that the registrations weren't made by the domain owners, they should be discussing such a huge incident with the security team. That discussion would naturally lead to them knowing about the specifics of this case, unless this case wasn't widely shared in the security team. So the options are: 1. Digital Ocean bans legitimate customers without warning or option for reconsideration; for no obvious reason 2. Big security incidents don't get reported to the security team 3. The responsible people thought that this was not a big security incident 4. This incident wasn't discussed in the security team 5. They knowingly banned a white hat hacker (who may or may not have gone too far) Of all those options, the last one is by far the one that looks best for Digital Ocean.
- treehau5 10y ago
  Because it can still cause real damage, even if the intentions were good.
- empath75 10y ago
  I wouldn't call redirecting 10s of thousands of accounts 'white-hat' hacking. He could have just done like, I dunno, 3, to prove his point.
diegorbaquero 10y ago
Great article! I'm saddened by DO's response and further wronging a white hat by banning you. Let's remember Linode offers 2x the RAM.
- putlake 10y ago
  Linode also does not verify domain ownership before you add a domain to their DNS. And they also use the same nameservers for all accounts. So I think they're in the same boat as DO as far as this "vulnerability" is concerned. I'm a long-time Linode customer and use them for all servers that run important services. I use DO for the less important stuff. But Linode has had a not-so-stellar record of data breaches.
- optforfon 10y ago
  No one remembers how Linode tried to cover up getting hacked?
  - linrocks 10y ago
    I never noticed anything like that.
- thegeekpirate 10y ago
  Linode has had _far_ larger issues than DO ever has. That's why I switched to DO in the first place.
  - nsgf 10y ago
    Same here. Please stay away from Linode. They used to be good. Now they are unprofessional and unethical.
    - akie 10y ago
      I'm a new Linode user (as of this week) and have been happy so far. What are these problems you talk about? I'd like to know before I commit further.
V8OaSsoA 10y ago
this post raises questions: Was there a realization into how legitimate users may be affected by this action? Was there a plan to remove those domains from their account after making and disclosing their proof of concept? Why not stop at 10 or 20, and then alert DO to the findings? 20 thousand was unnecessary.
- mandatory 10y ago
  Fair point, my relucatance to stop was mainly due to companies usually disreguarding reports unless I have strong proof. Stopping short of the full scope would've left it up to speculation as to the full amount of vulnerable domains. It was my plan to delete the domains (or at least null route them so others couldn't take them over with more malicious intent). However my account was banned before I could do so.
  - deleted 10y ago
    [deleted]
  - tombrossman 10y ago
    Have you had any contact with DO's support or security team prior to this? You are correct that companies frequently ignore or downplay reports but it's nice to at least make that first attempt for each new find (per company). This is only ever done as a courtesy, I'm not saying you must or even that you should have here. Remember, it's just another nerd out there somewhere who receives your report. Sometimes they are super stoked to have it and will be very responsive. This can be more satisfying than a public disclosure without first contacting the company.
  - Aeolun 10y ago
    That turned out to indeed be the case after their first response :/ How would it have ended if you hadn't reached out to one of their security people after being banned? They'd have given you the traffic, locked your account, and congratulated themselves on a job well done?
- jsmthrowaway 10y ago
  How does making a domain that does not resolve suddenly resolve negatively impact a user, again? The domain could not have possibly been operational before the circumstances that brought about this scenario, and there is no legitimate traffic they could possibly be receiving. DigitalOcean could certainly improve authentication here but there are dozens of authoritative services that do not, and this is not a new problem (but is a valuable reminder). I can think of four free providers off the top of my head that this trick would work against. Put another way, the domains were entirely unresolved and offline. Then they weren't. If anything, this is a nice lesson about keeping your zone and delegations clean, and I'm glad I read it. Nobody got hacked, nobody lost traffic, nobody was impacted. If there was sensitive traffic going to a non-resolving domain, I have more questions than answers. I agree adding the full set was probably a bit much, but you can make that point without misplaced concern for alleged harm. I'm not impressed with signing up for HN to hit someone like this and your far worse and flagged followup, particularly since it really looks like astroturfing. I guess take solace that the OP pretty much guaranteed he won't get the zones again, since they're for research. (Nice ninja edit and deletion.)
anilgulecha 10y ago
TO: ANY DIGITAL-OCEAN USER, This is an absolutely terrible response from DO. If I had anything hosted here, I'd move away ASAP. Seriously, do it.
- sattoshi 10y ago
  I should leave the most reliable host I had to date because someone was messing around with things he shouldn't be? DO knows people can do this, but they don't want people to do it. Remembering 2 recurring DNS servers is easier for bulk management than a bunch of different ones. You don't test services like that, it can negatively affect other users. Response wasn't perfect but it was reasonable.
  - timv 10y ago
    > Response wasn't perfect but it was reasonable I'm sorry, "We aware that we make it easy for about 20k domains to be directed to a malicious host, but we're not going to do anything about it" is reasonable? But of course, it's because Matt "was messing around with things he shouldn't be". It's all solved - we just need everyone to stop doing things that DO "don't want people to do".
    - MichaelGG 10y ago
      Meh the owners of the domains gave up control by pointing to someone else's nameservers.
      - Dylan16807 10y ago
        You've just condemned 99% of domains. You really think that's reasonable?
        ZoF 10y ago
        I do think it's reasonable. If the domain is added to the account there is no PoC, it's only for domains that have been removed from accounts but still have the nameserver values(meaning the domain is not being used at this point, there's no zone file if it isn't added to an account). So this is mostly only going to affect currently derelict domains. I'm not saying it isn't something to worry about, but I do think it's a reasonable solution.
flexd 10y ago
This doesn't help my impression of Digital Ocean at all (even if I am a paying customer currently). A few years ago you could impersonate Digital Ocean staff on their support pages with no effort. They grabbed the username from your email, so whatever you put in front of the @ becamse your username on the forums, visible to everyone. And the avatar came from one of those email->avatar services where you can sign up and set it to anything. So when I signed up with a username like digitalocean@mydomain.com, I ended up being called "digitalocean" on the support forums, and if I had wanted I could just change the avatar to the Digital Ocean logo and impersonate DO or anyone else. I tried reporting it but got pretty much the same answer as this guy (though I did not get banned). Luckily they fixed it like a year later. Great write-up, and interesting problem! I wonder if more hosting providers are vulnerable to the same problem.
- jsmthrowaway 10y ago
  I can think of at least Cloudflare (somewhat), Linode, and Hurricane Electric off the top of my head. Anybody who operates a well-known ns1 type of resolver. It's more a problem with zone hygiene than hosts, honestly.
  - deleted 10y ago
    [deleted]
  - stestagg 10y ago
    This seems like a great candidate for an ACME style protocol. I would totally support the ACME spec being expanded to cover this, and various other domain verification related issues
    - corobo 10y ago
      Or just claim your domain if you point it at a DNS service..
  - ilogik 10y ago
    Cloudflare does something similar to AWS. Each user gets different nameservers
ultramancool 10y ago
Wonder what else might be vulnerable to this... CloudFlare seems like it may, they only have a handful of nameservers in any case. Sort of hard to call it a vulnerability on DO's part though - more of an issue with the admins. I think most DNS services operate in this way, really, route53 may be the exception, not the rule.
i_have_to_speak 10y ago
TL;DR - If you own example.com and use DO as your nameserver, then anyone with a DO account can add DNS records for example.com.
- ultramancool 10y ago
  Not quite, only one person can, so if you add first, it's yours. Typically they have you add the domain and then you switch the DNS and in that configuration you wouldn't be vulnerable at all.
  - MichaelGG 10y ago
    It sounds like this only impacts people that weren't using their domain, eh? If I leave my NS pointing to some random provider and delete my account, I'm sorta handing off control... At first I thought it was an issue where DO was using their own DNS servers but letting you add domains. So if Microsoft.com wasn't registered, you'd register it then all DO customer traffic would get your DNS records. Some telcos have this. Sign up, say you're porting a number. Provide number of local bank and invalid port info. Account gets set up, telco routes their own calls to you. Port is rejected and customer service is asking for info. All while you are getting the bank's calls and forwarding them. Perfect MITM.
    - samuellb 10y ago
      Yes, security of number porting is too lax. My number was once ported by a telco (accidentally in this case). They did call me and I had to confirm that I would get a new SIM card and which size I needed, but since I was expecting a new SIM card from the very same telco I didn't understand that they were in fact porting my number. They forgot to say that. Luckily I was able to get my number back, after many calls to customer support (they managed to undo the porting when I wanted to check how it was going) and a couple of weeks of waiting.
dotBen 10y ago
An additional vector for this kind of attack is to create a zonefile for a subdomain off of a working, live domain administered by the same DNS server. EG if foo.com is a working site on your DNS provider, try creating a zonefile for bar.foo.com and see if you can create an A record to point to your own server. This used to be something shared web hosting services running CPanel/WHM were particularly susceptible to. Clearly, the risks here are both phishing/identity and cookie credential stealing.
djhworld 10y ago
Very interesting read, thanks. I'm surprised at the response from Digital Ocean, did you adequately explain what you had done? The first person that replied looks like he just skim read your email or didn't understand the fact you had sinkholed a lot of traffic.
- mandatory 10y ago
  I believe I was clear about it. However sometimes my writing can be unclear so perhaps it wasn't properly understood (I assume you're talking about their security team's response and not Trust & Safety?). Kind of sad about the massive amount of hate for DigitalOcean in this thread as their security team really seemed quite nice. Their support was just acting on an anomaly they had seen so shrugs.
  - djhworld 10y ago
    It was the guy who said "Thank you for sending this in. This is a known workflow within our platform. We are committed to always improving our customer’s experience and have been examining ways of minimizing the type of behavior you are describing." It sounds to me like he read your email about the vulnerability, dismissed it as "on the backlog" and skipped the bit about the fact you had sinkholed lots of traffic. We only have one side of the story though, so who knows. Lessons can be learnt.
djhworld 10y ago
Very interesting read, thanks. I'm surprised at the response from Digital Ocean, did you adequately explain what you had done? The first person that replied looks like he just skim read your email or didn't understand the fact you had sinkholed a lot of traffic.
tszming 10y ago
I think most of the providers (e.g. DO, Linode, CloudFlare etc) do not check the authority of DNS due to the chicken-and-egg problem. The AWS way to handle this issue is definitely awesome but the infrastructure required is not worth for those companies who are providing "free DNS service" as an add-on to their existing customers. Anyway, IMO, it is your fault if you point to a nameserver but not utilizing it.
- jsmthrowaway 10y ago
  The random nameservers are only accidentally a defense against this attack. They're avoiding SPOFs, including TLDs -- you never receive nameservers in the same TLD for example. It's a reliability and scaling consideration with this accidental benefit. Most admins don't think about a complete TLD failure. Amazon did.
  - tszming 10y ago
    >> accidental benefit. Agree >> Most admins don't think about a complete TLD failure. Amazon did. I think companies such as Google or Facebook did think that before, but I am not sure why they didn't follow this trick.
chinathrow 10y ago
Am I reading this right: The only defence AWS has against this type of attack is the random (?) grouping of four different NS?
- breakingcups 10y ago
  It's effective.
mixedbit 10y ago
Amazon S3 has similar problems. To host static website you need use your domain name as the S3 bucket name. Amazon does not verify ownership of your domain, and bucket names use global namespace. Someone can easily block you from using S3 static website hosting by adding a bucket with your domain name before you do. Also if you delete a bucket and do not change your DNS, someone can recreate the bucket and will be serving files from your domain.
- athrun 10y ago
  As a best practice, use a 'normal' name for your S3 bucket and then put Cloudfront in front of your S3 website. This way you are not limited by bucket names and you also avoid any SSL validation errors. Note: you can set Cloudfront's TTL to 0 if you don't need any caching.
  - mortar 10y ago
    This is certainly good practice as long as anyone contemplating this considers the cost implications versus serving from S3 directly.
    - athrun 10y ago
      Good point. It's actually generally cheaper to serve content through Cloudfront instead of S3 directly as bandwidth is less expensive with Cloudfront. However you are going to pay for Cloudfront requests in addition to S3's. In the vast majority of cases, it's insignificant compared to the savings you can achieve on the bandwidth side. (And at a certain level you can ask AWS to waive CF request fees entirely).
- nulagrithom 10y ago
  EDIT: Just tested it and looks like I'm wrong. Proxying with CloudFlare doesn't help either... Looks like I may have done this with CloudFront instead? That's not correct. The S3 bucket name is always prefixed. The format is: bucketname.region.amazonaws.com. https://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html https://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosti... To clarify, you're going to have to add a DNS record either way. Doesn't matter what you call your bucket. And no, you don't need to put CloudFlare in front of it for this.
supersan 10y ago
Banning his account was totally unjustified since he approached them first with the issue. A less ethical person could have tried to make money or sold this off on the back market. People like him should be rewarded not have their accounts banned. For all we know he just saved DO a lot of headache in sorting this issue had it gone wrong. I really wish the response from DO on this was different.
- corobo 10y ago
  Adding 20k domains to your account is probably enough to flag as abuse even if you own the domains. Next time the author should probably try just the one or two. Bonus points if they're their own domains.
  - AgentME 10y ago
    >Bonus points if they're their own domains. If the service doesn't understand the issue at all, then when you explain that they're your domains, then they'll probably just tell you it's working as intended and that users should be able to add their own domains.
    - corobo 10y ago
      Arseuming makes an arse out of you and me. Give the techies on the other end of security@ the benefit of the doubt on the first go. When they fail to understand that then feel free to go ahead and pick one that's not yours to prove the point. One though, not 20 thousand.
- mitoyarzun 10y ago
  > The main reason I did not reach out with the theory instead of the proof-of-concept was because I believed that it would be ignored due to lack of evidence (as is my experience with past disclosures) I think this was his mistake.
ben_jones 10y ago
I find myself asking WW$D where $ is any large tech company with a "good" reputation. What would Google have done? Lyft? Spotify? Blizzard? Use some imagination to apply a similarly dangerous security breach to these companies. I feel like this question yields better context to ethical arguments because it makes us aware of the cognitive biases and view things from a more abstract perspective.. EDIT: Is there a way to include plain asterisks in HN posts?
wrren 10y ago
As an aside, you can set up a security group in AWS that blocks inbound traffic on port 80 if you'd like to neuter the incoming requests.
- perlgeek 10y ago
  Or, like, simply shut down the web server.
  - irth 10y ago
    He'd still have to pay. By neutering it he means that nobody with bad intensions should be able to use the domains. It is neutered now, but if he was allowed to change DNS entry to point to localhost, he wouldn't have to pay Amazon.
nowprovision 10y ago
Bye bye digitalocean - account deletion request submitted 1178917. When you have reckless people like Cashan Stine (trust & safety specialist - WTF is that title? sounds like a road safety officer?) that close accounts due to a security report then it won't win any business from me or my clients.
- deleted 10y ago
  [deleted]
- eropple 10y ago
  That's not why his account was closed. His account was closed not for discovering a vulnerability, but for exploiting it. While his intentions might have been good (and I expect that they were!), that kind of behavior isn't.
  - dx034 10y ago
    He did not exploit it, he just provided proof. He did not make any money from the traffic and visitors just saw a white page.
    - eropple 10y ago
      That is exploiting the bug. That is literally exploiting the bug. In the same paragraph where you say he did not exploit the bug, you describe the peripherals of his exploit of the bug. If he was operating responsibly, he would have applied it to a domain he controlled and provided that as a proof of concept. Instead, he ganked twenty thousand domains. That is at best irresponsible and at worst malicious and DigitalOcean (who I am no fan of, for what it's worth) has no obligation to figure out, or even care, which is which before showing him the door.
    - hatsix 10y ago
      Doing it once, proof. Doing it 20,000 times... Exploit.
- dumbsecreport 10y ago
DangerousPie 10y ago
Something similar happened to me a few months ago with Cloudflare. I set up a new domain to use Cloudflare's nameservers but did not immediately get around to setting it up on the admin panel. By the time I wanted to add the domain, someone else had already grabbed it and set up some sort of spam page. Took a few emails to Cloudflare support to resolve this one. They also didn't seem to care much about the security implications when I questioned them about it. So this is far from a DO-specific issue...
- xxdesmus 10y ago
  "They also didn't seem to care much about the security implications when I questioned them about it." no one ever said that was the case.
jbb555 10y ago
"I was walking down the street and I noticed your house wasn't locked very well. So I stole all your stuff and put it in my own house. Now I'm in prison because of this so it's really hard for me to put it back." The article writer is an idiot. He deliberately stole accounts because he could. Just because he then decided to blame the provider because he was able to do this does't make it any more defensible. If I mug you in the street, should I then post that because I was able to do so it's all your fault? No. I'd go to prison....
- TheCowboy 10y ago
  Comparisons of events like this to violent crime often seem inaccurate. A better comparison is removing 10 people's lunch money from their school lockers, maybe due to a careless sequential scheme of creating combinations, and giving the money back. And doing this before talking to the principal or any teacher. Either way, he still should have contacted them before.
  - T2_t2 10y ago
    A better analogy is "I found a dollar on the ground". I'm not sure that is really a crime!
- SixSigma 10y ago
  > Theft is defined as the physical removal of an object that is capable of being stolen without the consent of the owner and with the intention of depriving the owner of it permanently.
- riskable 10y ago
  This is a really bad comparison. He didn't deny anyone access to anything or take anything at all. The real-world equivalent would be if he found a bunch of empty community/neighborhood whiteboards and drew a (trivially erased) line on them. The signs weren't there for him to use but they were empty (he didn't erase them) and the change he made was absolutely minimal. He just configured a bunch of deleted/unconfigured domains to point them to a blank web page running on his own server. The point is that he could have done all sorts of nefarious things with that redirection but he didn't. He made a harmless change to demonstrate the vulnerability. That's what white hats do. It's what they're supposed to do. We should thank him for his efforts not lambaste him for actually doing something about the problem.
gengkev 10y ago
Did anybody else click CrashChrome.com (or equivalent) in the sidebar?
- gshakir 10y ago
  I saw it but due to security concerns I did not click.
- whatthesmack 10y ago
  I was wondering the same thing! I did, and it crashed. Currently running 53.0.2785.80 beta (64-bit) on OS X.
- zubi 10y ago
  I clicked on it even before reading the article. Not only the chrome crashed, but also my notebook! I had to reboot the computer.
- rasz_pl 10y ago
  Vivaldi doesnt crash, but it is a resource starvation dos ( several threads go 100% cpu).
20yrs_no_equity 10y ago
I made the mistake of applying for a job there once. I was discriminated against. Despite being in a protected class, I was so surprised to be so obviously discriminated against by them. (I've interviewed a lot, I don't always get a follow up, this isn't sour grapes, this was very different.) But of course the HR people are careful to not say things that are overtly discriminatory. But when a company insists on a VIDEO call rather than a phone call (despite asking them to do the first one by phone since I was not in a location with good bandwidth at the time they wanted the call)... and then visibly reacts to your image when they first see it, and then pretty much blows you off, despite being well qualified for the position... yeah, it's not what they say.
deleted 10y ago
[deleted]
adanto6840 10y ago
This same thing happens with CloudFlare & is being actively exploited. We reported it to them within the last two weeks and we were told that it's expected behaviour and that they weren't going to do anything about it. I asked them to, at the absolute least, send an email notification to the prior-CloudFlare owner letting them know that the domain "your CF account used to control is now being controlled by a new CF account". Better yet, implement a domain ownership validation scheme. They told us that they wouldn't be making any changes. FWIW, on CloudFlare what happened to us was: we were moving registrars for ~100 domains, from GoDaddy to Route53. During this transition, the NS for the domains temporarily became blank; at this point CF automatically removed the domains from our CF account. The NS were then re-added to the domains on the Route53 side (<4 hours of 'no nameserver' time). Apparently there are people out there that are looking for domains that are pointed to CF and then attempting to add them to their own CF account (automated I'm sure) -- which CF lets them do without any verification once they've been auto-removed from your [the original CF account] account. Interestingly, the original account must be still stored in their system with the domain because we were able to re-add the domain to our original CF account without any verification; effectively "stealing the domains back" to our CF account, away from the thieve's CF account. In this case, the "attackers" (perhaps more appropriate, I call them 'malicious actors') were able to commandeer ~100 of our domains for ~2 months, for free; they redirected them to Russian websites, torrent sites, affiliate sites, etc. Again, this is being actively exploited on CloudFlare, at the direct expense of CF customers -- but, according to CF, it's not an issue...?!
- yoo1I 10y ago
  > according to CF, it's not an issue...?! According to CloudFlare, they are are a reverse proxy, and they are not responsible for anything. This has been their response to every issue that I've tried to bring up with them over any channel, including here on HN. CloudFlare just doesn't care.
  - cortesoft 10y ago
    But I read on their blog that they are saving the internet, like, everyday and twice on Sundays!
perlgeek 10y ago
On the topic of having to pay for traffic to the sinkhole server: how about just closing ports 80 and 443? Then you only get a SYN, and answer with a NACK, that's far less traffic than processing a complete HTTP(s) request.
jarland 10y ago
Hey Matthew, I just wanted to let you know that I really appreciate your feedback, as well as the feedback from the other commenters here. I understand that many here are concerned that banning the account seems, from this perspective, to have been an unjustified action. I do believe that there is a bit of a misunderstanding on the timeline of events here, as well as the source of the decision. To be clear, Cash supported the decision that I had made to ban the account in question, and there had been no communication between us and Matthew at this point. We began receiving a significant number of support requests to remove domains from this account, and I authorized the shutting down of this account as it was clear to me what was happening. I have been working with our engineers to see to the removal of the domains from the account as well. I apologize if our actions seemed at any point rude or inappropriate, it was definitely not my intention. I want nothing more than to look out for the safety and wellbeing of our customers, and I chose what I believed to be the best action. I do want you to know that if I was aware that a security researcher had been working on testing a theory, I might have acted differently. That can, however, impact the reason behind a white hat test. You generally want the company to see you as normal user, so that you can see how they act in return. We do shut down users who are intentionally causing problems for other users, and I do think that was made evident here. I do understand that opening a line of communication with Matthew may have been appropriate, and I consider that valuable feedback moving forward. <3 Jarland
- mandatory 10y ago
  Hey Jarland thanks for the thoughtful response. I don't think your actions were rude or inappropriate (what host would see this behavior and not act similarly?). I apologize that the discussion on the topic became so negative towards DigitalOcean (this might be my fault, I was merely trying to point out why I hadn't been able to delete the domains - not say that DigitalOcean was a bad company/etc). The disclaimer I added early on I thought would mitigate some of this negativity but apparently not quite. At the very least I'm happy that some people have seen this more as a study of this pattern of functionality being an issue instead of this being only a DigitalOcean problem. I've seen (and have been reached out by) multiple people realizing this affects their company as well which has been awesome to watch.
nvigier 10y ago
Thank you all for the conversation around this! The security team at DigitalOcean has been working to ensure that DO is a safe place for security researchers to identify issues on the Internet as well as at DigitalOcean - security is in everyone's interest. We encourage researchers to contact us when they want to use our platform for this type of work specifically so that we can avoid the types of pain that Matthew encountered while doing his experimentation. Feel free to reach out to security@digitalocean.com and we will be happy to help. Nick, DigitalOcean Security Director
devrelm 10y ago
I actually suggested this was possible on security.stackexchange.com a while back and was basically met with "meh". http://security.stackexchange.com/questions/49612/how-does-digitalocean-dns-verify-the-owner-of-a-domain http://security.stackexchange.com/questions/49612/how-does-d...
devin481 10y ago
https://news.ycombinator.com/news https://news.ycombinator.com/news
linrocks 10y ago
Get ready for attack posts by the DigitalOcean/Vultr crowd.
rasz_pl 10y ago
Same thing happened to me after reporting SQL injection (in 2015!) on Vivaldi website. Polite email and blocked account. Some companies do seem to prefer to learn about vulnerabilities from pastebin database dump.
fmichlick 10y ago
I don't think that setting up custom DNS for everyone as suggested by the author is quite as simple as it sounds. It's not enough to just come up with the custom nameservers. In order to use them in most TLDs they also need to be "registered" with the registry that operates the TLD. So let's say you have myDNSdomain.com. You get a new customer who owns NewCustomer.com and wants to you your DNS, so you create these nameservers for them: ns237.myDNSdomain.com ns2323.myDNSdomain.com In order for your new customer to be able to use those on their NewCustomer.com domain, you will need to go to your registrar and set up these nameservers. The registrar will then create the corresponding nameserver records with Verisign, the registry. Only then, the customer will be able to use the nameservers on his domain.
- iancarroll 10y ago
  Well, yeah. That process already happens when you set up your domain to use DigitalOcean, but the nameserver used during the setup isn't unique.

385 points by infosecau 10y ago

I will never stop being infuriated by responses like this from companies - how many more megaleaks have to happen before they realize that they need to embrace white hats, not ban their accounts, not sue them, not swat them / have them arrested, not silence them. Great find / writeup.

IanCal 10y ago
Why, when the author realised this was likely to be possible, didn't they get in touch with DO? Or try with a domain they knew was OK to use? Or at least just try with one domain. They took almost twenty thousand, sent all the requests to their own server and logged them. That's surely not your first step.
- mandatory 10y ago
  See my response below and the logs were secure removed shortly after (as stated in the blog post).
  - xuki 10y ago
    Not that I don't believe you, but you already lost white hat status in this case the moment you log traffic on 20k domains. Just do an attack on another domain you own is white hat, 20k domains is not.
    - mandatory 10y ago
      You're probably right about the logging being a bit too far, it was mainly my curiosity getting the best of me. One of my big assumptions was that all of these domains were just owned by one domain broker and this wasn't actually a systemic problem with the implemented importation methodology. I also thought it would be mild because if they had been deleted from an account they were likely no longer used (or so I had wrongfully assumed).
      - IanCal 10y ago
        I certainly don't attribute malice on your part, and I'm sorry if my comment came across this way. My point was (at least intended to be) that I'm not really surprised at DO banning you. I would be careful about doing something like this on a large scale, I would actually be surprised if this doesn't technically violate some laws. Please be careful, you don't want to end up trying to explain nameservers to a judge. [edit - and the rest of us don't want that either, even just selfishly I would like as many security researchers practicing their craft as possible, but also I don't want people being punished for trying to do the right thing]
- arkadiyt 10y ago
  Domains can only be added if they are not in another account, so he added only domains which were previously deleted but still pointed to DO nameservers (hence almost certainly not being used). And if DO was really concerned about the damage then they would have removed the added A records, but they didn't, they banned the author and continued to let all traffic go to his server.
  - IanCal 10y ago
    > (hence almost certainly not being used) Well they were still being accessed by real people. > And if DO was really concerned about the damage then they would have removed the added A records, but they didn't, they banned the author and continued to let all traffic go to his server. Both seem reasonable. And it does sound like the security team are actually doing something about it. > I reached out to DigitalOcean’s team to see if they can assist in deleting the domains from my account (sadly leaving them vulnerable again) or sinkholing the DNS to 127.0.0.1. I received a very helpful response from someone on the security team and it appears they will look into it. I assume these things happened from different departments. Banning an account because you saw it make 20k requests to your API adding domains seems pretty reasonable, and it was a few hours before they reached out. If you saw that activity, would you ban the account or leave it open hoping that they'd be doing something nice and reach out?
    - wongarsu 10y ago
      > Banning an account because you saw it make 20k requests to your API adding domains seems pretty reasonable, and it was a few hours before they reached out. If you saw that activity, would you ban the account or leave it open hoping that they'd be doing something nice and reach out? If I was a domain reseller, adding my 20k domains to digital ocean just to get banned without warning, explaination or option for reconsideration I would be rightfully upset. If they didn't want people adding large numbers of domains they could just have limited the feature instead of banning people who reach some arbitrary threshold. If on the other hand the department that executed the ban knew that the registrations weren't made by the domain owners, they should be discussing such a huge incident with the security team. That discussion would naturally lead to them knowing about the specifics of this case, unless this case wasn't widely shared in the security team. So the options are: 1. Digital Ocean bans legitimate customers without warning or option for reconsideration; for no obvious reason 2. Big security incidents don't get reported to the security team 3. The responsible people thought that this was not a big security incident 4. This incident wasn't discussed in the security team 5. They knowingly banned a white hat hacker (who may or may not have gone too far) Of all those options, the last one is by far the one that looks best for Digital Ocean.
treehau5 10y ago
Because it can still cause real damage, even if the intentions were good.
empath75 10y ago
I wouldn't call redirecting 10s of thousands of accounts 'white-hat' hacking. He could have just done like, I dunno, 3, to prove his point.

diegorbaquero 10y ago

Great article! I'm saddened by DO's response and further wronging a white hat by banning you. Let's remember Linode offers 2x the RAM.

putlake 10y ago
Linode also does not verify domain ownership before you add a domain to their DNS. And they also use the same nameservers for all accounts. So I think they're in the same boat as DO as far as this "vulnerability" is concerned. I'm a long-time Linode customer and use them for all servers that run important services. I use DO for the less important stuff. But Linode has had a not-so-stellar record of data breaches.
optforfon 10y ago
No one remembers how Linode tried to cover up getting hacked?
- linrocks 10y ago
  I never noticed anything like that.
thegeekpirate 10y ago
Linode has had _far_ larger issues than DO ever has. That's why I switched to DO in the first place.
- nsgf 10y ago
  Same here. Please stay away from Linode. They used to be good. Now they are unprofessional and unethical.
  - akie 10y ago
    I'm a new Linode user (as of this week) and have been happy so far. What are these problems you talk about? I'd like to know before I commit further.

V8OaSsoA 10y ago

this post raises questions: Was there a realization into how legitimate users may be affected by this action? Was there a plan to remove those domains from their account after making and disclosing their proof of concept? Why not stop at 10 or 20, and then alert DO to the findings? 20 thousand was unnecessary.

mandatory 10y ago
Fair point, my relucatance to stop was mainly due to companies usually disreguarding reports unless I have strong proof. Stopping short of the full scope would've left it up to speculation as to the full amount of vulnerable domains. It was my plan to delete the domains (or at least null route them so others couldn't take them over with more malicious intent). However my account was banned before I could do so.
- deleted 10y ago
  [deleted]
- tombrossman 10y ago
  Have you had any contact with DO's support or security team prior to this? You are correct that companies frequently ignore or downplay reports but it's nice to at least make that first attempt for each new find (per company). This is only ever done as a courtesy, I'm not saying you must or even that you should have here. Remember, it's just another nerd out there somewhere who receives your report. Sometimes they are super stoked to have it and will be very responsive. This can be more satisfying than a public disclosure without first contacting the company.
- Aeolun 10y ago
  That turned out to indeed be the case after their first response :/ How would it have ended if you hadn't reached out to one of their security people after being banned? They'd have given you the traffic, locked your account, and congratulated themselves on a job well done?
jsmthrowaway 10y ago
How does making a domain that does not resolve suddenly resolve negatively impact a user, again? The domain could not have possibly been operational before the circumstances that brought about this scenario, and there is no legitimate traffic they could possibly be receiving. DigitalOcean could certainly improve authentication here but there are dozens of authoritative services that do not, and this is not a new problem (but is a valuable reminder). I can think of four free providers off the top of my head that this trick would work against. Put another way, the domains were entirely unresolved and offline. Then they weren't. If anything, this is a nice lesson about keeping your zone and delegations clean, and I'm glad I read it. Nobody got hacked, nobody lost traffic, nobody was impacted. If there was sensitive traffic going to a non-resolving domain, I have more questions than answers. I agree adding the full set was probably a bit much, but you can make that point without misplaced concern for alleged harm. I'm not impressed with signing up for HN to hit someone like this and your far worse and flagged followup, particularly since it really looks like astroturfing. I guess take solace that the OP pretty much guaranteed he won't get the zones again, since they're for research. (Nice ninja edit and deletion.)

anilgulecha 10y ago

TO: ANY DIGITAL-OCEAN USER, This is an absolutely terrible response from DO. If I had anything hosted here, I'd move away ASAP. Seriously, do it.

sattoshi 10y ago
I should leave the most reliable host I had to date because someone was messing around with things he shouldn't be? DO knows people can do this, but they don't want people to do it. Remembering 2 recurring DNS servers is easier for bulk management than a bunch of different ones. You don't test services like that, it can negatively affect other users. Response wasn't perfect but it was reasonable.
- timv 10y ago
  > Response wasn't perfect but it was reasonable I'm sorry, "We aware that we make it easy for about 20k domains to be directed to a malicious host, but we're not going to do anything about it" is reasonable? But of course, it's because Matt "was messing around with things he shouldn't be". It's all solved - we just need everyone to stop doing things that DO "don't want people to do".
  - MichaelGG 10y ago
    Meh the owners of the domains gave up control by pointing to someone else's nameservers.
    - Dylan16807 10y ago
      You've just condemned 99% of domains. You really think that's reasonable?
      - ZoF 10y ago
        I do think it's reasonable. If the domain is added to the account there is no PoC, it's only for domains that have been removed from accounts but still have the nameserver values(meaning the domain is not being used at this point, there's no zone file if it isn't added to an account). So this is mostly only going to affect currently derelict domains. I'm not saying it isn't something to worry about, but I do think it's a reasonable solution.

flexd 10y ago

This doesn't help my impression of Digital Ocean at all (even if I am a paying customer currently). A few years ago you could impersonate Digital Ocean staff on their support pages with no effort. They grabbed the username from your email, so whatever you put in front of the @ becamse your username on the forums, visible to everyone. And the avatar came from one of those email->avatar services where you can sign up and set it to anything. So when I signed up with a username like digitalocean@mydomain.com, I ended up being called "digitalocean" on the support forums, and if I had wanted I could just change the avatar to the Digital Ocean logo and impersonate DO or anyone else. I tried reporting it but got pretty much the same answer as this guy (though I did not get banned). Luckily they fixed it like a year later. Great write-up, and interesting problem! I wonder if more hosting providers are vulnerable to the same problem.

jsmthrowaway 10y ago
I can think of at least Cloudflare (somewhat), Linode, and Hurricane Electric off the top of my head. Anybody who operates a well-known ns1 type of resolver. It's more a problem with zone hygiene than hosts, honestly.
- deleted 10y ago
  [deleted]
- stestagg 10y ago
  This seems like a great candidate for an ACME style protocol. I would totally support the ACME spec being expanded to cover this, and various other domain verification related issues
  - corobo 10y ago
    Or just claim your domain if you point it at a DNS service..
- ilogik 10y ago
  Cloudflare does something similar to AWS. Each user gets different nameservers

ultramancool 10y ago

Wonder what else might be vulnerable to this... CloudFlare seems like it may, they only have a handful of nameservers in any case. Sort of hard to call it a vulnerability on DO's part though - more of an issue with the admins. I think most DNS services operate in this way, really, route53 may be the exception, not the rule.

i_have_to_speak 10y ago

TL;DR - If you own example.com and use DO as your nameserver, then anyone with a DO account can add DNS records for example.com.

ultramancool 10y ago
Not quite, only one person can, so if you add first, it's yours. Typically they have you add the domain and then you switch the DNS and in that configuration you wouldn't be vulnerable at all.
- MichaelGG 10y ago
  It sounds like this only impacts people that weren't using their domain, eh? If I leave my NS pointing to some random provider and delete my account, I'm sorta handing off control... At first I thought it was an issue where DO was using their own DNS servers but letting you add domains. So if Microsoft.com wasn't registered, you'd register it then all DO customer traffic would get your DNS records. Some telcos have this. Sign up, say you're porting a number. Provide number of local bank and invalid port info. Account gets set up, telco routes their own calls to you. Port is rejected and customer service is asking for info. All while you are getting the bank's calls and forwarding them. Perfect MITM.
  - samuellb 10y ago
    Yes, security of number porting is too lax. My number was once ported by a telco (accidentally in this case). They did call me and I had to confirm that I would get a new SIM card and which size I needed, but since I was expecting a new SIM card from the very same telco I didn't understand that they were in fact porting my number. They forgot to say that. Luckily I was able to get my number back, after many calls to customer support (they managed to undo the porting when I wanted to check how it was going) and a couple of weeks of waiting.

dotBen 10y ago

An additional vector for this kind of attack is to create a zonefile for a subdomain off of a working, live domain administered by the same DNS server. EG if foo.com is a working site on your DNS provider, try creating a zonefile for bar.foo.com and see if you can create an A record to point to your own server. This used to be something shared web hosting services running CPanel/WHM were particularly susceptible to. Clearly, the risks here are both phishing/identity and cookie credential stealing.

djhworld 10y ago

Very interesting read, thanks. I'm surprised at the response from Digital Ocean, did you adequately explain what you had done? The first person that replied looks like he just skim read your email or didn't understand the fact you had sinkholed a lot of traffic.

mandatory 10y ago
I believe I was clear about it. However sometimes my writing can be unclear so perhaps it wasn't properly understood (I assume you're talking about their security team's response and not Trust & Safety?). Kind of sad about the massive amount of hate for DigitalOcean in this thread as their security team really seemed quite nice. Their support was just acting on an anomaly they had seen so shrugs.
- djhworld 10y ago
  It was the guy who said "Thank you for sending this in. This is a known workflow within our platform. We are committed to always improving our customer’s experience and have been examining ways of minimizing the type of behavior you are describing." It sounds to me like he read your email about the vulnerability, dismissed it as "on the backlog" and skipped the bit about the fact you had sinkholed lots of traffic. We only have one side of the story though, so who knows. Lessons can be learnt.

djhworld 10y ago

tszming 10y ago

I think most of the providers (e.g. DO, Linode, CloudFlare etc) do not check the authority of DNS due to the chicken-and-egg problem. The AWS way to handle this issue is definitely awesome but the infrastructure required is not worth for those companies who are providing "free DNS service" as an add-on to their existing customers. Anyway, IMO, it is your fault if you point to a nameserver but not utilizing it.

jsmthrowaway 10y ago
The random nameservers are only accidentally a defense against this attack. They're avoiding SPOFs, including TLDs -- you never receive nameservers in the same TLD for example. It's a reliability and scaling consideration with this accidental benefit. Most admins don't think about a complete TLD failure. Amazon did.
- tszming 10y ago
  >> accidental benefit. Agree >> Most admins don't think about a complete TLD failure. Amazon did. I think companies such as Google or Facebook did think that before, but I am not sure why they didn't follow this trick.

chinathrow 10y ago

Am I reading this right: The only defence AWS has against this type of attack is the random (?) grouping of four different NS?

breakingcups 10y ago
It's effective.

mixedbit 10y ago

Amazon S3 has similar problems. To host static website you need use your domain name as the S3 bucket name. Amazon does not verify ownership of your domain, and bucket names use global namespace. Someone can easily block you from using S3 static website hosting by adding a bucket with your domain name before you do. Also if you delete a bucket and do not change your DNS, someone can recreate the bucket and will be serving files from your domain.

athrun 10y ago
As a best practice, use a 'normal' name for your S3 bucket and then put Cloudfront in front of your S3 website. This way you are not limited by bucket names and you also avoid any SSL validation errors. Note: you can set Cloudfront's TTL to 0 if you don't need any caching.
- mortar 10y ago
  This is certainly good practice as long as anyone contemplating this considers the cost implications versus serving from S3 directly.
  - athrun 10y ago
    Good point. It's actually generally cheaper to serve content through Cloudfront instead of S3 directly as bandwidth is less expensive with Cloudfront. However you are going to pay for Cloudfront requests in addition to S3's. In the vast majority of cases, it's insignificant compared to the savings you can achieve on the bandwidth side. (And at a certain level you can ask AWS to waive CF request fees entirely).
nulagrithom 10y ago
EDIT: Just tested it and looks like I'm wrong. Proxying with CloudFlare doesn't help either... Looks like I may have done this with CloudFront instead? That's not correct. The S3 bucket name is always prefixed. The format is: bucketname.region.amazonaws.com. https://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html https://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosti... To clarify, you're going to have to add a DNS record either way. Doesn't matter what you call your bucket. And no, you don't need to put CloudFlare in front of it for this.

supersan 10y ago

Banning his account was totally unjustified since he approached them first with the issue. A less ethical person could have tried to make money or sold this off on the back market. People like him should be rewarded not have their accounts banned. For all we know he just saved DO a lot of headache in sorting this issue had it gone wrong. I really wish the response from DO on this was different.

corobo 10y ago
Adding 20k domains to your account is probably enough to flag as abuse even if you own the domains. Next time the author should probably try just the one or two. Bonus points if they're their own domains.
- AgentME 10y ago
  >Bonus points if they're their own domains. If the service doesn't understand the issue at all, then when you explain that they're your domains, then they'll probably just tell you it's working as intended and that users should be able to add their own domains.
  - corobo 10y ago
    Arseuming makes an arse out of you and me. Give the techies on the other end of security@ the benefit of the doubt on the first go. When they fail to understand that then feel free to go ahead and pick one that's not yours to prove the point. One though, not 20 thousand.
mitoyarzun 10y ago
> The main reason I did not reach out with the theory instead of the proof-of-concept was because I believed that it would be ignored due to lack of evidence (as is my experience with past disclosures) I think this was his mistake.

ben_jones 10y ago

I find myself asking WW$D where $ is any large tech company with a "good" reputation. What would Google have done? Lyft? Spotify? Blizzard? Use some imagination to apply a similarly dangerous security breach to these companies. I feel like this question yields better context to ethical arguments because it makes us aware of the cognitive biases and view things from a more abstract perspective.. EDIT: Is there a way to include plain asterisks in HN posts?

wrren 10y ago

As an aside, you can set up a security group in AWS that blocks inbound traffic on port 80 if you'd like to neuter the incoming requests.

perlgeek 10y ago
Or, like, simply shut down the web server.
- irth 10y ago
  He'd still have to pay. By neutering it he means that nobody with bad intensions should be able to use the domains. It is neutered now, but if he was allowed to change DNS entry to point to localhost, he wouldn't have to pay Amazon.

nowprovision 10y ago

Bye bye digitalocean - account deletion request submitted 1178917. When you have reckless people like Cashan Stine (trust & safety specialist - WTF is that title? sounds like a road safety officer?) that close accounts due to a security report then it won't win any business from me or my clients.

deleted 10y ago
[deleted]
eropple 10y ago
That's not why his account was closed. His account was closed not for discovering a vulnerability, but for exploiting it. While his intentions might have been good (and I expect that they were!), that kind of behavior isn't.
- dx034 10y ago
  He did not exploit it, he just provided proof. He did not make any money from the traffic and visitors just saw a white page.
  - eropple 10y ago
    That is exploiting the bug. That is literally exploiting the bug. In the same paragraph where you say he did not exploit the bug, you describe the peripherals of his exploit of the bug. If he was operating responsibly, he would have applied it to a domain he controlled and provided that as a proof of concept. Instead, he ganked twenty thousand domains. That is at best irresponsible and at worst malicious and DigitalOcean (who I am no fan of, for what it's worth) has no obligation to figure out, or even care, which is which before showing him the door.
  - hatsix 10y ago
    Doing it once, proof. Doing it 20,000 times... Exploit.
dumbsecreport 10y ago

DangerousPie 10y ago

Something similar happened to me a few months ago with Cloudflare. I set up a new domain to use Cloudflare's nameservers but did not immediately get around to setting it up on the admin panel. By the time I wanted to add the domain, someone else had already grabbed it and set up some sort of spam page. Took a few emails to Cloudflare support to resolve this one. They also didn't seem to care much about the security implications when I questioned them about it. So this is far from a DO-specific issue...

xxdesmus 10y ago
"They also didn't seem to care much about the security implications when I questioned them about it." no one ever said that was the case.

jbb555 10y ago

"I was walking down the street and I noticed your house wasn't locked very well. So I stole all your stuff and put it in my own house. Now I'm in prison because of this so it's really hard for me to put it back." The article writer is an idiot. He deliberately stole accounts because he could. Just because he then decided to blame the provider because he was able to do this does't make it any more defensible. If I mug you in the street, should I then post that because I was able to do so it's all your fault? No. I'd go to prison....

TheCowboy 10y ago
Comparisons of events like this to violent crime often seem inaccurate. A better comparison is removing 10 people's lunch money from their school lockers, maybe due to a careless sequential scheme of creating combinations, and giving the money back. And doing this before talking to the principal or any teacher. Either way, he still should have contacted them before.
- T2_t2 10y ago
  A better analogy is "I found a dollar on the ground". I'm not sure that is really a crime!
SixSigma 10y ago
> Theft is defined as the physical removal of an object that is capable of being stolen without the consent of the owner and with the intention of depriving the owner of it permanently.
riskable 10y ago
This is a really bad comparison. He didn't deny anyone access to anything or take anything at all. The real-world equivalent would be if he found a bunch of empty community/neighborhood whiteboards and drew a (trivially erased) line on them. The signs weren't there for him to use but they were empty (he didn't erase them) and the change he made was absolutely minimal. He just configured a bunch of deleted/unconfigured domains to point them to a blank web page running on his own server. The point is that he could have done all sorts of nefarious things with that redirection but he didn't. He made a harmless change to demonstrate the vulnerability. That's what white hats do. It's what they're supposed to do. We should thank him for his efforts not lambaste him for actually doing something about the problem.

gengkev 10y ago

Did anybody else click CrashChrome.com (or equivalent) in the sidebar?

gshakir 10y ago
I saw it but due to security concerns I did not click.
whatthesmack 10y ago
I was wondering the same thing! I did, and it crashed. Currently running 53.0.2785.80 beta (64-bit) on OS X.
zubi 10y ago
I clicked on it even before reading the article. Not only the chrome crashed, but also my notebook! I had to reboot the computer.
rasz_pl 10y ago
Vivaldi doesnt crash, but it is a resource starvation dos ( several threads go 100% cpu).

20yrs_no_equity 10y ago

I made the mistake of applying for a job there once. I was discriminated against. Despite being in a protected class, I was so surprised to be so obviously discriminated against by them. (I've interviewed a lot, I don't always get a follow up, this isn't sour grapes, this was very different.) But of course the HR people are careful to not say things that are overtly discriminatory. But when a company insists on a VIDEO call rather than a phone call (despite asking them to do the first one by phone since I was not in a location with good bandwidth at the time they wanted the call)... and then visibly reacts to your image when they first see it, and then pretty much blows you off, despite being well qualified for the position... yeah, it's not what they say.

deleted 10y ago

[deleted]

adanto6840 10y ago

This same thing happens with CloudFlare & is being actively exploited. We reported it to them within the last two weeks and we were told that it's expected behaviour and that they weren't going to do anything about it. I asked them to, at the absolute least, send an email notification to the prior-CloudFlare owner letting them know that the domain "your CF account used to control is now being controlled by a new CF account". Better yet, implement a domain ownership validation scheme. They told us that they wouldn't be making any changes. FWIW, on CloudFlare what happened to us was: we were moving registrars for ~100 domains, from GoDaddy to Route53. During this transition, the NS for the domains temporarily became blank; at this point CF automatically removed the domains from our CF account. The NS were then re-added to the domains on the Route53 side (<4 hours of 'no nameserver' time). Apparently there are people out there that are looking for domains that are pointed to CF and then attempting to add them to their own CF account (automated I'm sure) -- which CF lets them do without any verification once they've been auto-removed from your [the original CF account] account. Interestingly, the original account must be still stored in their system with the domain because we were able to re-add the domain to our original CF account without any verification; effectively "stealing the domains back" to our CF account, away from the thieve's CF account. In this case, the "attackers" (perhaps more appropriate, I call them 'malicious actors') were able to commandeer ~100 of our domains for ~2 months, for free; they redirected them to Russian websites, torrent sites, affiliate sites, etc. Again, this is being actively exploited on CloudFlare, at the direct expense of CF customers -- but, according to CF, it's not an issue...?!

yoo1I 10y ago
> according to CF, it's not an issue...?! According to CloudFlare, they are are a reverse proxy, and they are not responsible for anything. This has been their response to every issue that I've tried to bring up with them over any channel, including here on HN. CloudFlare just doesn't care.
- cortesoft 10y ago
  But I read on their blog that they are saving the internet, like, everyday and twice on Sundays!

perlgeek 10y ago

On the topic of having to pay for traffic to the sinkhole server: how about just closing ports 80 and 443? Then you only get a SYN, and answer with a NACK, that's far less traffic than processing a complete HTTP(s) request.

jarland 10y ago

Hey Matthew, I just wanted to let you know that I really appreciate your feedback, as well as the feedback from the other commenters here. I understand that many here are concerned that banning the account seems, from this perspective, to have been an unjustified action. I do believe that there is a bit of a misunderstanding on the timeline of events here, as well as the source of the decision. To be clear, Cash supported the decision that I had made to ban the account in question, and there had been no communication between us and Matthew at this point. We began receiving a significant number of support requests to remove domains from this account, and I authorized the shutting down of this account as it was clear to me what was happening. I have been working with our engineers to see to the removal of the domains from the account as well. I apologize if our actions seemed at any point rude or inappropriate, it was definitely not my intention. I want nothing more than to look out for the safety and wellbeing of our customers, and I chose what I believed to be the best action. I do want you to know that if I was aware that a security researcher had been working on testing a theory, I might have acted differently. That can, however, impact the reason behind a white hat test. You generally want the company to see you as normal user, so that you can see how they act in return. We do shut down users who are intentionally causing problems for other users, and I do think that was made evident here. I do understand that opening a line of communication with Matthew may have been appropriate, and I consider that valuable feedback moving forward. <3 Jarland

mandatory 10y ago
Hey Jarland thanks for the thoughtful response. I don't think your actions were rude or inappropriate (what host would see this behavior and not act similarly?). I apologize that the discussion on the topic became so negative towards DigitalOcean (this might be my fault, I was merely trying to point out why I hadn't been able to delete the domains - not say that DigitalOcean was a bad company/etc). The disclaimer I added early on I thought would mitigate some of this negativity but apparently not quite. At the very least I'm happy that some people have seen this more as a study of this pattern of functionality being an issue instead of this being only a DigitalOcean problem. I've seen (and have been reached out by) multiple people realizing this affects their company as well which has been awesome to watch.

nvigier 10y ago

Thank you all for the conversation around this! The security team at DigitalOcean has been working to ensure that DO is a safe place for security researchers to identify issues on the Internet as well as at DigitalOcean - security is in everyone's interest. We encourage researchers to contact us when they want to use our platform for this type of work specifically so that we can avoid the types of pain that Matthew encountered while doing his experimentation. Feel free to reach out to security@digitalocean.com and we will be happy to help. Nick, DigitalOcean Security Director

devrelm 10y ago

I actually suggested this was possible on security.stackexchange.com a while back and was basically met with "meh". http://security.stackexchange.com/questions/49612/how-does-digitalocean-dns-verify-the-owner-of-a-domain http://security.stackexchange.com/questions/49612/how-does-d...

devin481 10y ago

https://news.ycombinator.com/news https://news.ycombinator.com/news

linrocks 10y ago

Get ready for attack posts by the DigitalOcean/Vultr crowd.

rasz_pl 10y ago

Same thing happened to me after reporting SQL injection (in 2015!) on Vivaldi website. Polite email and blocked account. Some companies do seem to prefer to learn about vulnerabilities from pastebin database dump.

fmichlick 10y ago

I don't think that setting up custom DNS for everyone as suggested by the author is quite as simple as it sounds. It's not enough to just come up with the custom nameservers. In order to use them in most TLDs they also need to be "registered" with the registry that operates the TLD. So let's say you have myDNSdomain.com. You get a new customer who owns NewCustomer.com and wants to you your DNS, so you create these nameservers for them: ns237.myDNSdomain.com ns2323.myDNSdomain.com In order for your new customer to be able to use those on their NewCustomer.com domain, you will need to go to your registrar and set up these nameservers. The registrar will then create the corresponding nameserver records with Verisign, the registry. Only then, the customer will be able to use the nameservers on his domain.

iancarroll 10y ago
Well, yeah. That process already happens when you set up your domain to use DigitalOcean, but the nameserver used during the setup isn't unique.