30 ms·

When security goes right

336 points by cperciva 12y ago

deleted 12y ago
[deleted]
jacquesm 12y ago
Merry Christmas Colin! Nice to see this worked out. I'm confused as to why you're surprised they had working whois contacts, most real businesses do, it's usually the scammers, the spammers and people that are up to no good that use whois privacy, rarely you see that used for someone who genuinely needs some protection. If a company uses whois privacy I don't do business with them as a rule.
- sirclueless 12y ago
  I assume this only applies to internet tech companies that should know better. If the non-technical guy outsourcing a company's IT got upsold on whois privacy, I'm not about to hold it against them.
  - jacquesm 12y ago
    Real businesses do no not hide. Whois privacy for any real business is a dumb decision, your customers need to know who you are, where you reside and how they can reach you even when (or especially when!) your website is down. I don't really understand the way ICANN currently rhymes their 'whois data should be accurate' policy with the 'we allow the use of anonymization services' exception on that policy.
    - meowface 12y ago
      There are rumors that they plan on disallowing WHOIS privacy in the future.
      - keithpeter 12y ago
        http://www.nominet.org.uk/uk-domain-names/about-domain-names/domain-lookup-whois/opt-out http://www.nominet.org.uk/uk-domain-names/about-domain-names... In the UK whois privacy is only for non-trading individuals (i.e. for personal Web sites) and always has been I think.
        talideon 12y ago
        They're not alone in that: for instance, .eu and .be domains with registrant contacts without an organisation name set have only very minimal details published in WHOIS. Then again, it's totally up to ccTLD operators to decide on what details are published in WHOIS. gTLDs (those longer then two letters long) have to stick to ICANN's policies.
      - talideon 12y ago
        Doubtful. I know people in the ICANN registrar constituency, and to the best of my knowledge there have been no discussions of this. Now, what is the case is that there's been a crackdown, due to pressure from law enforcement agencies, on incorrect details being set on domains. However, this actually has no effect on what's published in WHOIS because the only actual requirement is that the registrar have the correct details for the domain on record and escrowed. So long as the registrar has correct details, the registrar can mask the detail in WHOIS however is applicable, so long as it's clear that WHOIS privacy is in place.
    - cbd1984 12y ago
      > Real businesses do no not hide. There's hiding and then there's hiding. If the number leads to a phone center the business contracted with and there's no real way out of that to the people who make decisions, is that not a form of hiding?
    - moe 12y ago
      your customers need to know who you are, where you reside and how they can reach you even when (or especially when!) your website is down. Except the overwhelming majority of customers has no idea what 'whois' even is and couldn't care less what info yours may or may not contain... The only people who do care are spammers and scammers. Whois-databases are a goldmine for them, and that's why whois-anonymization is a Very Good Idea™.
    - larrys 12y ago
      100% Agree. As a registrar (and as someone who personally has been in the business since the mid 90's) we don't even offer it. But yet not a day goes by (ok an exaggeration) where we get a legitimate business that thinks they need privacy because it's going to somehow protect them from spam or getting their domain stolen or they've read that it's the right thing to do. The large registrars push this as a profit center and/or some kind of benefit to enhance their offerings. There are reasons to want privacy of course but not in the case of a business with a business address that most likely (say the local cake shop?) already puts their address on their website. As a business, why wouldn't you want your contact info to be public? It's another piece of marketing. Even more absurd to me is people who own domains who clearly want to sell them (let's say they are listed on SEDO or Afternic etc.) and they have privacy on their whois record. I mean get a PO box if you don't want your home address and you don't have a business address. If your domain is so valuable or if you own many we are talking $100 per year approx for an address. Use a google voice number for the phone number. Lastly, lack of public info on ownership (so no trail of ownership at whois history) makes it much harder to prove you own the domain if something happens at the registrar. You are depending on them to have all their records in order. If they get hacked, go out of business and so on you could have a problem proving ownership. (However small it's not worth the risk). ICANN does require registrars to archive whois data (with Iron Mountain) however since we don't offer privacy I'm not sure whether they require the underlying ownership (if you want to call it that) to be archived. Also many of the early privacy programs actually put ownership in the registrar's hands but had a separate contract with the actual registrant (not sure if that is needed anymore).
      - talideon 12y ago
        > 100% Agree. As a registrar (and as someone who personally has been in the business since the mid 90's) we don't even offer it. But yet not a day goes by (ok an exaggeration) where we get a legitimate business that thinks they need privacy because it's going to somehow protect them from spam or getting their domain stolen or they've read that it's the right thing to do. Personally, I don't think WHOIS privacy is something a business ought to be using, but it's perfectly legitimate for private individuals to use it. > The large registrars push this as a profit center and/or some kind of benefit to enhance their offerings. It is, because it's something the registrar can do at little or no cost. Given registrars work at thin margins, there are good reasons why even smaller registrars tend to offer WHOIS privacy. > Lastly, lack of public info on ownership (so no trail of ownership at whois history) makes it much harder to prove you own the domain if something happens at the registrar. You are depending on them to have all their records in order. If they get hacked, go out of business and so on you could have a problem proving ownership. (However small it's not worth the risk). That's precisely what data escrow through Iron Mountain is intended for, and why ICANN have been enforcing RDE since the failure of RegisterFly. The key thing to keep in mind is that what is shown in WHOIS isn't necessarily what needs to be escrowed. A registrant is meant to provide accurate data to the registrar, and it's this data that has to be escrowed, but this data doesn't have to be what's published in WHOIS. This is why the only way to legitimately do WHOIS privacy is through the registrar of record. Anybody who's dumb enough to use a third party is asking for trouble. If you're not escrowing the underlying ownership information (that is, the real registrant, admin, tech and billing contact details), then you're in breach of the RAA. > Also many of the early privacy programs actually put ownership in the registrar's hands but had a separate contract with the actual registrant (not sure if that is needed anymore). That can be solved by making it clear in the WHOIS details that the details displayed in WHOIS are for an agent acting the actual registrant. The registrar I work for also forwards any details sent to the email address published in WHOIS with WHOIS privacy on to the actual contact behind the scenes. Also, the registrar is required to escrow the real details.
- geofft 12y ago
  It's not uncommon for the whois contact to be a non-private but unhelpful number for the company. If you call any of the numbers in, say, `whois google.com` or `whois dropbox.com`, I'd expect that at best you'd reach front-line technical support, and more likely a phone tree or a voicemail box, not a live human who runs their infrastructure.
  - jacquesm 12y ago
    Google's tech whois number is a different number than the one for their main contact and a different one for the DNS admin one. Dropbox uses the same number for all three. Neither company uses whois privacy. Agreed that for plenty of companies you'd end up in an IVR system but that seems to go for every kind of company these days, not just internet companies.
- cperciva 12y ago
  As other people have said, I've encountered lots of startups using whois privacy; I think it's often a decision made in the early days and never revisited. This is one of the reasons I wrote about this -- to point out to people that having non-"private" whois actually serves an important purpose.
  - steakejjs 12y ago
    I think "Never unchecked the checked-by-default whois checkbox when purchasing the domain" is more accurate than actually saying it is a decision. While a company should definitely have a whois page, most startups make it abundantly clear how to get ahold of someone at the company with large "Contact Us" buttons. I've never contacted a startup to report a security issue, gotten a response (some don't respond), and had it not be from someone in a position to have the issue worked on immediately.
- Fogest 12y ago
  I use whois privacy on my domain because it is for my personal site and I'd rather random people not getting my home address, especially since sometimes people get angry with me and will try to find data about me.
  - cperciva 12y ago
    ... and that's fine, because it's a personal site and it's unlikely someone will need to contact you urgently to report a problem. The tradeoffs would be different if you were a company providing a service to a large number of customers.
noonespecial 12y ago
Granted that this is Canada so is a bit different from where I live, I personally would still at this point fear malicious prosecution just enough that I would not have taken action. That's a little sad all by itself. Hats off to Colin for the (brave) good deed.
- dmix 12y ago
  As a Canadian I wouldn't be too worried about reporting a security vuln here. If I was in America on the other-hand... I'd 100% think twice.
  - sarciszewski 12y ago
    As someone who was bitten by the CFAA before, you are 100% correct in being concerned. If you're going to test, do it anonymously. If you find anything, don't report it. Anonymity and good OpSec are the only things that can guarantee to keep American hackers out of prison.
- cperciva 12y ago
  I'm not a lawyer, but I'm confident that I could not be successfully prosecuted under Canadian law. The law in Canada states that access to computer systems must be "fraudulent and without colour of right" in order to be criminal. I'm not familiar enough with US law to know if a prosecution could be possible there, but regardless of the law might say I think juries can be influenced by the clear intentions of the accused.
  - sarciszewski 12y ago
    You are blessed to not fall under American jurisdiction.
  - jacques_chester 12y ago
    A few months ago, working on a system, I noticed that the URLs seemed to be based directly on a fixed, sequential user ID. It was a system that, in this case, was being used by An Important Company You Have Probably Heard Of. I was about to see what happened if I changed the number, when I realised that I am a foreign citizen in a country with notoriously unpleasantly drafted laws about computer security. And then I stopped. It's almost certain that nothing would go awry, but the mere act of gently poking might be interpreted in an unfavourable light. Losing a work visa is very easy and I very much don't want to.
click170 12y ago
Novus is great, they offer some of the highest speeds available in the areas that you can get access. My biggest gripe with Novus (besides limited coverage) is they have no unlimited bandwidth offering like my current provider has. Despite that, I'm still considering switching to them for those awesome upload speeds. In light of this, I'm probably going to sign up. Last I heard, when you went over your bandwidth limit with Novus they cut you off to prevent overages. I really liked this because you have the freedom to call them and confirm that you're fine with additional charges and they'll immediately reconnect you, but I'm curious about if they still do this. Any current customers able to clarify?
- cperciva 12y ago
  According to their website you'll get an email when you hit 80% utilization, and you can upgrade plans (or buy extra bandwidth for $5 / 20 GB) at any time. Also if you tell them in advance they can set your account to automatically buy more bandwidth rather than shutting off when you hit the limit. Personally I've used about 20 GB in the past week, so I'm not worried about hitting my 500 GB cap.
- FeeTinesAMady 12y ago
  I think Teksavvy, also available in Vancouver, has a feature whereby data used between 2 am and 8 am doesn't count towards your cap, which is downright reasonable and makes me wonder why none of the other providers are honest enough to offer that. I mean, data used in that time bracket is going to be far less than in the peak hours even with the people who will save their heavy bandwidth use for then, so usage then really doesn't cost them any money. Novus doesn't appear to offer that. So personally, I'd switch to Teksavvy instead, and I will if I ever have any problems with Telus (but so far I haven't).
  - mook 12y ago
    I don't think that feature (Zap the Cap) is available in Western Canada though (I might be wrong). I think that's Eastern Canada. (Happy TekSavvy customer intentionally paying more than Novus would cost because they were doing nice things politically, and the 25Mb plan is ~$5 difference. Might switch if I get a faster plan, though.)
protomyth 12y ago
"First, they were using widely used open source code; if I hadn't been familiar with it, I wouldn't have noticed the problem. Bad guys will always dig deeper than unpaid good guys; so if you're going to benefit from having many eyeballs looking at what you're doing, it's much better if your bugs are shallow." That is a really interesting reason to go with the popular open source solution. I guess I don't always follow that advice, but I wonder why I didn't think of it as a security decision.
- chris_wot 12y ago
  That could be seen as a disadvantage also. I don't subscribe to that world view, but it's a legitimate concern.
  - protomyth 12y ago
    I'm not sure I do either. It does make quite a lot of sense, but then you get the "how do new things get adopted" problem. I would probably weigh much more heavily to it if I was a manager at a large company.
patio11 12y ago
As mentioned, a great five minute project for when you get back to work after the holiday is adding a /security page to any website you control which handles user data. All it needs is a monitored inbox, a promise to get back to security researchers, and a PGP key. If you want to make it a 15 minute project, write a bit of customer-facing "We take your security seriously. That's why we encrypt all data with bank-grade security..." copy above or adjacent to the researcher-focused payload. Good examples (I picked their disclosure pages rather than the security marketing pages) include: https://basecamp.com/security/response https://basecamp.com/security/response https://www.twilio.com/docs/security/disclosure https://www.twilio.com/docs/security/disclosure
- sarciszewski 12y ago
  I wholeheartedly agree. I go as far as to put my PGP key on my contact page and to have a public policy regarding security vulnerabilities in all of my projects related to my website: https://scott.arciszewski.me/disclosure/ https://scott.arciszewski.me/disclosure/
  - akerl_ 12y ago
    I'll admit to some confusion regarding your disclosure page. I accept that full disclosure is an option and it's clear that you prefer it, but having read the page I'm left with the impression that your primary goal is that I publicize any vulnerability to other people, and the secondary goal is that I tell you. It seems that even if I'm practicing full disclosure, telling the author at the same time I tell the world is a 1st order goal.
    - resonantcore 12y ago
      > I'm left with the impression that your primary goal is that I publicize any vulnerability to other people, and the secondary goal is that I tell you. That is correct. If the world knows, then Scott will also be informed. > It seems that even if I'm practicing full disclosure, telling the author at the same time I tell the world is a 1st order goal. That is your choice, you are not obligated to do so with Scott. If he makes a mistake, he wants hackers to call attention to it. He wants people to see his mistakes and how he responds. Maybe follow his example: to accept mistakes graciously, and immediately issue a patch that adequately addresses them. If he gets burned in the process, that's the price Scott is willing to pay to improve. Or maybe he's just cocky and is bluffing everyone because he thinks he's too good of a programmer to make a security-affecting mistake. Only way to find out is to audit his open source code and drop 0days onto Full Disclosure ;)
chris_wot 12y ago
The first comment was: "I don't see why this was a security problem in the first place. No personally identifiable data was disclosed. What does it matter if you can view anonymous traffic graphs from other customers?" I hate it when people say this sort of thing. That just indicates they can't see a potential exploit, not that it won't be one potential aspect of an attack. Honestly, attackers - regardless of their morality - will tend to look at things from a viewpoint others haven't imagined. It's best to give them as few avenues as possible.
- marvy 12y ago
  True, but the fact remains that this does not sound dangerous.
  - andreyf 12y ago
    I'm pretty sure that "have access to someone's account number once" => "have access to past and future traffic usage forever" is not good security practice regardless if there are or are not obvious vulnerabilities, and you really don't have to think very hard to construct a scenario where it can be abused.
  - yourad_io 12y ago
    Security isn't about mitigating danger. You're leaking information that you aren't supposed to. There is no benefit or legitimate use case for this leak; it is a leak. You gotta plug it.
  - frio 12y ago
    You could send large amounts of data to an IP in sporadic bursts, find which graph correlates, get their account number and use that as a starting point for social engineering the helpdesk staff. Elaborate, sure, but plausible. The more important part of the leak is all the account numbers anyway :).
  - thefreeman 12y ago
    A good one mentioned in the OP comments is deducing when people are on vacation due to their traffic usage (or rather lack there of). Also, we don't know what other information may have been listed there. What if they decided to add your name and IP address to the graph?
wtbob 12y ago
Why not just store each user's information in a file whose name is based on HMAC(some secret, user's account number)? That should be secure against enumeration attacks.
- cperciva 12y ago
  That would also work (once they fixed the directory index issue). But their code was written in PHP, so I didn't think it was worth trying to explain that option to them.
  - sarciszewski 12y ago
    Yeah, because all PHP programmers are idiots right? :(
    - cperciva 12y ago
      Let's just say that there's a strong correlation. ;-) EDIT: To elaborate: I'm definitely not saying that all PHP developers are idiots. But PHP is a much easier language to learn than many others, so it gains a lot of users at the low end of the ability spectrum. There are just as many talented and security-aware Windows users as there are talented and security-aware Debian users, but there are far more untalented and security-unaware Windows users.
      - sarciszewski 12y ago
        --$respect['cperciva']; The implication in your word choice came across as unfair and elitist. > EDIT: To elaborate: I'm definitely not saying that all PHP developers are idiots. But PHP is a much easier language to learn than many others, so it gains a lot of users at the low end of the ability spectrum. There are just as many talented and security-aware Windows users as there are talented and security-aware Debian users, but there are far more untalented and security-unaware Windows users. Okay, that's a fair statement. I thought you were calling ME an idiot :P
danieltillett 12y ago
My only concern here is how quickly support was making changes to production code. Should changes like this be made to a functioning system within 10 minutes of getting a verbal bug report?
- cperciva 12y ago
  I'd agree with you if they were making changes to their product code, but they weren't. This was just a monitoring interface, quite separate from their actual product (which is internet access, not RRDtool graphs).
  - danieltillett 12y ago
    I was expressing concern about the process not the particular changes made. I am pretty gung-ho and the sole person who makes changes to any customer facing areas and even I would take longer than this to address anything that was functioning.
    - cperciva 12y ago
      The only change they needed was "Options -Indexes". Hardly likely to break anything.
- ddoolin 12y ago
  We have pushed severe/security issues to production nearly as fast to a major bank. Granted we had the attention of an entire team but this ISP seems to be much smaller and these tools probably see a lot less use. Out of curiosity, how long do you think it would take to where you wouldn't be concerned? It seems hard to me to be really concerned without knowing exactly what changes were made, how many people were involved, and what kind of process they have in place for pushing code out the door.
  - danieltillett 12y ago
    I agree if the issue is critical that you want to get the fix out as soon as you can. I also agree with you that we don't really know the full background here, but I am concerned when this is being held up as a good way for a company to behave. I can only speak for myself, but I never push any non-critical changes out on a customer facing application in less than a day. I like to think about what I have done overnight just to make sure I am not breaking something. Even doing this I have still managed to break things on occasion :)
xupijack 12y ago
really insteresting
chubot 12y ago
A short time later they started checking Referer headers; but as I pointed out to them, sending a fake Referer header is easy, so that only helps against the most careless attackers. I suggested several options for solving this, and they chose the simplest: Use a cron job to delete all the generated graphs every minute. If I'm understanding correctly, I think this problem can be elegantly solved with Macaroons: http://research.google.com/pubs/pub41892.html http://research.google.com/pubs/pub41892.html And it's relatively common issue with auth on the web. I think Facebook has (or had) this problem too. You can generally right click and "copy link" to get a .jpg URL, and send around people's pictures without any auth. Basically the problem is when there are two web servers, a "dynamic" one with auth, and a static one that serves images. The static one is often a CDN. Macaroons are basically a simple technique for decentralized auth, involving HMAC chaining. In this setting, the static server would first give the dynamic server a macaroon M authorizing ALL pictures. At serving time, the dynamic server authorizes the user for a particular request. That request will have an <img src="acct123.jpg"> link. The dynamic server will take the macaroon M, and add a CAVEAT that the file must be "acct123.jpg", yielding Macaroon M2. The client gets the restricted macaroon M2 with the HTML, and sends it back to the static server to retrieve the .jpg. The server can 1) verify that M2 is derived from the original M, and 2) read the caveat from the dynamic server, proving that the user was authorized for the image acct123.jpg (and only that image). The HMAC chain is constructed so that the client can't remove the caveat and get access to all pictures. Basically what happened is that the static server DELEGATED auth logic for its resources to the dynamic server. In figure 2 of the paper, the static server would be TS (target service), and the dynamic server is IS (intermediate service). The static server still needs extra code for Macaroons, which existing CDNs and static servers don't currently have. It would be cool to have an Nginx plugin that does this. But the key point is that it is preserving the original intention behind the static/dynamic split: performance. In less performance sensitive context, you would have a web server perform custom auth logic, and then just read() the static file from disk and serve it over HTTP. This is likely to be many times slower than say Nginx. With the Macaroons, you can authorize ONCE in the dynamic server, and then PROVE to the static server that the auth decision was made. So all the .jpg requests can be fast and only hit the static server. The HMAC calculations are just hashing so they are cheap. It is symmetric crypto, with the shared HMAC secret. The paper has some other use cases and is definitely worth a read. I'm thinking about using this technique for a project. I'm interested in opinions from crypto/security folks.
fubarred 12y ago
Don't have a screen capture, but several insightful comments on the blog disappeared. Wtf?
- cperciva 12y ago
  I haven't deleted anything, and looking at them now I can't think of any which are missing. Disqus glitch maybe?
  - fubarred 12y ago
    Probably right, sorry for jumping to conclusions too hastily (a common antipattern on the ol' internets). Certainly if someone deleted a parent comment, there's several design decisions for comment services on how to handle child threads. One would hope that if a parent comment were deleted, it would leave a placeholder like HN, but it's doubtful that it's only design branch taken.
fubarred 12y ago
Re: whois. There's a need for extremely private registrations that still have a feedback channel. A guy I know has such a registrar, but I doubt there's any means of contacting any site's operator for technical, legal or other matters. (The whois postal address always lists somewhere Europe, but there's absolutely no published details. So it's a registrar that's about as private as allowable.)
mnarayan01 12y ago
Use a cron job to delete all the generated graphs every minute. That doesn't strike me as all that great of a solution, since (at least naively) this is going to result in a small (but presumably non-trivial) number of 404s on initial load. I guess if it's just a stopgap until a better solution could be implemented then it's fine.
- cperciva 12y ago
  If you mean the race between "create graph" / "graph is requested by browser" and deletion, that can be mitigated by not deleting any files which have been modified recently.
lukastsai 12y ago
a mobile readable version: https://getscroll.com/r/obvqb https://getscroll.com/r/obvqb

336 points by cperciva 12y ago

deleted 12y ago

[deleted]

Merry Christmas Colin! Nice to see this worked out. I'm confused as to why you're surprised they had working whois contacts, most real businesses do, it's usually the scammers, the spammers and people that are up to no good that use whois privacy, rarely you see that used for someone who genuinely needs some protection. If a company uses whois privacy I don't do business with them as a rule.

sirclueless 12y ago
I assume this only applies to internet tech companies that should know better. If the non-technical guy outsourcing a company's IT got upsold on whois privacy, I'm not about to hold it against them.
- jacquesm 12y ago
  Real businesses do no not hide. Whois privacy for any real business is a dumb decision, your customers need to know who you are, where you reside and how they can reach you even when (or especially when!) your website is down. I don't really understand the way ICANN currently rhymes their 'whois data should be accurate' policy with the 'we allow the use of anonymization services' exception on that policy.
  - meowface 12y ago
    There are rumors that they plan on disallowing WHOIS privacy in the future.
    - keithpeter 12y ago
      http://www.nominet.org.uk/uk-domain-names/about-domain-names/domain-lookup-whois/opt-out http://www.nominet.org.uk/uk-domain-names/about-domain-names... In the UK whois privacy is only for non-trading individuals (i.e. for personal Web sites) and always has been I think.
      - talideon 12y ago
        They're not alone in that: for instance, .eu and .be domains with registrant contacts without an organisation name set have only very minimal details published in WHOIS. Then again, it's totally up to ccTLD operators to decide on what details are published in WHOIS. gTLDs (those longer then two letters long) have to stick to ICANN's policies.
    - talideon 12y ago
      Doubtful. I know people in the ICANN registrar constituency, and to the best of my knowledge there have been no discussions of this. Now, what is the case is that there's been a crackdown, due to pressure from law enforcement agencies, on incorrect details being set on domains. However, this actually has no effect on what's published in WHOIS because the only actual requirement is that the registrar have the correct details for the domain on record and escrowed. So long as the registrar has correct details, the registrar can mask the detail in WHOIS however is applicable, so long as it's clear that WHOIS privacy is in place.
  - cbd1984 12y ago
    > Real businesses do no not hide. There's hiding and then there's hiding. If the number leads to a phone center the business contracted with and there's no real way out of that to the people who make decisions, is that not a form of hiding?
  - moe 12y ago
    your customers need to know who you are, where you reside and how they can reach you even when (or especially when!) your website is down. Except the overwhelming majority of customers has no idea what 'whois' even is and couldn't care less what info yours may or may not contain... The only people who do care are spammers and scammers. Whois-databases are a goldmine for them, and that's why whois-anonymization is a Very Good Idea™.
  - larrys 12y ago
    100% Agree. As a registrar (and as someone who personally has been in the business since the mid 90's) we don't even offer it. But yet not a day goes by (ok an exaggeration) where we get a legitimate business that thinks they need privacy because it's going to somehow protect them from spam or getting their domain stolen or they've read that it's the right thing to do. The large registrars push this as a profit center and/or some kind of benefit to enhance their offerings. There are reasons to want privacy of course but not in the case of a business with a business address that most likely (say the local cake shop?) already puts their address on their website. As a business, why wouldn't you want your contact info to be public? It's another piece of marketing. Even more absurd to me is people who own domains who clearly want to sell them (let's say they are listed on SEDO or Afternic etc.) and they have privacy on their whois record. I mean get a PO box if you don't want your home address and you don't have a business address. If your domain is so valuable or if you own many we are talking $100 per year approx for an address. Use a google voice number for the phone number. Lastly, lack of public info on ownership (so no trail of ownership at whois history) makes it much harder to prove you own the domain if something happens at the registrar. You are depending on them to have all their records in order. If they get hacked, go out of business and so on you could have a problem proving ownership. (However small it's not worth the risk). ICANN does require registrars to archive whois data (with Iron Mountain) however since we don't offer privacy I'm not sure whether they require the underlying ownership (if you want to call it that) to be archived. Also many of the early privacy programs actually put ownership in the registrar's hands but had a separate contract with the actual registrant (not sure if that is needed anymore).
    - talideon 12y ago
      > 100% Agree. As a registrar (and as someone who personally has been in the business since the mid 90's) we don't even offer it. But yet not a day goes by (ok an exaggeration) where we get a legitimate business that thinks they need privacy because it's going to somehow protect them from spam or getting their domain stolen or they've read that it's the right thing to do. Personally, I don't think WHOIS privacy is something a business ought to be using, but it's perfectly legitimate for private individuals to use it. > The large registrars push this as a profit center and/or some kind of benefit to enhance their offerings. It is, because it's something the registrar can do at little or no cost. Given registrars work at thin margins, there are good reasons why even smaller registrars tend to offer WHOIS privacy. > Lastly, lack of public info on ownership (so no trail of ownership at whois history) makes it much harder to prove you own the domain if something happens at the registrar. You are depending on them to have all their records in order. If they get hacked, go out of business and so on you could have a problem proving ownership. (However small it's not worth the risk). That's precisely what data escrow through Iron Mountain is intended for, and why ICANN have been enforcing RDE since the failure of RegisterFly. The key thing to keep in mind is that what is shown in WHOIS isn't necessarily what needs to be escrowed. A registrant is meant to provide accurate data to the registrar, and it's this data that has to be escrowed, but this data doesn't have to be what's published in WHOIS. This is why the only way to legitimately do WHOIS privacy is through the registrar of record. Anybody who's dumb enough to use a third party is asking for trouble. If you're not escrowing the underlying ownership information (that is, the real registrant, admin, tech and billing contact details), then you're in breach of the RAA. > Also many of the early privacy programs actually put ownership in the registrar's hands but had a separate contract with the actual registrant (not sure if that is needed anymore). That can be solved by making it clear in the WHOIS details that the details displayed in WHOIS are for an agent acting the actual registrant. The registrar I work for also forwards any details sent to the email address published in WHOIS with WHOIS privacy on to the actual contact behind the scenes. Also, the registrar is required to escrow the real details.
geofft 12y ago
It's not uncommon for the whois contact to be a non-private but unhelpful number for the company. If you call any of the numbers in, say, `whois google.com` or `whois dropbox.com`, I'd expect that at best you'd reach front-line technical support, and more likely a phone tree or a voicemail box, not a live human who runs their infrastructure.
- jacquesm 12y ago
  Google's tech whois number is a different number than the one for their main contact and a different one for the DNS admin one. Dropbox uses the same number for all three. Neither company uses whois privacy. Agreed that for plenty of companies you'd end up in an IVR system but that seems to go for every kind of company these days, not just internet companies.
cperciva 12y ago
As other people have said, I've encountered lots of startups using whois privacy; I think it's often a decision made in the early days and never revisited. This is one of the reasons I wrote about this -- to point out to people that having non-"private" whois actually serves an important purpose.
- steakejjs 12y ago
  I think "Never unchecked the checked-by-default whois checkbox when purchasing the domain" is more accurate than actually saying it is a decision. While a company should definitely have a whois page, most startups make it abundantly clear how to get ahold of someone at the company with large "Contact Us" buttons. I've never contacted a startup to report a security issue, gotten a response (some don't respond), and had it not be from someone in a position to have the issue worked on immediately.
Fogest 12y ago
I use whois privacy on my domain because it is for my personal site and I'd rather random people not getting my home address, especially since sometimes people get angry with me and will try to find data about me.
- cperciva 12y ago
  ... and that's fine, because it's a personal site and it's unlikely someone will need to contact you urgently to report a problem. The tradeoffs would be different if you were a company providing a service to a large number of customers.

noonespecial 12y ago

Granted that this is Canada so is a bit different from where I live, I personally would still at this point fear malicious prosecution just enough that I would not have taken action. That's a little sad all by itself. Hats off to Colin for the (brave) good deed.

dmix 12y ago
As a Canadian I wouldn't be too worried about reporting a security vuln here. If I was in America on the other-hand... I'd 100% think twice.
- sarciszewski 12y ago
  As someone who was bitten by the CFAA before, you are 100% correct in being concerned. If you're going to test, do it anonymously. If you find anything, don't report it. Anonymity and good OpSec are the only things that can guarantee to keep American hackers out of prison.
cperciva 12y ago
I'm not a lawyer, but I'm confident that I could not be successfully prosecuted under Canadian law. The law in Canada states that access to computer systems must be "fraudulent and without colour of right" in order to be criminal. I'm not familiar enough with US law to know if a prosecution could be possible there, but regardless of the law might say I think juries can be influenced by the clear intentions of the accused.
- sarciszewski 12y ago
  You are blessed to not fall under American jurisdiction.
- jacques_chester 12y ago
  A few months ago, working on a system, I noticed that the URLs seemed to be based directly on a fixed, sequential user ID. It was a system that, in this case, was being used by An Important Company You Have Probably Heard Of. I was about to see what happened if I changed the number, when I realised that I am a foreign citizen in a country with notoriously unpleasantly drafted laws about computer security. And then I stopped. It's almost certain that nothing would go awry, but the mere act of gently poking might be interpreted in an unfavourable light. Losing a work visa is very easy and I very much don't want to.

click170 12y ago

Novus is great, they offer some of the highest speeds available in the areas that you can get access. My biggest gripe with Novus (besides limited coverage) is they have no unlimited bandwidth offering like my current provider has. Despite that, I'm still considering switching to them for those awesome upload speeds. In light of this, I'm probably going to sign up. Last I heard, when you went over your bandwidth limit with Novus they cut you off to prevent overages. I really liked this because you have the freedom to call them and confirm that you're fine with additional charges and they'll immediately reconnect you, but I'm curious about if they still do this. Any current customers able to clarify?

cperciva 12y ago
According to their website you'll get an email when you hit 80% utilization, and you can upgrade plans (or buy extra bandwidth for $5 / 20 GB) at any time. Also if you tell them in advance they can set your account to automatically buy more bandwidth rather than shutting off when you hit the limit. Personally I've used about 20 GB in the past week, so I'm not worried about hitting my 500 GB cap.
FeeTinesAMady 12y ago
I think Teksavvy, also available in Vancouver, has a feature whereby data used between 2 am and 8 am doesn't count towards your cap, which is downright reasonable and makes me wonder why none of the other providers are honest enough to offer that. I mean, data used in that time bracket is going to be far less than in the peak hours even with the people who will save their heavy bandwidth use for then, so usage then really doesn't cost them any money. Novus doesn't appear to offer that. So personally, I'd switch to Teksavvy instead, and I will if I ever have any problems with Telus (but so far I haven't).
- mook 12y ago
  I don't think that feature (Zap the Cap) is available in Western Canada though (I might be wrong). I think that's Eastern Canada. (Happy TekSavvy customer intentionally paying more than Novus would cost because they were doing nice things politically, and the 25Mb plan is ~$5 difference. Might switch if I get a faster plan, though.)

protomyth 12y ago

"First, they were using widely used open source code; if I hadn't been familiar with it, I wouldn't have noticed the problem. Bad guys will always dig deeper than unpaid good guys; so if you're going to benefit from having many eyeballs looking at what you're doing, it's much better if your bugs are shallow." That is a really interesting reason to go with the popular open source solution. I guess I don't always follow that advice, but I wonder why I didn't think of it as a security decision.

chris_wot 12y ago
That could be seen as a disadvantage also. I don't subscribe to that world view, but it's a legitimate concern.
- protomyth 12y ago
  I'm not sure I do either. It does make quite a lot of sense, but then you get the "how do new things get adopted" problem. I would probably weigh much more heavily to it if I was a manager at a large company.

patio11 12y ago

As mentioned, a great five minute project for when you get back to work after the holiday is adding a /security page to any website you control which handles user data. All it needs is a monitored inbox, a promise to get back to security researchers, and a PGP key. If you want to make it a 15 minute project, write a bit of customer-facing "We take your security seriously. That's why we encrypt all data with bank-grade security..." copy above or adjacent to the researcher-focused payload. Good examples (I picked their disclosure pages rather than the security marketing pages) include: https://basecamp.com/security/response https://basecamp.com/security/response https://www.twilio.com/docs/security/disclosure https://www.twilio.com/docs/security/disclosure

sarciszewski 12y ago
I wholeheartedly agree. I go as far as to put my PGP key on my contact page and to have a public policy regarding security vulnerabilities in all of my projects related to my website: https://scott.arciszewski.me/disclosure/ https://scott.arciszewski.me/disclosure/
- akerl_ 12y ago
  I'll admit to some confusion regarding your disclosure page. I accept that full disclosure is an option and it's clear that you prefer it, but having read the page I'm left with the impression that your primary goal is that I publicize any vulnerability to other people, and the secondary goal is that I tell you. It seems that even if I'm practicing full disclosure, telling the author at the same time I tell the world is a 1st order goal.
  - resonantcore 12y ago
    > I'm left with the impression that your primary goal is that I publicize any vulnerability to other people, and the secondary goal is that I tell you. That is correct. If the world knows, then Scott will also be informed. > It seems that even if I'm practicing full disclosure, telling the author at the same time I tell the world is a 1st order goal. That is your choice, you are not obligated to do so with Scott. If he makes a mistake, he wants hackers to call attention to it. He wants people to see his mistakes and how he responds. Maybe follow his example: to accept mistakes graciously, and immediately issue a patch that adequately addresses them. If he gets burned in the process, that's the price Scott is willing to pay to improve. Or maybe he's just cocky and is bluffing everyone because he thinks he's too good of a programmer to make a security-affecting mistake. Only way to find out is to audit his open source code and drop 0days onto Full Disclosure ;)

chris_wot 12y ago

The first comment was: "I don't see why this was a security problem in the first place. No personally identifiable data was disclosed. What does it matter if you can view anonymous traffic graphs from other customers?" I hate it when people say this sort of thing. That just indicates they can't see a potential exploit, not that it won't be one potential aspect of an attack. Honestly, attackers - regardless of their morality - will tend to look at things from a viewpoint others haven't imagined. It's best to give them as few avenues as possible.

marvy 12y ago
True, but the fact remains that this does not sound dangerous.
- andreyf 12y ago
  I'm pretty sure that "have access to someone's account number once" => "have access to past and future traffic usage forever" is not good security practice regardless if there are or are not obvious vulnerabilities, and you really don't have to think very hard to construct a scenario where it can be abused.
- yourad_io 12y ago
  Security isn't about mitigating danger. You're leaking information that you aren't supposed to. There is no benefit or legitimate use case for this leak; it is a leak. You gotta plug it.
- frio 12y ago
  You could send large amounts of data to an IP in sporadic bursts, find which graph correlates, get their account number and use that as a starting point for social engineering the helpdesk staff. Elaborate, sure, but plausible. The more important part of the leak is all the account numbers anyway :).
- thefreeman 12y ago
  A good one mentioned in the OP comments is deducing when people are on vacation due to their traffic usage (or rather lack there of). Also, we don't know what other information may have been listed there. What if they decided to add your name and IP address to the graph?

wtbob 12y ago

Why not just store each user's information in a file whose name is based on HMAC(some secret, user's account number)? That should be secure against enumeration attacks.

cperciva 12y ago
That would also work (once they fixed the directory index issue). But their code was written in PHP, so I didn't think it was worth trying to explain that option to them.
- sarciszewski 12y ago
  Yeah, because all PHP programmers are idiots right? :(
  - cperciva 12y ago
    Let's just say that there's a strong correlation. ;-) EDIT: To elaborate: I'm definitely not saying that all PHP developers are idiots. But PHP is a much easier language to learn than many others, so it gains a lot of users at the low end of the ability spectrum. There are just as many talented and security-aware Windows users as there are talented and security-aware Debian users, but there are far more untalented and security-unaware Windows users.
    - sarciszewski 12y ago
      --$respect['cperciva']; The implication in your word choice came across as unfair and elitist. > EDIT: To elaborate: I'm definitely not saying that all PHP developers are idiots. But PHP is a much easier language to learn than many others, so it gains a lot of users at the low end of the ability spectrum. There are just as many talented and security-aware Windows users as there are talented and security-aware Debian users, but there are far more untalented and security-unaware Windows users. Okay, that's a fair statement. I thought you were calling ME an idiot :P

danieltillett 12y ago

My only concern here is how quickly support was making changes to production code. Should changes like this be made to a functioning system within 10 minutes of getting a verbal bug report?

cperciva 12y ago
I'd agree with you if they were making changes to their product code, but they weren't. This was just a monitoring interface, quite separate from their actual product (which is internet access, not RRDtool graphs).
- danieltillett 12y ago
  I was expressing concern about the process not the particular changes made. I am pretty gung-ho and the sole person who makes changes to any customer facing areas and even I would take longer than this to address anything that was functioning.
  - cperciva 12y ago
    The only change they needed was "Options -Indexes". Hardly likely to break anything.
ddoolin 12y ago
We have pushed severe/security issues to production nearly as fast to a major bank. Granted we had the attention of an entire team but this ISP seems to be much smaller and these tools probably see a lot less use. Out of curiosity, how long do you think it would take to where you wouldn't be concerned? It seems hard to me to be really concerned without knowing exactly what changes were made, how many people were involved, and what kind of process they have in place for pushing code out the door.
- danieltillett 12y ago
  I agree if the issue is critical that you want to get the fix out as soon as you can. I also agree with you that we don't really know the full background here, but I am concerned when this is being held up as a good way for a company to behave. I can only speak for myself, but I never push any non-critical changes out on a customer facing application in less than a day. I like to think about what I have done overnight just to make sure I am not breaking something. Even doing this I have still managed to break things on occasion :)

xupijack 12y ago

really insteresting

chubot 12y ago

A short time later they started checking Referer headers; but as I pointed out to them, sending a fake Referer header is easy, so that only helps against the most careless attackers. I suggested several options for solving this, and they chose the simplest: Use a cron job to delete all the generated graphs every minute. If I'm understanding correctly, I think this problem can be elegantly solved with Macaroons: http://research.google.com/pubs/pub41892.html http://research.google.com/pubs/pub41892.html And it's relatively common issue with auth on the web. I think Facebook has (or had) this problem too. You can generally right click and "copy link" to get a .jpg URL, and send around people's pictures without any auth. Basically the problem is when there are two web servers, a "dynamic" one with auth, and a static one that serves images. The static one is often a CDN. Macaroons are basically a simple technique for decentralized auth, involving HMAC chaining. In this setting, the static server would first give the dynamic server a macaroon M authorizing ALL pictures. At serving time, the dynamic server authorizes the user for a particular request. That request will have an <img src="acct123.jpg"> link. The dynamic server will take the macaroon M, and add a CAVEAT that the file must be "acct123.jpg", yielding Macaroon M2. The client gets the restricted macaroon M2 with the HTML, and sends it back to the static server to retrieve the .jpg. The server can 1) verify that M2 is derived from the original M, and 2) read the caveat from the dynamic server, proving that the user was authorized for the image acct123.jpg (and only that image). The HMAC chain is constructed so that the client can't remove the caveat and get access to all pictures. Basically what happened is that the static server DELEGATED auth logic for its resources to the dynamic server. In figure 2 of the paper, the static server would be TS (target service), and the dynamic server is IS (intermediate service). The static server still needs extra code for Macaroons, which existing CDNs and static servers don't currently have. It would be cool to have an Nginx plugin that does this. But the key point is that it is preserving the original intention behind the static/dynamic split: performance. In less performance sensitive context, you would have a web server perform custom auth logic, and then just read() the static file from disk and serve it over HTTP. This is likely to be many times slower than say Nginx. With the Macaroons, you can authorize ONCE in the dynamic server, and then PROVE to the static server that the auth decision was made. So all the .jpg requests can be fast and only hit the static server. The HMAC calculations are just hashing so they are cheap. It is symmetric crypto, with the shared HMAC secret. The paper has some other use cases and is definitely worth a read. I'm thinking about using this technique for a project. I'm interested in opinions from crypto/security folks.

fubarred 12y ago

Don't have a screen capture, but several insightful comments on the blog disappeared. Wtf?

cperciva 12y ago
I haven't deleted anything, and looking at them now I can't think of any which are missing. Disqus glitch maybe?
- fubarred 12y ago
  Probably right, sorry for jumping to conclusions too hastily (a common antipattern on the ol' internets). Certainly if someone deleted a parent comment, there's several design decisions for comment services on how to handle child threads. One would hope that if a parent comment were deleted, it would leave a placeholder like HN, but it's doubtful that it's only design branch taken.

fubarred 12y ago

Re: whois. There's a need for extremely private registrations that still have a feedback channel. A guy I know has such a registrar, but I doubt there's any means of contacting any site's operator for technical, legal or other matters. (The whois postal address always lists somewhere Europe, but there's absolutely no published details. So it's a registrar that's about as private as allowable.)

mnarayan01 12y ago

Use a cron job to delete all the generated graphs every minute. That doesn't strike me as all that great of a solution, since (at least naively) this is going to result in a small (but presumably non-trivial) number of 404s on initial load. I guess if it's just a stopgap until a better solution could be implemented then it's fine.

cperciva 12y ago
If you mean the race between "create graph" / "graph is requested by browser" and deletion, that can be mitigated by not deleting any files which have been modified recently.

lukastsai 12y ago

a mobile readable version: https://getscroll.com/r/obvqb https://getscroll.com/r/obvqb