12 ms·

Netgear Signed TLS Cert Private Key Disclosure

198 points by Maxious 7y ago

regecks 7y ago
6 days is nowhere near a justifiable timeframe for full disclosure. Even if you disagree with that, you should have first reported Key Compromises to Entrust and Comodo before publicly posting the private keys. They are bound by BRs and their own CPS to revoke certificates such as this one - and they would have done so promptly. This is not what you should do as a security researcher - delete the gist until the CAs have a chance to revoke it via OCSP.
- mjevans 7y ago
  I think in this case it's to force browser vendors (who have the most exploitable endpoints) and companies like Apple and Microsoft at the OS level, to blacklist the offending certs. Though the CAs in question should probably have an automated challenge-response system to which a timed signed reply of a given message of their choice causes a revocation of the key that signed the message. (As sufficient proof of "this is vuln, kill it now, ask questions after".)
  - toast0 7y ago
    Did this researcher even contact the CAs? Generally, they're pretty responsive and will revoke quickly; there are CA/Browser forum guidelines that address this; if nothing else, you can send them the private key to prove possession.
  - buraksarica 7y ago
    The OS level actions won't be fast. So this disclosure is closer to an irresponsible attitude. At least CAs should be informed.
    - philprx 7y ago
      I tend to differ: Netgear security is paid to manage security. They failed by not responding to these legitimate communications requests. The researcher are not paid. They did what could be done to really fix the problem. Of course you can always do better as a researcher, always, but consider time available and paid vs. pro bono time. Also consider all the people who probably found this before and may have sold it on black market, you’re attacking the wrong people.
- RL_Quine 7y ago
  > 6 days is nowhere near a justifiable timeframe for full disclosure. Of course it is, if you didn't get a response in the first 48 hours you're not going to, that's the way things like this tend to operate. The security contact for all companies is either a) inactive or b) very active. It's a detriment to everybody that for some reason it has been normalized that people should wait months or years for disclosing bugs. Full disclosure is the only way things get fixed. > This is not what you should do as a security researcher - delete the gist until the CAs have a chance to revoke it via OCSP. Github is archived in real time for the most part. Especially you'll notice that if you post a private key for a cryptocurrency address or credentials for AWS, it'll be stolen and used within seconds. There's some really good sets of information out there like https://www.gharchive.org/ https://www.gharchive.org/ which give you an idea of the sheet amount of data that github produces on a daily basis, and that's just the metadata and things like comments rather than actual git repository contents. The idea that you could delete something from there and have it actually "removed from the internet" is amusing.
- gdm85 7y ago
  There is another reason why responsible disclosure would have been better (and less short-sighted): vendor might make an internal audit, find root cause and invalidate a bunch of certificates at once (think about similarly leaked certificates due to a bug in some deployment tool). Now it's an open race with the bad guys, and surely a lot of them all at once; hardly an advantageous scenario for end users.
  - JshWright 7y ago
    That's an awfully big (and awfully optimistic) "might" in your first paragraph there...
    - StavrosK 7y ago
      Not to mention that all the bad guys had to do to get the private key was unpack the firmware image, so they've probably noticed long ago.
      - gdm85 7y ago
        If what you said were true then zero-days would be always ignored and not a concern.
        StavrosK 7y ago
        How does that follow from what I said?
    - gdm85 7y ago
      It's not optimistic, it's what a responsible vendor is supposed to do. By not following responsible disclosure the researcher has lifted the vendor from the possibility of organizing a proper response.
      - gdm85 7y ago
        By the way, I meant coordinated disclosure and I am not aware of Netgear not doing any at all, never.
- kick 7y ago
  Responsible disclosure is meant as a way to harm researchers and users for the benefit of companies. This is absolutely the right thing to do as a security researcher, especially given that Netgear prevents public disclosure, ever, if you submit a bug bounty. To make sure it stays online: https://web.archive.org/web/*/https://gist.github.com/nstarke/a611a19aab433555e91c656fe1f030a9 https://web.archive.org/web/*/https://gist.github.com/nstark...
- yardstick 7y ago
  “ Thursday, January 16 - Bugcrowd responds, but we are unable to establish a communications channel outside of the Netgear bug bounty programs.” I’m curious to what does this mean? They got a response form Bugcrowd but not Netgear themselves? Or they got through to Netgear but didn’t like what Netgear said so they went public? Something else?
  - ukd1 7y ago
    Guessing they meant they were/are unwilling to.
- beardog 7y ago
  I discovered this same thing in 2017. My bugcrowd report was shot down as dupe https://www.chaoswebs.net/blog/tls-key-reuse-on-popular-router-models.html https://www.chaoswebs.net/blog/tls-key-reuse-on-popular-rout...
- Thorrez 7y ago
  Full disclosure means 0 days generally.
- tinus_hn 7y ago
  Boo hoo. Netgear should have known better than to expose their keys like this in the first place, this is really amateur hour stuff. They were reached out to and didn’t respond. It’s their own fault and they have to make do with what they’ve been granted. Now they get to pick up the pieces.
  - devwastaken 7y ago
    Yep. Defending bad security products only leads to people buying more. Netgear has not and will not learn any lesson, and will continue to serve bad devices. This time the good to users comes from publicizing that.
- ryanlol 7y ago
  Full disclosure is always perfectly justifiable, unless you’re stealing someone else’s work.
- lima 7y ago
  It wasn't the researcher who disclosed the keys, it was the vendor who did, knowingly and purposefully (in their firmware updates).
- gsich 7y ago
  Who cares, this does not even warrant a responsible disclosure. Netgear can't fix that in a way that makes everyone happy. You can revoke the cert, then what? A firmware update with a new cert? Same problem, private key will still be on the device and in the firmware file. Self signed? You get users crying about the cert. HTTP only? You get users crying about "not secure" even though that might not be true in all cases. My guess is that people before the ones in the gist have tried to contact Netgear about this (since it's not hard to find) and chose to ignore it.
fulafel 7y ago
Can anyone make out what the funjsq.com is about?
- dylz 7y ago
  Chinese gaming VPN service or something, bypasses China IP blocks to allow them to play on NA/EU servers.
  - bflesch 7y ago
    I'm wondering why such a VPN service is included in the netgear firmware image? Wouldn't this resonate negatively with Chinese authorities?
    - lenova 7y ago
      Unless it was mandated/deployed by a state actor, looking to MITM/monitor VPN users within the country?
mschuster91 7y ago
The HTTPS cert is used for the router's login page - apparently putting an IP address on the box backside label confuses too many people. It makes sense to put it behind HTTPS because the browser will whine "this page is insecure"... but how is a router vendor supposed to include the neccessary certificate that won't get leaked? The only thing I can imagine here is a dedicated HSM chip... but that's overkill for a 10$ router?
- Erlich_Bachman 7y ago
  "How" is that they only inlude the private key for that specific router, and then that key/cert itself is certified by the real CA cert, the key for which is not kept on the router itself?
  - zyztem 7y ago
    That requires generating and burning keys to devices during manufacturing. This process will be hacked by NSA & all, exactly like Gemalto https://www.theguardian.com/us-news/2015/feb/19/nsa-gchq-sim-card-billions-cellphones-hacking https://www.theguardian.com/us-news/2015/feb/19/nsa-gchq-sim...
    - Erlich_Bachman 7y ago
      Haha that's a tall order... Constructing a scheme where NSA is an active agent in the threat model was not an original requirement :) You are welcome to introduce any way to produce any part of a router or a PC for that matter that would protect from NSA, it seems that the biggest players in the field are still working out and it is very much a work in progress. When you have an adversary that is able to intercept hardware in transit and spend endless amounts of dollars on devising clever hacks or undetectable hardware exploits, then yes, you're right, some TLS scheme, regardless of where the certs are, is not going to be enough.
  - rvnx 7y ago
    Delegated Credentials for TLS I guess ? but this requires internet access so barely a solution. https://engineering.fb.com/security/delegated-credentials/ https://engineering.fb.com/security/delegated-credentials/ I don't think that the fact that private keys for routerlogin.net are bundled with the router are an issue. It's very logical to do so, and BETTER than plain HTTP in most real-life scenarios. routerlogin.net is a local server (when you are using the router) and not a remote website so it's expected that you can trust the content as much as trustable your local network is.
zyztem 7y ago
This is a world after QUANTUMINSERT for you. Somehow everything should be encrypted but key management are still unsolved
beardog 7y ago
I discovered this same thing in 2017. My bugcrowd report was shot down as dupe https://www.chaoswebs.net/blog/tls-key-reuse-on-popular-router-models.html https://www.chaoswebs.net/blog/tls-key-reuse-on-popular-rout...
- ploxiln 7y ago
  The difference is that these newly discovered certs with private keys were signed by real Certificate Authorities trusted by browsers by default.
  - tinus_hn 7y ago
    So their security got worse. It’s a miracle people even bother attempting to do responsible disclosure to fools like this.
    - josteink 7y ago
      > so their security got worse Only because current browsers makes it increasingly hard to run medium security, user-trusted networking.
    - lima 7y ago
      How did security get worse?
politelemon 7y ago
Tuesday, January 14th 2020 - Initial Discovery Tuesday, January 14 2020 - Tweet sent attempting to establish communications with Netgear Wednesday, January 15 2020 - Reached out to Bugcrowd to attempt to establish communications. Thursday, January 16 - Bugcrowd responds, but we are unable to establish a communications channel outside of the Netgear bug bounty programs. Friday, Jaunary 17th - Conversation with bugcrowd proves inconclusive Sunday, January 19th - Feeling we have exhausted our disclosure avenues, we decide to publish I didn't understand, can someone explain this to me - I see the author tried to talk to Netgear, then it's BUgcrowd the next day... it's not clear to me what the relation between Netgear and Bugcrowd is.
- Maxious 7y ago
  Bugcrowd run bug bounty programs like https://bugcrowd.com/netgear https://bugcrowd.com/netgear on behalf of companies They explained earlier that if they entered the bug bounty program with Netgear, they would be agreeing to not disclose: "By submitting the security bug, you affirm that you have not disclosed and agree that you will not disclose the security bug to anyone other than NETGEAR." So it seems they were trying to get Bugcrowd as a intermediary to contact Netgear on their behalf to "establish a communications channel outside of the Netgear bug bounty programs"
  - politelemon 7y ago
    Thanks for the explanation!
bottled_poe 7y ago
Netgear has reliably demonstrated to me over more than 10 years that they are incapable of delivering secure products of an acceptable quality. I actively avoid this brand like the plague.
- shock 7y ago
  I have posted this in another thread, but I am posting it again because more people need to be aware of Netgear's practices: I recently bought a couple of Netgear Managed Switches (for Business)⁰ and in their datasheet they list "Local-only management" as a feature. Only after they arrived we discovered that you only get limited functionality in the Local-only management mode, you have to register the switches to your Netgear Cloud account to get access to the full functionality. Reading up on it, this was achieved only after a community outcry because in the prior firmware versions the switch would have to connect to the Netgear Cloud on every bootup. Needless to say I would not have bought the swiches if I had knew I needed to register them to Netgear Cloud to have access to the full functionality specified in the data sheet. If I had bought them as a consumer, not as a business, I would have returned them immediately. Netgear are now on our purchasing blacklist. ⓪ - the switches are Netgear GS-108Tv3
  - mdszy 7y ago
    >Netgear Managed Switches (for Business)⁰ >⓪ - the switches are Netgear GS-108Tv3 are we not allowed to use any sane method of doing footnotes? Who does this? Why would you even use footnotes in internet comments other than to specifically screw over anyone who uses a screen reader?
    - bmn__ 7y ago
      > are we not allowed to use any sane method of doing footnotes? There is no markup on HN for footnotes. https://news.ycombinator.com/formatdoc https://news.ycombinator.com/formatdoc > Who does this? Blame the people running the site, this is just a small inconvenience compared with other lack of features. > screw over anyone who uses a screen reader? As GP's footnote is plain text – not markup – there is no problem with screen readers (or any user-agent, really). Curious why you would think otherwise.
deleted 7y ago
[deleted]
CiPHPerCoder 7y ago
Some commenters are decrying that this post fails to meet the bar for "responsible disclosure". Please stop using that phrase. "Responsible disclosure". It's wrong and harmful, and the person who coined it agrees with me: https://adamcaudill.com/2015/11/19/responsible-disclosure-is-wrong/ https://adamcaudill.com/2015/11/19/responsible-disclosure-is... You want "coordinated disclosure" instead. Netgear doesn't do coordinated disclosure. They do non-disclosure. In the absence of a coordinated effort, full disclosure is the responsible thing to do.
- ryanlol 7y ago
  It should be noted that it’s essentially never the people who find bugs arguing for “responsible disclosure”. It sure is easy to tell others what to do with their work product when you have zero stake in the game. There exists a really easy solution to the purported problem of full disclosure, vendors could just offer significant enough financial compensation for non-disclosure.
  - RL_Quine 7y ago
    That isn't what disclosing these sort of vulnerabilities is about. The core reason for just going full disclosure is that the vendor has absolutely not incentive to fix any sort of bug that is kept private. The customer is impacted, never the vendor.
    - ryanlol 7y ago
      Nah, disclosing these vulnerabilities is usually about CV-padding and publicity . There aren’t many people out there hunting bugs just to be nice.
      - cipherboy 7y ago
        I happen to know one of the authors of this post (hey Tom!). He's actually a really nice, down to earth guy. Helped out with our college cyber defense programs and is a killer red teamer. Very patient in explaining how he got in and defaced your website, time and time again. Is this his best work? Nah, this is amateur hour on the part of Netgear. But am I glad it was him who found it? Definitely. Keep in mind, there really wasn't anything _to_ this vulnerability other than copying and pasting from one website (and a firmware zip) to another. They just added a little pretty formating and saved you the trouble of extracting the firmware.
josteink 7y ago
I guess I run a different firmware for my Netgear ReadyNAS 312. This is easy to inspect because the FW is Debian-based, and enabling inbound SSH is easy. Checking on mine, it has a cert only for the "nas.local"-domain, and the cert is stored in /etc/ssl/certs/ssl-cert-snakeoil.pem and /etc/ssl/private/ssl-cert-snakeoil.key respectively. Have to admit I love the naming of those files :) Edit: Obviously it is a different firmware. Mine is a NAS, this was a router. And “snakeoil” seems to be default debianism.
josteink 7y ago
To all the people shitting on Netgear and security in this thread, just how do you propose one deliver a secure network appliance to end customers which they can deploy on their network? And which is user-accessible to common users in modern browsers rejecting everything not touched by a proper CA? Really. Please educate the world with your ingenious insight. I’ll be waiting. The unavoidable truth is: You have to stuff the key in there somehow. There’s no way around that. Either browsers have to start accepting “local” certs more easily, or end-user network deployed appliances can’t have ssl. Take a pick.
- zzzcpan 7y ago
  On local networks TLS doesn't make things more secure, maybe even less in some scenarios given all the obtaining of certificates over public internet and leaking info to the public internet. So sane thing to do would be plain http. But really, browsers should stop the bullshit with https "security" and accept ssh-like behavior at least for local networks.
- pjc50 7y ago
  Define "secure". If you've "stuffed the key in there" in such a way that other people can get it out again, and it works across all routers of that model, then it's possible for an active attacker on your network to MITM your connection to your router. So it's slightly more secure than cleartext but not much; about the same as a random self-signed certificate. It is definitely a hard problem because there's no easy way to authenticate the router to the client, but on the other hand I'm not sure how important all of this is when it's on a local link anyway. A "best possible practice" solution would be to have the routers issued with individual certificates at factory programming time, and provide a rollover mechanism through the admin UI.
  - vbezhenar 7y ago
    AFAIK CA can't issue certificate for more than two years. So if you've bought the device two years after it was manufactured, you're getting error and it's even worse than HTTP for user.
zonidjan 7y ago
This is intentional on Netgear's part, and does not in any way degrade security compared to the alternative (an untrusted cert or no HTTPS). It is neither a bug nor a security issue.
gene91 7y ago
Is this cert revoked yet? Sure, Entrust is required to revoke it. It is bound by BRs. If it refuses, Entrust will get itself blacklisted by browsers. On the other hand, having the cert revoked will cost Netgear dearly. As a result, I wonder if Entrust might be dragging its feet on the revocation.
- isclever 7y ago
  mini-app.funjsq.com is revoked (https://decoder.link/result/418b8d20793d3f4daa4153752e45e78b87e8e293 https://decoder.link/result/418b8d20793d3f4daa4153752e45e78b...), can't check routerlogin.net/com as they didn't paste the public cert.
  - tgsovlerkhgsel 7y ago
    routerlogin is not revoked: https://crt.sh/?id=1955992027&opt=ocsp https://crt.sh/?id=1955992027&opt=ocsp
    - deleted 7y ago
      [deleted]
rodnim 7y ago
I don't envy Netgear. They sell home routers to people who hardly know the difference between HTTP and HTTPS. In this regard, I respect them for NOT making those people submit sensitive information by ignoring browser security warnings and/or accepting self signed certificates. I.e, not teaching bad habits. On the other hand, making private keys publicly available is obviously far from ideal. Damned if you do, damned if you don't. Again, I don't envy Netgear...
gene91 7y ago
Netgear picked a terrible and clearly unacceptable approach. I I read through this discussion, trying to understand if there exists a good solution. Using plain http isn't a good solution because it involves telling the user to ignore the "not secure" warnings by browsers. This seems like the best available solution. Without network connectivity, the traffic can not be tunneled through a remote server with proper cert. Nor can the box request a cert in realtime at the time of setup. Pre-provisioned cert won't work if the box is purchased more than 27 months after manufacturing (the current max validity period). For home network equipment, I suppose this isn't unusual. I see TLS-SRP mentioned in comments. But the commenter hinted at its limited adoption in browsers.
- jniedrauer 7y ago
  When I can control the client configuration, I just set up my own trusted root CA and then have the distributed systems automatically submit a CSR on initialization. This doesn't work when you have to support end user's web browsers though (unless you're somehow lucky enough to own a public CA and don't care about ICANN rules). It really does seem like there should be an encryption-only option for TLS. Maybe anything that resolves by IP address and not name should display the green padlock even if the SAN doesn't match the name?
  - gene91 7y ago
    Encryption-only TLS doesn't protect MITM. It prevents sniffing. At a higher level, TLS provides integrity and privacy when the network linked is not trustworthy. As a result, I'm not sure what your proposal achieves. Similarly, resolution by IP address does not provide privacy or integrity unless you trust the network link.
    - jniedrauer 7y ago
      The green padlock indicates two things about the server to the user: "This is really my name" and "no one can snoop on our conversation." It doesn't indicate anything about the trustworthiness of the server. The first one is meaningless in the context of an IP address. If a user is going to get pished by an IP address, then they could just as well get phished by 123.sketchydomain.com. The important part when communicating directly with a device by IP address, is that the payload is encrypted end to end. > Similarly, resolution by IP address does not provide privacy or integrity unless you trust the network link. That's a fair point. It's certainly no worse than our current situation where you have to trust a self signed certificate on a case-by-case basis. The router could forge a self signed certificate even more easily than it could forge a certificate signed by a trusted CA.
tgsovlerkhgsel 7y ago
The cert for www.routerlogin.net (Serial c1:a1:00:64:07:61:2c:07:00:00:00:00:50:f1:09:6a) isn't revoked yet: https://crt.sh/?id=1955992027&opt=ocsp https://crt.sh/?id=1955992027&opt=ocsp The reporters should have asked the CAs to revoke the certificate. If the CA doesn't do that within 24 hours, it's considered a violation of the rules that the CA needs to follow to stay in browsers.
- dlgeek 7y ago
  Anyone with access to the private key can submit a problem report to the CA, they're obligated to revoke if the key is exposed to a non-subscriber. I have submitted a report - they are obligated to revoke within 24 hours.
  - cjbprime 7y ago
    Thanks for doing that.
  - cjbprime 7y ago
    Looks like they've failed to perform the revocation in time -- other users reported to Entrust >24h ago.
    - dlgeek 7y ago
      They revoked it 7 minutes shy of the 24 hour deadline after my report. If other users reported it and they failed to act, that's a BR violation and they're required to provide an incident reports to the root programs via Mozilla's mechanisms. If they don't, and you have evidence of those reports, you should provide them to the root programs via the mozilla.dev.security.policy mailing list/group. There's already a thread about this issue, though not claiming that EnTrust was notified. (Based on timestamps, it's also possible they revoked in response to that thread).
      - cjbprime 7y ago
        Thanks. Here is the timestamp showing that it took more than 32 hours from reporting to revocation. https://twitter.com/FiloSottile/status/1219147543667453953?s=19 https://twitter.com/FiloSottile/status/1219147543667453953?s...
privateSFacct 7y ago
What’s a good netgear alternative for switches from 5 port to 48 port - too late for us now but for next buying wave
bilekas 7y ago
Name and Shame. Responsible disclosure ? How about reposonsible security pracitces first.
actionowl 7y ago
HTTP(S) might just be the wrong protocol for initial setup of a router! Maybe instead generate a password for each router and print it on the box along with the device's unique SSH Key fingerprint. Technical users can then SSH in and bootstrap the system (also generate/upload their own TLS certs if they want to use a browser and connect over HTTPS, etc). Non-technical users get an app and they can scan a QR code on the box which has the SSH user, password, and fingerprint for verification. The App connects to the router over SSH to do router setup. If an App is involved you wouldn't technically NEED to use SSH with the App but it was the first thing that came to mind.
- fulafel 7y ago
  I think https could work fine, just give each device a unique key and dns name.
  - actionowl 7y ago
    That approach has been mentioned in the comments. Some problems with it are: If a certificate is valid beyond 825 days Chrome will not honor it. So if you generate a key and cert at the factory for every device you run into the issue of the end-user getting a device with an expired cert. If you generate a unique key and self-signed cert users will complain and call support about the "insecure" site warning. Communicating upstream to a third party to get a cert won't work for all users (e.g. those that need to configure a static IP from their ISP.)
    - fulafel 7y ago
      You shouldn't plug something with more than 2 years of missing security updates to the internet anyway (or sell it). Reflash & restock time by then.
- xg15 7y ago
  > Non-technical users get an app ... So now I need an already-working smartphone to set up my internet connection.
xg15 7y ago
> These certificates are trusted by browsers on all platforms, but will surely be added to revocation lists shortly. So what about all the routers that were already sold and the customers that bought them? Are we ok with leaving all devices that are already in operation or that are currently being sold unconfigurable for non-technical users, just to protect against a highly hypothetical scenario of abusing routerlogin.net?
tapper82 7y ago
This is why I use OpenWrt. Shout out to the dude sticking up for screen reader users to! Thanks friend. Shit like footnotes and captchirs are the bane of my life!
mkeedlinger 7y ago
This reminds me of something I have been looking for for a long time. If you know any reasonable way to fix it please let me know! This "any CA can authenticate any domain" system is ridiculous. I manage my own CA, which is fine for my own self-hosted sites, but the issue is that it doesn't protect me from a cert made valid by some other CA. Is there any way I can whitelist my own CA such that IT ALONE will work for my domains? (note: this is a me-only thing. Don't need anyone to be able to validate these domains). PS, I've asked this question elsewhere [0] before. I did not find an adequate answer. [0]: https://security.stackexchange.com/questions/211401/can-you-whitelist-a-single-ca-for-a-domain https://security.stackexchange.com/questions/211401/can-you-...
- zzzcpan 7y ago
  See DNS CAA https://en.wikipedia.org/wiki/DNS_Certification_Authority_Authorization https://en.wikipedia.org/wiki/DNS_Certification_Authority_Au...
  - mkeedlinger 7y ago
    Thanks for the reply! This is very interesting, I will surely take a look. Only issue I see is that it seems to use DNS, and that's not exactly secure either. In my mind I am thinking of something that would be more theoretically secure from any remote attack (closer to a form of key-pinning maybe?)

198 points by Maxious 7y ago

6 days is nowhere near a justifiable timeframe for full disclosure. Even if you disagree with that, you should have first reported Key Compromises to Entrust and Comodo before publicly posting the private keys. They are bound by BRs and their own CPS to revoke certificates such as this one - and they would have done so promptly. This is not what you should do as a security researcher - delete the gist until the CAs have a chance to revoke it via OCSP.

mjevans 7y ago
I think in this case it's to force browser vendors (who have the most exploitable endpoints) and companies like Apple and Microsoft at the OS level, to blacklist the offending certs. Though the CAs in question should probably have an automated challenge-response system to which a timed signed reply of a given message of their choice causes a revocation of the key that signed the message. (As sufficient proof of "this is vuln, kill it now, ask questions after".)
- toast0 7y ago
  Did this researcher even contact the CAs? Generally, they're pretty responsive and will revoke quickly; there are CA/Browser forum guidelines that address this; if nothing else, you can send them the private key to prove possession.
- buraksarica 7y ago
  The OS level actions won't be fast. So this disclosure is closer to an irresponsible attitude. At least CAs should be informed.
  - philprx 7y ago
    I tend to differ: Netgear security is paid to manage security. They failed by not responding to these legitimate communications requests. The researcher are not paid. They did what could be done to really fix the problem. Of course you can always do better as a researcher, always, but consider time available and paid vs. pro bono time. Also consider all the people who probably found this before and may have sold it on black market, you’re attacking the wrong people.
RL_Quine 7y ago
> 6 days is nowhere near a justifiable timeframe for full disclosure. Of course it is, if you didn't get a response in the first 48 hours you're not going to, that's the way things like this tend to operate. The security contact for all companies is either a) inactive or b) very active. It's a detriment to everybody that for some reason it has been normalized that people should wait months or years for disclosing bugs. Full disclosure is the only way things get fixed. > This is not what you should do as a security researcher - delete the gist until the CAs have a chance to revoke it via OCSP. Github is archived in real time for the most part. Especially you'll notice that if you post a private key for a cryptocurrency address or credentials for AWS, it'll be stolen and used within seconds. There's some really good sets of information out there like https://www.gharchive.org/ https://www.gharchive.org/ which give you an idea of the sheet amount of data that github produces on a daily basis, and that's just the metadata and things like comments rather than actual git repository contents. The idea that you could delete something from there and have it actually "removed from the internet" is amusing.
gdm85 7y ago
There is another reason why responsible disclosure would have been better (and less short-sighted): vendor might make an internal audit, find root cause and invalidate a bunch of certificates at once (think about similarly leaked certificates due to a bug in some deployment tool). Now it's an open race with the bad guys, and surely a lot of them all at once; hardly an advantageous scenario for end users.
- JshWright 7y ago
  That's an awfully big (and awfully optimistic) "might" in your first paragraph there...
  - StavrosK 7y ago
    Not to mention that all the bad guys had to do to get the private key was unpack the firmware image, so they've probably noticed long ago.
    - gdm85 7y ago
      If what you said were true then zero-days would be always ignored and not a concern.
      - StavrosK 7y ago
        How does that follow from what I said?
  - gdm85 7y ago
    It's not optimistic, it's what a responsible vendor is supposed to do. By not following responsible disclosure the researcher has lifted the vendor from the possibility of organizing a proper response.
    - gdm85 7y ago
      By the way, I meant coordinated disclosure and I am not aware of Netgear not doing any at all, never.
kick 7y ago
Responsible disclosure is meant as a way to harm researchers and users for the benefit of companies. This is absolutely the right thing to do as a security researcher, especially given that Netgear prevents public disclosure, ever, if you submit a bug bounty. To make sure it stays online: https://web.archive.org/web/*/https://gist.github.com/nstarke/a611a19aab433555e91c656fe1f030a9 https://web.archive.org/web/*/https://gist.github.com/nstark...
yardstick 7y ago
“ Thursday, January 16 - Bugcrowd responds, but we are unable to establish a communications channel outside of the Netgear bug bounty programs.” I’m curious to what does this mean? They got a response form Bugcrowd but not Netgear themselves? Or they got through to Netgear but didn’t like what Netgear said so they went public? Something else?
- ukd1 7y ago
  Guessing they meant they were/are unwilling to.
beardog 7y ago
I discovered this same thing in 2017. My bugcrowd report was shot down as dupe https://www.chaoswebs.net/blog/tls-key-reuse-on-popular-router-models.html https://www.chaoswebs.net/blog/tls-key-reuse-on-popular-rout...
Thorrez 7y ago
Full disclosure means 0 days generally.
tinus_hn 7y ago
Boo hoo. Netgear should have known better than to expose their keys like this in the first place, this is really amateur hour stuff. They were reached out to and didn’t respond. It’s their own fault and they have to make do with what they’ve been granted. Now they get to pick up the pieces.
- devwastaken 7y ago
  Yep. Defending bad security products only leads to people buying more. Netgear has not and will not learn any lesson, and will continue to serve bad devices. This time the good to users comes from publicizing that.
ryanlol 7y ago
Full disclosure is always perfectly justifiable, unless you’re stealing someone else’s work.
lima 7y ago
It wasn't the researcher who disclosed the keys, it was the vendor who did, knowingly and purposefully (in their firmware updates).
gsich 7y ago
Who cares, this does not even warrant a responsible disclosure. Netgear can't fix that in a way that makes everyone happy. You can revoke the cert, then what? A firmware update with a new cert? Same problem, private key will still be on the device and in the firmware file. Self signed? You get users crying about the cert. HTTP only? You get users crying about "not secure" even though that might not be true in all cases. My guess is that people before the ones in the gist have tried to contact Netgear about this (since it's not hard to find) and chose to ignore it.

fulafel 7y ago

Can anyone make out what the funjsq.com is about?

dylz 7y ago
Chinese gaming VPN service or something, bypasses China IP blocks to allow them to play on NA/EU servers.
- bflesch 7y ago
  I'm wondering why such a VPN service is included in the netgear firmware image? Wouldn't this resonate negatively with Chinese authorities?
  - lenova 7y ago
    Unless it was mandated/deployed by a state actor, looking to MITM/monitor VPN users within the country?

mschuster91 7y ago

The HTTPS cert is used for the router's login page - apparently putting an IP address on the box backside label confuses too many people. It makes sense to put it behind HTTPS because the browser will whine "this page is insecure"... but how is a router vendor supposed to include the neccessary certificate that won't get leaked? The only thing I can imagine here is a dedicated HSM chip... but that's overkill for a 10$ router?

Erlich_Bachman 7y ago
"How" is that they only inlude the private key for that specific router, and then that key/cert itself is certified by the real CA cert, the key for which is not kept on the router itself?
- zyztem 7y ago
  That requires generating and burning keys to devices during manufacturing. This process will be hacked by NSA & all, exactly like Gemalto https://www.theguardian.com/us-news/2015/feb/19/nsa-gchq-sim-card-billions-cellphones-hacking https://www.theguardian.com/us-news/2015/feb/19/nsa-gchq-sim...
  - Erlich_Bachman 7y ago
    Haha that's a tall order... Constructing a scheme where NSA is an active agent in the threat model was not an original requirement :) You are welcome to introduce any way to produce any part of a router or a PC for that matter that would protect from NSA, it seems that the biggest players in the field are still working out and it is very much a work in progress. When you have an adversary that is able to intercept hardware in transit and spend endless amounts of dollars on devising clever hacks or undetectable hardware exploits, then yes, you're right, some TLS scheme, regardless of where the certs are, is not going to be enough.
- rvnx 7y ago
  Delegated Credentials for TLS I guess ? but this requires internet access so barely a solution. https://engineering.fb.com/security/delegated-credentials/ https://engineering.fb.com/security/delegated-credentials/ I don't think that the fact that private keys for routerlogin.net are bundled with the router are an issue. It's very logical to do so, and BETTER than plain HTTP in most real-life scenarios. routerlogin.net is a local server (when you are using the router) and not a remote website so it's expected that you can trust the content as much as trustable your local network is.

zyztem 7y ago

This is a world after QUANTUMINSERT for you. Somehow everything should be encrypted but key management are still unsolved

beardog 7y ago

I discovered this same thing in 2017. My bugcrowd report was shot down as dupe https://www.chaoswebs.net/blog/tls-key-reuse-on-popular-router-models.html https://www.chaoswebs.net/blog/tls-key-reuse-on-popular-rout...

ploxiln 7y ago
The difference is that these newly discovered certs with private keys were signed by real Certificate Authorities trusted by browsers by default.
- tinus_hn 7y ago
  So their security got worse. It’s a miracle people even bother attempting to do responsible disclosure to fools like this.
  - josteink 7y ago
    > so their security got worse Only because current browsers makes it increasingly hard to run medium security, user-trusted networking.
  - lima 7y ago
    How did security get worse?

politelemon 7y ago

Tuesday, January 14th 2020 - Initial Discovery Tuesday, January 14 2020 - Tweet sent attempting to establish communications with Netgear Wednesday, January 15 2020 - Reached out to Bugcrowd to attempt to establish communications. Thursday, January 16 - Bugcrowd responds, but we are unable to establish a communications channel outside of the Netgear bug bounty programs. Friday, Jaunary 17th - Conversation with bugcrowd proves inconclusive Sunday, January 19th - Feeling we have exhausted our disclosure avenues, we decide to publish I didn't understand, can someone explain this to me - I see the author tried to talk to Netgear, then it's BUgcrowd the next day... it's not clear to me what the relation between Netgear and Bugcrowd is.

Maxious 7y ago
Bugcrowd run bug bounty programs like https://bugcrowd.com/netgear https://bugcrowd.com/netgear on behalf of companies They explained earlier that if they entered the bug bounty program with Netgear, they would be agreeing to not disclose: "By submitting the security bug, you affirm that you have not disclosed and agree that you will not disclose the security bug to anyone other than NETGEAR." So it seems they were trying to get Bugcrowd as a intermediary to contact Netgear on their behalf to "establish a communications channel outside of the Netgear bug bounty programs"
- politelemon 7y ago
  Thanks for the explanation!

bottled_poe 7y ago

Netgear has reliably demonstrated to me over more than 10 years that they are incapable of delivering secure products of an acceptable quality. I actively avoid this brand like the plague.

shock 7y ago
I have posted this in another thread, but I am posting it again because more people need to be aware of Netgear's practices: I recently bought a couple of Netgear Managed Switches (for Business)⁰ and in their datasheet they list "Local-only management" as a feature. Only after they arrived we discovered that you only get limited functionality in the Local-only management mode, you have to register the switches to your Netgear Cloud account to get access to the full functionality. Reading up on it, this was achieved only after a community outcry because in the prior firmware versions the switch would have to connect to the Netgear Cloud on every bootup. Needless to say I would not have bought the swiches if I had knew I needed to register them to Netgear Cloud to have access to the full functionality specified in the data sheet. If I had bought them as a consumer, not as a business, I would have returned them immediately. Netgear are now on our purchasing blacklist. ⓪ - the switches are Netgear GS-108Tv3
- mdszy 7y ago
  >Netgear Managed Switches (for Business)⁰ >⓪ - the switches are Netgear GS-108Tv3 are we not allowed to use any sane method of doing footnotes? Who does this? Why would you even use footnotes in internet comments other than to specifically screw over anyone who uses a screen reader?
  - bmn__ 7y ago
    > are we not allowed to use any sane method of doing footnotes? There is no markup on HN for footnotes. https://news.ycombinator.com/formatdoc https://news.ycombinator.com/formatdoc > Who does this? Blame the people running the site, this is just a small inconvenience compared with other lack of features. > screw over anyone who uses a screen reader? As GP's footnote is plain text – not markup – there is no problem with screen readers (or any user-agent, really). Curious why you would think otherwise.

deleted 7y ago

[deleted]

CiPHPerCoder 7y ago

Some commenters are decrying that this post fails to meet the bar for "responsible disclosure". Please stop using that phrase. "Responsible disclosure". It's wrong and harmful, and the person who coined it agrees with me: https://adamcaudill.com/2015/11/19/responsible-disclosure-is-wrong/ https://adamcaudill.com/2015/11/19/responsible-disclosure-is... You want "coordinated disclosure" instead. Netgear doesn't do coordinated disclosure. They do non-disclosure. In the absence of a coordinated effort, full disclosure is the responsible thing to do.

ryanlol 7y ago
It should be noted that it’s essentially never the people who find bugs arguing for “responsible disclosure”. It sure is easy to tell others what to do with their work product when you have zero stake in the game. There exists a really easy solution to the purported problem of full disclosure, vendors could just offer significant enough financial compensation for non-disclosure.
- RL_Quine 7y ago
  That isn't what disclosing these sort of vulnerabilities is about. The core reason for just going full disclosure is that the vendor has absolutely not incentive to fix any sort of bug that is kept private. The customer is impacted, never the vendor.
  - ryanlol 7y ago
    Nah, disclosing these vulnerabilities is usually about CV-padding and publicity . There aren’t many people out there hunting bugs just to be nice.
    - cipherboy 7y ago
      I happen to know one of the authors of this post (hey Tom!). He's actually a really nice, down to earth guy. Helped out with our college cyber defense programs and is a killer red teamer. Very patient in explaining how he got in and defaced your website, time and time again. Is this his best work? Nah, this is amateur hour on the part of Netgear. But am I glad it was him who found it? Definitely. Keep in mind, there really wasn't anything _to_ this vulnerability other than copying and pasting from one website (and a firmware zip) to another. They just added a little pretty formating and saved you the trouble of extracting the firmware.

josteink 7y ago

I guess I run a different firmware for my Netgear ReadyNAS 312. This is easy to inspect because the FW is Debian-based, and enabling inbound SSH is easy. Checking on mine, it has a cert only for the "nas.local"-domain, and the cert is stored in /etc/ssl/certs/ssl-cert-snakeoil.pem and /etc/ssl/private/ssl-cert-snakeoil.key respectively. Have to admit I love the naming of those files :) Edit: Obviously it is a different firmware. Mine is a NAS, this was a router. And “snakeoil” seems to be default debianism.

josteink 7y ago

To all the people shitting on Netgear and security in this thread, just how do you propose one deliver a secure network appliance to end customers which they can deploy on their network? And which is user-accessible to common users in modern browsers rejecting everything not touched by a proper CA? Really. Please educate the world with your ingenious insight. I’ll be waiting. The unavoidable truth is: You have to stuff the key in there somehow. There’s no way around that. Either browsers have to start accepting “local” certs more easily, or end-user network deployed appliances can’t have ssl. Take a pick.

zzzcpan 7y ago
On local networks TLS doesn't make things more secure, maybe even less in some scenarios given all the obtaining of certificates over public internet and leaking info to the public internet. So sane thing to do would be plain http. But really, browsers should stop the bullshit with https "security" and accept ssh-like behavior at least for local networks.
pjc50 7y ago
Define "secure". If you've "stuffed the key in there" in such a way that other people can get it out again, and it works across all routers of that model, then it's possible for an active attacker on your network to MITM your connection to your router. So it's slightly more secure than cleartext but not much; about the same as a random self-signed certificate. It is definitely a hard problem because there's no easy way to authenticate the router to the client, but on the other hand I'm not sure how important all of this is when it's on a local link anyway. A "best possible practice" solution would be to have the routers issued with individual certificates at factory programming time, and provide a rollover mechanism through the admin UI.
- vbezhenar 7y ago
  AFAIK CA can't issue certificate for more than two years. So if you've bought the device two years after it was manufactured, you're getting error and it's even worse than HTTP for user.

zonidjan 7y ago

This is intentional on Netgear's part, and does not in any way degrade security compared to the alternative (an untrusted cert or no HTTPS). It is neither a bug nor a security issue.

gene91 7y ago

Is this cert revoked yet? Sure, Entrust is required to revoke it. It is bound by BRs. If it refuses, Entrust will get itself blacklisted by browsers. On the other hand, having the cert revoked will cost Netgear dearly. As a result, I wonder if Entrust might be dragging its feet on the revocation.

isclever 7y ago
mini-app.funjsq.com is revoked (https://decoder.link/result/418b8d20793d3f4daa4153752e45e78b87e8e293 https://decoder.link/result/418b8d20793d3f4daa4153752e45e78b...), can't check routerlogin.net/com as they didn't paste the public cert.
- tgsovlerkhgsel 7y ago
  routerlogin is not revoked: https://crt.sh/?id=1955992027&opt=ocsp https://crt.sh/?id=1955992027&opt=ocsp
  - deleted 7y ago
    [deleted]

rodnim 7y ago

I don't envy Netgear. They sell home routers to people who hardly know the difference between HTTP and HTTPS. In this regard, I respect them for NOT making those people submit sensitive information by ignoring browser security warnings and/or accepting self signed certificates. I.e, not teaching bad habits. On the other hand, making private keys publicly available is obviously far from ideal. Damned if you do, damned if you don't. Again, I don't envy Netgear...

gene91 7y ago

Netgear picked a terrible and clearly unacceptable approach. I I read through this discussion, trying to understand if there exists a good solution. Using plain http isn't a good solution because it involves telling the user to ignore the "not secure" warnings by browsers. This seems like the best available solution. Without network connectivity, the traffic can not be tunneled through a remote server with proper cert. Nor can the box request a cert in realtime at the time of setup. Pre-provisioned cert won't work if the box is purchased more than 27 months after manufacturing (the current max validity period). For home network equipment, I suppose this isn't unusual. I see TLS-SRP mentioned in comments. But the commenter hinted at its limited adoption in browsers.

jniedrauer 7y ago
When I can control the client configuration, I just set up my own trusted root CA and then have the distributed systems automatically submit a CSR on initialization. This doesn't work when you have to support end user's web browsers though (unless you're somehow lucky enough to own a public CA and don't care about ICANN rules). It really does seem like there should be an encryption-only option for TLS. Maybe anything that resolves by IP address and not name should display the green padlock even if the SAN doesn't match the name?
- gene91 7y ago
  Encryption-only TLS doesn't protect MITM. It prevents sniffing. At a higher level, TLS provides integrity and privacy when the network linked is not trustworthy. As a result, I'm not sure what your proposal achieves. Similarly, resolution by IP address does not provide privacy or integrity unless you trust the network link.
  - jniedrauer 7y ago
    The green padlock indicates two things about the server to the user: "This is really my name" and "no one can snoop on our conversation." It doesn't indicate anything about the trustworthiness of the server. The first one is meaningless in the context of an IP address. If a user is going to get pished by an IP address, then they could just as well get phished by 123.sketchydomain.com. The important part when communicating directly with a device by IP address, is that the payload is encrypted end to end. > Similarly, resolution by IP address does not provide privacy or integrity unless you trust the network link. That's a fair point. It's certainly no worse than our current situation where you have to trust a self signed certificate on a case-by-case basis. The router could forge a self signed certificate even more easily than it could forge a certificate signed by a trusted CA.

tgsovlerkhgsel 7y ago

The cert for www.routerlogin.net (Serial c1:a1:00:64:07:61:2c:07:00:00:00:00:50:f1:09:6a) isn't revoked yet: https://crt.sh/?id=1955992027&opt=ocsp https://crt.sh/?id=1955992027&opt=ocsp The reporters should have asked the CAs to revoke the certificate. If the CA doesn't do that within 24 hours, it's considered a violation of the rules that the CA needs to follow to stay in browsers.

dlgeek 7y ago
Anyone with access to the private key can submit a problem report to the CA, they're obligated to revoke if the key is exposed to a non-subscriber. I have submitted a report - they are obligated to revoke within 24 hours.
- cjbprime 7y ago
  Thanks for doing that.
- cjbprime 7y ago
  Looks like they've failed to perform the revocation in time -- other users reported to Entrust >24h ago.
  - dlgeek 7y ago
    They revoked it 7 minutes shy of the 24 hour deadline after my report. If other users reported it and they failed to act, that's a BR violation and they're required to provide an incident reports to the root programs via Mozilla's mechanisms. If they don't, and you have evidence of those reports, you should provide them to the root programs via the mozilla.dev.security.policy mailing list/group. There's already a thread about this issue, though not claiming that EnTrust was notified. (Based on timestamps, it's also possible they revoked in response to that thread).
    - cjbprime 7y ago
      Thanks. Here is the timestamp showing that it took more than 32 hours from reporting to revocation. https://twitter.com/FiloSottile/status/1219147543667453953?s=19 https://twitter.com/FiloSottile/status/1219147543667453953?s...

privateSFacct 7y ago

What’s a good netgear alternative for switches from 5 port to 48 port - too late for us now but for next buying wave

bilekas 7y ago

Name and Shame. Responsible disclosure ? How about reposonsible security pracitces first.

actionowl 7y ago

HTTP(S) might just be the wrong protocol for initial setup of a router! Maybe instead generate a password for each router and print it on the box along with the device's unique SSH Key fingerprint. Technical users can then SSH in and bootstrap the system (also generate/upload their own TLS certs if they want to use a browser and connect over HTTPS, etc). Non-technical users get an app and they can scan a QR code on the box which has the SSH user, password, and fingerprint for verification. The App connects to the router over SSH to do router setup. If an App is involved you wouldn't technically NEED to use SSH with the App but it was the first thing that came to mind.

fulafel 7y ago
I think https could work fine, just give each device a unique key and dns name.
- actionowl 7y ago
  That approach has been mentioned in the comments. Some problems with it are: If a certificate is valid beyond 825 days Chrome will not honor it. So if you generate a key and cert at the factory for every device you run into the issue of the end-user getting a device with an expired cert. If you generate a unique key and self-signed cert users will complain and call support about the "insecure" site warning. Communicating upstream to a third party to get a cert won't work for all users (e.g. those that need to configure a static IP from their ISP.)
  - fulafel 7y ago
    You shouldn't plug something with more than 2 years of missing security updates to the internet anyway (or sell it). Reflash & restock time by then.
xg15 7y ago
> Non-technical users get an app ... So now I need an already-working smartphone to set up my internet connection.

xg15 7y ago

> These certificates are trusted by browsers on all platforms, but will surely be added to revocation lists shortly. So what about all the routers that were already sold and the customers that bought them? Are we ok with leaving all devices that are already in operation or that are currently being sold unconfigurable for non-technical users, just to protect against a highly hypothetical scenario of abusing routerlogin.net?

tapper82 7y ago

This is why I use OpenWrt. Shout out to the dude sticking up for screen reader users to! Thanks friend. Shit like footnotes and captchirs are the bane of my life!

mkeedlinger 7y ago

This reminds me of something I have been looking for for a long time. If you know any reasonable way to fix it please let me know! This "any CA can authenticate any domain" system is ridiculous. I manage my own CA, which is fine for my own self-hosted sites, but the issue is that it doesn't protect me from a cert made valid by some other CA. Is there any way I can whitelist my own CA such that IT ALONE will work for my domains? (note: this is a me-only thing. Don't need anyone to be able to validate these domains). PS, I've asked this question elsewhere [0] before. I did not find an adequate answer. [0]: https://security.stackexchange.com/questions/211401/can-you-whitelist-a-single-ca-for-a-domain https://security.stackexchange.com/questions/211401/can-you-...

zzzcpan 7y ago
See DNS CAA https://en.wikipedia.org/wiki/DNS_Certification_Authority_Authorization https://en.wikipedia.org/wiki/DNS_Certification_Authority_Au...
- mkeedlinger 7y ago
  Thanks for the reply! This is very interesting, I will surely take a look. Only issue I see is that it seems to use DNS, and that's not exactly secure either. In my mind I am thinking of something that would be more theoretically secure from any remote attack (closer to a form of key-pinning maybe?)