6 ms·

Hardcoded Password Found in Cisco Enterprise Software Again

191 points by willvarfar 8y ago

sschueller 8y ago
At least this time they them selves found the security vulnerabilities and not someone else. I hope these internal audits are now a normal part of their business.
- willvarfar 8y ago
  Are there any recent ones found by others? From the article: > Let's not criticize Cisco > The company discovered these flaws following as part of its massive series of internal audits it started back in December 2015. > At the time, security researchers found a backdoor account in Juniper software that could decrypt VPN traffic, and Cisco decided to hunt and root out any similar backdoors before attackers found them first. > The company discovered many backdoors and hardcoded accounts in the past two years as part of internal audits and has received some pretty unfair criticism for its efforts. > The most recent backdoors Cisco discovered was in March, when the company's engineers discovered two —one in Cisco's Prime Collaboration Provisioning (PCP) platform, and one in the IOS XE operating system.
  - userbinator 8y ago
    The question is who put them there... and more interestingly, do the updates replace the backdoors with different, more hidden ones? To put it bluntly, even backdoor accounts need their credentials changed once in a while.
    - ptero 8y ago
      Most backdoors in commercial software are due to stupidity, not ill intent: software engineers putting a debug access point "I will remove it later before it goes into production" or business side having weird ideas "it would be so much simpler if we could log in and fix/reconfigure it for consumers". Finding those only to replace them with a more elaborate bug seems to be ill intent and unlikely for me. The bigger problem IMO is that the mentality above is still prevalent: for every two backdoors found a new one may be put in by another do-gooder.
      - _jal 8y ago
        Cisco is the Oracle of the networking world. I'm sure it is mostly incompetence, with only the occasional state-actor poisoning the well. But I see little evidence Cisco gives a damn or has any intention of actually attempting to regain some of the squandered trust. Which is fine, from my perspective; for the most part I like other vendors' kit quite a bit more anyway. I do think product liability law needs to kick in somewhere. If you have some Tourettes-like inability to stop writing back doors in security products, there should be some outside pressure to maybe get some help.
        giancarlostoro 8y ago
        Is Cisco as evil as Oracle? Because ouch. Oracle is just plain evil. I have no reason to compare Cisco to Oracle yet. I usually prefer TP Link since they're usually effective and affordable to boot.
    - davidu 8y ago
      Absolutely not.
  - retSava 8y ago
    > has received some pretty unfair criticism for its efforts. Ha, sure, one could say that the effort itself is commendable (internal audits and exploit hunts), but why does it feel it's a little like applauding a hitman for cleaning up the bloody mess they made in someones house?
- athenot 8y ago
  > I hope these internal audits are now a normal part of their business. They now are, at 3 stages: - During design, - Before early field trial, - Before production. Stuff like this is bad for everyone so we're spending a lot of effort upstream to ensure these things don't happen again. And yes it's embarrassing when we find existing products that have these vulns but there's a no-blame attitude that's really helping get things addressed instead of ignoring them (which is the default in large organizations). The faster we can find and address the errors of the past, the better off everyone is.
  - EpicEng 8y ago
    >but there's a no-blame attitude that's really helping get things addressed instead of ignoring them (which is the default in large organizations) Ok, but you guys have been screwing the pooch in this regard for decades. What went wrong the last 20+ times?
adrianN 8y ago
Why do people still buy Cisco products? Leaving hardcoded passwords in their software shows such massive incompetence that I don't understand how they can still sell anything.
- ptero 8y ago
  They still sell high performance products. And (most important) I suspect the alternatives are no better with respect to security; they just get less scrutiny and thus are less in the news.
- exelius 8y ago
  The entire telecom industry has been asking this question for the last 20 years. They have finally broken Cisco’s hold over the last 5 years by just building their own hardware a la Facebook and Google. Turns out it’s cheaper that way anyway, and they were already having to hold cisco’s hand through meeting spec lists, so it’s not much extra work for them to own the IP.
  - ra1n85 8y ago
    It’s only cheaper if you’re operating at large scale. For the vast majority of organizations, Cisco/Juniper/Palo Alto/Arista are the only cost effective options.
    - exelius 8y ago
      Right, but at the low end, there are better options than Cisco (all 3 you mentioned really). At the high end (carrier grade), there weren’t necessarily. They’re not going it alone; ONOS [1] is basically the open source carrier-grade networking platform. Think of it as Kubernetes for SDN packet routing that they’re building custom network integration solutions (think routers, bulk fiber termination, RF, satellite, laser, WiFi mesh) on top of. Then get Chinese manufacturers to build the hardware (which doesn’t have anything proprietary in it — all white box with Intel or ARM CPUs). Incredibly powerful and flexible, and they have all the heavy hitters involved (Google, Intel, Samsung, AT&T, Comcast, etc.) It will be a game-changer for telecom in another 5 or 6 years once the management tools mature and commercial solutions based on it are developed. [1] https://www.onosproject.org/ https://www.onosproject.org/
gabcoh 8y ago
Sort of funny that right under the section titled: >Let's not criticize Cisco I get a nice Cisco add. I know that the author doesn’t choose the ads but it’s still pretty funny.
ausjke 8y ago
Time to bring back mid-to-high-end open source routers. Something like Vyos: https://vyos.io/ https://vyos.io/ For low-end you always have LEDE or Openwrt http://openwrt.org http://openwrt.org
- Kenji 8y ago
  OpenWRT is fantastic. Critical security patches come within days of the vulnerability being disclosed, the thing compiles with little hassle on Linux Ubuntu and it runs stable and it's very lean and fast.
DannyB2 8y ago
Cisco seriously needs to fix this and make sure that next time it is much harder to find their hardcoded passwords. /s
kuon 8y ago
I started learning networking with Cisco IOS nearly 25 years ago, but I hate Cisco on many level (licensing, hardware price, support experience). They pushed the technology forward, there is no denying to that, but they need to be disturbed a lot more than what is happening now. I have been using PCEngines for small routers running OpenBSD. But I am still waiting for larger hardware (ethernet switch, telco routers...) to be able to run an open source software. I know facebook/google started building their own "open" hardware, but it is not generally available.
- xiconfjs 8y ago
  Take a look at EdgeCore Switches [1] and Cumulus [2] or Pica8 [3]. Should be interesting for you. [1] https://www.edge-core.com/productsKind.php?cls=1 https://www.edge-core.com/productsKind.php?cls=1 [2] https://cumulusnetworks.com/products/cumulus-linux/ https://cumulusnetworks.com/products/cumulus-linux/ [3] https://www.pica8.com/ https://www.pica8.com/
  - kuon 8y ago
    Those EdgeCore switches looks like a good "start". Thanks.
- monocasa 8y ago
  I wonder if we would have had something better though, if Cisco hadn't gotten their early start on blatantly and unashamedly stolen IP. EDIT: For those downvoting: https://www.aol.com/article/finance/2013/12/10/the-real-story-of-ciscos-creation/20785939/ https://www.aol.com/article/finance/2013/12/10/the-real-stor... Cisco got started by blatantly stealing the Stanford Blue Box code and board layout, replying with "well sue me then" when Stanford found out, and put so many legal roadblocks up that Stanford finally said "fine, whatever, just give us some proceeds".
- ausjke 8y ago
  RouterOS on Mikrotik, pfsense on x86, VyOS(debian) on x86 are all good options. It's just hard to find off-the-shelf parts to build your 1u/2u router/switches running open source though, software is there, but no such hardware, the best you get is some random formats(e.g. mini-itx) and you can not really put them on the rack.
Clubber 8y ago
It's difficult to believe a company like Cisco would release something like this on accident. Maybe I'm just being naive.
- tboyd47 8y ago
  Like many companies its size, it does not operate like a single company but like many companies. I don't know what you are implying is the true cause, but developers make these kinds of bad decisions all the time in companies of all size. Most likely explanation to me is a calculated risk on the part of some middle manager.
noslenac 8y ago
As a business owner, this kind of thing is exactly why we as a company are completely vendor/brand/technology agnostic. Too many companies get in a rut and will be loyalists no matter what happens even when there are better products out there. I understand that switching is not always easy or even the right answer, but if Cisco were worried about people/companies leaving to go to a competitor, they may correct the problems that allow this kind of thing to keep occurring. Fear of losing good customers may cause them to invest more in a better QA process. Just my two cents. I'm not anti-Cisco or pro-Cisco but do like our customers to have the best possible solution for their environments and needs.
paulie_a 8y ago
Wtf Cisco. Get your shitty software in order. This is getting absurd how frequently this happens. I will never trust Cisco gearfor anything
phyzome 8y ago
Ah, what an evergreen headline.
NelsonMinar 8y ago
I don't understand how this continues to happen. Is it because it's so hard to sue a company like Cisco for negligence?
Rotdhizon 8y ago
Can anyone elaborate on how these situations occur? What happened during the construction of this software? Did someone put in the credentials as a test account and forget to remove it? Did someone sneak it in? Is it a standard practice? I assume quite a few people worked on that software, I can't imagine a whole department just up and forgot they coded in a backdoor.
- smacktoward 8y ago
  "Just hard-code the credentials in for now, we'll go back and put in a more sophisticated system for managing them later." RON HOWARD VOICE: They didn't.
Animats 8y ago
"Let's not criticize Cisco" No, let's criticize Cisco. Those backdoors didn't get there by themselves. They had help. Do they have source control adequate to tell them where those came from? Who put them in? Was the FBI informed? Were people arrested? Who was fired? If not, why not?

191 points by willvarfar 8y ago

At least this time they them selves found the security vulnerabilities and not someone else. I hope these internal audits are now a normal part of their business.

willvarfar 8y ago
Are there any recent ones found by others? From the article: > Let's not criticize Cisco > The company discovered these flaws following as part of its massive series of internal audits it started back in December 2015. > At the time, security researchers found a backdoor account in Juniper software that could decrypt VPN traffic, and Cisco decided to hunt and root out any similar backdoors before attackers found them first. > The company discovered many backdoors and hardcoded accounts in the past two years as part of internal audits and has received some pretty unfair criticism for its efforts. > The most recent backdoors Cisco discovered was in March, when the company's engineers discovered two —one in Cisco's Prime Collaboration Provisioning (PCP) platform, and one in the IOS XE operating system.
- userbinator 8y ago
  The question is who put them there... and more interestingly, do the updates replace the backdoors with different, more hidden ones? To put it bluntly, even backdoor accounts need their credentials changed once in a while.
  - ptero 8y ago
    Most backdoors in commercial software are due to stupidity, not ill intent: software engineers putting a debug access point "I will remove it later before it goes into production" or business side having weird ideas "it would be so much simpler if we could log in and fix/reconfigure it for consumers". Finding those only to replace them with a more elaborate bug seems to be ill intent and unlikely for me. The bigger problem IMO is that the mentality above is still prevalent: for every two backdoors found a new one may be put in by another do-gooder.
    - _jal 8y ago
      Cisco is the Oracle of the networking world. I'm sure it is mostly incompetence, with only the occasional state-actor poisoning the well. But I see little evidence Cisco gives a damn or has any intention of actually attempting to regain some of the squandered trust. Which is fine, from my perspective; for the most part I like other vendors' kit quite a bit more anyway. I do think product liability law needs to kick in somewhere. If you have some Tourettes-like inability to stop writing back doors in security products, there should be some outside pressure to maybe get some help.
      - giancarlostoro 8y ago
        Is Cisco as evil as Oracle? Because ouch. Oracle is just plain evil. I have no reason to compare Cisco to Oracle yet. I usually prefer TP Link since they're usually effective and affordable to boot.
  - davidu 8y ago
    Absolutely not.
- retSava 8y ago
  > has received some pretty unfair criticism for its efforts. Ha, sure, one could say that the effort itself is commendable (internal audits and exploit hunts), but why does it feel it's a little like applauding a hitman for cleaning up the bloody mess they made in someones house?
athenot 8y ago
> I hope these internal audits are now a normal part of their business. They now are, at 3 stages: - During design, - Before early field trial, - Before production. Stuff like this is bad for everyone so we're spending a lot of effort upstream to ensure these things don't happen again. And yes it's embarrassing when we find existing products that have these vulns but there's a no-blame attitude that's really helping get things addressed instead of ignoring them (which is the default in large organizations). The faster we can find and address the errors of the past, the better off everyone is.
- EpicEng 8y ago
  >but there's a no-blame attitude that's really helping get things addressed instead of ignoring them (which is the default in large organizations) Ok, but you guys have been screwing the pooch in this regard for decades. What went wrong the last 20+ times?

adrianN 8y ago

Why do people still buy Cisco products? Leaving hardcoded passwords in their software shows such massive incompetence that I don't understand how they can still sell anything.

ptero 8y ago
They still sell high performance products. And (most important) I suspect the alternatives are no better with respect to security; they just get less scrutiny and thus are less in the news.
exelius 8y ago
The entire telecom industry has been asking this question for the last 20 years. They have finally broken Cisco’s hold over the last 5 years by just building their own hardware a la Facebook and Google. Turns out it’s cheaper that way anyway, and they were already having to hold cisco’s hand through meeting spec lists, so it’s not much extra work for them to own the IP.
- ra1n85 8y ago
  It’s only cheaper if you’re operating at large scale. For the vast majority of organizations, Cisco/Juniper/Palo Alto/Arista are the only cost effective options.
  - exelius 8y ago
    Right, but at the low end, there are better options than Cisco (all 3 you mentioned really). At the high end (carrier grade), there weren’t necessarily. They’re not going it alone; ONOS [1] is basically the open source carrier-grade networking platform. Think of it as Kubernetes for SDN packet routing that they’re building custom network integration solutions (think routers, bulk fiber termination, RF, satellite, laser, WiFi mesh) on top of. Then get Chinese manufacturers to build the hardware (which doesn’t have anything proprietary in it — all white box with Intel or ARM CPUs). Incredibly powerful and flexible, and they have all the heavy hitters involved (Google, Intel, Samsung, AT&T, Comcast, etc.) It will be a game-changer for telecom in another 5 or 6 years once the management tools mature and commercial solutions based on it are developed. [1] https://www.onosproject.org/ https://www.onosproject.org/

gabcoh 8y ago

Sort of funny that right under the section titled: >Let's not criticize Cisco I get a nice Cisco add. I know that the author doesn’t choose the ads but it’s still pretty funny.

ausjke 8y ago

Time to bring back mid-to-high-end open source routers. Something like Vyos: https://vyos.io/ https://vyos.io/ For low-end you always have LEDE or Openwrt http://openwrt.org http://openwrt.org

Kenji 8y ago
OpenWRT is fantastic. Critical security patches come within days of the vulnerability being disclosed, the thing compiles with little hassle on Linux Ubuntu and it runs stable and it's very lean and fast.

DannyB2 8y ago

Cisco seriously needs to fix this and make sure that next time it is much harder to find their hardcoded passwords. /s

kuon 8y ago

I started learning networking with Cisco IOS nearly 25 years ago, but I hate Cisco on many level (licensing, hardware price, support experience). They pushed the technology forward, there is no denying to that, but they need to be disturbed a lot more than what is happening now. I have been using PCEngines for small routers running OpenBSD. But I am still waiting for larger hardware (ethernet switch, telco routers...) to be able to run an open source software. I know facebook/google started building their own "open" hardware, but it is not generally available.

xiconfjs 8y ago
Take a look at EdgeCore Switches [1] and Cumulus [2] or Pica8 [3]. Should be interesting for you. [1] https://www.edge-core.com/productsKind.php?cls=1 https://www.edge-core.com/productsKind.php?cls=1 [2] https://cumulusnetworks.com/products/cumulus-linux/ https://cumulusnetworks.com/products/cumulus-linux/ [3] https://www.pica8.com/ https://www.pica8.com/
- kuon 8y ago
  Those EdgeCore switches looks like a good "start". Thanks.
monocasa 8y ago
I wonder if we would have had something better though, if Cisco hadn't gotten their early start on blatantly and unashamedly stolen IP. EDIT: For those downvoting: https://www.aol.com/article/finance/2013/12/10/the-real-story-of-ciscos-creation/20785939/ https://www.aol.com/article/finance/2013/12/10/the-real-stor... Cisco got started by blatantly stealing the Stanford Blue Box code and board layout, replying with "well sue me then" when Stanford found out, and put so many legal roadblocks up that Stanford finally said "fine, whatever, just give us some proceeds".
ausjke 8y ago
RouterOS on Mikrotik, pfsense on x86, VyOS(debian) on x86 are all good options. It's just hard to find off-the-shelf parts to build your 1u/2u router/switches running open source though, software is there, but no such hardware, the best you get is some random formats(e.g. mini-itx) and you can not really put them on the rack.

Clubber 8y ago

It's difficult to believe a company like Cisco would release something like this on accident. Maybe I'm just being naive.

tboyd47 8y ago
Like many companies its size, it does not operate like a single company but like many companies. I don't know what you are implying is the true cause, but developers make these kinds of bad decisions all the time in companies of all size. Most likely explanation to me is a calculated risk on the part of some middle manager.

noslenac 8y ago

As a business owner, this kind of thing is exactly why we as a company are completely vendor/brand/technology agnostic. Too many companies get in a rut and will be loyalists no matter what happens even when there are better products out there. I understand that switching is not always easy or even the right answer, but if Cisco were worried about people/companies leaving to go to a competitor, they may correct the problems that allow this kind of thing to keep occurring. Fear of losing good customers may cause them to invest more in a better QA process. Just my two cents. I'm not anti-Cisco or pro-Cisco but do like our customers to have the best possible solution for their environments and needs.

paulie_a 8y ago

Wtf Cisco. Get your shitty software in order. This is getting absurd how frequently this happens. I will never trust Cisco gearfor anything

phyzome 8y ago

Ah, what an evergreen headline.

NelsonMinar 8y ago

I don't understand how this continues to happen. Is it because it's so hard to sue a company like Cisco for negligence?

Rotdhizon 8y ago

Can anyone elaborate on how these situations occur? What happened during the construction of this software? Did someone put in the credentials as a test account and forget to remove it? Did someone sneak it in? Is it a standard practice? I assume quite a few people worked on that software, I can't imagine a whole department just up and forgot they coded in a backdoor.

smacktoward 8y ago
"Just hard-code the credentials in for now, we'll go back and put in a more sophisticated system for managing them later." RON HOWARD VOICE: They didn't.

Animats 8y ago

"Let's not criticize Cisco" No, let's criticize Cisco. Those backdoors didn't get there by themselves. They had help. Do they have source control adequate to tell them where those came from? Who put them in? Was the FBI informed? Were people arrested? Who was fired? If not, why not?