5 ms·

I work for a hardware vendor and their customers are basically the same as Cisco's. So here is what happened with us - we had some hardcoded root password in th

by Yizahi 3y ago

I work for a hardware vendor and their customers are basically the same as Cisco's. So here is what happened with us - we had some hardcoded root password in the equipment during early development stages. And it was also known to some of the customers support staff for debugging. Then some of our tech leads raised this issue and discussed that we had to change the root password because it was a correct thing to do and don't wait for some breach in the field. After some usual months of bureaucracy someone did the push and actually changed the password in all versions. Can you already guess what has happened next? :) The biggest customer threw a gigantic hysterical fit and practically demanded to return the old root password which they knew. We just shrugged and complied. It's not like it will be ours network which can be breached. If they want to operate insecurely, they are welcome. PS: just saying, no need to involve NSA in something that is better and more often explained by stupidity. :)

kwanbix 3y ago
Did you try to explain why this is a serious security issue?
- Yizahi 3y ago
  Our lead developers tried, and they are way smarter than me). No luck, for some reason customers don't consider this a vulnerability. Even ones who take security more seriously and do some external pentesting on our equipment.
  - deadbunny 3y ago
    I've found having smart people explain things can be a detriment. Sometimes it needs to be the dumbest explanation to get people to understand.
    - posix86 3y ago
      Smart people explain things with big mental steps they deem obvious (true or not), and less smart people disagree with these steps but are unable to point the finger on them.
    - Aeolun 3y ago
      Something like handing out keys to your villa to every subcontractor that ever visits your house?
throwawayqqq11 3y ago
Would be better to have such default creds parameterized by your build system. So every dev can choose theirs and you can choose unique ones per customer.
jsiepkes 3y ago
There is nothing as permanent as a temporary fix...and worse people will become dependent on the specific inner workings of the temporary fix. Once given...
- accrual 3y ago
  Relevant XKCD: "Every change breaks someone's workflow." https://xkcd.com/1172/ https://xkcd.com/1172/
kogepathic 3y ago
> The biggest customer threw a gigantic hysterical fit and practically demanded to return the old root password which they knew. We just shrugged and complied. It's not like it will be ours network which can be breached. If they want to operate insecurely, they are welcome. Did a PM or sales engineer from your org ever sit down with the customer to figure out what they were doing with root access that your solution lacked? My naive read here is that the customer saw it as easier to keep root access because your product didn't expose what they wanted or did it in a cumbersome way.
- masklinn 3y ago
  They were providing root either way, the only change I assume was switching from the same root on all machines to a per-machine root password. I can understand how that would be a pain in the ass if your legacy architecture (e.g. administration and reporting tooling, etc…) is predicated upon a unified root, you probably don’t even have a way to report the information so that machines and passwords could be paired. Assuming a good way to provision keys is provided, I’d make the insecure unified root password into a paid option (with lots of warnings) the price of which would increase every year. This way the existing workflow of the customer keeps working short-term, but you CY the hell out of your A in case of breach at the customer’s, and at one point the line item will get large enough that the customer allocates time & money to fix their shit.
  - true_religion 3y ago
    Charging for insecure systems incentivizes selling those same systems to customers. Soon you’ll have sales guys regaling how this unified root password system is extremely convenient and you’re one of the few companies who provides this single-signature-security-system (S4 compliance). Next thing you know someone will gain an software patent on this to ensure no one else can provide it.
    - perfmode 3y ago
      But charging for it also disincentivizes customers from asking for it.
mrtksn 3y ago
What’s their complaint? Do they have a workflow that requires the root password all the time and it creates too much of a management to not have QWERTY1234 as root password?
throw0101c 3y ago
You could use the excuse that you are complying with the law per "California passes law that bans default passwords in connected devices": * https://techcrunch.com/2018/10/05/california-passes-law-that-bans-default-passwords-in-connected-devices/?guccounter=1 https://techcrunch.com/2018/10/05/california-passes-law-that...
- hhh 3y ago
  Is complying with the law an excuse these days?
  - bigbillheck 3y ago
    Always has been.
- Yizahi 3y ago
  Oh, I'm pretty sure we can just say whatever, even "we have decided and it will be so". There is only a tiny problem when a customer is bigger (and richer) than the vendor by an order of magnitude or more :) . They tend to control a lot, with such power imbalance.
  - hirsin 3y ago
    Honestly doesn't get better in the reverse either. At Microsoft I'd sit in on calls with banks and other "big names", watching them pound the table and argue their way, usually involving something dangerous and borderline malicious use of the product, all the while thinking "we could buy your company with our cash on hand". I think the key was that they're used to being the big dog in the room, and Microsoft made a point of not correcting them when that behavior was seen.
TheLoafOfBread 3y ago
Or instead of changing out of the blue what exists today and is probably soaked into whole following ecosystem around the product and would be massive PITA to change it as well, you could have create parallel way for login (i.e. via certificate) and tell customer that password access will be deprecated in 1-2 years, so they have time to update their tooling.
trustingtrust 3y ago
Can confirm the hardware vendor that I work for does the same. root password is given and customers are asked to change. No one wants to. Heck I would prefer if they disabled root login completely and used keys instead. But thats now it is. Some customers even enable login from web with those passwords. The stupidity is crazy.
ajsnigrutin 3y ago
Meh, just prompt them to set/change it the first time they boot, and after that, it's their fault :)
- Pathogen-David 3y ago
  Yeah, this is the only real solution.
segmondy 3y ago
You are conflating two separate types of passwords. Hardcoded password is different from default passwords. You can't change a hardcoded password, it's always part of the source code/image. A default password can be changed. Both can be backdoors if kept secret, but a hardcoded password from the onset is a backdoor.
- mecsred 3y ago
  Do you mean can't be changed on a live system? It's very easy to change a hard-coded password, just recompile the source and flash. That could definitely be what OP is referring to if they pushed OTA firmware upgrades.
NoMoreNicksLeft 3y ago
Wait. Do I understand this correctly? Are you saying you removed that root password, such that it wasn't possible to log in with another (but different) root password? Or are you saying that you merely changed it, to one the customer no longer knew, but was the same password across all of your install base, for every customer? How would that be any improvement, if the latter?
- Yizahi 3y ago
  It was at that moment a simple change of the password. The benefit was that people outside of our company wouldn't have access (thus not sharing it further), and then as a future improvement we could have implemented some better key scheme. But while customers want to have root access it is not an option. Theoretically we could implement a scheme with individual passwords or keys per device (as someone suggested in the comments above), but such thing is not being prioritized by our company, so that's our fault.
  - NoMoreNicksLeft 3y ago
    Jesus fuck. This isn't a Latvian company, by chance, is it? Blink twice if yes.
  - pabs3 3y ago
    The right scheme would be to require the user to set their own password on first login, and require that to happen before the device would work.
throwawaaarrgh 3y ago
In this case, only one version out of three (the middle one) had this vuln, so it's very likely it was introduced by mistake. But your scenario happens all the time too.
chasd00 3y ago
I hope you guys got an indemnification agreement or something else that make you not liable when the inevitable happens but, knowing how these things work, I doubt it.
kermatt 3y ago
Unfortunately folding to an unreasonable (in terms of security) request like this emboldens them to be stupid in the future as well. The customer is usually wrong.
codedokode 3y ago
> practically demanded to return the old root password You could set that password just for that one customer, and there is no problem.
phendrenad2 3y ago
So you are intentionally including a secret backdoor in your software that is only known to some customers? Yikes!

by Yizahi 3y ago

kwanbix 3y ago

Did you try to explain why this is a serious security issue?

Yizahi 3y ago
Our lead developers tried, and they are way smarter than me). No luck, for some reason customers don't consider this a vulnerability. Even ones who take security more seriously and do some external pentesting on our equipment.
- deadbunny 3y ago
  I've found having smart people explain things can be a detriment. Sometimes it needs to be the dumbest explanation to get people to understand.
  - posix86 3y ago
    Smart people explain things with big mental steps they deem obvious (true or not), and less smart people disagree with these steps but are unable to point the finger on them.
  - Aeolun 3y ago
    Something like handing out keys to your villa to every subcontractor that ever visits your house?

throwawayqqq11 3y ago

Would be better to have such default creds parameterized by your build system. So every dev can choose theirs and you can choose unique ones per customer.

jsiepkes 3y ago

There is nothing as permanent as a temporary fix...and worse people will become dependent on the specific inner workings of the temporary fix. Once given...

accrual 3y ago
Relevant XKCD: "Every change breaks someone's workflow." https://xkcd.com/1172/ https://xkcd.com/1172/

kogepathic 3y ago

> The biggest customer threw a gigantic hysterical fit and practically demanded to return the old root password which they knew. We just shrugged and complied. It's not like it will be ours network which can be breached. If they want to operate insecurely, they are welcome. Did a PM or sales engineer from your org ever sit down with the customer to figure out what they were doing with root access that your solution lacked? My naive read here is that the customer saw it as easier to keep root access because your product didn't expose what they wanted or did it in a cumbersome way.

masklinn 3y ago
They were providing root either way, the only change I assume was switching from the same root on all machines to a per-machine root password. I can understand how that would be a pain in the ass if your legacy architecture (e.g. administration and reporting tooling, etc…) is predicated upon a unified root, you probably don’t even have a way to report the information so that machines and passwords could be paired. Assuming a good way to provision keys is provided, I’d make the insecure unified root password into a paid option (with lots of warnings) the price of which would increase every year. This way the existing workflow of the customer keeps working short-term, but you CY the hell out of your A in case of breach at the customer’s, and at one point the line item will get large enough that the customer allocates time & money to fix their shit.
- true_religion 3y ago
  Charging for insecure systems incentivizes selling those same systems to customers. Soon you’ll have sales guys regaling how this unified root password system is extremely convenient and you’re one of the few companies who provides this single-signature-security-system (S4 compliance). Next thing you know someone will gain an software patent on this to ensure no one else can provide it.
  - perfmode 3y ago
    But charging for it also disincentivizes customers from asking for it.

mrtksn 3y ago

What’s their complaint? Do they have a workflow that requires the root password all the time and it creates too much of a management to not have QWERTY1234 as root password?

throw0101c 3y ago

You could use the excuse that you are complying with the law per "California passes law that bans default passwords in connected devices": * https://techcrunch.com/2018/10/05/california-passes-law-that-bans-default-passwords-in-connected-devices/?guccounter=1 https://techcrunch.com/2018/10/05/california-passes-law-that...

hhh 3y ago
Is complying with the law an excuse these days?
- bigbillheck 3y ago
  Always has been.
Yizahi 3y ago
Oh, I'm pretty sure we can just say whatever, even "we have decided and it will be so". There is only a tiny problem when a customer is bigger (and richer) than the vendor by an order of magnitude or more :) . They tend to control a lot, with such power imbalance.
- hirsin 3y ago
  Honestly doesn't get better in the reverse either. At Microsoft I'd sit in on calls with banks and other "big names", watching them pound the table and argue their way, usually involving something dangerous and borderline malicious use of the product, all the while thinking "we could buy your company with our cash on hand". I think the key was that they're used to being the big dog in the room, and Microsoft made a point of not correcting them when that behavior was seen.

TheLoafOfBread 3y ago

Or instead of changing out of the blue what exists today and is probably soaked into whole following ecosystem around the product and would be massive PITA to change it as well, you could have create parallel way for login (i.e. via certificate) and tell customer that password access will be deprecated in 1-2 years, so they have time to update their tooling.

trustingtrust 3y ago

Can confirm the hardware vendor that I work for does the same. root password is given and customers are asked to change. No one wants to. Heck I would prefer if they disabled root login completely and used keys instead. But thats now it is. Some customers even enable login from web with those passwords. The stupidity is crazy.

ajsnigrutin 3y ago

Meh, just prompt them to set/change it the first time they boot, and after that, it's their fault :)

Pathogen-David 3y ago
Yeah, this is the only real solution.

segmondy 3y ago

You are conflating two separate types of passwords. Hardcoded password is different from default passwords. You can't change a hardcoded password, it's always part of the source code/image. A default password can be changed. Both can be backdoors if kept secret, but a hardcoded password from the onset is a backdoor.

mecsred 3y ago
Do you mean can't be changed on a live system? It's very easy to change a hard-coded password, just recompile the source and flash. That could definitely be what OP is referring to if they pushed OTA firmware upgrades.

NoMoreNicksLeft 3y ago

Wait. Do I understand this correctly? Are you saying you removed that root password, such that it wasn't possible to log in with another (but different) root password? Or are you saying that you merely changed it, to one the customer no longer knew, but was the same password across all of your install base, for every customer? How would that be any improvement, if the latter?

Yizahi 3y ago
It was at that moment a simple change of the password. The benefit was that people outside of our company wouldn't have access (thus not sharing it further), and then as a future improvement we could have implemented some better key scheme. But while customers want to have root access it is not an option. Theoretically we could implement a scheme with individual passwords or keys per device (as someone suggested in the comments above), but such thing is not being prioritized by our company, so that's our fault.
- NoMoreNicksLeft 3y ago
  Jesus fuck. This isn't a Latvian company, by chance, is it? Blink twice if yes.
- pabs3 3y ago
  The right scheme would be to require the user to set their own password on first login, and require that to happen before the device would work.

throwawaaarrgh 3y ago

In this case, only one version out of three (the middle one) had this vuln, so it's very likely it was introduced by mistake. But your scenario happens all the time too.

chasd00 3y ago

I hope you guys got an indemnification agreement or something else that make you not liable when the inevitable happens but, knowing how these things work, I doubt it.

kermatt 3y ago

Unfortunately folding to an unreasonable (in terms of security) request like this emboldens them to be stupid in the future as well. The customer is usually wrong.

codedokode 3y ago

> practically demanded to return the old root password You could set that password just for that one customer, and there is no problem.

phendrenad2 3y ago

So you are intentionally including a secret backdoor in your software that is only known to some customers? Yikes!