4 ms·

I really don't get how companies so large do stupid things like this. Hashing and salting passwords isn't some newly introduced advanced rocket science, it's l

I really don't get how companies so large do stupid things like this. Hashing and salting passwords isn't some newly introduced advanced rocket science, it's literally a 101-level "obvious" thing. How can a huge corporation like Meta/Facebook can do this is beyond my imagination.

progbits 2y ago
It's unbelievable how little most developers care about security. At this point I've given up on educating them since it went nowhere, instead I'm locking down permissions to things like firewall and secret vault so random people don't fuck it up.
- wahnfrieden 2y ago
  Their managers don't incentivize them spending time on it, and their PM will fight security tickets they don't understand the need for. Most devs have little autonomy at orgs today and operate under a strict hierarchy of command at the ticket level.
  - pbhjpbhj 2y ago
    Presumably, "we're been storing 600 Million passwords in plaintext" is understandable to their PMs given its understandable to complete laymen. Aren't FAANG companies supposed to employ the very brightest minds. Hard to imagine this wasn't done on purpose.
- throwawaysleep 2y ago
  Using the terms, wallet, threat, or prestige, make me as a non aultristic person care about the security of the systems my employer owns.
- radicalbyte 2y ago
  I don't see that in capable developers (who usually end up pushed to "backend") but it is absolutely endemic in frontend and extremely problematic in organisations which have full stack created frontend-first. All too often even the super seniors / leads have a very limit knowledge of security (or performance, reliability etc).
  - deleted 2y ago
    [deleted]
Negitivefrags 2y ago
The usual way this happens is accidentally logging passwords. Or even other cases where passwords happen to be included in something else. It can happen more easily than you think. Like for example, if you collect server side crash dumps, are you really taking care that there is no sensitive information sitting in the memory image stored in them?
- grayhatter 2y ago
  the word you should use is carelessly, as in carelessly logging passwords. When working with data that you can reasonably expect to contain secrets, you should behave as if it does contain secrets. It worries me that you mention you're aware that server side crash dumps may contain sensitive data, but you also speak as if it's reasonable to not protect them knowing they do, or they might. I'd hope or expect anyone would mention or at least imply that it'd be negligent to behave so recklessly with someone else's secrets.
  - bongodongobob 2y ago
    What you're saying makes sense for a small startup with 20 people. Have you worked in an org of 10000+ that's been around for 30 years? Where the people who built the systems no longer work there? Where the people who maintain them are 3+ career cycles removed? Things get so baked in "one doesn't simply make changes" because no one person understands how everything works, so you do your best and keep closing tickets. "Hey we should really investigate whether or not passwords are included in memory in our 2000 severs across the globe!" "Lol ok maybe when you close out the 30 tickets on your plate."
    - grayhatter 2y ago
      > Have you worked in an org of 10000+ that's been around for 30 years? yes. Would you still make this argument for something else known to be dangerous? Hey this gigantic lathe doesn't have a safety shut off switch. Ahh yeah the guy that did the wiring for that left years ago so we just don't touch it. It's fine as long as no one puts their hands near it when it's running. Hey, isn't this freshly dug 20 foot hole supposed to have an escape route, and side wall reinforcements so it doesn't collapse on someone? lol ok new guy, let me know when you've finished pouring that concrete and then we'll look at that idea. To go back to the example; Taking steps to protect them could be as simple as restricting who can access core dumps, enforcing they don't get stored unencrypted, and the only people who can copy and inspect them have are already in a trusted role where they can get root on that production server. Or an even better option would be restricting the service that can read clear text passwords. These machines don't crash, and when they do, we don't write a coredump. (but this trusted team can start if we see the system crashing suddenly) > Things get so baked in "one doesn't simply make changes" because no one person understands how everything works, so you do your best and keep closing tickets. This is a super disappointing attitude. It's hard, and it's the way we've always done it, so that means it's impossible, or not worth the effort? Yeah, I'm not likely to buy into that mentality. I believe I can fix things and have a positive impact. It's one thing to say it's impossible because you don't understand how to do it, that's wrong, but I guess if that's what you need to believe to sleep at night, I can pretend to understand. It's another thing entirely if you don't have the autonomy at work to try to improve something you know to be broken, and a risk to users, and the company. If that's actually what you meant to describe, you might wanna consider if you can find another job. You're clearly not an idiot, and there's plenty of companies that wont treat you like a code monkey.
rjzzleep 2y ago
Most of the times it's when people reinvent stuff from scratch. That's why I'm always careful when people implement their own framework in the next shiny thing. Lack of hashing, CSRF, basic XSS issues happen all the time in rust, golang and especially the node.js web crowd.
bongodongobob 2y ago
I'm honestly never surprised by any of this stuff. I've done some contracting and file access is always a shitshow. Picture this: > Intern or contractor gets hired. > Someone runs a script to create the user because permissions have turned into a rats nest that no human can understand > No one knows how the script works anymore, it's probably outdated and only does 60% of the job > User is added to a quagmire of groups > I cant access the Citrix apps > "Oh you don't have access and I'm waiting on that app owner to give you access, give me a few days" > Ok it works but I don't have a login for "the program I need" > "Looks like there's a licensing issue, give me a few days" > Ok it works but I don't have credentials for the database connection > "It's legitimately complicated, give me a few days" > IT gets tired of fucking with the back and forth on this 13 day old ticket "Just give him everything, so he can get X done" > Get access > Finally. Let's just dump this out into another file/folder/location so I don't have to go through hoops again and I can actually do my job. Now that dump is in a remote profile folder or /temp folder somewhere in a 100TB blob that will be backed up for 7 years or more. I've been on both sides.
- ChoHag 2y ago
  [dead]
- bongodongobob 2y ago
  To the comment below about measuring security... Usually this comes from putting security theater over pragmatism. Everything is so granular it's impossible to figure out what people need so you fuck around with it for a day and then give up and just give em local admin because actual work needs to be done. You can't get away from the fact that work requires write access. All those groups and policies are meaningless when the rubber hits the road. In the end you just have to hire people who can be trusted to do the right thing and not burn the house down.
tjpnz 2y ago
When you select candidates based on whether they know how to invert a BST and other trivia it's not terribly surprisingly.
- FartyMcFarter 2y ago
  How would you select candidates to make sure they avoid this kind of security bug?
  - Arbortheus 2y ago
    “You are building an account authentication system and need to store user passwords, what design measures would you take to ensure this would be done securely?” Follow up question. “What measures would you take to ensure this complies with relevant data protection regulations?” If they give a somewhat competent answer, you at least know they’re the sort of developer that can critically analyse this sort of thing.
    - tjpnz 2y ago
      >What measures would you take to ensure this complies with relevant data protection regulations? We hire devs to work on safety critical systems and do gauge for familiarity with standards applicable to our industry. This happens regardless of whether you're mid career or a new grad. I don't think it unreasonable to expect similar of devs working on systems where there are security or privacy considerations (which might as well be everything).
  - deleted 2y ago
    [deleted]
  - pbhjpbhj 2y ago
    Avoiding the bug in the first place is not the big issue, surely. An employee can leave a door open to a secure area, if you employ security people they should have worked to mitigate that, and to catch it if it did happen. Well "we want to run an audit to find if any passwords are stored in plaintext in our file systems". Sounds like a problem any highschooler could answer? The idea of an audit might need a person who has done a remedial level of security work.
vanjajaja1 2y ago
it happens because you have one component logging everything for traceability, and it hosts the interconnect between 2 other components which need to communicate passwords. thus the password accidentally slips into a plain text log file generally there are tools that search for & flag PII logging, if it slips through the tools its because there are layers of indirection involved
fulafel 2y ago
Big organizations just have a lot of bureucracy attempting to codify common sense. I bet they have paper saying you shouldn't do this.
herval 2y ago
An accidental print statement on the login page. That’s all it takes.

progbits 2y ago

It's unbelievable how little most developers care about security. At this point I've given up on educating them since it went nowhere, instead I'm locking down permissions to things like firewall and secret vault so random people don't fuck it up.

wahnfrieden 2y ago
Their managers don't incentivize them spending time on it, and their PM will fight security tickets they don't understand the need for. Most devs have little autonomy at orgs today and operate under a strict hierarchy of command at the ticket level.
- pbhjpbhj 2y ago
  Presumably, "we're been storing 600 Million passwords in plaintext" is understandable to their PMs given its understandable to complete laymen. Aren't FAANG companies supposed to employ the very brightest minds. Hard to imagine this wasn't done on purpose.
throwawaysleep 2y ago
Using the terms, wallet, threat, or prestige, make me as a non aultristic person care about the security of the systems my employer owns.
radicalbyte 2y ago
I don't see that in capable developers (who usually end up pushed to "backend") but it is absolutely endemic in frontend and extremely problematic in organisations which have full stack created frontend-first. All too often even the super seniors / leads have a very limit knowledge of security (or performance, reliability etc).
- deleted 2y ago
  [deleted]

Negitivefrags 2y ago

The usual way this happens is accidentally logging passwords. Or even other cases where passwords happen to be included in something else. It can happen more easily than you think. Like for example, if you collect server side crash dumps, are you really taking care that there is no sensitive information sitting in the memory image stored in them?

grayhatter 2y ago
the word you should use is carelessly, as in carelessly logging passwords. When working with data that you can reasonably expect to contain secrets, you should behave as if it does contain secrets. It worries me that you mention you're aware that server side crash dumps may contain sensitive data, but you also speak as if it's reasonable to not protect them knowing they do, or they might. I'd hope or expect anyone would mention or at least imply that it'd be negligent to behave so recklessly with someone else's secrets.
- bongodongobob 2y ago
  What you're saying makes sense for a small startup with 20 people. Have you worked in an org of 10000+ that's been around for 30 years? Where the people who built the systems no longer work there? Where the people who maintain them are 3+ career cycles removed? Things get so baked in "one doesn't simply make changes" because no one person understands how everything works, so you do your best and keep closing tickets. "Hey we should really investigate whether or not passwords are included in memory in our 2000 severs across the globe!" "Lol ok maybe when you close out the 30 tickets on your plate."
  - grayhatter 2y ago
    > Have you worked in an org of 10000+ that's been around for 30 years? yes. Would you still make this argument for something else known to be dangerous? Hey this gigantic lathe doesn't have a safety shut off switch. Ahh yeah the guy that did the wiring for that left years ago so we just don't touch it. It's fine as long as no one puts their hands near it when it's running. Hey, isn't this freshly dug 20 foot hole supposed to have an escape route, and side wall reinforcements so it doesn't collapse on someone? lol ok new guy, let me know when you've finished pouring that concrete and then we'll look at that idea. To go back to the example; Taking steps to protect them could be as simple as restricting who can access core dumps, enforcing they don't get stored unencrypted, and the only people who can copy and inspect them have are already in a trusted role where they can get root on that production server. Or an even better option would be restricting the service that can read clear text passwords. These machines don't crash, and when they do, we don't write a coredump. (but this trusted team can start if we see the system crashing suddenly) > Things get so baked in "one doesn't simply make changes" because no one person understands how everything works, so you do your best and keep closing tickets. This is a super disappointing attitude. It's hard, and it's the way we've always done it, so that means it's impossible, or not worth the effort? Yeah, I'm not likely to buy into that mentality. I believe I can fix things and have a positive impact. It's one thing to say it's impossible because you don't understand how to do it, that's wrong, but I guess if that's what you need to believe to sleep at night, I can pretend to understand. It's another thing entirely if you don't have the autonomy at work to try to improve something you know to be broken, and a risk to users, and the company. If that's actually what you meant to describe, you might wanna consider if you can find another job. You're clearly not an idiot, and there's plenty of companies that wont treat you like a code monkey.

rjzzleep 2y ago

Most of the times it's when people reinvent stuff from scratch. That's why I'm always careful when people implement their own framework in the next shiny thing. Lack of hashing, CSRF, basic XSS issues happen all the time in rust, golang and especially the node.js web crowd.

bongodongobob 2y ago

I'm honestly never surprised by any of this stuff. I've done some contracting and file access is always a shitshow. Picture this: > Intern or contractor gets hired. > Someone runs a script to create the user because permissions have turned into a rats nest that no human can understand > No one knows how the script works anymore, it's probably outdated and only does 60% of the job > User is added to a quagmire of groups > I cant access the Citrix apps > "Oh you don't have access and I'm waiting on that app owner to give you access, give me a few days" > Ok it works but I don't have a login for "the program I need" > "Looks like there's a licensing issue, give me a few days" > Ok it works but I don't have credentials for the database connection > "It's legitimately complicated, give me a few days" > IT gets tired of fucking with the back and forth on this 13 day old ticket "Just give him everything, so he can get X done" > Get access > Finally. Let's just dump this out into another file/folder/location so I don't have to go through hoops again and I can actually do my job. Now that dump is in a remote profile folder or /temp folder somewhere in a 100TB blob that will be backed up for 7 years or more. I've been on both sides.

ChoHag 2y ago
[dead]
bongodongobob 2y ago
To the comment below about measuring security... Usually this comes from putting security theater over pragmatism. Everything is so granular it's impossible to figure out what people need so you fuck around with it for a day and then give up and just give em local admin because actual work needs to be done. You can't get away from the fact that work requires write access. All those groups and policies are meaningless when the rubber hits the road. In the end you just have to hire people who can be trusted to do the right thing and not burn the house down.

tjpnz 2y ago

When you select candidates based on whether they know how to invert a BST and other trivia it's not terribly surprisingly.

FartyMcFarter 2y ago
How would you select candidates to make sure they avoid this kind of security bug?
- Arbortheus 2y ago
  “You are building an account authentication system and need to store user passwords, what design measures would you take to ensure this would be done securely?” Follow up question. “What measures would you take to ensure this complies with relevant data protection regulations?” If they give a somewhat competent answer, you at least know they’re the sort of developer that can critically analyse this sort of thing.
  - tjpnz 2y ago
    >What measures would you take to ensure this complies with relevant data protection regulations? We hire devs to work on safety critical systems and do gauge for familiarity with standards applicable to our industry. This happens regardless of whether you're mid career or a new grad. I don't think it unreasonable to expect similar of devs working on systems where there are security or privacy considerations (which might as well be everything).
- deleted 2y ago
  [deleted]
- pbhjpbhj 2y ago
  Avoiding the bug in the first place is not the big issue, surely. An employee can leave a door open to a secure area, if you employ security people they should have worked to mitigate that, and to catch it if it did happen. Well "we want to run an audit to find if any passwords are stored in plaintext in our file systems". Sounds like a problem any highschooler could answer? The idea of an audit might need a person who has done a remedial level of security work.

vanjajaja1 2y ago

it happens because you have one component logging everything for traceability, and it hosts the interconnect between 2 other components which need to communicate passwords. thus the password accidentally slips into a plain text log file generally there are tools that search for & flag PII logging, if it slips through the tools its because there are layers of indirection involved

fulafel 2y ago

Big organizations just have a lot of bureucracy attempting to codify common sense. I bet they have paper saying you shouldn't do this.

herval 2y ago

An accidental print statement on the login page. That’s all it takes.