6 ms·

Amazing that even within the last decade a site as large as LinkedIn could be storing unsalted passwords. How does anyone fail at this in the modern era?

mschuster91 1y ago
> How does anyone fail at this in the modern era? Most probably some ancient legacy mainframe or whatnot other integration that nobody really has the time and budget to clean up and migrate to something more modern. The larger the company, the larger the risk for ossification of anything deemed "business critical" because even a minuscule outage of one hour now is six if not seven figures worth of "lost" time.
- fwip 1y ago
  LinkedIn isn't old enough to have anything ancient. It was launched in 2003, and even then you'd get laughed at for suggesting storing passwords in plaintext.
  - Sohcahtoa82 1y ago
    Plaintext, sure, but it was certainly common still to use SHA-256 which is very quickly cracked if your password is short.
  - bongodongobob 1y ago
    Doesn't mean that the infra is still ancient. What I see a lot is tech debt from migrations. Lots of times both the old and new systems have to work together for a period of time, so you leave certain legacy protocols and flags in place for the transition period and then the new system is never fully "updated" to the new standards. Pre win2k AD, file path lengths, encryption protocols, etc etc. Sure, the new system is "up to date" but the old compatibility settings remain.
    - malfist 1y ago
      This is also how feature flag services become mission critical because everything gets launched behind feature flags that never get cleaned up
korm 1y ago
They must have not asked enough Leetcode Hard questions in interviews.
- Svoka 1y ago
  I am stealing this. Made my day :)
hoseyor 1y ago
For all the talk of AI Slop, I don’t hear much about the fact that we have been suffering from Outsourced Slop for decades now. I suspect that is how this kind of thing also fail at LinkedIn. I say that based on my experience dealing with outsourcing companies and the product they produce through outsourced programmers. It’s really just been a similar problem as with AI code, that without strong and competent management that can set intelligent expectations and requirements and test for them, you will surely get what appears to all the business and leadership types like an equivalent product, without any sense that it’s slop underneath the surface.
- deleted 1y ago
  [deleted]
- sally_glance 1y ago
  I'm on board with the cheap offshore and bad incentives motiv, but feel this has to be augmented with a mention of the senior cowboy coder (who just went into retirement). Most likely in the future these stereotypes will be joined by vibe coders and AI-powered juniors, but as someone working this industry for a couple of decades give or take - we've learned how to deal with these by now.
  - ohhnoodont 1y ago
    > the senior cowboy coder (who just went into retirement) They just went into retirement?
- ohhnoodont 1y ago
  I've seen coworkers at Big Tech Co™ make huge security blunders despite attending prestigious universities (Berkeley, Stanford, etc) and having 5+ years of industry experience. No LLM slop required. Just rushing to meet deadlines while requirements shift rapidly enough that details get overlooked.
miki123211 1y ago
It's actually really easy to do unintentionally. For an intervening middleware, a password field in a JSON object is just like any other field in a JSON object. You may have some kind of logging / tracking / analytics somewhere that logs request bodies. You don't even have to engage in marketing shenanigans for that to be a problem, an abuse prevention system (which is definitely a necessity at their scale) is enough. Storing unsalted passwords in the "passwords database" is uncommon. Storing request logs from e.g. the Android app's API gateway, and forgetting to mark the `password` field in the forgot password flow as sensitive? Not so uncommon.
- jeffparsons 1y ago
  A company as big as LinkedIn should have bots continually accessing their site with unique generated passwords etc., and then be searching for those secrets in logging pipelines, bytes on disk, etc. to see where they get leaked. I know much smaller companies that do this. Yes, it's easy to fuck up. But a responsible company implements mitigations. And LinkedIn can absolutely afford to do much more.
  - knowitnone 1y ago
    that would require hiring a security personnel which they can't afford to do. /s
    - _heimdall 1y ago
      I think the new approach is to "hire" LLM agents to do the job, unless the hiring manager can prove they exhausted all ways an LLM could possibly have done the task.
  - some_random 1y ago
    There are so many things that companies as big as linkedin should be doing but aren't :(
  - marcus_holmes 1y ago
    I always picture a random middle manager in $large_organisation being told about something like this, and then they work out the angles and try to find the benefit. If the method works, and it shows that the logging feature Fred got so much credit for is storing passwords, what are the political implications of that? Can our intrepid middle manager steal some of Fred's glory? Or is Fred an ally and it should be carefully handled? Or do they sit on it and wait until an opportune moment to destroy Fred? This is the kind of reasoning process I think goes on, because I've seen very few large organisations make actually-good technical decisions.
paranoidrobot 1y ago
LinkedIn at one point were continually pressuring people into handing over their email credentials in the name of making it easy to find your contacts. So yeah, LinkedIn have never been exactly a bastion of IT Security.
- dewey 1y ago
  They (and the users) have a very real use case for that, just like a contacts app needs all of that. The problem is not keeping it safe.
  - rpigab 1y ago
    No user ever had a real use case for seeing a button that says "invite X" that doesn't send an invite on the platform, but instead sends an email to X who doesn't have a Linkedin account. And if you decline, it asks you again. Two times using different wording.
    - dewey 1y ago
      You'll be surprised how many features "tech" people think nobody uses (Like a share button on a website), are actually very popular. That's likely the reason that feature still exists as everything is most likely A/B tested to death. I was not only talking about that though, but also that they can build shadow profiles and recommend people to you that way.
DaSHacka 1y ago
Same company that requires you upload a biometric scan of your face paired with your passport for ""verification"" (despite not needing it on signup) if you want to enable MFA, btw ;-) On a related note, I no longer have an active linkedin account.
qingcharles 1y ago
I worked for a company with millions of users that had plaintext passwords in the DB. The login had been rolled from scratch in the days before you could get decent, tested off-the-shelf code for their particular stack. There were always so many fires to put out and projects to keep the wages being paid that it never got looked at. It got bought by Microsoft and eventually they just consumed the whole thing somehow, so it's gone now. It did allow me to cheekily run a SQL GROUP BY once to see what the most common passwords were, though. Top password was actually "trustno1" IIRC, followed by all the usual suspects, e.g. abcdefg, 12345678 etc. (there were no meaningful password rules)
nedt 1y ago
IIRC linkedin was one of the breaches where I got a spam email to my linkedin address, told them and they were like "can't be us - must be you who has been hacked". And then later "ah yeah was us, but no personal data was stolen". Like email address is not personal - lucky me for having a catch all domain and being able to just block the address I had used with linkedin.

mschuster91 1y ago

> How does anyone fail at this in the modern era? Most probably some ancient legacy mainframe or whatnot other integration that nobody really has the time and budget to clean up and migrate to something more modern. The larger the company, the larger the risk for ossification of anything deemed "business critical" because even a minuscule outage of one hour now is six if not seven figures worth of "lost" time.

fwip 1y ago
LinkedIn isn't old enough to have anything ancient. It was launched in 2003, and even then you'd get laughed at for suggesting storing passwords in plaintext.
- Sohcahtoa82 1y ago
  Plaintext, sure, but it was certainly common still to use SHA-256 which is very quickly cracked if your password is short.
- bongodongobob 1y ago
  Doesn't mean that the infra is still ancient. What I see a lot is tech debt from migrations. Lots of times both the old and new systems have to work together for a period of time, so you leave certain legacy protocols and flags in place for the transition period and then the new system is never fully "updated" to the new standards. Pre win2k AD, file path lengths, encryption protocols, etc etc. Sure, the new system is "up to date" but the old compatibility settings remain.
  - malfist 1y ago
    This is also how feature flag services become mission critical because everything gets launched behind feature flags that never get cleaned up

korm 1y ago

They must have not asked enough Leetcode Hard questions in interviews.

Svoka 1y ago
I am stealing this. Made my day :)

hoseyor 1y ago

For all the talk of AI Slop, I don’t hear much about the fact that we have been suffering from Outsourced Slop for decades now. I suspect that is how this kind of thing also fail at LinkedIn. I say that based on my experience dealing with outsourcing companies and the product they produce through outsourced programmers. It’s really just been a similar problem as with AI code, that without strong and competent management that can set intelligent expectations and requirements and test for them, you will surely get what appears to all the business and leadership types like an equivalent product, without any sense that it’s slop underneath the surface.

deleted 1y ago
[deleted]
sally_glance 1y ago
I'm on board with the cheap offshore and bad incentives motiv, but feel this has to be augmented with a mention of the senior cowboy coder (who just went into retirement). Most likely in the future these stereotypes will be joined by vibe coders and AI-powered juniors, but as someone working this industry for a couple of decades give or take - we've learned how to deal with these by now.
- ohhnoodont 1y ago
  > the senior cowboy coder (who just went into retirement) They just went into retirement?
ohhnoodont 1y ago
I've seen coworkers at Big Tech Co™ make huge security blunders despite attending prestigious universities (Berkeley, Stanford, etc) and having 5+ years of industry experience. No LLM slop required. Just rushing to meet deadlines while requirements shift rapidly enough that details get overlooked.

miki123211 1y ago

It's actually really easy to do unintentionally. For an intervening middleware, a password field in a JSON object is just like any other field in a JSON object. You may have some kind of logging / tracking / analytics somewhere that logs request bodies. You don't even have to engage in marketing shenanigans for that to be a problem, an abuse prevention system (which is definitely a necessity at their scale) is enough. Storing unsalted passwords in the "passwords database" is uncommon. Storing request logs from e.g. the Android app's API gateway, and forgetting to mark the `password` field in the forgot password flow as sensitive? Not so uncommon.

jeffparsons 1y ago
A company as big as LinkedIn should have bots continually accessing their site with unique generated passwords etc., and then be searching for those secrets in logging pipelines, bytes on disk, etc. to see where they get leaked. I know much smaller companies that do this. Yes, it's easy to fuck up. But a responsible company implements mitigations. And LinkedIn can absolutely afford to do much more.
- knowitnone 1y ago
  that would require hiring a security personnel which they can't afford to do. /s
  - _heimdall 1y ago
    I think the new approach is to "hire" LLM agents to do the job, unless the hiring manager can prove they exhausted all ways an LLM could possibly have done the task.
- some_random 1y ago
  There are so many things that companies as big as linkedin should be doing but aren't :(
- marcus_holmes 1y ago
  I always picture a random middle manager in $large_organisation being told about something like this, and then they work out the angles and try to find the benefit. If the method works, and it shows that the logging feature Fred got so much credit for is storing passwords, what are the political implications of that? Can our intrepid middle manager steal some of Fred's glory? Or is Fred an ally and it should be carefully handled? Or do they sit on it and wait until an opportune moment to destroy Fred? This is the kind of reasoning process I think goes on, because I've seen very few large organisations make actually-good technical decisions.

paranoidrobot 1y ago

LinkedIn at one point were continually pressuring people into handing over their email credentials in the name of making it easy to find your contacts. So yeah, LinkedIn have never been exactly a bastion of IT Security.

dewey 1y ago
They (and the users) have a very real use case for that, just like a contacts app needs all of that. The problem is not keeping it safe.
- rpigab 1y ago
  No user ever had a real use case for seeing a button that says "invite X" that doesn't send an invite on the platform, but instead sends an email to X who doesn't have a Linkedin account. And if you decline, it asks you again. Two times using different wording.
  - dewey 1y ago
    You'll be surprised how many features "tech" people think nobody uses (Like a share button on a website), are actually very popular. That's likely the reason that feature still exists as everything is most likely A/B tested to death. I was not only talking about that though, but also that they can build shadow profiles and recommend people to you that way.

DaSHacka 1y ago

Same company that requires you upload a biometric scan of your face paired with your passport for ""verification"" (despite not needing it on signup) if you want to enable MFA, btw ;-) On a related note, I no longer have an active linkedin account.

qingcharles 1y ago

I worked for a company with millions of users that had plaintext passwords in the DB. The login had been rolled from scratch in the days before you could get decent, tested off-the-shelf code for their particular stack. There were always so many fires to put out and projects to keep the wages being paid that it never got looked at. It got bought by Microsoft and eventually they just consumed the whole thing somehow, so it's gone now. It did allow me to cheekily run a SQL GROUP BY once to see what the most common passwords were, though. Top password was actually "trustno1" IIRC, followed by all the usual suspects, e.g. abcdefg, 12345678 etc. (there were no meaningful password rules)

nedt 1y ago

IIRC linkedin was one of the breaches where I got a spam email to my linkedin address, told them and they were like "can't be us - must be you who has been hacked". And then later "ah yeah was us, but no personal data was stolen". Like email address is not personal - lucky me for having a catch all domain and being able to just block the address I had used with linkedin.