7 ms·

Bit.ly compromised – More Details

60 points by shdon 12y ago

dewey 12y ago
"If you registered, logged in or changed your password after January 8th, 2014, your password was converted to be hashed with BCrypt and HMAC using a unique salt. Before that, it was salted MD5." Are they saying the passwords were hashed with MD5 before January? Because that'd be a bit scary.
- camus2 12y ago
  You'd be surprised how much "PHP login" tutorial use md5 for password encryption. And then people are surprised these devs dont known better? it's because of all the mis-eductive content the web cant get rid of.
  - krapp 12y ago
    Also, many startups consider security to be a waste of time, since you don't need a secure site to generate revenue (arguably, appearing secure has more value in that sense than actually being secure.) Also I still see people now and then arguing that salted md5 is perfectly secure and that using a slower hash or any hash at all either doesn't add value or actually harms value. So they may not know better, or they may know better and simply not care.
    - balls187 12y ago
      > Also, many startups consider security to be a waste of time Do you have evidence to support this? I wouldn't say that security is considered a waste of time. It is, however is a trade-off[1] between money, time, and other resources, things that, generally speaking, start-ups tend to be short on. I worked for a startup that provided multifactor authentication using biometrics (amongst others), and almost all of our business came after publicized (and not publicized) breaches, from both large and small firms. [1]https://www.schneier.com/essay-155.html https://www.schneier.com/essay-155.html
      - krapp 12y ago
        We may be talking about the same phenomenon from different points of view. I have evidence but I really don't want to name names - but anecdotally, I have pointed out massive issues (like not even escaping user-provided variables in inline SQL queries) to clients, and seen it dismissed as less important than just getting live. Maybe my comment was a bit too biased and hyperbolic, but I've certainly encountered that attitude.
  - voltagex_ 12y ago
    I once registered wrongtutorial.com to call out incorrect and dangerous tutorials. I couldn't write enough content to get it off the ground. Do you think it'd still be a good idea?
    - nknighthb 12y ago
      A more general form of tutorial review/correction site would be more interesting. Actually, a browser add-on to load commentary/annotations for known tutorials on the fly might be nice. All sorts of practical problems that would have to be overcome, though (like changing content and getting people to use the add-on in the first place).
    - aaronem 12y ago
      > I couldn't write enough content to get it off the ground. Poke around w3schools. That should give you more than enough initial grist for your mill.
      - id 12y ago
        http://www.w3fools.com/ http://www.w3fools.com/
    - camus2 12y ago
      Definetly! I love this idea!
- codecondo 12y ago
  It's not scary, it's the truth, and it's bad. But, what can you do?
- Dylan16807 12y ago
  Salted MD5/SHA is basically the standard even though it's awful, and the flaws in MD5's algorithm don't really apply to the password case.
  - tptacek 12y ago
    The flaws in MD5 don't really apply to passwords, but the flaws in salted MD5/SHAx sure do.
- KMag 12y ago
  It's still a bit scary that they're apparently using some ad-hoc KDF (key derivation function) built from bcrypt and HMAC. bcrypt is a decent KDF, but there are several ways to make a bad KDF from a combination of bcrypt and HMAC. For instance, concatenating bcrypt(salt, pw) and HMAC-SHA512(salt, pw) makes guessing passwords way too fast. I'm hoping that all they've done is use bcrypt(salt, HMAC-SHA256(salt, pw)) to get around the bcrypt 72 character limit. However, using scrypt would have been better.
  - nialo 12y ago
    why does concatenating bcrypt(salt, pw) with HMAC(salt, pw) make guessing fast? Intuitively it seems like you'd still need to guess the bcrypt portion, making it no worse than just bcrypt.
    - euank 12y ago
      Why not just ignore the bcrypt portion? "Intuitively" it seems to me like you could just un-concatonate the two strings and guess at them separately, working on whichever you know to be easier.
  - reedloden 12y ago
    Hopefully, the 'salt' passed to HMAC-SHA256 is a pepper (preferably stored outside the database), not the same user salt as what bcrypt uses. As for scrypt, it definitely has great potential, but I still feel it's too new and not as well understood to be used 100% correctly in production by most folks (see also http://blog.ircmaxell.com/2014/03/why-i-dont-recommend-scrypt.html http://blog.ircmaxell.com/2014/03/why-i-dont-recommend-scryp...).
  - azinman2 12y ago
    What's wrong with bcrypt?
michaelfeathers 12y ago
I've never been clear about what value-add registration brings to a link-shortener, so I've just used bit.ly without registration. I'm kind of happy about that now.
- edraferi 12y ago
  it allows you to track stats more easily. for example, you can link bit.ly to Buffer and get a unique bit mark for each link/platform pair. then you can track engagement separately for each platform.
  - brownbat 12y ago
    Couldn't you just have an IFTTT pull anything you bitly into a bucket of links to watch stats on? Why does bitly need to know who you are, or what links you submitted, if you can just remember (maybe through automation) which links are yours, and bitly or buffer can monitor each link individually for stats? My guess is that logins probably help bitly track usage patterns, improve the product, and keep an eye out for ways to increase their value by measuring engagement with their userbase. Maybe it was just "everyone's adding a social layer" peer pressure. Not sure. There are security costs to "use passwords everywhere, for everything," but they are more diffuse. We've ended up with a system where security best practices would recommend using hundreds of distinct passwords that are each too complex to memorize (so you can dump them in a locker or support one login services, each creating a single point of failure, or you can use a similar password or mental password algorithm for each site, it's all tradeoffs). I don't think there's an obviously better solution, I think it's just a tragedy of the commons we were probably inevitably going to trip over sooner or later.
    - peteforde 12y ago
      What percentage of the internet would even understand what you just suggested, much less know how to do it? Also, you'd have to see a massive amount of value in not signing up to Bitly to offset the hours you'd spend creating a vastly inferior hack job on your own.
theboss 12y ago
"Security Team"?. I'm honestly curious. I've never seen startups that aren't security startups hiring a security team. Is this a real team or just their developers doing an extra job?
- bullseye 12y ago
  I don't fault them for it if they don't. "The Bitly security team learned of the potential compromise..." sounds much better than "our programmer, Joe, got an email..."
  - addandsubtract 12y ago
    Good guy Joe. Developer by day, security team by night.
- nknighthb 12y ago
  Preferably even the smallest company would have at least a designated "security guy", even if they don't spend most of their time directly on security issues. If there are two, there's your "security team". Reality, of course, is that most companies don't give much thought to security at all until it bites them in the ass, and security competency is rare enough that even a designated "security guy" could be worse than useless.
nknighthb 12y ago
Missing step: Asymmetric encryption of off-site backups with passphrase-protected private keys kept offline.
kator 12y ago
Is the root issue here that they stored remote server database passwords in their source code? I'm always hesitant to have source code with hard coded passwords, I know at some point you have to have them "somewhere" but anything you can do to reduce attack vectors for them is helpful. What do people think should be best practices for storing sensitive configuration data? I know some people will have just a config repo with very limited access but that still seems dangerous at scale when that ends up being a lot of people etc.
- nknighthb 12y ago
  I have yet to find any solution to this problem that I like. A separate config repo is least bad of the various bad options. The others have all been error-prone and failed to play nicely with version control. If someone could make git store all on-disk data in encrypted form with an ssh-agent-like solution for on-the-fly (de|en)cryption, that would seem ideal.
  - jarrett 12y ago
    > If someone could make git store all on-disk data in encrypted form with an ssh-agent-like solution for on-the-fly (de|en)cryption, that would seem ideal. You could probably roll your own. A decent middleground would be to keep the data in the central repo encrypted, but keep it decrypted on the engineers' workstations. Though a better approach is just to not check in sensitive values at all.
    - nknighthb 12y ago
      "Roll your own" is exactly what causes trouble. It's never right and always painful. We need a general, widely-vetted solution that doesn't make me want to yank my hair out. "Don't check in sensitive values" is not an option. Credentials must be kept track of and maintained somewhere. Storing them in an ad-hoc manner without version control doesn't just invite trouble, it demands it. We have password managers for desktops and mobile devices. We need an equivalent for servers that accounts for the fact that the credentials must be highly available, multiple people must be able to use them (whether or not they get to directly view them), some of them will be technicians with limited training following someone else's procedures, and regardless of knowledge and skill, there will be half-asleep people taking emergency actions at 3AM when shit has hit the fan.
druiid 12y ago
Chef supports encrypted data-buckets where you decrypt only on the server. Puppet supports hiera data with encryption as well (via plugins). There's no reason anyone should be storing password information in source, especially API keys which I'm guessing this was (S3 information more than likely?). I think about the extent of private information I'd be okay with storing in source is DB user/pass information because that will generally require compromising the machine more generally first and by that point they already had access to your DB even if the password/username wasn't stored in source. Essentially this will serve as a wake-up call to the Bitly guys and hopefully they'll take some steps to deploy this stuff via CM or similar in the future.
- thefreeman 12y ago
  I read it a bit differently. I don't think they store their credentials directly in their source code repository. I think they are saying they have a separate repository specifically for credentials. It seems like a bit of a strange way to manage sensitive information, but not really any worse then a lot of other things I've seen.
ForHackernews 12y ago
Link-shorteners are a cancer. We have enough of a problem with historical links going dead, without interposing a third party. When bit.ly goes down, what happens to all their links?
- helium 12y ago
  By that rationale, any website is a cancer.
- barsonme 12y ago
  True, but for example if I'm posting an article I wrote it's much more efficient to save space using my erclgrgn.us short link because it's usually ~20 chars which is less than half of what the full URL is. Unfortunately, sometimes brevity is a necessity. edit: original comment mentioned twitter, then I remembered twitter automatically reduces a URL's chars to 20 or so.
  - ForHackernews 12y ago
    Yeah, or websites could just allow actual links. I don't see why hackernews (and twitter, etc.) can't just render <a href="http://somereallylongdomain.com/stupidlylongpath/?what=thisisgettingridiculous">I'm http://somereallylongdomain.com/stupidlylongpath/?what=thisi... short!</a>
    - maxerickson 12y ago
      Because of griefer trolls. Forcing links to show the destination is basically 0 effort on the system side and pretty much completely undermines them. (that only answers the HN part, I'm guessing that is much of the reasoning, I don't know it for a fact)
tedd4u 12y ago
For security stuff, if you can't find a well-tested implementation to re-use, I always suggest referring to the OWASP Cheat Series [1] - there's a nice one just for password storage [2]. [1] https://www.owasp.org/index.php/OWASP_Cheat_Sheet_Series https://www.owasp.org/index.php/OWASP_Cheat_Sheet_Series [2] https://www.owasp.org/index.php/Password_Storage_Cheat_Sheet https://www.owasp.org/index.php/Password_Storage_Cheat_Sheet

60 points by shdon 12y ago

dewey 12y ago

"If you registered, logged in or changed your password after January 8th, 2014, your password was converted to be hashed with BCrypt and HMAC using a unique salt. Before that, it was salted MD5." Are they saying the passwords were hashed with MD5 before January? Because that'd be a bit scary.

camus2 12y ago
You'd be surprised how much "PHP login" tutorial use md5 for password encryption. And then people are surprised these devs dont known better? it's because of all the mis-eductive content the web cant get rid of.
- krapp 12y ago
  Also, many startups consider security to be a waste of time, since you don't need a secure site to generate revenue (arguably, appearing secure has more value in that sense than actually being secure.) Also I still see people now and then arguing that salted md5 is perfectly secure and that using a slower hash or any hash at all either doesn't add value or actually harms value. So they may not know better, or they may know better and simply not care.
  - balls187 12y ago
    > Also, many startups consider security to be a waste of time Do you have evidence to support this? I wouldn't say that security is considered a waste of time. It is, however is a trade-off[1] between money, time, and other resources, things that, generally speaking, start-ups tend to be short on. I worked for a startup that provided multifactor authentication using biometrics (amongst others), and almost all of our business came after publicized (and not publicized) breaches, from both large and small firms. [1]https://www.schneier.com/essay-155.html https://www.schneier.com/essay-155.html
    - krapp 12y ago
      We may be talking about the same phenomenon from different points of view. I have evidence but I really don't want to name names - but anecdotally, I have pointed out massive issues (like not even escaping user-provided variables in inline SQL queries) to clients, and seen it dismissed as less important than just getting live. Maybe my comment was a bit too biased and hyperbolic, but I've certainly encountered that attitude.
- voltagex_ 12y ago
  I once registered wrongtutorial.com to call out incorrect and dangerous tutorials. I couldn't write enough content to get it off the ground. Do you think it'd still be a good idea?
  - nknighthb 12y ago
    A more general form of tutorial review/correction site would be more interesting. Actually, a browser add-on to load commentary/annotations for known tutorials on the fly might be nice. All sorts of practical problems that would have to be overcome, though (like changing content and getting people to use the add-on in the first place).
  - aaronem 12y ago
    > I couldn't write enough content to get it off the ground. Poke around w3schools. That should give you more than enough initial grist for your mill.
    - id 12y ago
      http://www.w3fools.com/ http://www.w3fools.com/
  - camus2 12y ago
    Definetly! I love this idea!
codecondo 12y ago
It's not scary, it's the truth, and it's bad. But, what can you do?
Dylan16807 12y ago
Salted MD5/SHA is basically the standard even though it's awful, and the flaws in MD5's algorithm don't really apply to the password case.
- tptacek 12y ago
  The flaws in MD5 don't really apply to passwords, but the flaws in salted MD5/SHAx sure do.
KMag 12y ago
It's still a bit scary that they're apparently using some ad-hoc KDF (key derivation function) built from bcrypt and HMAC. bcrypt is a decent KDF, but there are several ways to make a bad KDF from a combination of bcrypt and HMAC. For instance, concatenating bcrypt(salt, pw) and HMAC-SHA512(salt, pw) makes guessing passwords way too fast. I'm hoping that all they've done is use bcrypt(salt, HMAC-SHA256(salt, pw)) to get around the bcrypt 72 character limit. However, using scrypt would have been better.
- nialo 12y ago
  why does concatenating bcrypt(salt, pw) with HMAC(salt, pw) make guessing fast? Intuitively it seems like you'd still need to guess the bcrypt portion, making it no worse than just bcrypt.
  - euank 12y ago
    Why not just ignore the bcrypt portion? "Intuitively" it seems to me like you could just un-concatonate the two strings and guess at them separately, working on whichever you know to be easier.
- reedloden 12y ago
  Hopefully, the 'salt' passed to HMAC-SHA256 is a pepper (preferably stored outside the database), not the same user salt as what bcrypt uses. As for scrypt, it definitely has great potential, but I still feel it's too new and not as well understood to be used 100% correctly in production by most folks (see also http://blog.ircmaxell.com/2014/03/why-i-dont-recommend-scrypt.html http://blog.ircmaxell.com/2014/03/why-i-dont-recommend-scryp...).
- azinman2 12y ago
  What's wrong with bcrypt?

michaelfeathers 12y ago

I've never been clear about what value-add registration brings to a link-shortener, so I've just used bit.ly without registration. I'm kind of happy about that now.

edraferi 12y ago
it allows you to track stats more easily. for example, you can link bit.ly to Buffer and get a unique bit mark for each link/platform pair. then you can track engagement separately for each platform.
- brownbat 12y ago
  Couldn't you just have an IFTTT pull anything you bitly into a bucket of links to watch stats on? Why does bitly need to know who you are, or what links you submitted, if you can just remember (maybe through automation) which links are yours, and bitly or buffer can monitor each link individually for stats? My guess is that logins probably help bitly track usage patterns, improve the product, and keep an eye out for ways to increase their value by measuring engagement with their userbase. Maybe it was just "everyone's adding a social layer" peer pressure. Not sure. There are security costs to "use passwords everywhere, for everything," but they are more diffuse. We've ended up with a system where security best practices would recommend using hundreds of distinct passwords that are each too complex to memorize (so you can dump them in a locker or support one login services, each creating a single point of failure, or you can use a similar password or mental password algorithm for each site, it's all tradeoffs). I don't think there's an obviously better solution, I think it's just a tragedy of the commons we were probably inevitably going to trip over sooner or later.
  - peteforde 12y ago
    What percentage of the internet would even understand what you just suggested, much less know how to do it? Also, you'd have to see a massive amount of value in not signing up to Bitly to offset the hours you'd spend creating a vastly inferior hack job on your own.

theboss 12y ago

"Security Team"?. I'm honestly curious. I've never seen startups that aren't security startups hiring a security team. Is this a real team or just their developers doing an extra job?

bullseye 12y ago
I don't fault them for it if they don't. "The Bitly security team learned of the potential compromise..." sounds much better than "our programmer, Joe, got an email..."
- addandsubtract 12y ago
  Good guy Joe. Developer by day, security team by night.
nknighthb 12y ago
Preferably even the smallest company would have at least a designated "security guy", even if they don't spend most of their time directly on security issues. If there are two, there's your "security team". Reality, of course, is that most companies don't give much thought to security at all until it bites them in the ass, and security competency is rare enough that even a designated "security guy" could be worse than useless.

nknighthb 12y ago

Missing step: Asymmetric encryption of off-site backups with passphrase-protected private keys kept offline.

kator 12y ago

Is the root issue here that they stored remote server database passwords in their source code? I'm always hesitant to have source code with hard coded passwords, I know at some point you have to have them "somewhere" but anything you can do to reduce attack vectors for them is helpful. What do people think should be best practices for storing sensitive configuration data? I know some people will have just a config repo with very limited access but that still seems dangerous at scale when that ends up being a lot of people etc.

nknighthb 12y ago
I have yet to find any solution to this problem that I like. A separate config repo is least bad of the various bad options. The others have all been error-prone and failed to play nicely with version control. If someone could make git store all on-disk data in encrypted form with an ssh-agent-like solution for on-the-fly (de|en)cryption, that would seem ideal.
- jarrett 12y ago
  > If someone could make git store all on-disk data in encrypted form with an ssh-agent-like solution for on-the-fly (de|en)cryption, that would seem ideal. You could probably roll your own. A decent middleground would be to keep the data in the central repo encrypted, but keep it decrypted on the engineers' workstations. Though a better approach is just to not check in sensitive values at all.
  - nknighthb 12y ago
    "Roll your own" is exactly what causes trouble. It's never right and always painful. We need a general, widely-vetted solution that doesn't make me want to yank my hair out. "Don't check in sensitive values" is not an option. Credentials must be kept track of and maintained somewhere. Storing them in an ad-hoc manner without version control doesn't just invite trouble, it demands it. We have password managers for desktops and mobile devices. We need an equivalent for servers that accounts for the fact that the credentials must be highly available, multiple people must be able to use them (whether or not they get to directly view them), some of them will be technicians with limited training following someone else's procedures, and regardless of knowledge and skill, there will be half-asleep people taking emergency actions at 3AM when shit has hit the fan.

druiid 12y ago

Chef supports encrypted data-buckets where you decrypt only on the server. Puppet supports hiera data with encryption as well (via plugins). There's no reason anyone should be storing password information in source, especially API keys which I'm guessing this was (S3 information more than likely?). I think about the extent of private information I'd be okay with storing in source is DB user/pass information because that will generally require compromising the machine more generally first and by that point they already had access to your DB even if the password/username wasn't stored in source. Essentially this will serve as a wake-up call to the Bitly guys and hopefully they'll take some steps to deploy this stuff via CM or similar in the future.

thefreeman 12y ago
I read it a bit differently. I don't think they store their credentials directly in their source code repository. I think they are saying they have a separate repository specifically for credentials. It seems like a bit of a strange way to manage sensitive information, but not really any worse then a lot of other things I've seen.

ForHackernews 12y ago

Link-shorteners are a cancer. We have enough of a problem with historical links going dead, without interposing a third party. When bit.ly goes down, what happens to all their links?

helium 12y ago
By that rationale, any website is a cancer.
barsonme 12y ago
True, but for example if I'm posting an article I wrote it's much more efficient to save space using my erclgrgn.us short link because it's usually ~20 chars which is less than half of what the full URL is. Unfortunately, sometimes brevity is a necessity. edit: original comment mentioned twitter, then I remembered twitter automatically reduces a URL's chars to 20 or so.
- ForHackernews 12y ago
  Yeah, or websites could just allow actual links. I don't see why hackernews (and twitter, etc.) can't just render <a href="http://somereallylongdomain.com/stupidlylongpath/?what=thisisgettingridiculous">I'm http://somereallylongdomain.com/stupidlylongpath/?what=thisi... short!</a>
  - maxerickson 12y ago
    Because of griefer trolls. Forcing links to show the destination is basically 0 effort on the system side and pretty much completely undermines them. (that only answers the HN part, I'm guessing that is much of the reasoning, I don't know it for a fact)

tedd4u 12y ago

For security stuff, if you can't find a well-tested implementation to re-use, I always suggest referring to the OWASP Cheat Series [1] - there's a nice one just for password storage [2]. [1] https://www.owasp.org/index.php/OWASP_Cheat_Sheet_Series https://www.owasp.org/index.php/OWASP_Cheat_Sheet_Series [2] https://www.owasp.org/index.php/Password_Storage_Cheat_Sheet https://www.owasp.org/index.php/Password_Storage_Cheat_Sheet