3 ms·

>Last question first: large, but not Facebook large (I spent 10 years consulting on this kind of thing). I just spent 3 years living it for 50h/week. Hence why

>Last question first: large, but not Facebook large (I spent 10 years consulting on this kind of thing). I just spent 3 years living it for 50h/week. Hence why I am taking a vacation. >If it's there because that's how password hashing worked before your team got to it, that makes a lot more sense. That. It wasn't even me, it was done before I arrived, but it was done by a team of geeks with a tremendous nose for making the best of the database that they had available to them without pulling the old password-migration "log in with one password, parallel-encrypt with a new algorithm, and save the new hashes" - thing, because some of those billion people might never log in again for years. You would never stop migrating people. I remember internal pasword algorithm migrations at Sun, at least there you could force the matter for 10,000..40,000 people. But you can't force everyone to migrate at FB scale. >But then: it's not a "layer" of the onion so much as a sheen of dirt that needs to be washed off the onion. :) You can take that approach, but - again - when will you finish the task? Whereas wrapping one algorithm in the next is a finite task which is completable in a reasonable amount of time. >I get that your ...Facebook's... >auth problem is huge. Yuge. So big you wouldn't believe it. I totally believe you. No, wait, I don't believe you, that's how big I know your authN problem to be: unbelievably huge. Well channeled. :-) >But here's the thing: you're already scrypting passwords. We're not debating whether you can use expensive password hashes. You already use expensive password hashes. I'm saying: the model where the KMS does a small bit of the password hash step and defers the heavy lifting to front-end servers seems like a suboptimal way to structure this: >* You have to bill cycles from the front end to do it Yes. 0.1% of frontend cycles. <blank expression> And? >* You can't change password hashing without updating all the front-end servers ...which happens three times a day, weekdays, and is moving to moreso. >* It's harder to track usage because it's spread across a zillion machines "Facebook has tools for that sort of thing": http://conferences.oreilly.com/strata/stratany2012/public/schedule/detail/25540 http://conferences.oreilly.com/strata/stratany2012/public/sc... ...and to be honest it's not hard to do anyway. >* You're more constrained in how you scale it (for instance, if you wanted to double or triple the work factor) because whatever your new scheme is, it has to fit with the existing front-end resources. Yes. For a site with wildly heterogeneous architectures in front-end deployments, I can see how that might be a concern; but even AWS leads people to standardise on having approximately-the-same-kinds-of-hardware-doing-approximately-the-same-things. >I'm not saying "wow, it's dumb that you ...Facebook... >built it this way". I'm saying, if other people are reading this thread thinking about how to do it: >* DO split authentication out into its own service ...or some component of it... >* DON'T have that authentication service be "HMAC as a service" and then do scrypt on your front-end service Why not? >YOUR MOVE, ALEC MUFFETT. I keep going until you unfriend me on Facebook so I can't see you wincing about these posts. Wince? > I've assessed Facebook-large variants of this, though. Did they buy it?

tptacek 10y ago
Sorry for the delay. A Jazzercise class suddenly appeared in the coworking space I work out of, so I fled, and then I had to give a talk about Starfighter. Responses: (when I say "your" let's just stipulate I mean Facebook) * Your password validation overhead is .1% of current front end resources, but could be ratcheted up, and would be easier to ratchet up if they weren't shared by other things. * I totally understand why you keep the old MD5 cruft around --- but would add that it's cruft that would be even less obvious if it lived behind an authentication server. * I think it's safer, simpler, cleaner, probably easier to scale, and definitely easier to change authentication if it lives in its own service rather than being implemented (in part) on a generic application server. As usage shifts from HTML front-end to all API, you might even be able to keep app servers from even seeing passwords. * By "assessed", I mean, worked on other people's systems at this scale. So I guess I'd wrap up with a question: if you had this to do over again, from scratch, the way you wanted to, would you have app servers do a password hash and then entangle it somehow with an HMAC operation from a crypto service, or would you have the whole password hash done on the crypto service directly?
- alecmuffett 10y ago
  tl;dr for me: Putting the authentication service into a nice tidy centralised box does not actually achieve much, and may have architectural downsides. Not the least of which is: if it's wholly in a service, then you have to authenticate the service; that's not such a big step from "if the hashed passwords are stored in a directory, then you have to authenticate the directory" of course - but if we were to equate the two systems because of the need to authenticate the {directory, service} then the service-based solution still has the downside of being a CPU hotspot and a potential single point of failure. We're much better at distributing directories of data which is self-protected / needs no special treatment, than we are at building humongous scalable "secure" services with an enormous TCB and a physically enormous attack surface / footprint. Yes. If I was doing this, sure, I would do this again. Curiously I am a big fan of password hashing rather than all-singing, all-dancing authentication services. General notes: http://dropsafe.crypticide.com/muffett-passwords http://dropsafe.crypticide.com/muffett-passwords

tptacek 10y ago

Sorry for the delay. A Jazzercise class suddenly appeared in the coworking space I work out of, so I fled, and then I had to give a talk about Starfighter. Responses: (when I say "your" let's just stipulate I mean Facebook) * Your password validation overhead is .1% of current front end resources, but could be ratcheted up, and would be easier to ratchet up if they weren't shared by other things. * I totally understand why you keep the old MD5 cruft around --- but would add that it's cruft that would be even less obvious if it lived behind an authentication server. * I think it's safer, simpler, cleaner, probably easier to scale, and definitely easier to change authentication if it lives in its own service rather than being implemented (in part) on a generic application server. As usage shifts from HTML front-end to all API, you might even be able to keep app servers from even seeing passwords. * By "assessed", I mean, worked on other people's systems at this scale. So I guess I'd wrap up with a question: if you had this to do over again, from scratch, the way you wanted to, would you have app servers do a password hash and then entangle it somehow with an HMAC operation from a crypto service, or would you have the whole password hash done on the crypto service directly?

alecmuffett 10y ago
tl;dr for me: Putting the authentication service into a nice tidy centralised box does not actually achieve much, and may have architectural downsides. Not the least of which is: if it's wholly in a service, then you have to authenticate the service; that's not such a big step from "if the hashed passwords are stored in a directory, then you have to authenticate the directory" of course - but if we were to equate the two systems because of the need to authenticate the {directory, service} then the service-based solution still has the downside of being a CPU hotspot and a potential single point of failure. We're much better at distributing directories of data which is self-protected / needs no special treatment, than we are at building humongous scalable "secure" services with an enormous TCB and a physically enormous attack surface / footprint. Yes. If I was doing this, sure, I would do this again. Curiously I am a big fan of password hashing rather than all-singing, all-dancing authentication services. General notes: http://dropsafe.crypticide.com/muffett-passwords http://dropsafe.crypticide.com/muffett-passwords