3 ms·

>No, according to your talk, Facebook's solution is that password validation is pushed out to the front-end servers, Yes, although I don't work for Facebook an

>No, according to your talk, Facebook's solution is that password validation is pushed out to the front-end servers, Yes, although I don't work for Facebook any more, that was and probably still is the case. >who use back-end KMS services to do some (but not all) of the crypto. In fact, the backend service does a tiny amount of the crypto. >I think this is a suboptimal approach. Of course you do, it's not like I've not argued with you before, Thomas. :-) >Can you tell me what the benefit of your layered approach ...Facebook's layered approach... >is over simply adding to your KMS servers the APIs for validate(user, password) and change(user, oldpassword, newpassword)? If you want your hash to be ... hashed, rather than lodged in some questionable silicon, not putting a fat crypto load onto the backend avoids the "thundering herd" problem when some fraction of 1.7 billion people want to log in. >If your KMS service did password validation directly, you wouldn't need any of the layers in this architecture. Quite, and if everyone flew instead of drove, we wouldn't need cars with layers of bumpers, crumplezones, seatbelts and airbags; but it's merely shifting the problem for 1.7 billion people. Have I mentioned "scale"? I should mention scale. Scale is a thing. >HMAC would add nothing (it would be tautological, since anyone who could directly attack the hashes must have also owned up the KMS service). Yes. Fucking huge KMS service. HUGE. Trumpiness levels of -HUGE- and eating lots of power and redundancy. 1.7 billion people. That's a lot. Like 0.01% of it is 170,000 people. All logging in together. All over the world. >I still don't totally understand the MD5 step. Yeah, but I'm not betting that Mark's/whomever's coding at the time was focused on the future of password authentication. >You could just use scrypt and nothing else Yes, but where's the fun in that? >and you could probably ratchet the work factor up because you wouldn't be billing the front end servers for those cycles. The frontend servers are approximately precisely where you want the cost/chokepoint. There's a metric fucktonne of them and they are closest to the request, so by definition they are scaled to the load. >I generally like, have recommended, and have built a few times the "software HSM" KMS approach you're describing here --- but only for "seal/unseal" and "sign/verify" APIs. Cool. What's your biggest deployment?

tptacek 10y ago
Last question first: large, but not Facebook large (I spent 10 years consulting on this kind of thing).† You comment early in the talk that the MD5 step is somehow helpful for password dumps. That was the bit I didn't follow. If it's there because that's how password hashing worked before your team got to it, that makes a lot more sense. But then: it's not a "layer" of the onion so much as a sheen of dirt that needs to be washed off the onion. :) I get that your auth problem is huge. Yuge. So big you wouldn't believe it. I totally believe you. No, wait, I don't believe you, that's how big I know your authN problem to be: unbelievably huge. But here's the thing: you're already scrypting passwords. We're not debating whether you can use expensive password hashes. You already use expensive password hashes. I'm saying: the model where the KMS does a small bit of the password hash step and defers the heavy lifting to front-end servers seems like a suboptimal way to structure this: * You have to bill cycles from the front end to do it * You can't change password hashing without updating all the front-end servers * It's harder to track usage because it's spread across a zillion machines * You're more constrained in how you scale it (for instance, if you wanted to double or triple the work factor) because whatever your new scheme is, it has to fit with the existing front-end resources. I'm not saying "wow, it's dumb that you built it this way". I'm saying, if other people are reading this thread thinking about how to do it: * DO split authentication out into its own service * DON'T have that authentication service be "HMAC as a service" and then do scrypt on your front-end service YOUR MOVE, ALEC MUFFETT. I keep going until you unfriend me on Facebook so I can't see you wincing about these posts. † I've assessed Facebook-large variants of this, though.
- alecmuffett 10y ago
  >Last question first: large, but not Facebook large (I spent 10 years consulting on this kind of thing). I just spent 3 years living it for 50h/week. Hence why I am taking a vacation. >If it's there because that's how password hashing worked before your team got to it, that makes a lot more sense. That. It wasn't even me, it was done before I arrived, but it was done by a team of geeks with a tremendous nose for making the best of the database that they had available to them without pulling the old password-migration "log in with one password, parallel-encrypt with a new algorithm, and save the new hashes" - thing, because some of those billion people might never log in again for years. You would never stop migrating people. I remember internal pasword algorithm migrations at Sun, at least there you could force the matter for 10,000..40,000 people. But you can't force everyone to migrate at FB scale. >But then: it's not a "layer" of the onion so much as a sheen of dirt that needs to be washed off the onion. :) You can take that approach, but - again - when will you finish the task? Whereas wrapping one algorithm in the next is a finite task which is completable in a reasonable amount of time. >I get that your ...Facebook's... >auth problem is huge. Yuge. So big you wouldn't believe it. I totally believe you. No, wait, I don't believe you, that's how big I know your authN problem to be: unbelievably huge. Well channeled. :-) >But here's the thing: you're already scrypting passwords. We're not debating whether you can use expensive password hashes. You already use expensive password hashes. I'm saying: the model where the KMS does a small bit of the password hash step and defers the heavy lifting to front-end servers seems like a suboptimal way to structure this: >* You have to bill cycles from the front end to do it Yes. 0.1% of frontend cycles. <blank expression> And? >* You can't change password hashing without updating all the front-end servers ...which happens three times a day, weekdays, and is moving to moreso. >* It's harder to track usage because it's spread across a zillion machines "Facebook has tools for that sort of thing": http://conferences.oreilly.com/strata/stratany2012/public/schedule/detail/25540 http://conferences.oreilly.com/strata/stratany2012/public/sc... ...and to be honest it's not hard to do anyway. >* You're more constrained in how you scale it (for instance, if you wanted to double or triple the work factor) because whatever your new scheme is, it has to fit with the existing front-end resources. Yes. For a site with wildly heterogeneous architectures in front-end deployments, I can see how that might be a concern; but even AWS leads people to standardise on having approximately-the-same-kinds-of-hardware-doing-approximately-the-same-things. >I'm not saying "wow, it's dumb that you ...Facebook... >built it this way". I'm saying, if other people are reading this thread thinking about how to do it: >* DO split authentication out into its own service ...or some component of it... >* DON'T have that authentication service be "HMAC as a service" and then do scrypt on your front-end service Why not? >YOUR MOVE, ALEC MUFFETT. I keep going until you unfriend me on Facebook so I can't see you wincing about these posts. Wince? > I've assessed Facebook-large variants of this, though. Did they buy it?

tptacek 10y ago

Last question first: large, but not Facebook large (I spent 10 years consulting on this kind of thing).† You comment early in the talk that the MD5 step is somehow helpful for password dumps. That was the bit I didn't follow. If it's there because that's how password hashing worked before your team got to it, that makes a lot more sense. But then: it's not a "layer" of the onion so much as a sheen of dirt that needs to be washed off the onion. :) I get that your auth problem is huge. Yuge. So big you wouldn't believe it. I totally believe you. No, wait, I don't believe you, that's how big I know your authN problem to be: unbelievably huge. But here's the thing: you're already scrypting passwords. We're not debating whether you can use expensive password hashes. You already use expensive password hashes. I'm saying: the model where the KMS does a small bit of the password hash step and defers the heavy lifting to front-end servers seems like a suboptimal way to structure this: * You have to bill cycles from the front end to do it * You can't change password hashing without updating all the front-end servers * It's harder to track usage because it's spread across a zillion machines * You're more constrained in how you scale it (for instance, if you wanted to double or triple the work factor) because whatever your new scheme is, it has to fit with the existing front-end resources. I'm not saying "wow, it's dumb that you built it this way". I'm saying, if other people are reading this thread thinking about how to do it: * DO split authentication out into its own service * DON'T have that authentication service be "HMAC as a service" and then do scrypt on your front-end service YOUR MOVE, ALEC MUFFETT. I keep going until you unfriend me on Facebook so I can't see you wincing about these posts. † I've assessed Facebook-large variants of this, though.

alecmuffett 10y ago
>Last question first: large, but not Facebook large (I spent 10 years consulting on this kind of thing). I just spent 3 years living it for 50h/week. Hence why I am taking a vacation. >If it's there because that's how password hashing worked before your team got to it, that makes a lot more sense. That. It wasn't even me, it was done before I arrived, but it was done by a team of geeks with a tremendous nose for making the best of the database that they had available to them without pulling the old password-migration "log in with one password, parallel-encrypt with a new algorithm, and save the new hashes" - thing, because some of those billion people might never log in again for years. You would never stop migrating people. I remember internal pasword algorithm migrations at Sun, at least there you could force the matter for 10,000..40,000 people. But you can't force everyone to migrate at FB scale. >But then: it's not a "layer" of the onion so much as a sheen of dirt that needs to be washed off the onion. :) You can take that approach, but - again - when will you finish the task? Whereas wrapping one algorithm in the next is a finite task which is completable in a reasonable amount of time. >I get that your ...Facebook's... >auth problem is huge. Yuge. So big you wouldn't believe it. I totally believe you. No, wait, I don't believe you, that's how big I know your authN problem to be: unbelievably huge. Well channeled. :-) >But here's the thing: you're already scrypting passwords. We're not debating whether you can use expensive password hashes. You already use expensive password hashes. I'm saying: the model where the KMS does a small bit of the password hash step and defers the heavy lifting to front-end servers seems like a suboptimal way to structure this: >* You have to bill cycles from the front end to do it Yes. 0.1% of frontend cycles. <blank expression> And? >* You can't change password hashing without updating all the front-end servers ...which happens three times a day, weekdays, and is moving to moreso. >* It's harder to track usage because it's spread across a zillion machines "Facebook has tools for that sort of thing": http://conferences.oreilly.com/strata/stratany2012/public/schedule/detail/25540 http://conferences.oreilly.com/strata/stratany2012/public/sc... ...and to be honest it's not hard to do anyway. >* You're more constrained in how you scale it (for instance, if you wanted to double or triple the work factor) because whatever your new scheme is, it has to fit with the existing front-end resources. Yes. For a site with wildly heterogeneous architectures in front-end deployments, I can see how that might be a concern; but even AWS leads people to standardise on having approximately-the-same-kinds-of-hardware-doing-approximately-the-same-things. >I'm not saying "wow, it's dumb that you ...Facebook... >built it this way". I'm saying, if other people are reading this thread thinking about how to do it: >* DO split authentication out into its own service ...or some component of it... >* DON'T have that authentication service be "HMAC as a service" and then do scrypt on your front-end service Why not? >YOUR MOVE, ALEC MUFFETT. I keep going until you unfriend me on Facebook so I can't see you wincing about these posts. Wince? > I've assessed Facebook-large variants of this, though. Did they buy it?