12 ms·

API Tokens: A Tedious Survey

387 points by enobrev 5y ago

simonw 5y ago
This post reminded me a little of https://blog.thea.codes/building-a-stateless-api-proxy/ https://blog.thea.codes/building-a-stateless-api-proxy/ from 2019, an absolutely brilliant hack that attempted to make up for GitHub's lack of finely-grained API tokens. The key idea there is to build your own custom proxy for the GitHub API, then issue tokens for it which are actually encrypted bundles of the full-permission API token plus a set of rules about what the proxy should allow it to do - only allow a GET to paths that match "/gists/.*" for example. It's somewhat similar to Biscuits storing a Datalog program "to evaluate whether a token allows an operation."
greysteil 5y ago
Since the author is on here - reckon you could beat up on simple random tokens a _little_ bit more? In particular how easy they are to identify and prevent leaking (easily fixed by adding a prefix). I work on secret scanning at GitHub. When token issuers use easily identifiable formats for their tokens we can easily spot them when they're accidentally committed. We can then work with the token issuer to automatically alert them of those leaks. A good example is AWS - if you commit an AWS key and secret to a public GitHub repo we will tell AWS about it and they will tell you about it (and quarantine the exposed keys) within a few seconds. We work with dozens of other token issuers too, though - some of the latest we added were Linear, PlanetScale and Ionic. The above relies on tokens being identifiable - we can't send hundreds of partners everything that looks like 32 hex chars. In future we want to be able to do even more sophisticated things, like ask users for confirmation before the push code that contains secrets. We recently changed our own token pattern for that reason. GitHub secret scanning program: https://docs.github.com/en/developers/overview/secret-scanning-partner-program https://docs.github.com/en/developers/overview/secret-scanni... GitHub's updated token format: https://github.blog/2021-04-05-behind-githubs-new-authentication-token-formats/ https://github.blog/2021-04-05-behind-githubs-new-authentica...
- tptacek 5y ago
  I agree with you. Someone brought this up on Twitter and I'm kicking myself for not remembering to include the notion of adding identifiable markings to sensitive tokens (I'd do it not, but I'd feel like I was plagiarizing). And it's a noodly and somewhat incoherent notion of "safety" I'm using here, because of course, random tokens are unconstrained bearer tokens --- authenticated requests, CATs, Macaroons, and Biscuits all address that weakness. I'm biased by my concern over cryptographic implementation mistakes. It's a neat property of Macaroons (and maybe Biscuits) that you can come up with sane configurations where checking a Macaroon into source code can be, if not totally safe, at least not a major incident. I wish I'd thought of that, too, since I think "checking the token into source control" is a more vivid example than "emailing tokens" or "passing them around".
  - tialaramex 5y ago
    Surely you needn't feel like you're plagiarizing, just give credit. The credit won't be any less deserved next week or next year, even if you "Know" you'd definitely have fixed this without prompting, nobody else knows that, and so it looks like you avoided giving credit where it was due, so, don't do that. Nobody's asking for royalties, so "Shout out to @SomeoneOnTwitter for reminding me" is enough.
    - tptacek 5y ago
      I will probably get around to doing that. If it was an error in the post I'd feel differently, but this is just another good idea I forgot to include.
      - greysteil 5y ago
        Thanks for doing the good work of encouraging people to mint better tokens!
      - gmmeyer 5y ago
        tbh I actually do think this is an error. I worked someone where we didn't do this and I think of it as a bug rather than a missing feature. It makes it very difficult to identify the token if it's ever leaked without checking it against our database!
tialaramex 5y ago
As a feedback on the visuals, I'm sure you know what this scorecard means. I'm sure I could figure out what this scorecard means if I put a bunch of effort in. The fact that "figuring out what it means" is a problem worth talking about is all you need to know. If the idea was "LOL, joke scorecard" then, I guess joke is on me, otherwise, if you revise this I'd recommend to decide what one thing the scorecard is supposed to communicate, then visualise only that and accept that people will need to read the rest of the text to know more.
- tptacek 5y ago
  It's like 90% a joke (it's the old Consumer Reports scale, where the red-white-dot is "best" and black-white-dot is "worst"). It's definitely not important to the content in the post.
- staticassertion 5y ago
  I also was unclear on what the scorecard was. From elsewhere in this thread it's something to do with "Consumer Reports", which I've never heard of. Wikipedia helped with the rest. "Consumer Reports graphs formerly used a modified form of Harvey balls for qualitative comparison. The round ideograms were arranged from best to worst. On the left of the diagram, the red circle indicated the highest rating, the half red and white circle was the second highest rating, the white circle was neutral, the half black circle was the second-lowest rating, and the entirely black circle was the lowest rating possible"
jopsen 5y ago
Why all the hate for JWTs? Just pick a crypto scheme and the JWT is just an encoding that makes it easier to use. (It's just a more convenient way of rolling your own scheme) That said: random tokens have a lot going for them :)
- CiPHPerCoder 5y ago
  > Why all the hate for JWTs? > Just pick a crypto scheme and the JWT is just an encoding that makes it easier to use. That's not what JWT is, but I can understand why someone would be misled into believing that. JWT isn't just an encoding format, it also includes a crypto algorithm negotiation protocol that lets the attacker choose the algorithm. Even if you strictly allow-list which algorithm you want to support, you can accidentally bypass this control in many libraries if you support the `kid` (key ID) header. [1] It also allows attackers to completely strip the security. [2] [3] Put shortly, JWT is a gun aimed directly at your foot. That's why there's so much hate for JWTs. [1] https://github.com/firebase/php-jwt/issues/351 https://github.com/firebase/php-jwt/issues/351 [2] https://paragonie.com/blog/2017/03/jwt-json-web-tokens-is-bad-standard-that-everyone-should-avoid https://paragonie.com/blog/2017/03/jwt-json-web-tokens-is-ba... [3] https://www.howmanydayssinceajwtalgnonevuln.com/ https://www.howmanydayssinceajwtalgnonevuln.com/
  - daigoba66 5y ago
    > crypto algorithm negotiation If you control both sides, then you can ignore this part or do it out-of-band. Though, if you control both sides, then you can use literally anything else too.
    - jerf 5y ago
      But that's the problem with JWTs, the whole "if you..." part. You want fewer of those rather than more in your crypto code, and JWT has too many. That's the whole problem.
  - benatkin 5y ago
    > Even if you strictly allow-list which algorithm you want to support This is the norm. As for the key ID attack, this sounds like it's just a trick to know where the private key is located? It shouldn't be publicly accessible. https://blog.pentesteracademy.com/hacking-jwt-tokens-kid-claim-misuse-key-leak-e7fce9a10a9c https://blog.pentesteracademy.com/hacking-jwt-tokens-kid-cla...
betaby 5y ago
Simple random tokens are essentially passwords. Thus one should store them 'salted' in the authentication database. Is my understanding correct?
- simonw 5y ago
  I don't see the need to salt them in most cases. The reason you hash and store passwords with a salt is that there's a very good chance that a user will have used that same password somewhere else. As such, it's important that you make it as hard as possible for you - or a co-worker - or someone who gets hold of a dump of your database - to gain access to the original password. Random tokens were generated by you. They are only valid against your own service. If someone bad gets hold of them, all they can do is make API calls against your service until that token is revoked. So salting and hashing them doesn't win you much. That said, if a lot of people at your company have unfettered access to run queries against your database it may be a good idea to store your random tokens in a way that prevents people from copying them out and abusing them. But if you're storing any private data at all and you have that kind of a culture you have bigger problems you need to solve.
  - tptacek 5y ago
    If you're worried about database timing attacks (I'm mostly not) then there's a good thread about that here: https://twitter.com/colmmacc/status/1430282652821585920 https://twitter.com/colmmacc/status/1430282652821585920
- deleted 5y ago
  [deleted]
keithwinstein 5y ago
I'd love to hear tptacek's view on the pros and cons of "authenticated connections," where the client uses asymmetric crypto to establish an authenticated-as-the-principal bytestream, and then issues a bunch of unsigned requests over that bytestream. In practice, I'm talking about (1) "SSH public-key authentication to GitHub" (or to an EC2 or GCE instance), or (2) using FIDO2/WebAuthn (or TLS client certs...) to authenticate an HTTP connection or browsing session. One major pro is that this is a lot more widely implemented and standardized than "authenticated requests" -- there are lots of reasonable implementations of SSH pubkey authentication or FIDO2/WebAuthn, compared with trying to sign an individual HTTP request. And that maturity brings robustness and security, e.g., I like that I can have my SSH private key (or my FIDO2 private key) on a tamper-resistant hardware device and not worry about some rogue software stealing my private key, whereas trying to find a hardware device to hold my AWS credentials and do the AWSv4 signature is a different story. Any bearer-token-based scheme seems... a lot scarier in that sense. I'm less clear on the practical cons (assuming "authenticated requests" is not available). With SSH keys or TLS client certs, I could imagine it's annoying because only the terminus of the secure connection can verify that the client is who it claims to be, and maybe it's annoying to have your SSH/TLS termination component directly talk to a user database to authenticate individual clients and then attest that it did so to the rest of the backend. But... somehow GitHub/GitLab/Bitbucket all manage to do this at scale, and with WebAuthn/FIDO2, any component can be the one to make the challenge. (I suppose only recently can you make a "user presence not required" challenge, so maybe that's why nobody uses this in the non-presence setting...?)
- tptacek 5y ago
  I have actually a lot to say about this, in the context of mTLS; it's a post I'm working on. I am generally a fan of the mTLS approach for very simple topologies, like "central Consul cluster nobody can talk to without the right client certificate". I am much less a fan of it for the general inter-service authentication problem.
FreakLegion 5y ago
I get the criticism of ECDH-ES with JWT in vivo, but not the linked tweets' broadside against it in vitro. Are we not just talking about ECIES?
- tptacek 5y ago
  It's all the things that had to go wrong all at once. P-curves, not Curve25519 (where curve point validation is less important). Static-ephemeral ECDH, so there's a key to target. Long-term durable keys. I'm not making an argument against ECIES (though I guess I'd push not to use the P-curves).
catchmeifyoucan 5y ago
What about revokable JWT tokens? Isn't that similar to the random token approach, except each token holds some additional meaning, and revokable.
- tptacek 5y ago
  Yes. Revocation is painful with all of the stateless approaches; it's not on its own a reason to avoid JWT. On the other hand, the folkloric draw of JWT is that it's stateless, and they're only stateless if you can revoke them without issuing SQL queries.
  - initplus 5y ago
    A downside of stateful auth is the extra DB round trip on every request. Different revocation techniques like periodically distributing a revocation list to your auth services can resolve that part of the issue.
RustyRussell 5y ago
Because standards are always improved by creating new options, I should note runes here (simplified macaroons with Python implementation): https://pypi.org/project/runes/ https://pypi.org/project/runes/ Disclosure: I'm the author
Scaevolus 5y ago
Google's internal mutual authentication/encryption might be of interest. https://cloud.google.com/security/encryption-in-transit/application-layer-transport-security https://cloud.google.com/security/encryption-in-transit/appl... They show how far you can drive Protocol Buffer Tokens.
monocasa 5y ago
I'm curious where kerberos tickets fit into this. A little out of vogue, but it'd be neat to hear how exactly they're long in the tooth.
- tptacek 5y ago
  Nobody sane uses Kerberos to do IAM for public APIs. But people use them for inter-service authentication, as the post mentions. It links to an article I wrote a couple years ago that considers Kerberos in the context of a variety of other inter-service security tools.
  - monocasa 5y ago
    > Nobody sane uses Kerberos to do IAM for public APIs. Why if you don't mind? I'm actually looking at a major proprietary protocol used by hundreds of millions of users that use Kerberos as the fundamental cryptographic attestation of identity and roles. Like clients all connect to a VPN who's bridge uses the Kerberos ticket to whitelist backend services accessible from that connection. They're basically being used the same as if an API gateway understood oath claims and could stop whole classes of client calls at the perimeter.
    - tptacek 5y ago
      Because it's very complicated and fussy (no HTTP API client framework has K5 built in) and, if you're going to force your clients to use a nonstandard authentication protocol, you can do better than Kerberos. A private CA, mTLS, and an authenticated role-based certificate issuer probably does a better job across the board. Facebook talks a little bit about the tradeoffs here in the paper linked to the post; note that they could have used K5 instead of CATs, and the stuff that CATs does is in some ways a response to the limitations of K5.
ignoramous 5y ago
Very consistent with what tptacek has been advocating for on news.yc over the years. The authn mechanism we use is closer to Keybase's NIST (non-interactive session tokens)[0] that are a mix of AWS-style Bearer Tokens and the usual Random Tokens. Of course, the problems around "logistics" (public-key cryptography)[1] are a real nightmare as the post points out. We exchange these tokens between devices (if needed) over password-authenticated channels (using CPACE [2]). [0] https://keybase.io/docs/api/1.0/nist https://keybase.io/docs/api/1.0/nist [1] As examples, see what goes on when a Keybase user associates a new device: https://book.keybase.io/docs/crypto/key-exchange https://book.keybase.io/docs/crypto/key-exchange or when SQRL user revokes compromised keys: https://www.grc.com/sqrl/idlock.htm https://www.grc.com/sqrl/idlock.htm [2] https://github.com/jedisct1/cpace https://github.com/jedisct1/cpace
- cormacrelf 5y ago
  Is NIST NIST-compliant?
  - ignoramous 5y ago
    Nothing to do with NIST.gov, that's just unfortunate naming coincidence. Re: Compliance: Keybase, if am not wrong, has pretty much rolled out their own crypto here. At least in one instance they were subject to criticism for rolling out a key-wrapping scheme viz. TripleSec: https://news.ycombinator.com/item?id=9655245 https://news.ycombinator.com/item?id=9655245
alex-olivier 5y ago
I tend to fall in the trap of once you have authenticated the person and passing tokens around that can hold claims/role for authorization, you will eventually reach the point where the tokens are getting so bloated due to the complexity of access controls. AuthN is the next big challenge especially in a multi-tenant/multi-enterprise SaaS type platform https://cerbos.dev/blog/the-hidden-costs-of-user-authorization https://cerbos.dev/blog/the-hidden-costs-of-user-authorizati...
- alex-olivier 5y ago
  ...and by that I mean authorization is the next big challenge especially in a multi-tenant/multi-enterprise SaaS type platform
  - tptacek 5y ago
    Authorization is authz, for what it's worth. "authn" means authentication.
traceroute66 5y ago
You know, what I'd like to see more than standardised API tokens to make scanning easier is actual addressing of the underlying problem. For example, we had a pentest done on a website and the pentester got all excited because they found some AWS tokens. Trouble is, they would be worthless to anyone external because we were making good use of AWS IAM to lock them down with ACLs, Roles etc. So it was effectively a non-event. What happened to the old concept of layered security ? Why should discovery or leakage of an API key automatically give the attacker all the keys to the castle ? In my ideal world, all cloud and API service operators would have the equivalent to AWS IAM and preferably would enforce its usage (i.e. "here's your API key, but it won't work until you set some layered security")
- innomatics 5y ago
  This very good practice and I am pleased that the one time I got the dreaded message from GitHub and AWS, that I'd done the unthinkable (on a public repo), the keys were only for accessing a single junk dev S3 bucket - phew! But no amount of layering makes the problem go away. Sometimes god-like keys are unavoidable. Those needed by Terraform etc are an example that comes to mind.
mwcampbell 5y ago
> I continue to believe that boring, trustworthy random tokens are underrated, and that people burn a lot of complexity chasing statelessness they can't achieve and won’t need, because token databases for most systems outside of Facebook aren’t hard to scale. Then why did you decide to go with macaroons rather than random tokens? Do you know for sure that fly.io needs that statelessness?
- mrkurt 5y ago
  We need statelessness for service reliability more than we need it for scale. There are a number of services that people use within our infrastructure that need to continue working even when a centralized DB is unavailable. Logs are a good example. We expose logs as a NATS service, people auth and subscribe to their logs. It's useful to terminate auth right there both because it's faster and because it continues to work when the internet burps.
- tptacek 5y ago
  I think this is a good question. I'm writing up our Macaroon implementation as I roll it out --- it'll be a few weeks before I publish --- and, if I've done it right, I'll have answered that question clearly.
sudhirj 5y ago
Ironically my main takeaway is that given the pros and cons, JWT seems like the best balance of security and convenience, assuming you always hardcode/verify the algo. Lots of library support, allows Macaroon type policy setup using the JSON (but no chaining). It's not recommended in the article, but it seems like knowing what's out there has made JWT's usefulness clearer by making its downsides more obvious.
tptacek 5y ago
The one thing I'm not super comfortable about here is my PASETO take. My attitude going in was that PASETO has a lot of boosters and not a lot of critical takes. I can beat up on Macaroons because we're using them, and I'm going to follow up with a post about what our Macaroons like like. I'm not doing that with PASETO. So, like, I stand by it, but take it for what it's worth. Later I turned the post into a handy chart. Let me know if you want a poster of it. https://twitter.com/tqbf/status/1430303808945524739 https://twitter.com/tqbf/status/1430303808945524739
- tingletech 5y ago
  is there a key to decode the scores on the chart?
  - tptacek 5y ago
    This seemed funny until I realized everyone else isn't 40+ years old and former Consumer Reports subscribers. The red-with-white-dot thing is "best".
    - mbesto 5y ago
      Ya, I can't parse this whatsoever. What is the difference between red and black? Thick circle? Half circle? No circle? Original harvey balls might be better here: https://en.wikipedia.org/wiki/Harvey_balls https://en.wikipedia.org/wiki/Harvey_balls PS - thanks for putting together.
      - tptacek 5y ago
        You'll have to take that up with Consumer Reports. :) The chart isn't very serious.
      - nwsm 5y ago
        Here's some help (left to right is worst to best): https://www.pentagram.com/work/consumer-reports#22596 https://www.pentagram.com/work/consumer-reports#22596 I still don't know where the black with white dot goes. I guess it's one better than the empty circle in the middle.
zrail 5y ago
There's an additional nuance to opaque random-ish tokens that can be helpful in high-traffic situations. You can essentially encode some, for lack of a better word, "routing" information (shard, region, etc) into the token when you generate it. It's still random, you still verify the whole token with your database, but you can extract the routing info and pass it to the correct backend from a mostly-stateless frontend.
mooreds 5y ago
This was great. A really fair survey of various token methods. Plus plenty of liveliness, not boring at all. Thanks, OP! One thing that I wish was addressed more was language/library support. It gets casual references a couple of times, but for an average developer (as I consider myself) a set of robust, supported open source libraries that help me use a token is so important (not write an implementation, but use in a project that just wants to use the tokens safely). I don't have anything but anecdata, but I feel like most software is going to be in the 'just want to use it' category, rather than the 'need to implement it'. This is where the standards like OAuth and JWT win right now. That doesn't mean they always will, but in my experience, that's the current situation.
- CiPHPerCoder 5y ago
  For PASETO, the quick guide to library support is https://paseto.io https://paseto.io
simonw 5y ago
This is great. One thing that's worth remembering about randomly generated tokens is that it's important to always use safe comparison methods when comparing them to the stored one - otherwise you could be vulnerable to timing attacks. In Python you can use secrets.compare_digest(a, b) for this: https://docs.python.org/3/library/secrets.html#secrets.compare_digest https://docs.python.org/3/library/secrets.html#secrets.compa...
- SilverRed 5y ago
  Could you elaborate on what you mean by timing attacks? What would go wrong if you did a basic params[:token] == dbvalue ?
  - pests 5y ago
    Most naive string comparison methods (including language expressions like above) will usually compare one letter at a time until a mismatch is found. This allows an attacker to build up the correct password one character at a time. The correct letter will take longer to check then the incorrect.
    - quelltext 5y ago
      But in the "real world" wouldn't any such time difference be very small and fluctuations in network delay and measurement error so large in comparison that this isn't typically exploitable? Sure, better be safe and use a safe comparison method and that's simple enough, but still, how realistic is such an attack over the public internet? I'm aware of "practical" results: http://crypto.stanford.edu/~dabo/papers/ssl-timing.pdf http://crypto.stanford.edu/~dabo/papers/ssl-timing.pdf But the experiment setup was on a LAN, with AFAICT a single server, no other traffic to that server, and other such conditions.
      - simonw 5y ago
        The trick with timing attacks is that you don't measure the time taken for a single request. You send the same request thousands of times and take an average of the response time, which lets you pick away at the secret one character at a time.
deleted 5y ago
[deleted]

387 points by enobrev 5y ago

This post reminded me a little of https://blog.thea.codes/building-a-stateless-api-proxy/ https://blog.thea.codes/building-a-stateless-api-proxy/ from 2019, an absolutely brilliant hack that attempted to make up for GitHub's lack of finely-grained API tokens. The key idea there is to build your own custom proxy for the GitHub API, then issue tokens for it which are actually encrypted bundles of the full-permission API token plus a set of rules about what the proxy should allow it to do - only allow a GET to paths that match "/gists/.*" for example. It's somewhat similar to Biscuits storing a Datalog program "to evaluate whether a token allows an operation."

greysteil 5y ago

Since the author is on here - reckon you could beat up on simple random tokens a _little_ bit more? In particular how easy they are to identify and prevent leaking (easily fixed by adding a prefix). I work on secret scanning at GitHub. When token issuers use easily identifiable formats for their tokens we can easily spot them when they're accidentally committed. We can then work with the token issuer to automatically alert them of those leaks. A good example is AWS - if you commit an AWS key and secret to a public GitHub repo we will tell AWS about it and they will tell you about it (and quarantine the exposed keys) within a few seconds. We work with dozens of other token issuers too, though - some of the latest we added were Linear, PlanetScale and Ionic. The above relies on tokens being identifiable - we can't send hundreds of partners everything that looks like 32 hex chars. In future we want to be able to do even more sophisticated things, like ask users for confirmation before the push code that contains secrets. We recently changed our own token pattern for that reason. GitHub secret scanning program: https://docs.github.com/en/developers/overview/secret-scanning-partner-program https://docs.github.com/en/developers/overview/secret-scanni... GitHub's updated token format: https://github.blog/2021-04-05-behind-githubs-new-authentication-token-formats/ https://github.blog/2021-04-05-behind-githubs-new-authentica...

tptacek 5y ago
I agree with you. Someone brought this up on Twitter and I'm kicking myself for not remembering to include the notion of adding identifiable markings to sensitive tokens (I'd do it not, but I'd feel like I was plagiarizing). And it's a noodly and somewhat incoherent notion of "safety" I'm using here, because of course, random tokens are unconstrained bearer tokens --- authenticated requests, CATs, Macaroons, and Biscuits all address that weakness. I'm biased by my concern over cryptographic implementation mistakes. It's a neat property of Macaroons (and maybe Biscuits) that you can come up with sane configurations where checking a Macaroon into source code can be, if not totally safe, at least not a major incident. I wish I'd thought of that, too, since I think "checking the token into source control" is a more vivid example than "emailing tokens" or "passing them around".
- tialaramex 5y ago
  Surely you needn't feel like you're plagiarizing, just give credit. The credit won't be any less deserved next week or next year, even if you "Know" you'd definitely have fixed this without prompting, nobody else knows that, and so it looks like you avoided giving credit where it was due, so, don't do that. Nobody's asking for royalties, so "Shout out to @SomeoneOnTwitter for reminding me" is enough.
  - tptacek 5y ago
    I will probably get around to doing that. If it was an error in the post I'd feel differently, but this is just another good idea I forgot to include.
    - greysteil 5y ago
      Thanks for doing the good work of encouraging people to mint better tokens!
    - gmmeyer 5y ago
      tbh I actually do think this is an error. I worked someone where we didn't do this and I think of it as a bug rather than a missing feature. It makes it very difficult to identify the token if it's ever leaked without checking it against our database!

tialaramex 5y ago

As a feedback on the visuals, I'm sure you know what this scorecard means. I'm sure I could figure out what this scorecard means if I put a bunch of effort in. The fact that "figuring out what it means" is a problem worth talking about is all you need to know. If the idea was "LOL, joke scorecard" then, I guess joke is on me, otherwise, if you revise this I'd recommend to decide what one thing the scorecard is supposed to communicate, then visualise only that and accept that people will need to read the rest of the text to know more.

tptacek 5y ago
It's like 90% a joke (it's the old Consumer Reports scale, where the red-white-dot is "best" and black-white-dot is "worst"). It's definitely not important to the content in the post.
staticassertion 5y ago
I also was unclear on what the scorecard was. From elsewhere in this thread it's something to do with "Consumer Reports", which I've never heard of. Wikipedia helped with the rest. "Consumer Reports graphs formerly used a modified form of Harvey balls for qualitative comparison. The round ideograms were arranged from best to worst. On the left of the diagram, the red circle indicated the highest rating, the half red and white circle was the second highest rating, the white circle was neutral, the half black circle was the second-lowest rating, and the entirely black circle was the lowest rating possible"

jopsen 5y ago

Why all the hate for JWTs? Just pick a crypto scheme and the JWT is just an encoding that makes it easier to use. (It's just a more convenient way of rolling your own scheme) That said: random tokens have a lot going for them :)

CiPHPerCoder 5y ago
> Why all the hate for JWTs? > Just pick a crypto scheme and the JWT is just an encoding that makes it easier to use. That's not what JWT is, but I can understand why someone would be misled into believing that. JWT isn't just an encoding format, it also includes a crypto algorithm negotiation protocol that lets the attacker choose the algorithm. Even if you strictly allow-list which algorithm you want to support, you can accidentally bypass this control in many libraries if you support the `kid` (key ID) header. [1] It also allows attackers to completely strip the security. [2] [3] Put shortly, JWT is a gun aimed directly at your foot. That's why there's so much hate for JWTs. [1] https://github.com/firebase/php-jwt/issues/351 https://github.com/firebase/php-jwt/issues/351 [2] https://paragonie.com/blog/2017/03/jwt-json-web-tokens-is-bad-standard-that-everyone-should-avoid https://paragonie.com/blog/2017/03/jwt-json-web-tokens-is-ba... [3] https://www.howmanydayssinceajwtalgnonevuln.com/ https://www.howmanydayssinceajwtalgnonevuln.com/
- daigoba66 5y ago
  > crypto algorithm negotiation If you control both sides, then you can ignore this part or do it out-of-band. Though, if you control both sides, then you can use literally anything else too.
  - jerf 5y ago
    But that's the problem with JWTs, the whole "if you..." part. You want fewer of those rather than more in your crypto code, and JWT has too many. That's the whole problem.
- benatkin 5y ago
  > Even if you strictly allow-list which algorithm you want to support This is the norm. As for the key ID attack, this sounds like it's just a trick to know where the private key is located? It shouldn't be publicly accessible. https://blog.pentesteracademy.com/hacking-jwt-tokens-kid-claim-misuse-key-leak-e7fce9a10a9c https://blog.pentesteracademy.com/hacking-jwt-tokens-kid-cla...

betaby 5y ago

Simple random tokens are essentially passwords. Thus one should store them 'salted' in the authentication database. Is my understanding correct?

simonw 5y ago
I don't see the need to salt them in most cases. The reason you hash and store passwords with a salt is that there's a very good chance that a user will have used that same password somewhere else. As such, it's important that you make it as hard as possible for you - or a co-worker - or someone who gets hold of a dump of your database - to gain access to the original password. Random tokens were generated by you. They are only valid against your own service. If someone bad gets hold of them, all they can do is make API calls against your service until that token is revoked. So salting and hashing them doesn't win you much. That said, if a lot of people at your company have unfettered access to run queries against your database it may be a good idea to store your random tokens in a way that prevents people from copying them out and abusing them. But if you're storing any private data at all and you have that kind of a culture you have bigger problems you need to solve.
- tptacek 5y ago
  If you're worried about database timing attacks (I'm mostly not) then there's a good thread about that here: https://twitter.com/colmmacc/status/1430282652821585920 https://twitter.com/colmmacc/status/1430282652821585920
deleted 5y ago
[deleted]

keithwinstein 5y ago

I'd love to hear tptacek's view on the pros and cons of "authenticated connections," where the client uses asymmetric crypto to establish an authenticated-as-the-principal bytestream, and then issues a bunch of unsigned requests over that bytestream. In practice, I'm talking about (1) "SSH public-key authentication to GitHub" (or to an EC2 or GCE instance), or (2) using FIDO2/WebAuthn (or TLS client certs...) to authenticate an HTTP connection or browsing session. One major pro is that this is a lot more widely implemented and standardized than "authenticated requests" -- there are lots of reasonable implementations of SSH pubkey authentication or FIDO2/WebAuthn, compared with trying to sign an individual HTTP request. And that maturity brings robustness and security, e.g., I like that I can have my SSH private key (or my FIDO2 private key) on a tamper-resistant hardware device and not worry about some rogue software stealing my private key, whereas trying to find a hardware device to hold my AWS credentials and do the AWSv4 signature is a different story. Any bearer-token-based scheme seems... a lot scarier in that sense. I'm less clear on the practical cons (assuming "authenticated requests" is not available). With SSH keys or TLS client certs, I could imagine it's annoying because only the terminus of the secure connection can verify that the client is who it claims to be, and maybe it's annoying to have your SSH/TLS termination component directly talk to a user database to authenticate individual clients and then attest that it did so to the rest of the backend. But... somehow GitHub/GitLab/Bitbucket all manage to do this at scale, and with WebAuthn/FIDO2, any component can be the one to make the challenge. (I suppose only recently can you make a "user presence not required" challenge, so maybe that's why nobody uses this in the non-presence setting...?)

tptacek 5y ago
I have actually a lot to say about this, in the context of mTLS; it's a post I'm working on. I am generally a fan of the mTLS approach for very simple topologies, like "central Consul cluster nobody can talk to without the right client certificate". I am much less a fan of it for the general inter-service authentication problem.

FreakLegion 5y ago

I get the criticism of ECDH-ES with JWT in vivo, but not the linked tweets' broadside against it in vitro. Are we not just talking about ECIES?

tptacek 5y ago
It's all the things that had to go wrong all at once. P-curves, not Curve25519 (where curve point validation is less important). Static-ephemeral ECDH, so there's a key to target. Long-term durable keys. I'm not making an argument against ECIES (though I guess I'd push not to use the P-curves).

catchmeifyoucan 5y ago

What about revokable JWT tokens? Isn't that similar to the random token approach, except each token holds some additional meaning, and revokable.

tptacek 5y ago
Yes. Revocation is painful with all of the stateless approaches; it's not on its own a reason to avoid JWT. On the other hand, the folkloric draw of JWT is that it's stateless, and they're only stateless if you can revoke them without issuing SQL queries.
- initplus 5y ago
  A downside of stateful auth is the extra DB round trip on every request. Different revocation techniques like periodically distributing a revocation list to your auth services can resolve that part of the issue.

RustyRussell 5y ago

Because standards are always improved by creating new options, I should note runes here (simplified macaroons with Python implementation): https://pypi.org/project/runes/ https://pypi.org/project/runes/ Disclosure: I'm the author

Scaevolus 5y ago

Google's internal mutual authentication/encryption might be of interest. https://cloud.google.com/security/encryption-in-transit/application-layer-transport-security https://cloud.google.com/security/encryption-in-transit/appl... They show how far you can drive Protocol Buffer Tokens.

monocasa 5y ago

I'm curious where kerberos tickets fit into this. A little out of vogue, but it'd be neat to hear how exactly they're long in the tooth.

tptacek 5y ago
Nobody sane uses Kerberos to do IAM for public APIs. But people use them for inter-service authentication, as the post mentions. It links to an article I wrote a couple years ago that considers Kerberos in the context of a variety of other inter-service security tools.
- monocasa 5y ago
  > Nobody sane uses Kerberos to do IAM for public APIs. Why if you don't mind? I'm actually looking at a major proprietary protocol used by hundreds of millions of users that use Kerberos as the fundamental cryptographic attestation of identity and roles. Like clients all connect to a VPN who's bridge uses the Kerberos ticket to whitelist backend services accessible from that connection. They're basically being used the same as if an API gateway understood oath claims and could stop whole classes of client calls at the perimeter.
  - tptacek 5y ago
    Because it's very complicated and fussy (no HTTP API client framework has K5 built in) and, if you're going to force your clients to use a nonstandard authentication protocol, you can do better than Kerberos. A private CA, mTLS, and an authenticated role-based certificate issuer probably does a better job across the board. Facebook talks a little bit about the tradeoffs here in the paper linked to the post; note that they could have used K5 instead of CATs, and the stuff that CATs does is in some ways a response to the limitations of K5.

ignoramous 5y ago

Very consistent with what tptacek has been advocating for on news.yc over the years. The authn mechanism we use is closer to Keybase's NIST (non-interactive session tokens)[0] that are a mix of AWS-style Bearer Tokens and the usual Random Tokens. Of course, the problems around "logistics" (public-key cryptography)[1] are a real nightmare as the post points out. We exchange these tokens between devices (if needed) over password-authenticated channels (using CPACE [2]). [0] https://keybase.io/docs/api/1.0/nist https://keybase.io/docs/api/1.0/nist [1] As examples, see what goes on when a Keybase user associates a new device: https://book.keybase.io/docs/crypto/key-exchange https://book.keybase.io/docs/crypto/key-exchange or when SQRL user revokes compromised keys: https://www.grc.com/sqrl/idlock.htm https://www.grc.com/sqrl/idlock.htm [2] https://github.com/jedisct1/cpace https://github.com/jedisct1/cpace

cormacrelf 5y ago
Is NIST NIST-compliant?
- ignoramous 5y ago
  Nothing to do with NIST.gov, that's just unfortunate naming coincidence. Re: Compliance: Keybase, if am not wrong, has pretty much rolled out their own crypto here. At least in one instance they were subject to criticism for rolling out a key-wrapping scheme viz. TripleSec: https://news.ycombinator.com/item?id=9655245 https://news.ycombinator.com/item?id=9655245

alex-olivier 5y ago

I tend to fall in the trap of once you have authenticated the person and passing tokens around that can hold claims/role for authorization, you will eventually reach the point where the tokens are getting so bloated due to the complexity of access controls. AuthN is the next big challenge especially in a multi-tenant/multi-enterprise SaaS type platform https://cerbos.dev/blog/the-hidden-costs-of-user-authorization https://cerbos.dev/blog/the-hidden-costs-of-user-authorizati...

alex-olivier 5y ago
...and by that I mean authorization is the next big challenge especially in a multi-tenant/multi-enterprise SaaS type platform
- tptacek 5y ago
  Authorization is authz, for what it's worth. "authn" means authentication.

traceroute66 5y ago

You know, what I'd like to see more than standardised API tokens to make scanning easier is actual addressing of the underlying problem. For example, we had a pentest done on a website and the pentester got all excited because they found some AWS tokens. Trouble is, they would be worthless to anyone external because we were making good use of AWS IAM to lock them down with ACLs, Roles etc. So it was effectively a non-event. What happened to the old concept of layered security ? Why should discovery or leakage of an API key automatically give the attacker all the keys to the castle ? In my ideal world, all cloud and API service operators would have the equivalent to AWS IAM and preferably would enforce its usage (i.e. "here's your API key, but it won't work until you set some layered security")

innomatics 5y ago
This very good practice and I am pleased that the one time I got the dreaded message from GitHub and AWS, that I'd done the unthinkable (on a public repo), the keys were only for accessing a single junk dev S3 bucket - phew! But no amount of layering makes the problem go away. Sometimes god-like keys are unavoidable. Those needed by Terraform etc are an example that comes to mind.

mwcampbell 5y ago

> I continue to believe that boring, trustworthy random tokens are underrated, and that people burn a lot of complexity chasing statelessness they can't achieve and won’t need, because token databases for most systems outside of Facebook aren’t hard to scale. Then why did you decide to go with macaroons rather than random tokens? Do you know for sure that fly.io needs that statelessness?

mrkurt 5y ago
We need statelessness for service reliability more than we need it for scale. There are a number of services that people use within our infrastructure that need to continue working even when a centralized DB is unavailable. Logs are a good example. We expose logs as a NATS service, people auth and subscribe to their logs. It's useful to terminate auth right there both because it's faster and because it continues to work when the internet burps.
tptacek 5y ago
I think this is a good question. I'm writing up our Macaroon implementation as I roll it out --- it'll be a few weeks before I publish --- and, if I've done it right, I'll have answered that question clearly.

sudhirj 5y ago

Ironically my main takeaway is that given the pros and cons, JWT seems like the best balance of security and convenience, assuming you always hardcode/verify the algo. Lots of library support, allows Macaroon type policy setup using the JSON (but no chaining). It's not recommended in the article, but it seems like knowing what's out there has made JWT's usefulness clearer by making its downsides more obvious.

tptacek 5y ago

The one thing I'm not super comfortable about here is my PASETO take. My attitude going in was that PASETO has a lot of boosters and not a lot of critical takes. I can beat up on Macaroons because we're using them, and I'm going to follow up with a post about what our Macaroons like like. I'm not doing that with PASETO. So, like, I stand by it, but take it for what it's worth. Later I turned the post into a handy chart. Let me know if you want a poster of it. https://twitter.com/tqbf/status/1430303808945524739 https://twitter.com/tqbf/status/1430303808945524739

tingletech 5y ago
is there a key to decode the scores on the chart?
- tptacek 5y ago
  This seemed funny until I realized everyone else isn't 40+ years old and former Consumer Reports subscribers. The red-with-white-dot thing is "best".
  - mbesto 5y ago
    Ya, I can't parse this whatsoever. What is the difference between red and black? Thick circle? Half circle? No circle? Original harvey balls might be better here: https://en.wikipedia.org/wiki/Harvey_balls https://en.wikipedia.org/wiki/Harvey_balls PS - thanks for putting together.
    - tptacek 5y ago
      You'll have to take that up with Consumer Reports. :) The chart isn't very serious.
    - nwsm 5y ago
      Here's some help (left to right is worst to best): https://www.pentagram.com/work/consumer-reports#22596 https://www.pentagram.com/work/consumer-reports#22596 I still don't know where the black with white dot goes. I guess it's one better than the empty circle in the middle.

zrail 5y ago

There's an additional nuance to opaque random-ish tokens that can be helpful in high-traffic situations. You can essentially encode some, for lack of a better word, "routing" information (shard, region, etc) into the token when you generate it. It's still random, you still verify the whole token with your database, but you can extract the routing info and pass it to the correct backend from a mostly-stateless frontend.

mooreds 5y ago

This was great. A really fair survey of various token methods. Plus plenty of liveliness, not boring at all. Thanks, OP! One thing that I wish was addressed more was language/library support. It gets casual references a couple of times, but for an average developer (as I consider myself) a set of robust, supported open source libraries that help me use a token is so important (not write an implementation, but use in a project that just wants to use the tokens safely). I don't have anything but anecdata, but I feel like most software is going to be in the 'just want to use it' category, rather than the 'need to implement it'. This is where the standards like OAuth and JWT win right now. That doesn't mean they always will, but in my experience, that's the current situation.

CiPHPerCoder 5y ago
For PASETO, the quick guide to library support is https://paseto.io https://paseto.io

simonw 5y ago

This is great. One thing that's worth remembering about randomly generated tokens is that it's important to always use safe comparison methods when comparing them to the stored one - otherwise you could be vulnerable to timing attacks. In Python you can use secrets.compare_digest(a, b) for this: https://docs.python.org/3/library/secrets.html#secrets.compare_digest https://docs.python.org/3/library/secrets.html#secrets.compa...

SilverRed 5y ago
Could you elaborate on what you mean by timing attacks? What would go wrong if you did a basic params[:token] == dbvalue ?
- pests 5y ago
  Most naive string comparison methods (including language expressions like above) will usually compare one letter at a time until a mismatch is found. This allows an attacker to build up the correct password one character at a time. The correct letter will take longer to check then the incorrect.
  - quelltext 5y ago
    But in the "real world" wouldn't any such time difference be very small and fluctuations in network delay and measurement error so large in comparison that this isn't typically exploitable? Sure, better be safe and use a safe comparison method and that's simple enough, but still, how realistic is such an attack over the public internet? I'm aware of "practical" results: http://crypto.stanford.edu/~dabo/papers/ssl-timing.pdf http://crypto.stanford.edu/~dabo/papers/ssl-timing.pdf But the experiment setup was on a LAN, with AFAICT a single server, no other traffic to that server, and other such conditions.
    - simonw 5y ago
      The trick with timing attacks is that you don't measure the time taken for a single request. You send the same request thousands of times and take an average of the response time, which lets you pick away at the secret one character at a time.

deleted 5y ago

[deleted]