9 ms·

Building a Stateless API Proxy

152 points by panarky 7y ago

siscia 7y ago
Is there a reason why they don't encrypt both the token and the permissions? This would remove the needs of a signature altogether.
- theacodes 7y ago
  Because JWT is well specified and supported and it's useful in our case for proxy users to be able to inspect the token.
- yardstick 7y ago
  You still need an integrity check or signature on the encrypted data, otherwise it’s potentially possible for someone to tamper with the ciphertext to change specific parts, such as the permissions. If you are encrypting using an AEAD cipher like AES-GCM or ChaCha20-Poly1305 then it is already built in. But AES-CBC and others need an explicit verification on top.
  - nkrisc 7y ago
    Ok I'm a bit of a cryptography noob, but are you saying it's possible to alter the encrypted token such that when decrypted by the private key the permissions are changed, in this example, but the real GitHub token is not? Edit: nevermind, I think I've just misunderstood the process outlined in the article. I confused the tokens.
    - deleted 7y ago
      [deleted]
pm90 7y ago
Slightly OT: Github has API rate limits that often slaughter us (we use Jenkins to scan our Github org), and will get worse in the future as we move/create more repos. Could this be used to alleviate that? I believe that would need some caching, but ... I don't know how caches work exactly, and how I would go about implementing that here...
- kolektiv 7y ago
  Yes, you could potentially do something like that. A cache is only in essence "I keep a local store, if someone asks for something which is already in the local store, I give them that - if not, I go and get it, put it in the local store for the next person that wants it, and then give them that". Of course, having things expire (knowing when to remove them from the cache) is not a trivial problem, but there's a lot of prior art out there (look for "cache expiry") and there's also quite a lot built in to HTTP to ensure that HTTP content can be cached well (proxies are a very common thing - look in to "HTTP cache headers"). Of course, in your case, it might be debatable in terms of utility - if you're trying to replace a scan, you probably are trying to get results as "up to date" as you can - which would be prevented if the cache avoided doing that! The cache would have the same rate limits, and so you would be just as well off adjusting the frequency you currently scan. Of course, if you can't control that frequency (perhaps multiple uncoordinated scanners) a proxy is one way to give you that control - so maybe useful!
- pseudobry 7y ago
  Take a look at Maintner, a tool made by the Go team to load and store GitHub metadata, particularly issues, comments, reviews, and events: https://godoc.org/golang.org/x/build/maintner https://godoc.org/golang.org/x/build/maintner. It handles backoff, holds all data in memory, and supports backing up data to GCS. My team is successfully using it to track ~200k issues/PRs across ~300 repos. We wrapped our deployment in a minimal API to give our infra blazing fast access to that data without needing to worry about GitHub auth or rate limits (since Maintner handles that). And, we use the proxy described in the article.
- deforciant 7y ago
  shameless plug: I initially created webhookrelay for just this case - Jenkins & Github :) Instead of polling, you can start builds on webhook request. It can create one-way tunnels so your Jenkins is not exposed to the internet (for PRs https://webhookrelay.com/blog/2019/04/17/automated-github-pull-request-builds-on-jenkins/ https://webhookrelay.com/blog/2019/04/17/automated-github-pu...). Open source client for webhook forwarding is also available! Now the project evolved into way more, but still, started from Jenkins, polling and waiting for the build to start after pushing changes.
dnet 7y ago
Using Elliptic Curve cryptography would've resulted in much smaller signatures, libsodium is considered secure and has bindings to most sane/modern environments: https://download.libsodium.org/doc/ https://download.libsodium.org/doc/ JWT also has its fair share of security issues in itself: https://paragonie.com/blog/2017/03/jwt-json-web-tokens-is-bad-standard-that-everyone-should-avoid https://paragonie.com/blog/2017/03/jwt-json-web-tokens-is-ba...
- icebraining 7y ago
  Not saying the standard is good, but in this case, the author is specifying a fixed signature algorithm, so that problem doesn't apply, as far as I can tell.
- 1_player 7y ago
  That JWT link sounds like hyperbole. Can anybody chime in on whether JWT is absolutely broken as stated in the article, or, while it has some issues, the author likes being a bit too dramatic?
  - weego 7y ago
    The root of most of the points in the article is "when you use JWT for the wrong use case it is bad" which is self-evident. Other than that it is hyperbole imo.
  - pillfill 7y ago
    Unnecessarly and overly dramatic, essentially arguing that because JWT/JOSE is insecure because it can be used in an insecure manner.
  - lvh 7y ago
    JWT is multiple layers of bad. My favorite summary is that it has poor implementations of a harebrained scheme designed to solve a problem you don't have. The idea that I need to read the header, which is unauthenticated, to parse the token violates the Cryptographic Doom Principle. Has that led to vulnerabilities? Of course it has: I just said it violates the Cryptographic Doom Principle. The idea that it has everything plus the kitchen sink -- even for drastically different behavior and opinions on how the world works, from symmetric encryption to asymmetric signing and multiple implementations of each at that, is anathema to modern cryptographic design. Wireguard has one scheme and it does a lot more complicated stuff than "encrypt a session token". JWT's saving grace here is that few people implement all of it. And ... that's ... cool? Until they do, of course. You can argue that something is an implementation problem and not a spec problem. Some issues definitely are, but if every major implementation has the same damn bug, then I think it's a spec problem. Unauthenticated headers are a spec problem. PKCS1V15 enc is a spec problem. The fact that an implementation can patch around it doesn't make it not a spec problem. I'm sitting on several more vulns in ~every JWT library that are, to cryptographers, literally too boring to publish even though one of them is _key recovery_. Other posters have said that it's silly to say that merely the ability to use it unsafely is a problem. But good crypto looks exactly like bad crypto while you're doing it, and there's good crypto that doesn't have that set of problems, so why would you ever choose the poor design? (Don't use JWT.)
kolektiv 7y ago
Just a quick supportive comment on this - it's a really nice piece of writing, introducing a topic which tends to cause people to glaze over, and doing it in a very approachable and easy to pick up way - and that takes some skill and effort!
- dinkleberg 7y ago
  Agreed! I really like how they introduce the topic incrementally, eventually getting to JWT, but not throwing that at you from the get go.
- max23_ 7y ago
  I am having the same thought too after reading through it. Good job on the step by step explanation together with the illustrations.
- theacodes 7y ago
  Thank you so much, I really appreciate the kind words!
- tamasnet 7y ago
  +1. Really a well structured and written article, with a good mix of clear prose & graphics to keep it interesting. I'm in awe of folks who make the time for the kind of effort this requires.
mgraczyk 7y ago
Cool stuff! I use a similar thing in my own servers to generate short-lived or one-time access to individual requests. Nice to be able to share an url or `curl` command with somebody to see what I am seeing or for ad-hoc permission grants.
sleepychu 7y ago
How does the proxy get the initial token? Do you hand it a GH token and get back your magic token?
- simonw 7y ago
  Yes, it has an API you can call that will encrypt the token with the proxy's key. Implementation here: https://github.com/theacodes/magic-github-proxy/blob/master/src/magicproxy/async_proxy.py#L32 https://github.com/theacodes/magic-github-proxy/blob/master/...
lvh 7y ago
Firstly, great writing. Secondly, magic proxies are awesome. And pyca/cryptography instead of something terrible like pycryptodome! All great stuff. Some critique of the crypto bits, since I'm a crypto person. 1. Do you actually need asymmetric cryptography for this? It seems like at some point the proxy has full authority, and it could just encrypt the token for you symmetrically? (This is valuable because symmetric crypto is a lot less precarious than asymmetric crypto, see next point) 2. Please don't use PKCS1v15 padding for encryption in new systems. It's been known to be busted for about 20 years now. We have workarounds, and they may well be deployed in the exact context you're using it. But they keep breaking, because we just have them to keep the infinite amount of already-deployed software running, not because we think it's fundamentally fixable. This is also the textbook example of a vulnerable service: one that takes ciphertexts and decrypts them on demand. With PKCSv15, I can modify the ciphertext so that _the way you treat the modified ciphertext_ tells me something about how to make private key operations. And in this setup, that means I get the real token, so that sounds bad. The good news is that you've successfully designed around it by adding a signature, so I don't think it's mountable... but. Please, no more PKCS1v15 :-( 3. It feels a little awkward to use JWT for the outside signature but not the inside encryption. But less JWT is a good thing :) Concrete suggestion: use Fernet (you're already using pyca/cryptography) or libsodium's secretbox and then all of the crypto problems go away. You keep the security engineering dilemma of stateful v stateless proxy (do I want the real token in the proxy at all?) -- but that's another argument.
- theacodes 7y ago
  This great feedback. I'm definitely not a crypto expert. I'm happy to update the post with a different padding algorithm if you want to suggest one. I'm a little hesitant on swapping out RSA because I intentionally picked something relatively simple and familiar to folks, but yeah, will still do some research. Others had suggested ECC which I think is totally worth noting in the post one way or another. Thank you!
  - lvh 7y ago
    The answer to your question is OAEP but I feel like I'd still be doing you a disservice there because I am convinced the answer ought to be box/secretbox or Fernet or AESGCM maybe -- but that hinges on my question about asymmetric cryptography which is elsewhere in the thread :)
0xdeadbeefbabe 7y ago
This begs the question why not be github? I bet some permutation of this idea occurs to the next developer.
twblalock 7y ago
As the article mentioned, revocation is a problem with the stateless approach. I've never seen a way to revoke individual tokens statelessly -- you either need to maintain state about the valid tokens, or maintain state about a revocation list.
- greiskul 7y ago
  I don't think it's possible to do revocation in a stateless manner. The token that you are revoking was once valid, and when you decide to invalidate it, this change of state needs to be persisted somewhere. But yeah, if revocation is rare and only few tokens are invalid at any given time (which can be easy by adding an expiry field to tokens), keeping a revocation list is the way to go.
languagehacker 7y ago
This is great! A couple of years ago I needed to write a quick-and-dirty proxy in Golang to get around some draconian security policies placed on us by the team running our GitHub: Enterprise server. We released it as an open-source project here: https://github.com/electric-it/hubbard https://github.com/electric-it/hubbard
ricardbejarano 7y ago
I built something similar for the Amazon SES (Simple Email Service) for my cron jobs and other private applications to use. https://github.com/ricardbejarano/postino https://github.com/ricardbejarano/postino
kureikain 7y ago
Anyone knows what tool is used to generate/ those images in the post?
- theacodes 7y ago
  I hand-drew the illustrations using an iPad pro & photoshop. The code highlighting is using Sphinx + witchhazel.thea.codes.

152 points by panarky 7y ago

Is there a reason why they don't encrypt both the token and the permissions? This would remove the needs of a signature altogether.

theacodes 7y ago
Because JWT is well specified and supported and it's useful in our case for proxy users to be able to inspect the token.
yardstick 7y ago
You still need an integrity check or signature on the encrypted data, otherwise it’s potentially possible for someone to tamper with the ciphertext to change specific parts, such as the permissions. If you are encrypting using an AEAD cipher like AES-GCM or ChaCha20-Poly1305 then it is already built in. But AES-CBC and others need an explicit verification on top.
- nkrisc 7y ago
  Ok I'm a bit of a cryptography noob, but are you saying it's possible to alter the encrypted token such that when decrypted by the private key the permissions are changed, in this example, but the real GitHub token is not? Edit: nevermind, I think I've just misunderstood the process outlined in the article. I confused the tokens.
  - deleted 7y ago
    [deleted]

pm90 7y ago

Slightly OT: Github has API rate limits that often slaughter us (we use Jenkins to scan our Github org), and will get worse in the future as we move/create more repos. Could this be used to alleviate that? I believe that would need some caching, but ... I don't know how caches work exactly, and how I would go about implementing that here...

kolektiv 7y ago
Yes, you could potentially do something like that. A cache is only in essence "I keep a local store, if someone asks for something which is already in the local store, I give them that - if not, I go and get it, put it in the local store for the next person that wants it, and then give them that". Of course, having things expire (knowing when to remove them from the cache) is not a trivial problem, but there's a lot of prior art out there (look for "cache expiry") and there's also quite a lot built in to HTTP to ensure that HTTP content can be cached well (proxies are a very common thing - look in to "HTTP cache headers"). Of course, in your case, it might be debatable in terms of utility - if you're trying to replace a scan, you probably are trying to get results as "up to date" as you can - which would be prevented if the cache avoided doing that! The cache would have the same rate limits, and so you would be just as well off adjusting the frequency you currently scan. Of course, if you can't control that frequency (perhaps multiple uncoordinated scanners) a proxy is one way to give you that control - so maybe useful!
pseudobry 7y ago
Take a look at Maintner, a tool made by the Go team to load and store GitHub metadata, particularly issues, comments, reviews, and events: https://godoc.org/golang.org/x/build/maintner https://godoc.org/golang.org/x/build/maintner. It handles backoff, holds all data in memory, and supports backing up data to GCS. My team is successfully using it to track ~200k issues/PRs across ~300 repos. We wrapped our deployment in a minimal API to give our infra blazing fast access to that data without needing to worry about GitHub auth or rate limits (since Maintner handles that). And, we use the proxy described in the article.
deforciant 7y ago
shameless plug: I initially created webhookrelay for just this case - Jenkins & Github :) Instead of polling, you can start builds on webhook request. It can create one-way tunnels so your Jenkins is not exposed to the internet (for PRs https://webhookrelay.com/blog/2019/04/17/automated-github-pull-request-builds-on-jenkins/ https://webhookrelay.com/blog/2019/04/17/automated-github-pu...). Open source client for webhook forwarding is also available! Now the project evolved into way more, but still, started from Jenkins, polling and waiting for the build to start after pushing changes.

dnet 7y ago

Using Elliptic Curve cryptography would've resulted in much smaller signatures, libsodium is considered secure and has bindings to most sane/modern environments: https://download.libsodium.org/doc/ https://download.libsodium.org/doc/ JWT also has its fair share of security issues in itself: https://paragonie.com/blog/2017/03/jwt-json-web-tokens-is-bad-standard-that-everyone-should-avoid https://paragonie.com/blog/2017/03/jwt-json-web-tokens-is-ba...

icebraining 7y ago
Not saying the standard is good, but in this case, the author is specifying a fixed signature algorithm, so that problem doesn't apply, as far as I can tell.
1_player 7y ago
That JWT link sounds like hyperbole. Can anybody chime in on whether JWT is absolutely broken as stated in the article, or, while it has some issues, the author likes being a bit too dramatic?
- weego 7y ago
  The root of most of the points in the article is "when you use JWT for the wrong use case it is bad" which is self-evident. Other than that it is hyperbole imo.
- pillfill 7y ago
  Unnecessarly and overly dramatic, essentially arguing that because JWT/JOSE is insecure because it can be used in an insecure manner.
- lvh 7y ago
  JWT is multiple layers of bad. My favorite summary is that it has poor implementations of a harebrained scheme designed to solve a problem you don't have. The idea that I need to read the header, which is unauthenticated, to parse the token violates the Cryptographic Doom Principle. Has that led to vulnerabilities? Of course it has: I just said it violates the Cryptographic Doom Principle. The idea that it has everything plus the kitchen sink -- even for drastically different behavior and opinions on how the world works, from symmetric encryption to asymmetric signing and multiple implementations of each at that, is anathema to modern cryptographic design. Wireguard has one scheme and it does a lot more complicated stuff than "encrypt a session token". JWT's saving grace here is that few people implement all of it. And ... that's ... cool? Until they do, of course. You can argue that something is an implementation problem and not a spec problem. Some issues definitely are, but if every major implementation has the same damn bug, then I think it's a spec problem. Unauthenticated headers are a spec problem. PKCS1V15 enc is a spec problem. The fact that an implementation can patch around it doesn't make it not a spec problem. I'm sitting on several more vulns in ~every JWT library that are, to cryptographers, literally too boring to publish even though one of them is _key recovery_. Other posters have said that it's silly to say that merely the ability to use it unsafely is a problem. But good crypto looks exactly like bad crypto while you're doing it, and there's good crypto that doesn't have that set of problems, so why would you ever choose the poor design? (Don't use JWT.)

kolektiv 7y ago

Just a quick supportive comment on this - it's a really nice piece of writing, introducing a topic which tends to cause people to glaze over, and doing it in a very approachable and easy to pick up way - and that takes some skill and effort!

dinkleberg 7y ago
Agreed! I really like how they introduce the topic incrementally, eventually getting to JWT, but not throwing that at you from the get go.
max23_ 7y ago
I am having the same thought too after reading through it. Good job on the step by step explanation together with the illustrations.
theacodes 7y ago
Thank you so much, I really appreciate the kind words!
tamasnet 7y ago
+1. Really a well structured and written article, with a good mix of clear prose & graphics to keep it interesting. I'm in awe of folks who make the time for the kind of effort this requires.

mgraczyk 7y ago

Cool stuff! I use a similar thing in my own servers to generate short-lived or one-time access to individual requests. Nice to be able to share an url or `curl` command with somebody to see what I am seeing or for ad-hoc permission grants.

sleepychu 7y ago

How does the proxy get the initial token? Do you hand it a GH token and get back your magic token?

simonw 7y ago
Yes, it has an API you can call that will encrypt the token with the proxy's key. Implementation here: https://github.com/theacodes/magic-github-proxy/blob/master/src/magicproxy/async_proxy.py#L32 https://github.com/theacodes/magic-github-proxy/blob/master/...

lvh 7y ago

Firstly, great writing. Secondly, magic proxies are awesome. And pyca/cryptography instead of something terrible like pycryptodome! All great stuff. Some critique of the crypto bits, since I'm a crypto person. 1. Do you actually need asymmetric cryptography for this? It seems like at some point the proxy has full authority, and it could just encrypt the token for you symmetrically? (This is valuable because symmetric crypto is a lot less precarious than asymmetric crypto, see next point) 2. Please don't use PKCS1v15 padding for encryption in new systems. It's been known to be busted for about 20 years now. We have workarounds, and they may well be deployed in the exact context you're using it. But they keep breaking, because we just have them to keep the infinite amount of already-deployed software running, not because we think it's fundamentally fixable. This is also the textbook example of a vulnerable service: one that takes ciphertexts and decrypts them on demand. With PKCSv15, I can modify the ciphertext so that _the way you treat the modified ciphertext_ tells me something about how to make private key operations. And in this setup, that means I get the real token, so that sounds bad. The good news is that you've successfully designed around it by adding a signature, so I don't think it's mountable... but. Please, no more PKCS1v15 :-( 3. It feels a little awkward to use JWT for the outside signature but not the inside encryption. But less JWT is a good thing :) Concrete suggestion: use Fernet (you're already using pyca/cryptography) or libsodium's secretbox and then all of the crypto problems go away. You keep the security engineering dilemma of stateful v stateless proxy (do I want the real token in the proxy at all?) -- but that's another argument.

theacodes 7y ago
This great feedback. I'm definitely not a crypto expert. I'm happy to update the post with a different padding algorithm if you want to suggest one. I'm a little hesitant on swapping out RSA because I intentionally picked something relatively simple and familiar to folks, but yeah, will still do some research. Others had suggested ECC which I think is totally worth noting in the post one way or another. Thank you!
- lvh 7y ago
  The answer to your question is OAEP but I feel like I'd still be doing you a disservice there because I am convinced the answer ought to be box/secretbox or Fernet or AESGCM maybe -- but that hinges on my question about asymmetric cryptography which is elsewhere in the thread :)

0xdeadbeefbabe 7y ago

This begs the question why not be github? I bet some permutation of this idea occurs to the next developer.

twblalock 7y ago

As the article mentioned, revocation is a problem with the stateless approach. I've never seen a way to revoke individual tokens statelessly -- you either need to maintain state about the valid tokens, or maintain state about a revocation list.

greiskul 7y ago
I don't think it's possible to do revocation in a stateless manner. The token that you are revoking was once valid, and when you decide to invalidate it, this change of state needs to be persisted somewhere. But yeah, if revocation is rare and only few tokens are invalid at any given time (which can be easy by adding an expiry field to tokens), keeping a revocation list is the way to go.

languagehacker 7y ago

This is great! A couple of years ago I needed to write a quick-and-dirty proxy in Golang to get around some draconian security policies placed on us by the team running our GitHub: Enterprise server. We released it as an open-source project here: https://github.com/electric-it/hubbard https://github.com/electric-it/hubbard

ricardbejarano 7y ago

I built something similar for the Amazon SES (Simple Email Service) for my cron jobs and other private applications to use. https://github.com/ricardbejarano/postino https://github.com/ricardbejarano/postino

kureikain 7y ago

Anyone knows what tool is used to generate/ those images in the post?

theacodes 7y ago
I hand-drew the illustrations using an iPad pro & photoshop. The code highlighting is using Sphinx + witchhazel.thea.codes.