13 ms·

Macaroons Escalated Quickly

263 points by mkeeter 3y ago

ssl232 3y ago
TIL macarons [1] are also known as macaroons [2]. Where I'm from, macaroons are dense, sweet coconut treats and the French sweets are known as macarons. But from the Wikipedia article I see macaroons mean many different things to many different people. [1] https://en.wikipedia.org/wiki/Macaron https://en.wikipedia.org/wiki/Macaron [2] https://en.wikipedia.org/wiki/Macaroon https://en.wikipedia.org/wiki/Macaroon
- cmcconomy 3y ago
  it messed with my head to hear that the english world interprets an "entrée" as a main course
  - ssl232 3y ago
    American English, but not British. Dinner here is a starter, main course and dessert. "Entrée" messes with my head as well, since it very much sounds like a starter.
    - bdsa 3y ago
      It's French for starter, even
    - KineticLensman 3y ago
      > Dinner here is a starter, main course and dessert. Yep. Check out the online menus in English restaurants and pubs. 'Mains' is the typical term used.
      - wiredfool 3y ago
        It's also the electrical supply. What a country.
        ssl232 3y ago
        And the water supply. And the central building on farms (in Scotland, at least). It's a silly country.
  - dboreham 3y ago
    Don't even think about "a la mode" then.
    - aidenn0 3y ago
      I speak French, not Italian!
- gonzus 3y ago
  Switzerland has the similar Luxemburgerli, which my Swiss friends who reside there (as opposed to me, a Swiss person who lives somewhere else) swear is NOT the same as a macaron... I have never understood the purported differences.
- deleted 3y ago
  [deleted]
- demondemidi 3y ago
  I was just about to complain. Thanks for nipping it in the bud!
- ISV_Damocles 3y ago
  This feels relevant to the conversation: https://twitter.com/BigSpiderBack/status/864309383106240512/photo/1 https://twitter.com/BigSpiderBack/status/864309383106240512/...
- enriquto 3y ago
  > TIL macarons [1] are also known as macaroons [2] Not to forget macarrons! Where I'm from, macarrons [3] is a kind of pasta (penne). [3] https://ca.wikipedia.org/wiki/Macarrons https://ca.wikipedia.org/wiki/Macarrons
  - madcaptenor 3y ago
    In English "macaroni" is a kind of pasta. (I'm being deliberately vague because which pasta it is seems to vary.)
    - sph 3y ago
      And the original name for that type of pasta in Italian is "maccheroni" (which are straight tubes, unlike the ones used abroad for "mac and cheese")
  - pasc1878 3y ago
    Or in English macaroni https://en.wikipedia.org/wiki/Macaroni https://en.wikipedia.org/wiki/Macaroni
- CoastalCoder 3y ago
  > Where I'm from, macaroons are dense, sweet coconut treats and the French sweets are known as macarons. Same for me (New England, USA). Both are delicious, but in my dialect they are not the same thing.
  - bageler 3y ago
    This is not a dialect thing, this is a factual thing. They are different, and anyone that uses macarOOn to mean a sweetly filled meringue sandwich is wrong.
- Nition 3y ago
  Oh funny, I actually came to this comment thread to point out that they'd accidentally used a picture of macarons instead of macaroons in the blog. But I guess not necessarily!
plucas 3y ago
[flagged]
- deleted 3y ago
  [deleted]
larsrc 3y ago
[flagged]
xyzzy_plugh 3y ago
> Until now! I dragged Fly.io into implementing them. Suckers! Let's be real: they knew you would do this eventually. It was inevitable.
asplake 3y ago
Typo/bug: pretty sure variable 'oldTag' should be 'oldTail'
- tptacek 3y ago
  Thanks! They're basically interchangeable --- HMAC produces a "MAC tag", and in the context of the Macaroon protocol the last visible tag is called a "tail". But I'm guessing I screwed up editing the code when I switched everything to "tail". :)
DonsDiscountGas 3y ago
The title and graphic suggest an article about the most delicious treat known to humans, and the post is about software. My disappointment is immeasurable and my day is ruined.
- chankstein38 3y ago
  Yeah I clicked here thinking this would be an interesting story about why macarons are so popular these days or something. When I saw a code block I closed immediately. I'm a programmer but I don't care about every random, poorly-named library or framework that exists.
  - apetresc 3y ago
    To be fair, it's not a library or framework, it's a format for certain types of security tokens. They're kind of like fancier cookies, hence the name "macaroon". Not so poorly-named, in my opinion (to the extent that "cookies" was ever a sensible name). Just the right level of whimsy while still calling back to what they are.
    - bbarn 3y ago
      A macaroon is a mixed up ball of coconut and egg whites. A macaron is the delicious fancy sandwich cookie.
      - dataflow 3y ago
        And for completeness, a Macron is the president.
        amenghra 3y ago
        Not to be confused with micron, which is the size of most politicians’ accomplishments after they get elected.
        rsynnott 3y ago
matthewaveryusa 3y ago
>and escape hatches like Mutation (for our GraphQL API). Can you explain in a bit more detail what you mean by escape hatches (sounds dangerously fun)
- tptacek 3y ago
  The Organization -> Apps -> Volumes (whatever) structure models our problem domain and is a sort of coherent abstraction. If we get that abstraction wrong, it's a pain to retrofit changes. Caveats like "Mutation", on the other hand, are simply strings representing calls you can make in our API; they're not a model of anything. I don't think we use `Mutation` for anything right now, but it's there in case we (or you) ever need a token that authorizes a particular set of GQL calls that don't fit the model we came up with.
jzelinskie 3y ago
It's probably more relevant to your previous blog post that did a survey of technologies, but I'm curious if y'all ever considered a fully centralized approach. Is there something about Fly that benefits from a decentralized approach? Disclosure: I work on SpiceDB alongside maintainers like the pymacaroons author
- tptacek 3y ago
  Can you define "centralized" and "decentralized" for me? What would a "fully centralized" approach to this problem look like?
  - jzelinskie 3y ago
    Sure thing! These terms may not be the best way to communicate these concepts. Would love your take on that, too. I used the term decentralized to describe claim-based token-passing authorization strategies because the data defining their access is being source from various services, attenuated, and passed around with the request. I use the term centralized to describe something where all the data defining access and the policies are sourced from one service. I personally work on a Google-inspired system, but there are plenty of policy engines using a similar approach.
    - tptacek 3y ago
      Couple thoughts. * With regards to the "core Macaroons" we implement with first-party caveats, we conceptually are centralized, in that there's a single source of truth about which apps belong to which orgs &c that caveats are evaluated against. * The basic challenge we have is providing a comparable experience in every data center we're operating in around the world, so a "purely" centralized model would be problematic for us; people in Sydney would get a much worse experience than people in Ashburn. We get around this with LiteFS, to an extent. This post undersells how important LiteFS is to the deployment model here. * With respect to having, like, a Zanzibar-like access matrix dealy: part of the thing here is not wanting to have to provide a single coherent permissions model to our users, because our problem domain is so broad and we will get it wrong. Providing orthogonal tools to let users express things seems like the better strategy to us? But there are upsides to both approaches. * Of course, one of the big wins here, that we have a plugin interface for authorization, depends on some measure of decentralization. I feel like this is kind of a scattershot response, but hopefully I hit some of what you were talking about, and you can let me know if I've missed the mark completely.
X-Istence 3y ago
Macaroons are also implemented and used by pypi.org's implementation named Warehouse: https://warehouse.pypa.io/development/token-scanning.html https://warehouse.pypa.io/development/token-scanning.html Also see: https://pypitoken.readthedocs.io/en/latest/ https://pypitoken.readthedocs.io/en/latest/
- woodruffw 3y ago
  Yes, although PyPI doesn't currently do much attenuation or delegation with them (this is largely my fault, since I didn't fully understand their power when picking them for the implementation). That's been slowly changing, however -- as of a few months ago, PyPI issues slightly more compact API tokens that make better use of discrete caveats. They're also used on the Trusted Publishing[1] side to make the API token self-expiring. [1]: https://docs.pypi.org/trusted-publishers/ https://docs.pypi.org/trusted-publishers/
iou 3y ago
lol, the picture in the post is of a macaron, not a macaroon https://www.foodnetwork.com/recipes/packages/baking-guide/macarons-vs-macaroons-whats-the-difference https://www.foodnetwork.com/recipes/packages/baking-guide/ma...
- deleted 3y ago
  [deleted]
sergioisidoro 3y ago
So, are Macaroons are like a "blockchain", where each block is an attenuation (ie reducing permission and scopes) of the permission of the original token? So a bearer can get a token, reduce its scope (without talking to the server) and pass it down to other less privileged entities. That's cool! I kind of had to look into the paper linked to really understand what they are. I feel the article could have given a bit more context on it :)
- dwaite 3y ago
  Somewhat. It is symmetric key based, so only the initial issuer can do verification. It is not part of a distributed data store, so there can be multiple copies (and derivatives) of a macaroon floating around. Basically, a trusted service creates an initial atom and shares it with the initial party. That trusted atom can be copied and shared. Anyone with a copy can amend additional information to it, and share those copies. Based on the copy you get, you can't remove or alter any information received. So it works best for representing restrictions - maybe my user agent gets a token that gives full access for a project. It shares a version with another service endpoint which is restricted to read-only access. That service endpoint sees it needs to kick off a background process, so it creates a version which also adds on a 4 hour time limit. None of these changes are necessarily authenticated - I don't know that it specifically was the user agent that shared the token with the service or added the read-only restriction, at least with the core tech. You have to create a domain language to specify all of these things. If you want restrictions to be authenticated, you need to describe how to sign them individually with PKI. While HMAC is fast, you still have the whole system rooted in a single trusted service that can mint any messages it wants or lie about the validity of a given message. So while it is indeed a chain of cryptographic elements, it is missing the multi-party validation and operation transparency which are the common selling points of distributed ledgers.
  - tptacek 3y ago
    Biscuits have single roots of trust too. The (superficial) distinction is that Biscuits can be verified (completely) on systems that don't hold the root keys. You can't do that with a Macaroon, but, as we explain in this post, you don't have to. I don't think we'd meaningfully gain security from Biscuits --- though we would potentially get flexibility from Datalog, and our deployment story might have been a little bit simpler. (We already need hardware-isolated secret stores, so the deloyment win is probably marginal for us, but it's a bigger deal for other shops). Purely symmetric cryptography is one of the features of Macaroons. It's a reason cryptography hipsters like it, not a reason to dunk on it.
jszymborski 3y ago
This post was written exceedingly well. Few posts execute humour and whimsy without coming off as insincere or just not very funny (I'm guilty myself). Bravo!
- robbles 3y ago
  The tone is entertaining, but some of the snarkiness around the code is bit frustrating at times. e.g. > # do i really need to say I'm not serious about this? Can you just come out and say what you mean here? Like, presumably it's bad, insecure code, but can't you just spell it out for the benefit of your audience? I'm probably being really picky here - just find this kind of developer in-crowd signalling to be really irritating and counterproductive.
  - tptacek 3y ago
    The Python code here is essentially just pseudocode. In reality, if you build a Python Macaroon implementation, you'd pull in pyca/cryptography and use an actual AEAD, rather than rolling your own authenticated cipher out of pure HMAC. But the point is that this isn't real code, just enough to make the concepts concrete.
    - byproxy 3y ago
      I can always appreciate that these types of stripped-down examples are merely for illustrative, conceptual purposes...but the ignoramus in me would also appreciate links to fleshed-out examples that take into account the shortcomings of the simpler example.
      - tptacek 3y ago
        Our actual Macaroon code is linked at the bottom of the article.
    - robbles 3y ago
      That's a great explanation, and pretty much exactly what I'd love to see instead of the original slightly mysterious comment :)
wereHamster 3y ago
I wonder why they went chose macaroons over biscuits. The later is only mentioned once in that article, but doesn't talk about why they were not selected.
- tptacek 3y ago
  I like Biscuits, wrote about them before, and recorded a podcast episode with Geoffroy Couprie about them†. The short answers: * When we started working on this design, there was still some uncertainty about the elliptic curve chaining construction (as in: they hadn't, or maybe had just started to, work out which one they were going to use). * We do a lot of verifications, and the prospect of running lots of 25519 computations every time we have to check a token was daunting; by comparison, HMAC is essentially "free". * I'm not especially scared of Ed25519, but asymmetric signature cryptography is much more complex than computing a MAC. Like I said in the post, one of the great charms of Macaroons is how much mileage it gets out of a construction that everyone understands, and that doesn't have sharp edges. * Geoffroy sold me on the value of using a logic language to express caveats, but that was much more complicated than what we felt we needed. Another charm of Macaroons is that they're rigidly coherent; just an unordered collection of true/false predicates, loop over them and if any fail reject the request. I feel safe expressing things with Macaroon caveats --- not in the sense of "the token implementation doesn't have bugs", I'm sure Biscuit's Datalog is fine, but rather in the "I'm not going to mess this up as a user" sense. * We needed implementations in a bunch of different languages. Biscuits have them, but owning the implementation ourselves derisked things for us. We could have owned an implementation of Biscuits, but I'm not smart enough to do that well. † https://securitycryptographywhatever.com/2022/01/29/biscuits-with-geoffroy-couprie/ https://securitycryptographywhatever.com/2022/01/29/biscuits...
  - crdrost 3y ago
    Oh my goodness -- I had this idea!! When I first read a blog post titled "Macaroons are better than cookies" I started thinking about the operational side and I really didn't like the fact that you had to make a network round-trip to some auth-service to validate the macaroon--because that's what the state of the art already was, and it kinda stunk to pay that delay on every single query. (Of course I understood that you could horizontally scale the machines doing auth and load-balance and all that, but I worked at smaller companies where you had a small rack in a colo space or an air-conditioned closet, creating VMs to do all of that ate into a relatively limited resource and made sysadmins raise eyebrows.) I tried for the longest time to put together a solution with hash functions and private-key crypto; I think the thing that bugged me the most was that I never was able to prove that you couldn't do it with those primitives. Eventually I had something similar with public-key crypto but at the time public-key usually meant RSA, so we were talking 1024-bit signatures for each step. Also I think the idea was that you'd furnish a zero-knowledge proof that you knew the key, which was hella complex. Two things then happened: first, if you're talking PKI then you're talking about revocation too; I convinced myself that revocation was necessary for these sorts of credentials (and I now always feel vindicated when places are like "log out all other instances?") but I also convinced myself that revocation required the network hop in any case (which is not true, revocation is actually pretty infrequent and you can push revoked serials to services via event bus and in most cases the latency to ingest those is not a huge issue). So I convinced myself that what I was looking for was ultimately a purely academic exercise, was the first thing. And the second was that I managed to simplify my PKI-plus-hashes-plus-zero-knowledge craziness to just what Biscuits are: "I'm signing a public key plus caveats," (but I didn't add a query language, I was just going to use a map of well-known-keys to values, "valid_before: 123...890"), "I'm going to transmit this to the recipient with its corresponding private key, which they will strip off and generate a new key to mint a new Biscuit." And then RSA was big and slow and I forgot all about this until this HN thread. Ed25519 is the perfect way to fix that -- that and distributed revocation. I still don't see myself using it in practice but it's nice that a library exists if I do... thanks for the walk down memory lane, I will queue up the podcast episode, maybe for tonight or tomorrow.
michelpp 3y ago
I like the "solve the now" perspective here, and having code examples is very helpful to understand some of the rational behind the approach. Having read your previous "tedious survey"[0] post on various token formats, I generally agree with a lot of your conclusions. Curious though about your thought process wrt macaroons vs biscuits. To me the one major downside of macaroons has always been the single shared root symmetric key. Many use cases are addressed by third party attenuation, but then there are the problems like key rotation, having to do online verification, no built in encryption, no peer-to-peer support through an "untrusted" fly.io, and no third party token verification without decryption like in signcryption[1] schemes. Of course this is traded off by having to do PK issuance and management so I can see the simplicity of it. Is fly.io scoping this pretty hard to just auth tokens with third party attenuation, or do you see further development and maybe moving to other token systems like biscuit when/if the need arises to address those known issues? fwiw I've done a bit of research work myself on a token format using signcryption [2] where I explored addressing some of these ideas (but not the attenuation side of it yet, which I get is a big deal here). [0] https://fly.io/blog/api-tokens-a-tedious-survey/ https://fly.io/blog/api-tokens-a-tedious-survey/ [1] https://github.com/jedisct1/libsodium-signcryption https://github.com/jedisct1/libsodium-signcryption [2] https://github.com/michelp/pgsodium/blob/feat/signcryption-tokens/sql/pgsodium--3.1.7--3.2.0.sql https://github.com/michelp/pgsodium/blob/feat/signcryption-t...
- tptacek 3y ago
  We're not going to use Biscuits. The "single root symmetric key" thing is a systems design challenge for us, not a cryptographic one: we simply isolate the key handling on a software-HSM-like verifier service, running on dedicated hardware. If you're coding authz logic for Fly.io's platform code, your experience will be that you can manipulate and check Macaroon caveats without ever having access to the keys themselves, because we split Macaroon-checking into "verification" and "caveat clearing". The threats in this design that would force us to do abrupt key rotations and mass token invalidation are thus pretty much the same as the ones we'd face with Biscuits; the "trusted code base" of key-handling code is comparably small. More detail here: https://github.com/superfly/macaroon/blob/main/macaroon-thought.md https://github.com/superfly/macaroon/blob/main/macaroon-thou...
tptacek 3y ago
Back in 2021, when I wrote the API token survey post, I had some, uh, caveats about Macaroons, and linked to a talk from a developer at Chain who had gone through the experience of deploying them. Don't get me wrong, it's still an excellent talk†, but after working on implementing them for a year or two I find it much less damning than I did at the time. One problem Chain seemed to run into was coupling between their services. They had a fast Golang "ledger" and a slow Rails "dashboard", and found that Macaroons forced all their requests to loop through the "dashboard". This was because they'd opted to do short-lived Macaroon tokens that needed to be reissued every 5 minutes (eminently sane), and only their dashboard could generate the proper token? Another problem they had with Macaroons: their Rails dashboard has an RBAC permissons interface. But once they've issued a Macaroon, users can't change permissions with the interface anymore. But that's a problem with all stateless tokens, not just Macaroons; in fact, Macaroons probably help here, because if you're presenting a Macaroon to the dashboard interface in the first place, the RBAC interface can just attenuate it for you and hand it back to you. I still think this is a design that mostly makes sense only if you have a problem domain for which both attenuation and delegation make sense. Most basic CRUD apps don't have these problems, and using Macaroons would be a mistake for them. † https://www.youtube.com/watch?v=MZFv62qz8RU https://www.youtube.com/watch?v=MZFv62qz8RU
- leoqa 3y ago
  I'm curious what your thoughts are on implementing a sane authorization system in 2024. You mentioned writing roles onto macroons but what does your policy look like? Is your surface area simple enough that it's not a concern? I've been on various security teams with disjoint product-facing authz, internal authz and service authz policy engines / mechanisms etc. Additionally, authz gets baked into service code and product interfaces, so it's hard to change later.
  - tptacek 3y ago
    The idea is that Macaroons are a low-level IAM language, designed close to the components that they pertain to, encoded directly into the tokens, and that higher-level IAM policies "compile down" into those tokens.
elbasti 3y ago
What I really want to know is how fly.io has recruited such talent. @tptacek (who wrote this article), @chrismccord (creator of elixir Phoenix/Liveview), and others. Very impressive, but also slightly worrying since some parts of fly.io have such terrible DX/documentation. I've only recently started using them, but the documentation around billing and databases is shockingly bad for a team that is clearly so deeply stacked.
- chubot 3y ago
  Part of it is probably that smart people want to work on harder and more ambitious problems, which YC advisors seem to say a lot. And also smart people want to work together As I understand it, fly.io built a cloud from scratch, from the hardware up. That's HARD. It's typically something only well-funded companies like Amazon / Google / Microsoft do, with billions of dollars in products to justify it. Cloudflare is another company that pretty much did it, and it seems to have taken similarly ambitious engineering (although maybe theirs isn't fully general purpose) https://fly.io/docs/about/security/ https://fly.io/docs/about/security/ - We run on our own hardware deployed in secure data centers like Equinix Most cloud companies in the 2010's built on top of other clouds like AWS. For example, Heroku and I'm pretty sure dotCloud, the company that became Docker. I always thought that was a weird design, but it definitely saves you a lot of time and hassle. At the cost of a lot of software complexity and platform risk. Even so, there were at least 20 or 30 of these companies, and most of them didn't make it. Google App Engine (one of the first PaaS like Heroku, circa ~2007) also built on top of Google's data centers and Borg of course. They didn't manage their own hardware -- they used another cloud for that! From the outside, "hosting" may all look kind of similar, but there are worlds of difference under the hood. --- I'll also say that the subject of this blog post -- cloud security -- is extremely hard. FWIW I joined the team that published the Macaroons paper ~10 years ago, and I tried to implement Macaroons as a side project then (for about a week, it wasn't very serious). I was looking for something simpler and more elegant, but there's no real magic bullet, and I think this post kinda concedes that toward the end. But when you have really hard problems, it's not surprising if the solution is really complex! The problem can actually be infinitely hard, because you have adversaries that are both more powerful, and that adapt. Google learned this the hard way ~10 years ago when it learned that BOTH the United States government and the Chinese government had successfully attacked it :-/ People seem to forget about these incidents: https://en.wikipedia.org/wiki/Operation_Aurora https://en.wikipedia.org/wiki/Operation_Aurora https://en.wikipedia.org/wiki/MUSCULAR https://en.wikipedia.org/wiki/MUSCULAR https://www.washingtonpost.com/world/national-security/nsa-infiltrates-links-to-yahoo-google-data-centers-worldwide-snowden-documents-say/2013/10/30/e51d661e-4166-11e3-8b74-d89d714ca4dd_story.html https://www.washingtonpost.com/world/national-security/nsa-i... So basically, if you are successful, then it earns you more problems :)
windlep 3y ago
I remember when I saw a presentation by the macaroon authors a few years back, there were pending patents that Google filed around them. While the authors claimed Google wouldn't sue anyone, I'm always a bit skeptical about such claims. I thought macaroons would be helpful for some of my use-cases, but since I now knew there were patents that'd be wilful infringement so I didn't bother. I can't find the patents now, so perhaps they were rejected or withdrawn. I had assumed that was why macaroons hadn't caught on more widely. Edit: Found the patent: https://patents.google.com/patent/US9397990B1/ https://patents.google.com/patent/US9397990B1/
- everybodyknows 3y ago
  The Pythonish pseudo-code, rendered to an image (figure 7) -- is that common nowadays? Though the patent I see is dated 2013.
- jimmyl02 3y ago
  maybe this is similar to google's patenting of dropout for neural networks? you can never know but so far there haven't been many adverse effects and they claim that they patent it so others can't maliciously patent and enforce it. dropout patent link: https://patents.google.com/patent/US9406017B2/en https://patents.google.com/patent/US9406017B2/en
  - windlep 3y ago
    That was what the authors claimed when I asked them about the macaroon patent. It'd be nice if Google had a legal document associated with patents they never plan to enforce, or the constraints around when they might enforce them (e.g. only against patent trolls) that a company could rely on.
    - xyzzy_plugh 3y ago
      I don't disagree but maintaining an arsenal of defensive parents is Enterprise IP legal 101. All the big companies do this. The goal is to avoid litigation by mutually assured destruction. At least that's what they tell you. Many projects grant you patents as part of, for example, an OSS project's license. It's not unusual to include a clause that voids any such terms of you litigate over your own parents, for example -- hence the mutually assured destruction. Can you imagine what would happen if Google and Facebook tried to duke out some dumb software patent in court? A waste for all parties. Lastly Google has a wealth of resources here: https://google.github.io/opencasebook/patents/#patents-in-open-source https://google.github.io/opencasebook/patents/#patents-in-op...
8organicbits 3y ago
> we want Macaroon tokens to be safe to transmit between users How does this work for auditing? If user A gives user B a token (perhaps after adding a caveat) how does this system audit log determine who did what? Does that require a third party system?
- tptacek 3y ago
  You can get as many independent Macaroons from us as you want, so while there are ways to handle this with third-party caveats (and never talking to our servers to get another token again), the simpler thing for token provenance with our system is the same as it would be with any other: just issue multiple tokens.
- dwaite 3y ago
  There is no centralized auditing of derivative macaroons, but there could be of verification of those macaroons - you still require a centralized service which knows the core HMAC secret to give a thumbs up or thumbs down. To that end you can define auditing information even when there aren't really caveats - user A appending that they are giving a version of the macaroon to user B with no restrictions on use. Since there isn't a domain language, you have to define all this. JSON can be useful to have an extensible notation for describing this, although you need to up-front declare the difference between information which can be ignored and caveats which much result in failure if not understood.
  - tptacek 3y ago
    Note also that in our scheme, if Alice attenuates a token and gives it to Bob, Bob will still have to hit our login service to "activate" the token (that's the word I should have used in the post and I'm kicking myself for not thinking to express it that way) by clearing the third-party auth caveat we put on every token. So it's not as if we have no auditing of token delegation. Still: if only for revocation purposes, I'd probably just mint a new token for e.g. a contractor (and certainly for every user). The nice thing about the system is that you can edit the tokens after we give them to you so you're not stuck with our role definitions, which I think is a significant win, but doing all of IAM without talking to our servers more than once, while a cute technical stunt, is probably not all that valuable to users.
fovc 3y ago
Surprised to see JSON here. I recall discussing JWT with tptacek a few years ago, and one of the concerns was the {a: x, a: y} ambiguity in JSON parsers. Was that a concern with this design? My other reaction was that this sounds scary: > a token with no caveats restricts nothing. It’s a god-mode token. Don’t honor it. I guess it's technically "fail closed" but seems kind of brittle
- tptacek 3y ago
  I think it becomes more obvious later in the post that the Python code here is for illustrative purposes only. We don't use JSON. Our actual tokens are strictly typed. But also: it doesn't matter that much, because all of the predicates in a Macaroon are evaluated independently. There's no opportunity to confuse a previous caveat with a subsequent one. That's one of the strengths of having a rigidly coherent design like Macaroons, rather than just a bag of keys and values like JWTs do.
  - fovc 3y ago
    Ah thanks for clarifying! I knew the code was illustrative but didn't realize the JSON part was too
- hinkley 3y ago
  The XML digital signature spec had that problem in spades. It took us about five times as long to button it down as it did to implement. By the time I was done with that project, the document for partners and vendors (if anyone wanted to implement their own instead of using ours) was several times longer than the spec, what with all the extra MUST and MUST NOT situations. Which is not that hard when you're dealing with swiss cheese. Document.findById and Element.findById being able to return different (non-null) results being the most egregious one I can remember.
  - dwaite 3y ago
    XML is a disaster here all of its own, but is at least unambiguous at the parser level. JSON says "if your document does this, parser behavior is undefined". So you need to declare that the parser used should behave in particular ways. Both are best solved a robust specification and supplemental test vectors. If you define meta rules (e.g. JSON parsers must either fail or return the last object property), you still retain the value you hoped for by using something like JSON or XML in the first place.
Arelius 3y ago
These look pretty useful cool... I'm curious a bit though, here, and in other posts in HN, I often see that HMAC, a symmetric key is preferred, going as far as suggesting, in JWT, that other algorithms should not be implemented. Why is that? What are the problems with say RSA? (Ignorning that I'm not sure Asymmetric keys work with Macaroons design at all) From my perspective, Asymmetric keys have been a great boon, in that I can keep the private key, secured on my single auth server, but then freely distribute the public key to the edge, greatly increasing responsiveness, and reducing the bottleneck on the Auth server. Is there some security concern I've been missing?
evancordell 3y ago
> The community that formed around building open source “standard” Macaroons decided to use untyped opaque blobs to represent candidates. I assume "candidates" was supposed to be "caveats" - and as an author of a "standard" macaroon implementation, I completely agree that this is the biggest downfall of Macaroons. With no common caveat language (and no independent "dischargers") it really limits their use to within a single org. And at that point you're basically asking everyone to invent their own token format anyway. Though I don't personally use them much anymore - I think the use-cases for Macaroons are much more limited if you have a Zanzibar! - I appreciate seeing Macaroon discussions pop up and this post and the related discussions it linked out to were a great read.
- xyzzy_plugh 3y ago
  Zanzibar and macaroons are actually pretty complimentary.
bethecloud 3y ago
Storj is also leveraging macaroons. Great write-up on decentralized access control here: https://medium.com/@kleffew/what-is-capability-based-security-227c6e5483a5 https://medium.com/@kleffew/what-is-capability-based-securit...
ijustwanttovote 3y ago
Small detail, there's a picture of macarons and not macaroons. A macaron is a sandwich-like cookie that's filled with jam, ganache, or buttercream. A macaroon is a drop cookie made using shredded coconut.
- arnarbi 3y ago
  Macaroon is just the English word for the French word macaron. ducks for cover
  - beAbU 3y ago
    Macaroon is a completely different confectionery: https://en.m.wikipedia.org/wiki/Macaroon https://en.m.wikipedia.org/wiki/Macaroon
    - IshKebab 3y ago
      Sure, but Macarons (the burger-looking things) are also sometimes known as "macaroons". Yeah. Examples: * https://missmacaroon.co.uk/ https://missmacaroon.co.uk/ * https://www.floristgrays.co.uk/design-202300010/valentines-macaroons.htm https://www.floristgrays.co.uk/design-202300010/valentines-m... * https://www.parisiennesouthwell.com/product-page/macaroons-vegan-gluten-free https://www.parisiennesouthwell.com/product-page/macaroons-v...
      - RussianCow 3y ago
        The fact that people say it doesn't make it any less wrong. :)
        recursive 3y ago
        If enough people do it for long enough, it does. After all, that's how we got the rest of the words.
        3y ago
denton-scratch 3y ago
[flagged]
- tptacek 3y ago
  Good note.
beeks 3y ago
it seems to me that macaroons could get quite big in order to adequately describe allowed resources and permissions? Especially if the API is broad? Much bigger than JWTs.
- tptacek 3y ago
  The resource descriptions are pretty parsimonious; they're binary-encoded MsgPack integers, for the most part. But the cryptography eats up a lot of bytes. They're bigger than JWTs, but probably by a factor less than 2 (I don't know the median JWT size). Basic take: unless you can golf your tokens down to a single terminal line, it doesn't much matter how much bigger or smaller they are. These are all smaller than X.509 documents.
  - beeks 3y ago
    Ah i see I'm (perhaps naively) imagining an exhaustive list of resources in every macaroon, but i guess the permissive evaluation of caveats would help here since it should mean only the most restricted users would have many caveats. Also I suppose if the caveat evaluation code can deal with wildcards then it can be reduced further. ok OK! I'll have to play around and see how these look in practice. We do have a good use-case for this with both delegation and attenuation... sadly our identity provider is married to JWTs and moving off it would take a lot of work. Anyways. Great Post! Certainly got me thinking, and now i'm tempted to just blend this with our existing system of JWTs :smirk:
    - tptacek 3y ago
      The modal caveat is probably (in vibes-based notation): (Organization 4567 ops=read,write,create,control) Followed by the second modal: (Organization 4567 ops=read,write,create,control) (Apps (App 345 ops=read,write,create,control)) You can make a _much_ more complicated token. And, for instance, our deploy tokens for CI/CD systems are pretty complicated, and they even have a disjunctive caveat, but most people are at most probably just going to lock their tokens down to a particular app, or turn them into read-only or start/stop-only tokens. (We modeled everything just to be safe, and also because that stuff is super valuable for service tokens, when we ourselves want to take platform actions on behalf of the user and have it be traceable back to a user authorization. I feel like I did not say enough about how much I like where service tokens are pointing for us.)
xbar 3y ago
I am not a customer, but I do like the fly.io "voice."
RustyRussell 3y ago
Compulsory plug when someone uses macaroons: using an HMAC is overkill. See https://github.com/rustyrussell/runes https://github.com/rustyrussell/runes for a simpler alternative and implementation (this has C and Python, but there's also a Rust implementation because why not?) However, the "no db access" property has proven to be untenable in practice. Users end up wanting to see what runes are issued, blacklist them, know when they were last used, and have rate limits. The last two are a killer, requiring some state to be kept (unless your system allows you to return a modified rune to the user, which is a different workflow from normal bearer creds).
- RustyRussell 3y ago
  Oops, I read further: you did use unique ids, and blacklists! I did that too. Now you're not db-lookup-free on use, so you need to think hard about what all this actually gains :(
- BoppreH 3y ago
  Huh, I never thought about using length extension attacks as a protocol feature. However, you still need the initial secret, analogous to the HMAC key, and it's not a complicated algorithm to justify much effort in removing. And don't forget that HMAC is more resistant to collision than the underlying hash. For example, I think it's still unknown if HMAC-MD5 is weak, even though MD5 definitely is.
djha-skin 3y ago
Macaroons are a cookie[1], Macarons are a French dessert[2]. I have a pet peeve about this mix-up in particular, and they seem to have mixed them up, calling the French thing macaroons. 1: https://www.savoryexperiments.com/almond-coconut-macaroons/ https://www.savoryexperiments.com/almond-coconut-macaroons/ 2: https://sallysbakingaddiction.com/french-macarons/ https://sallysbakingaddiction.com/french-macarons/
- TheRealPomax 3y ago
  Ultimately, of course, it doesn't really matter as long as the person selling them knows what you're asking for. Which they will. https://www.youtube.com/watch?v=nzcHeO43kgE https://www.youtube.com/watch?v=nzcHeO43kgE
  - hahajk 3y ago
    Wow, thank you. That was a real broccoli = kale moment for me.
- justworkout 3y ago
  Macaroons and macarons are both cookies. They're both desserts as well.
- mtlmtlmtlmtl 3y ago
  It's basically the same word though(they have the same etymology). In Norwegian they're called coconut and French "makron", respectively.
  - danielvinson 3y ago
    Macaroons are the coconut cookie. Macarons are the fluffy egg white ones. Very different things.
    - mtlmtlmtlmtl 3y ago
      But the word macaroon is derived from macaron. Obviously they are not the same thing, I never said that, did I?
alexissantos 3y ago
Off topic, but related: The art you see on Fly posts? It's made by the inimitable Annie Rugyt: https://annieruygtillustration.com/ https://annieruygtillustration.com/ She's wonderful, and has a knack for making brands come alive with illustration. Remember the RethinkDB mascot? That was her too!
- tptacek 3y ago
  She's a full-time member of the team. This is the least of the art she's queued up for the prodsec team's posts this quarter. :) Her big project right now is mascotizing Frankie the Balloon, for swag.
ramchip 3y ago
I'm curious why the connection from Rails is "mTLS-y" rather than actual mTLS. The macaroon repo mentions a Noise transport. Maybe to avoid dealing with X.509 certificates by distributing trusted public keys via LiteFS? > We didn’t use the pre-existing public implementation because we were warned not to. [...] Macaroons decided to use untyped opaque blobs to represent caveats. We need things to be as rigidly unambiguous as they can be. Another place where the implementations are limiting is in the nonce for discharge macaroons: it has to match the "challenge" exactly. I think the fly.io implementation makes the nonce a structured object that includes the challenge / key ID as a field, and extracts that field during third party caveat verification, which is nice. It makes it possible to include a random nonce (in the cryptographic sense) or revocation ID for instance. I experimented recently with discharges containing service-specific info, e.g. a discharge from an auth service could contain the user's name (for logging), etc. It felt dangerous though, because there's potential for confusion on which service is allowed to provide what info - if we have a 3P caveat for an auth service and a revocation checking service, we want to be sure that the user profile info comes from the auth service, not the other one, and it gets complicated to encode that in the token. Maybe this is what "proof" macaroons solve? They're not mentioned in the blog post or macaroon-thought.md, but seemed to be about making positive statements on something.
- paulmd 3y ago
  This has been a perpetual problem we’re trying to solve at our org and while I was on that team I was pushing for the idea of macaroons (we just use JWT and we suffer many headaches because of it). My suggestion at the time was macaroons with some hierarchal PKI - servers themselves have a macaroon that delegates the permission to sign tokens that meet certain criteria, delegated by us the root organization. So in most situations a typical key hierarchy would look like “root key -> signing servers -> normal services -> bearer token”. If a token is stateless then this just becomes another caveat and signature - show me that you have the authority to sign this key under this attenuation. Signing server? You should only be signing keys for key services. Data service? You should only be issuing tokens with X caveat. Etc. If you want revocation though it’s not really stateless - at minimum you have to store a list of revoked tokens until the parent tokens that issued them have expired. In fact you would want to build this as a routine capability on every single level - signing services should rotate their own keys (this is that “clearing caveats” thing) and issue their own revocations, and then the assertion is that any signing from that key is invalid from that moment forward. Similarly, if a browser session wants to refresh its token, it needs to commit to never using the old one again, for the lifespan of the IAM token, and we need to track that for the lifespan of the IAM token. But there is a defined TTL after which you can clear the revocation - and it’s the lifespan of the token used to issue it. Which is a very Redis-with-cross-region-replication sort of task, or perhaps Postgres for persistent backing. The good news is that there is no logical split-brain, the mere fact that another region issued a revocation is ipso facto all you need to know, when and why it happened is kinda irrelevant to the fact that the token is now revoked. In other words - signing servers probably have an IAM representing them as an instance, with their own internal secret/private key, which is used to issue their own "bearer token" (public key) which rotates periodically. And when they do a clearing etc they are the authority that signs that the new macaroon is valid. And after some time their own bearer token is revoked (eg after 2pm any signings are invalid) and they rotate to signing with a new bearer token. If you are going to rely on propagating revocation efficiently it needs to be a routine fact that’s occurring regularly, right? And if you don’t see a revocation from a signing server on some expected timeframe, it is probably in fact a sign of split-brain occurring… you’ve lost a server or a region and it’s in fact dubious to continue accepting that token anyway. So this sort of leads to an inherent “valid -> historically valid but not for new tokens -> expired” lifecycle that mirrors the IAM/bearer model that clients also use. This actually closes the "fail open" nature of the revocation - it is impossible to have a "split brain" in the usual sense of revocation not propagating to a region etc, because you know that a signing-server key isn't valid for more than 15 minutes anyway etc. And perhaps you could even add a "parentKey" field and use the bearer token itself as a carrier of these revocations (if the user has a new key and it's validly signed, then SOMEONE must have revoked the old key...) although I haven't rigorously thought that one through. I am drawn to this revocation idea like a moth to a flame despite knowing how much complexity is going to live in making sure that service doesn’t ever go down or lose data or go split-brain. But if you are doing this “clearing” idea at least you don’t end up with a giant tree of historical caveats and sub-signatures. On the other hand the reality is that really nobody wants true stateless especially for browser tokens… “log me out everywhere” is practically table stakes and that implies some kind of revocation. And revocation is implicitly state - it’s either valid or revoked, that’s state even if you aren’t mutating the token itself, you’re still mutating the validity of the token. Even if it’s a global “user revoked all tokens at 2pm” it’s still mutating state etc. The idea of revocations lets the “happy path” be stateless, and handle this smaller amount of replication of revocations etc, but otoh it’s still very load-bearing and revocations fail-open. But since the token is limited in time and scope anyway, that’s potentially a valid tradeoff… But I haven't actually implemented any of this, so, take it for what it's worth.

263 points by mkeeter 3y ago

TIL macarons [1] are also known as macaroons [2]. Where I'm from, macaroons are dense, sweet coconut treats and the French sweets are known as macarons. But from the Wikipedia article I see macaroons mean many different things to many different people. [1] https://en.wikipedia.org/wiki/Macaron https://en.wikipedia.org/wiki/Macaron [2] https://en.wikipedia.org/wiki/Macaroon https://en.wikipedia.org/wiki/Macaroon

cmcconomy 3y ago
it messed with my head to hear that the english world interprets an "entrée" as a main course
- ssl232 3y ago
  American English, but not British. Dinner here is a starter, main course and dessert. "Entrée" messes with my head as well, since it very much sounds like a starter.
  - bdsa 3y ago
    It's French for starter, even
  - KineticLensman 3y ago
    > Dinner here is a starter, main course and dessert. Yep. Check out the online menus in English restaurants and pubs. 'Mains' is the typical term used.
    - wiredfool 3y ago
      It's also the electrical supply. What a country.
      - ssl232 3y ago
        And the water supply. And the central building on farms (in Scotland, at least). It's a silly country.
- dboreham 3y ago
  Don't even think about "a la mode" then.
  - aidenn0 3y ago
    I speak French, not Italian!
gonzus 3y ago
Switzerland has the similar Luxemburgerli, which my Swiss friends who reside there (as opposed to me, a Swiss person who lives somewhere else) swear is NOT the same as a macaron... I have never understood the purported differences.
deleted 3y ago
[deleted]
demondemidi 3y ago
I was just about to complain. Thanks for nipping it in the bud!
ISV_Damocles 3y ago
This feels relevant to the conversation: https://twitter.com/BigSpiderBack/status/864309383106240512/photo/1 https://twitter.com/BigSpiderBack/status/864309383106240512/...
enriquto 3y ago
> TIL macarons [1] are also known as macaroons [2] Not to forget macarrons! Where I'm from, macarrons [3] is a kind of pasta (penne). [3] https://ca.wikipedia.org/wiki/Macarrons https://ca.wikipedia.org/wiki/Macarrons
- madcaptenor 3y ago
  In English "macaroni" is a kind of pasta. (I'm being deliberately vague because which pasta it is seems to vary.)
  - sph 3y ago
    And the original name for that type of pasta in Italian is "maccheroni" (which are straight tubes, unlike the ones used abroad for "mac and cheese")
- pasc1878 3y ago
  Or in English macaroni https://en.wikipedia.org/wiki/Macaroni https://en.wikipedia.org/wiki/Macaroni
CoastalCoder 3y ago
> Where I'm from, macaroons are dense, sweet coconut treats and the French sweets are known as macarons. Same for me (New England, USA). Both are delicious, but in my dialect they are not the same thing.
- bageler 3y ago
  This is not a dialect thing, this is a factual thing. They are different, and anyone that uses macarOOn to mean a sweetly filled meringue sandwich is wrong.
Nition 3y ago
Oh funny, I actually came to this comment thread to point out that they'd accidentally used a picture of macarons instead of macaroons in the blog. But I guess not necessarily!

plucas 3y ago

[flagged]

deleted 3y ago
[deleted]

larsrc 3y ago

[flagged]

xyzzy_plugh 3y ago

> Until now! I dragged Fly.io into implementing them. Suckers! Let's be real: they knew you would do this eventually. It was inevitable.

asplake 3y ago

Typo/bug: pretty sure variable 'oldTag' should be 'oldTail'

tptacek 3y ago
Thanks! They're basically interchangeable --- HMAC produces a "MAC tag", and in the context of the Macaroon protocol the last visible tag is called a "tail". But I'm guessing I screwed up editing the code when I switched everything to "tail". :)

DonsDiscountGas 3y ago

The title and graphic suggest an article about the most delicious treat known to humans, and the post is about software. My disappointment is immeasurable and my day is ruined.

chankstein38 3y ago
Yeah I clicked here thinking this would be an interesting story about why macarons are so popular these days or something. When I saw a code block I closed immediately. I'm a programmer but I don't care about every random, poorly-named library or framework that exists.
- apetresc 3y ago
  To be fair, it's not a library or framework, it's a format for certain types of security tokens. They're kind of like fancier cookies, hence the name "macaroon". Not so poorly-named, in my opinion (to the extent that "cookies" was ever a sensible name). Just the right level of whimsy while still calling back to what they are.
  - bbarn 3y ago
    A macaroon is a mixed up ball of coconut and egg whites. A macaron is the delicious fancy sandwich cookie.
    - dataflow 3y ago
      And for completeness, a Macron is the president.
      - amenghra 3y ago
        Not to be confused with micron, which is the size of most politicians’ accomplishments after they get elected.
      - rsynnott 3y ago

matthewaveryusa 3y ago

>and escape hatches like Mutation (for our GraphQL API). Can you explain in a bit more detail what you mean by escape hatches (sounds dangerously fun)

tptacek 3y ago
The Organization -> Apps -> Volumes (whatever) structure models our problem domain and is a sort of coherent abstraction. If we get that abstraction wrong, it's a pain to retrofit changes. Caveats like "Mutation", on the other hand, are simply strings representing calls you can make in our API; they're not a model of anything. I don't think we use `Mutation` for anything right now, but it's there in case we (or you) ever need a token that authorizes a particular set of GQL calls that don't fit the model we came up with.

jzelinskie 3y ago

It's probably more relevant to your previous blog post that did a survey of technologies, but I'm curious if y'all ever considered a fully centralized approach. Is there something about Fly that benefits from a decentralized approach? Disclosure: I work on SpiceDB alongside maintainers like the pymacaroons author

tptacek 3y ago
Can you define "centralized" and "decentralized" for me? What would a "fully centralized" approach to this problem look like?
- jzelinskie 3y ago
  Sure thing! These terms may not be the best way to communicate these concepts. Would love your take on that, too. I used the term decentralized to describe claim-based token-passing authorization strategies because the data defining their access is being source from various services, attenuated, and passed around with the request. I use the term centralized to describe something where all the data defining access and the policies are sourced from one service. I personally work on a Google-inspired system, but there are plenty of policy engines using a similar approach.
  - tptacek 3y ago
    Couple thoughts. * With regards to the "core Macaroons" we implement with first-party caveats, we conceptually are centralized, in that there's a single source of truth about which apps belong to which orgs &c that caveats are evaluated against. * The basic challenge we have is providing a comparable experience in every data center we're operating in around the world, so a "purely" centralized model would be problematic for us; people in Sydney would get a much worse experience than people in Ashburn. We get around this with LiteFS, to an extent. This post undersells how important LiteFS is to the deployment model here. * With respect to having, like, a Zanzibar-like access matrix dealy: part of the thing here is not wanting to have to provide a single coherent permissions model to our users, because our problem domain is so broad and we will get it wrong. Providing orthogonal tools to let users express things seems like the better strategy to us? But there are upsides to both approaches. * Of course, one of the big wins here, that we have a plugin interface for authorization, depends on some measure of decentralization. I feel like this is kind of a scattershot response, but hopefully I hit some of what you were talking about, and you can let me know if I've missed the mark completely.

X-Istence 3y ago

Macaroons are also implemented and used by pypi.org's implementation named Warehouse: https://warehouse.pypa.io/development/token-scanning.html https://warehouse.pypa.io/development/token-scanning.html Also see: https://pypitoken.readthedocs.io/en/latest/ https://pypitoken.readthedocs.io/en/latest/

woodruffw 3y ago
Yes, although PyPI doesn't currently do much attenuation or delegation with them (this is largely my fault, since I didn't fully understand their power when picking them for the implementation). That's been slowly changing, however -- as of a few months ago, PyPI issues slightly more compact API tokens that make better use of discrete caveats. They're also used on the Trusted Publishing[1] side to make the API token self-expiring. [1]: https://docs.pypi.org/trusted-publishers/ https://docs.pypi.org/trusted-publishers/

iou 3y ago

lol, the picture in the post is of a macaron, not a macaroon https://www.foodnetwork.com/recipes/packages/baking-guide/macarons-vs-macaroons-whats-the-difference https://www.foodnetwork.com/recipes/packages/baking-guide/ma...

deleted 3y ago
[deleted]

sergioisidoro 3y ago

So, are Macaroons are like a "blockchain", where each block is an attenuation (ie reducing permission and scopes) of the permission of the original token? So a bearer can get a token, reduce its scope (without talking to the server) and pass it down to other less privileged entities. That's cool! I kind of had to look into the paper linked to really understand what they are. I feel the article could have given a bit more context on it :)

dwaite 3y ago
Somewhat. It is symmetric key based, so only the initial issuer can do verification. It is not part of a distributed data store, so there can be multiple copies (and derivatives) of a macaroon floating around. Basically, a trusted service creates an initial atom and shares it with the initial party. That trusted atom can be copied and shared. Anyone with a copy can amend additional information to it, and share those copies. Based on the copy you get, you can't remove or alter any information received. So it works best for representing restrictions - maybe my user agent gets a token that gives full access for a project. It shares a version with another service endpoint which is restricted to read-only access. That service endpoint sees it needs to kick off a background process, so it creates a version which also adds on a 4 hour time limit. None of these changes are necessarily authenticated - I don't know that it specifically was the user agent that shared the token with the service or added the read-only restriction, at least with the core tech. You have to create a domain language to specify all of these things. If you want restrictions to be authenticated, you need to describe how to sign them individually with PKI. While HMAC is fast, you still have the whole system rooted in a single trusted service that can mint any messages it wants or lie about the validity of a given message. So while it is indeed a chain of cryptographic elements, it is missing the multi-party validation and operation transparency which are the common selling points of distributed ledgers.
- tptacek 3y ago
  Biscuits have single roots of trust too. The (superficial) distinction is that Biscuits can be verified (completely) on systems that don't hold the root keys. You can't do that with a Macaroon, but, as we explain in this post, you don't have to. I don't think we'd meaningfully gain security from Biscuits --- though we would potentially get flexibility from Datalog, and our deployment story might have been a little bit simpler. (We already need hardware-isolated secret stores, so the deloyment win is probably marginal for us, but it's a bigger deal for other shops). Purely symmetric cryptography is one of the features of Macaroons. It's a reason cryptography hipsters like it, not a reason to dunk on it.

jszymborski 3y ago

This post was written exceedingly well. Few posts execute humour and whimsy without coming off as insincere or just not very funny (I'm guilty myself). Bravo!

robbles 3y ago
The tone is entertaining, but some of the snarkiness around the code is bit frustrating at times. e.g. > # do i really need to say I'm not serious about this? Can you just come out and say what you mean here? Like, presumably it's bad, insecure code, but can't you just spell it out for the benefit of your audience? I'm probably being really picky here - just find this kind of developer in-crowd signalling to be really irritating and counterproductive.
- tptacek 3y ago
  The Python code here is essentially just pseudocode. In reality, if you build a Python Macaroon implementation, you'd pull in pyca/cryptography and use an actual AEAD, rather than rolling your own authenticated cipher out of pure HMAC. But the point is that this isn't real code, just enough to make the concepts concrete.
  - byproxy 3y ago
    I can always appreciate that these types of stripped-down examples are merely for illustrative, conceptual purposes...but the ignoramus in me would also appreciate links to fleshed-out examples that take into account the shortcomings of the simpler example.
    - tptacek 3y ago
      Our actual Macaroon code is linked at the bottom of the article.
  - robbles 3y ago
    That's a great explanation, and pretty much exactly what I'd love to see instead of the original slightly mysterious comment :)

wereHamster 3y ago

I wonder why they went chose macaroons over biscuits. The later is only mentioned once in that article, but doesn't talk about why they were not selected.

tptacek 3y ago
I like Biscuits, wrote about them before, and recorded a podcast episode with Geoffroy Couprie about them†. The short answers: * When we started working on this design, there was still some uncertainty about the elliptic curve chaining construction (as in: they hadn't, or maybe had just started to, work out which one they were going to use). * We do a lot of verifications, and the prospect of running lots of 25519 computations every time we have to check a token was daunting; by comparison, HMAC is essentially "free". * I'm not especially scared of Ed25519, but asymmetric signature cryptography is much more complex than computing a MAC. Like I said in the post, one of the great charms of Macaroons is how much mileage it gets out of a construction that everyone understands, and that doesn't have sharp edges. * Geoffroy sold me on the value of using a logic language to express caveats, but that was much more complicated than what we felt we needed. Another charm of Macaroons is that they're rigidly coherent; just an unordered collection of true/false predicates, loop over them and if any fail reject the request. I feel safe expressing things with Macaroon caveats --- not in the sense of "the token implementation doesn't have bugs", I'm sure Biscuit's Datalog is fine, but rather in the "I'm not going to mess this up as a user" sense. * We needed implementations in a bunch of different languages. Biscuits have them, but owning the implementation ourselves derisked things for us. We could have owned an implementation of Biscuits, but I'm not smart enough to do that well. † https://securitycryptographywhatever.com/2022/01/29/biscuits-with-geoffroy-couprie/ https://securitycryptographywhatever.com/2022/01/29/biscuits...
- crdrost 3y ago
  Oh my goodness -- I had this idea!! When I first read a blog post titled "Macaroons are better than cookies" I started thinking about the operational side and I really didn't like the fact that you had to make a network round-trip to some auth-service to validate the macaroon--because that's what the state of the art already was, and it kinda stunk to pay that delay on every single query. (Of course I understood that you could horizontally scale the machines doing auth and load-balance and all that, but I worked at smaller companies where you had a small rack in a colo space or an air-conditioned closet, creating VMs to do all of that ate into a relatively limited resource and made sysadmins raise eyebrows.) I tried for the longest time to put together a solution with hash functions and private-key crypto; I think the thing that bugged me the most was that I never was able to prove that you couldn't do it with those primitives. Eventually I had something similar with public-key crypto but at the time public-key usually meant RSA, so we were talking 1024-bit signatures for each step. Also I think the idea was that you'd furnish a zero-knowledge proof that you knew the key, which was hella complex. Two things then happened: first, if you're talking PKI then you're talking about revocation too; I convinced myself that revocation was necessary for these sorts of credentials (and I now always feel vindicated when places are like "log out all other instances?") but I also convinced myself that revocation required the network hop in any case (which is not true, revocation is actually pretty infrequent and you can push revoked serials to services via event bus and in most cases the latency to ingest those is not a huge issue). So I convinced myself that what I was looking for was ultimately a purely academic exercise, was the first thing. And the second was that I managed to simplify my PKI-plus-hashes-plus-zero-knowledge craziness to just what Biscuits are: "I'm signing a public key plus caveats," (but I didn't add a query language, I was just going to use a map of well-known-keys to values, "valid_before: 123...890"), "I'm going to transmit this to the recipient with its corresponding private key, which they will strip off and generate a new key to mint a new Biscuit." And then RSA was big and slow and I forgot all about this until this HN thread. Ed25519 is the perfect way to fix that -- that and distributed revocation. I still don't see myself using it in practice but it's nice that a library exists if I do... thanks for the walk down memory lane, I will queue up the podcast episode, maybe for tonight or tomorrow.

michelpp 3y ago

I like the "solve the now" perspective here, and having code examples is very helpful to understand some of the rational behind the approach. Having read your previous "tedious survey"[0] post on various token formats, I generally agree with a lot of your conclusions. Curious though about your thought process wrt macaroons vs biscuits. To me the one major downside of macaroons has always been the single shared root symmetric key. Many use cases are addressed by third party attenuation, but then there are the problems like key rotation, having to do online verification, no built in encryption, no peer-to-peer support through an "untrusted" fly.io, and no third party token verification without decryption like in signcryption[1] schemes. Of course this is traded off by having to do PK issuance and management so I can see the simplicity of it. Is fly.io scoping this pretty hard to just auth tokens with third party attenuation, or do you see further development and maybe moving to other token systems like biscuit when/if the need arises to address those known issues? fwiw I've done a bit of research work myself on a token format using signcryption [2] where I explored addressing some of these ideas (but not the attenuation side of it yet, which I get is a big deal here). [0] https://fly.io/blog/api-tokens-a-tedious-survey/ https://fly.io/blog/api-tokens-a-tedious-survey/ [1] https://github.com/jedisct1/libsodium-signcryption https://github.com/jedisct1/libsodium-signcryption [2] https://github.com/michelp/pgsodium/blob/feat/signcryption-tokens/sql/pgsodium--3.1.7--3.2.0.sql https://github.com/michelp/pgsodium/blob/feat/signcryption-t...

tptacek 3y ago
We're not going to use Biscuits. The "single root symmetric key" thing is a systems design challenge for us, not a cryptographic one: we simply isolate the key handling on a software-HSM-like verifier service, running on dedicated hardware. If you're coding authz logic for Fly.io's platform code, your experience will be that you can manipulate and check Macaroon caveats without ever having access to the keys themselves, because we split Macaroon-checking into "verification" and "caveat clearing". The threats in this design that would force us to do abrupt key rotations and mass token invalidation are thus pretty much the same as the ones we'd face with Biscuits; the "trusted code base" of key-handling code is comparably small. More detail here: https://github.com/superfly/macaroon/blob/main/macaroon-thought.md https://github.com/superfly/macaroon/blob/main/macaroon-thou...

tptacek 3y ago

Back in 2021, when I wrote the API token survey post, I had some, uh, caveats about Macaroons, and linked to a talk from a developer at Chain who had gone through the experience of deploying them. Don't get me wrong, it's still an excellent talk†, but after working on implementing them for a year or two I find it much less damning than I did at the time. One problem Chain seemed to run into was coupling between their services. They had a fast Golang "ledger" and a slow Rails "dashboard", and found that Macaroons forced all their requests to loop through the "dashboard". This was because they'd opted to do short-lived Macaroon tokens that needed to be reissued every 5 minutes (eminently sane), and only their dashboard could generate the proper token? Another problem they had with Macaroons: their Rails dashboard has an RBAC permissons interface. But once they've issued a Macaroon, users can't change permissions with the interface anymore. But that's a problem with all stateless tokens, not just Macaroons; in fact, Macaroons probably help here, because if you're presenting a Macaroon to the dashboard interface in the first place, the RBAC interface can just attenuate it for you and hand it back to you. I still think this is a design that mostly makes sense only if you have a problem domain for which both attenuation and delegation make sense. Most basic CRUD apps don't have these problems, and using Macaroons would be a mistake for them. † https://www.youtube.com/watch?v=MZFv62qz8RU https://www.youtube.com/watch?v=MZFv62qz8RU

leoqa 3y ago
I'm curious what your thoughts are on implementing a sane authorization system in 2024. You mentioned writing roles onto macroons but what does your policy look like? Is your surface area simple enough that it's not a concern? I've been on various security teams with disjoint product-facing authz, internal authz and service authz policy engines / mechanisms etc. Additionally, authz gets baked into service code and product interfaces, so it's hard to change later.
- tptacek 3y ago
  The idea is that Macaroons are a low-level IAM language, designed close to the components that they pertain to, encoded directly into the tokens, and that higher-level IAM policies "compile down" into those tokens.

elbasti 3y ago

What I really want to know is how fly.io has recruited such talent. @tptacek (who wrote this article), @chrismccord (creator of elixir Phoenix/Liveview), and others. Very impressive, but also slightly worrying since some parts of fly.io have such terrible DX/documentation. I've only recently started using them, but the documentation around billing and databases is shockingly bad for a team that is clearly so deeply stacked.

chubot 3y ago
Part of it is probably that smart people want to work on harder and more ambitious problems, which YC advisors seem to say a lot. And also smart people want to work together As I understand it, fly.io built a cloud from scratch, from the hardware up. That's HARD. It's typically something only well-funded companies like Amazon / Google / Microsoft do, with billions of dollars in products to justify it. Cloudflare is another company that pretty much did it, and it seems to have taken similarly ambitious engineering (although maybe theirs isn't fully general purpose) https://fly.io/docs/about/security/ https://fly.io/docs/about/security/ - We run on our own hardware deployed in secure data centers like Equinix Most cloud companies in the 2010's built on top of other clouds like AWS. For example, Heroku and I'm pretty sure dotCloud, the company that became Docker. I always thought that was a weird design, but it definitely saves you a lot of time and hassle. At the cost of a lot of software complexity and platform risk. Even so, there were at least 20 or 30 of these companies, and most of them didn't make it. Google App Engine (one of the first PaaS like Heroku, circa ~2007) also built on top of Google's data centers and Borg of course. They didn't manage their own hardware -- they used another cloud for that! From the outside, "hosting" may all look kind of similar, but there are worlds of difference under the hood. --- I'll also say that the subject of this blog post -- cloud security -- is extremely hard. FWIW I joined the team that published the Macaroons paper ~10 years ago, and I tried to implement Macaroons as a side project then (for about a week, it wasn't very serious). I was looking for something simpler and more elegant, but there's no real magic bullet, and I think this post kinda concedes that toward the end. But when you have really hard problems, it's not surprising if the solution is really complex! The problem can actually be infinitely hard, because you have adversaries that are both more powerful, and that adapt. Google learned this the hard way ~10 years ago when it learned that BOTH the United States government and the Chinese government had successfully attacked it :-/ People seem to forget about these incidents: https://en.wikipedia.org/wiki/Operation_Aurora https://en.wikipedia.org/wiki/Operation_Aurora https://en.wikipedia.org/wiki/MUSCULAR https://en.wikipedia.org/wiki/MUSCULAR https://www.washingtonpost.com/world/national-security/nsa-infiltrates-links-to-yahoo-google-data-centers-worldwide-snowden-documents-say/2013/10/30/e51d661e-4166-11e3-8b74-d89d714ca4dd_story.html https://www.washingtonpost.com/world/national-security/nsa-i... So basically, if you are successful, then it earns you more problems :)

windlep 3y ago

I remember when I saw a presentation by the macaroon authors a few years back, there were pending patents that Google filed around them. While the authors claimed Google wouldn't sue anyone, I'm always a bit skeptical about such claims. I thought macaroons would be helpful for some of my use-cases, but since I now knew there were patents that'd be wilful infringement so I didn't bother. I can't find the patents now, so perhaps they were rejected or withdrawn. I had assumed that was why macaroons hadn't caught on more widely. Edit: Found the patent: https://patents.google.com/patent/US9397990B1/ https://patents.google.com/patent/US9397990B1/

everybodyknows 3y ago
The Pythonish pseudo-code, rendered to an image (figure 7) -- is that common nowadays? Though the patent I see is dated 2013.
jimmyl02 3y ago
maybe this is similar to google's patenting of dropout for neural networks? you can never know but so far there haven't been many adverse effects and they claim that they patent it so others can't maliciously patent and enforce it. dropout patent link: https://patents.google.com/patent/US9406017B2/en https://patents.google.com/patent/US9406017B2/en
- windlep 3y ago
  That was what the authors claimed when I asked them about the macaroon patent. It'd be nice if Google had a legal document associated with patents they never plan to enforce, or the constraints around when they might enforce them (e.g. only against patent trolls) that a company could rely on.
  - xyzzy_plugh 3y ago
    I don't disagree but maintaining an arsenal of defensive parents is Enterprise IP legal 101. All the big companies do this. The goal is to avoid litigation by mutually assured destruction. At least that's what they tell you. Many projects grant you patents as part of, for example, an OSS project's license. It's not unusual to include a clause that voids any such terms of you litigate over your own parents, for example -- hence the mutually assured destruction. Can you imagine what would happen if Google and Facebook tried to duke out some dumb software patent in court? A waste for all parties. Lastly Google has a wealth of resources here: https://google.github.io/opencasebook/patents/#patents-in-open-source https://google.github.io/opencasebook/patents/#patents-in-op...

8organicbits 3y ago

> we want Macaroon tokens to be safe to transmit between users How does this work for auditing? If user A gives user B a token (perhaps after adding a caveat) how does this system audit log determine who did what? Does that require a third party system?

tptacek 3y ago
You can get as many independent Macaroons from us as you want, so while there are ways to handle this with third-party caveats (and never talking to our servers to get another token again), the simpler thing for token provenance with our system is the same as it would be with any other: just issue multiple tokens.
dwaite 3y ago
There is no centralized auditing of derivative macaroons, but there could be of verification of those macaroons - you still require a centralized service which knows the core HMAC secret to give a thumbs up or thumbs down. To that end you can define auditing information even when there aren't really caveats - user A appending that they are giving a version of the macaroon to user B with no restrictions on use. Since there isn't a domain language, you have to define all this. JSON can be useful to have an extensible notation for describing this, although you need to up-front declare the difference between information which can be ignored and caveats which much result in failure if not understood.
- tptacek 3y ago
  Note also that in our scheme, if Alice attenuates a token and gives it to Bob, Bob will still have to hit our login service to "activate" the token (that's the word I should have used in the post and I'm kicking myself for not thinking to express it that way) by clearing the third-party auth caveat we put on every token. So it's not as if we have no auditing of token delegation. Still: if only for revocation purposes, I'd probably just mint a new token for e.g. a contractor (and certainly for every user). The nice thing about the system is that you can edit the tokens after we give them to you so you're not stuck with our role definitions, which I think is a significant win, but doing all of IAM without talking to our servers more than once, while a cute technical stunt, is probably not all that valuable to users.

fovc 3y ago

Surprised to see JSON here. I recall discussing JWT with tptacek a few years ago, and one of the concerns was the {a: x, a: y} ambiguity in JSON parsers. Was that a concern with this design? My other reaction was that this sounds scary: > a token with no caveats restricts nothing. It’s a god-mode token. Don’t honor it. I guess it's technically "fail closed" but seems kind of brittle

tptacek 3y ago
I think it becomes more obvious later in the post that the Python code here is for illustrative purposes only. We don't use JSON. Our actual tokens are strictly typed. But also: it doesn't matter that much, because all of the predicates in a Macaroon are evaluated independently. There's no opportunity to confuse a previous caveat with a subsequent one. That's one of the strengths of having a rigidly coherent design like Macaroons, rather than just a bag of keys and values like JWTs do.
- fovc 3y ago
  Ah thanks for clarifying! I knew the code was illustrative but didn't realize the JSON part was too
hinkley 3y ago
The XML digital signature spec had that problem in spades. It took us about five times as long to button it down as it did to implement. By the time I was done with that project, the document for partners and vendors (if anyone wanted to implement their own instead of using ours) was several times longer than the spec, what with all the extra MUST and MUST NOT situations. Which is not that hard when you're dealing with swiss cheese. Document.findById and Element.findById being able to return different (non-null) results being the most egregious one I can remember.
- dwaite 3y ago
  XML is a disaster here all of its own, but is at least unambiguous at the parser level. JSON says "if your document does this, parser behavior is undefined". So you need to declare that the parser used should behave in particular ways. Both are best solved a robust specification and supplemental test vectors. If you define meta rules (e.g. JSON parsers must either fail or return the last object property), you still retain the value you hoped for by using something like JSON or XML in the first place.

Arelius 3y ago

These look pretty useful cool... I'm curious a bit though, here, and in other posts in HN, I often see that HMAC, a symmetric key is preferred, going as far as suggesting, in JWT, that other algorithms should not be implemented. Why is that? What are the problems with say RSA? (Ignorning that I'm not sure Asymmetric keys work with Macaroons design at all) From my perspective, Asymmetric keys have been a great boon, in that I can keep the private key, secured on my single auth server, but then freely distribute the public key to the edge, greatly increasing responsiveness, and reducing the bottleneck on the Auth server. Is there some security concern I've been missing?

evancordell 3y ago

> The community that formed around building open source “standard” Macaroons decided to use untyped opaque blobs to represent candidates. I assume "candidates" was supposed to be "caveats" - and as an author of a "standard" macaroon implementation, I completely agree that this is the biggest downfall of Macaroons. With no common caveat language (and no independent "dischargers") it really limits their use to within a single org. And at that point you're basically asking everyone to invent their own token format anyway. Though I don't personally use them much anymore - I think the use-cases for Macaroons are much more limited if you have a Zanzibar! - I appreciate seeing Macaroon discussions pop up and this post and the related discussions it linked out to were a great read.

xyzzy_plugh 3y ago
Zanzibar and macaroons are actually pretty complimentary.

bethecloud 3y ago

Storj is also leveraging macaroons. Great write-up on decentralized access control here: https://medium.com/@kleffew/what-is-capability-based-security-227c6e5483a5 https://medium.com/@kleffew/what-is-capability-based-securit...

ijustwanttovote 3y ago

Small detail, there's a picture of macarons and not macaroons. A macaron is a sandwich-like cookie that's filled with jam, ganache, or buttercream. A macaroon is a drop cookie made using shredded coconut.

arnarbi 3y ago
Macaroon is just the English word for the French word macaron. ducks for cover
- beAbU 3y ago
  Macaroon is a completely different confectionery: https://en.m.wikipedia.org/wiki/Macaroon https://en.m.wikipedia.org/wiki/Macaroon
  - IshKebab 3y ago
    Sure, but Macarons (the burger-looking things) are also sometimes known as "macaroons". Yeah. Examples: * https://missmacaroon.co.uk/ https://missmacaroon.co.uk/ * https://www.floristgrays.co.uk/design-202300010/valentines-macaroons.htm https://www.floristgrays.co.uk/design-202300010/valentines-m... * https://www.parisiennesouthwell.com/product-page/macaroons-vegan-gluten-free https://www.parisiennesouthwell.com/product-page/macaroons-v...
    - RussianCow 3y ago
      The fact that people say it doesn't make it any less wrong. :)
      - recursive 3y ago
        If enough people do it for long enough, it does. After all, that's how we got the rest of the words.
        3y ago

denton-scratch 3y ago

[flagged]

tptacek 3y ago
Good note.

beeks 3y ago

it seems to me that macaroons could get quite big in order to adequately describe allowed resources and permissions? Especially if the API is broad? Much bigger than JWTs.

tptacek 3y ago
The resource descriptions are pretty parsimonious; they're binary-encoded MsgPack integers, for the most part. But the cryptography eats up a lot of bytes. They're bigger than JWTs, but probably by a factor less than 2 (I don't know the median JWT size). Basic take: unless you can golf your tokens down to a single terminal line, it doesn't much matter how much bigger or smaller they are. These are all smaller than X.509 documents.
- beeks 3y ago
  Ah i see I'm (perhaps naively) imagining an exhaustive list of resources in every macaroon, but i guess the permissive evaluation of caveats would help here since it should mean only the most restricted users would have many caveats. Also I suppose if the caveat evaluation code can deal with wildcards then it can be reduced further. ok OK! I'll have to play around and see how these look in practice. We do have a good use-case for this with both delegation and attenuation... sadly our identity provider is married to JWTs and moving off it would take a lot of work. Anyways. Great Post! Certainly got me thinking, and now i'm tempted to just blend this with our existing system of JWTs :smirk:
  - tptacek 3y ago
    The modal caveat is probably (in vibes-based notation): (Organization 4567 ops=read,write,create,control) Followed by the second modal: (Organization 4567 ops=read,write,create,control) (Apps (App 345 ops=read,write,create,control)) You can make a _much_ more complicated token. And, for instance, our deploy tokens for CI/CD systems are pretty complicated, and they even have a disjunctive caveat, but most people are at most probably just going to lock their tokens down to a particular app, or turn them into read-only or start/stop-only tokens. (We modeled everything just to be safe, and also because that stuff is super valuable for service tokens, when we ourselves want to take platform actions on behalf of the user and have it be traceable back to a user authorization. I feel like I did not say enough about how much I like where service tokens are pointing for us.)

xbar 3y ago

I am not a customer, but I do like the fly.io "voice."

RustyRussell 3y ago

Compulsory plug when someone uses macaroons: using an HMAC is overkill. See https://github.com/rustyrussell/runes https://github.com/rustyrussell/runes for a simpler alternative and implementation (this has C and Python, but there's also a Rust implementation because why not?) However, the "no db access" property has proven to be untenable in practice. Users end up wanting to see what runes are issued, blacklist them, know when they were last used, and have rate limits. The last two are a killer, requiring some state to be kept (unless your system allows you to return a modified rune to the user, which is a different workflow from normal bearer creds).

RustyRussell 3y ago
Oops, I read further: you did use unique ids, and blacklists! I did that too. Now you're not db-lookup-free on use, so you need to think hard about what all this actually gains :(
BoppreH 3y ago
Huh, I never thought about using length extension attacks as a protocol feature. However, you still need the initial secret, analogous to the HMAC key, and it's not a complicated algorithm to justify much effort in removing. And don't forget that HMAC is more resistant to collision than the underlying hash. For example, I think it's still unknown if HMAC-MD5 is weak, even though MD5 definitely is.

djha-skin 3y ago

Macaroons are a cookie[1], Macarons are a French dessert[2]. I have a pet peeve about this mix-up in particular, and they seem to have mixed them up, calling the French thing macaroons. 1: https://www.savoryexperiments.com/almond-coconut-macaroons/ https://www.savoryexperiments.com/almond-coconut-macaroons/ 2: https://sallysbakingaddiction.com/french-macarons/ https://sallysbakingaddiction.com/french-macarons/

TheRealPomax 3y ago
Ultimately, of course, it doesn't really matter as long as the person selling them knows what you're asking for. Which they will. https://www.youtube.com/watch?v=nzcHeO43kgE https://www.youtube.com/watch?v=nzcHeO43kgE
- hahajk 3y ago
  Wow, thank you. That was a real broccoli = kale moment for me.
justworkout 3y ago
Macaroons and macarons are both cookies. They're both desserts as well.
mtlmtlmtlmtl 3y ago
It's basically the same word though(they have the same etymology). In Norwegian they're called coconut and French "makron", respectively.
- danielvinson 3y ago
  Macaroons are the coconut cookie. Macarons are the fluffy egg white ones. Very different things.
  - mtlmtlmtlmtl 3y ago
    But the word macaroon is derived from macaron. Obviously they are not the same thing, I never said that, did I?

alexissantos 3y ago

Off topic, but related: The art you see on Fly posts? It's made by the inimitable Annie Rugyt: https://annieruygtillustration.com/ https://annieruygtillustration.com/ She's wonderful, and has a knack for making brands come alive with illustration. Remember the RethinkDB mascot? That was her too!

tptacek 3y ago
She's a full-time member of the team. This is the least of the art she's queued up for the prodsec team's posts this quarter. :) Her big project right now is mascotizing Frankie the Balloon, for swag.

ramchip 3y ago

I'm curious why the connection from Rails is "mTLS-y" rather than actual mTLS. The macaroon repo mentions a Noise transport. Maybe to avoid dealing with X.509 certificates by distributing trusted public keys via LiteFS? > We didn’t use the pre-existing public implementation because we were warned not to. [...] Macaroons decided to use untyped opaque blobs to represent caveats. We need things to be as rigidly unambiguous as they can be. Another place where the implementations are limiting is in the nonce for discharge macaroons: it has to match the "challenge" exactly. I think the fly.io implementation makes the nonce a structured object that includes the challenge / key ID as a field, and extracts that field during third party caveat verification, which is nice. It makes it possible to include a random nonce (in the cryptographic sense) or revocation ID for instance. I experimented recently with discharges containing service-specific info, e.g. a discharge from an auth service could contain the user's name (for logging), etc. It felt dangerous though, because there's potential for confusion on which service is allowed to provide what info - if we have a 3P caveat for an auth service and a revocation checking service, we want to be sure that the user profile info comes from the auth service, not the other one, and it gets complicated to encode that in the token. Maybe this is what "proof" macaroons solve? They're not mentioned in the blog post or macaroon-thought.md, but seemed to be about making positive statements on something.

paulmd 3y ago
This has been a perpetual problem we’re trying to solve at our org and while I was on that team I was pushing for the idea of macaroons (we just use JWT and we suffer many headaches because of it). My suggestion at the time was macaroons with some hierarchal PKI - servers themselves have a macaroon that delegates the permission to sign tokens that meet certain criteria, delegated by us the root organization. So in most situations a typical key hierarchy would look like “root key -> signing servers -> normal services -> bearer token”. If a token is stateless then this just becomes another caveat and signature - show me that you have the authority to sign this key under this attenuation. Signing server? You should only be signing keys for key services. Data service? You should only be issuing tokens with X caveat. Etc. If you want revocation though it’s not really stateless - at minimum you have to store a list of revoked tokens until the parent tokens that issued them have expired. In fact you would want to build this as a routine capability on every single level - signing services should rotate their own keys (this is that “clearing caveats” thing) and issue their own revocations, and then the assertion is that any signing from that key is invalid from that moment forward. Similarly, if a browser session wants to refresh its token, it needs to commit to never using the old one again, for the lifespan of the IAM token, and we need to track that for the lifespan of the IAM token. But there is a defined TTL after which you can clear the revocation - and it’s the lifespan of the token used to issue it. Which is a very Redis-with-cross-region-replication sort of task, or perhaps Postgres for persistent backing. The good news is that there is no logical split-brain, the mere fact that another region issued a revocation is ipso facto all you need to know, when and why it happened is kinda irrelevant to the fact that the token is now revoked. In other words - signing servers probably have an IAM representing them as an instance, with their own internal secret/private key, which is used to issue their own "bearer token" (public key) which rotates periodically. And when they do a clearing etc they are the authority that signs that the new macaroon is valid. And after some time their own bearer token is revoked (eg after 2pm any signings are invalid) and they rotate to signing with a new bearer token. If you are going to rely on propagating revocation efficiently it needs to be a routine fact that’s occurring regularly, right? And if you don’t see a revocation from a signing server on some expected timeframe, it is probably in fact a sign of split-brain occurring… you’ve lost a server or a region and it’s in fact dubious to continue accepting that token anyway. So this sort of leads to an inherent “valid -> historically valid but not for new tokens -> expired” lifecycle that mirrors the IAM/bearer model that clients also use. This actually closes the "fail open" nature of the revocation - it is impossible to have a "split brain" in the usual sense of revocation not propagating to a region etc, because you know that a signing-server key isn't valid for more than 15 minutes anyway etc. And perhaps you could even add a "parentKey" field and use the bearer token itself as a carrier of these revocations (if the user has a new key and it's validly signed, then SOMEONE must have revoked the old key...) although I haven't rigorously thought that one through. I am drawn to this revocation idea like a moth to a flame despite knowing how much complexity is going to live in making sure that service doesn’t ever go down or lose data or go split-brain. But if you are doing this “clearing” idea at least you don’t end up with a giant tree of historical caveats and sub-signatures. On the other hand the reality is that really nobody wants true stateless especially for browser tokens… “log me out everywhere” is practically table stakes and that implies some kind of revocation. And revocation is implicitly state - it’s either valid or revoked, that’s state even if you aren’t mutating the token itself, you’re still mutating the validity of the token. Even if it’s a global “user revoked all tokens at 2pm” it’s still mutating state etc. The idea of revocations lets the “happy path” be stateless, and handle this smaller amount of replication of revocations etc, but otoh it’s still very load-bearing and revocations fail-open. But since the token is limited in time and scope anyway, that’s potentially a valid tradeoff… But I haven't actually implemented any of this, so, take it for what it's worth.