5 ms·

Himeji: A scalable centralized system for authorization at Airbnb

34 points by kiyanwang 5y ago

softveda 5y ago
I don't know why a packaged solution (OSS or commercial) doesn't exists for this kind of Authorization. Many companies suffer in silence and cobble together some home grow stuff. The last place I worked and looked at this exact problem the only standard that was available was XACML and it was too complex for our need within the project timeframe so we went with some simple DB based solution
- jitl 5y ago
  Here’s an open source Zanzibar implementation: https://github.com/ory/keto https://github.com/ory/keto Ory appears to offer a complete stack for the entire AuthN/AuthZ space: https://www.ory.sh https://www.ory.sh (I haven’t used or evaluated this software)
  - bradleyjg 5y ago
    I remember reading the Zanzibar paper (highly recommended) and it depended heavily on spanner—-especially its consistency latency guarantees—-for its nice properties. This looks more like an (in progress) api compatible competitor than a open source clone.
    - jakemoshenko 5y ago
      At Authzed, our implementation builds on top of CockroachDB (inspired by spanner) to get those same latency and consistency guarantees. As you are someone else who has read the paper, we would love to get your feedback on how well we've captured the spirit.
      - bradleyjg 5y ago
        Do you have a white paper or something? I saw some good blog posts, but I’m interested particularly in the differences with Zanzibar especially those that are the necessary result of the differences between cockroach and spanner.
        jakemoshenko 5y ago
        We don't have a white paper. I wrote about my particular interpretation of the Zanzibar paper[1]. As far as the differences due to cockroach vs spanner. There are only a few minor ones. Without TrueTime cockroachdb relies on their hybrid logical clocks[2] which at the very least require more storage. The Zanzibar paper talks about separating tuples into separate databases by namespace, and shards [3§3.1.1]. We haven't had to do that yet, but we are far from operating at Google's scale. Finally, cockroachdb's CHANGEFEED API[4] combined with our single tuple table have allowed us to skip storing a separate changelog table (for now). [1]: https://authzed.com/blog/what-is-zanzibar/ https://authzed.com/blog/what-is-zanzibar/ [2]: https://www.cockroachlabs.com/blog/living-without-atomic-clocks/ https://www.cockroachlabs.com/blog/living-without-atomic-clo... [3]: https://research.google/pubs/pub48190/ https://research.google/pubs/pub48190/ [4]: https://www.cockroachlabs.com/docs/v20.2/changefeed-for.html https://www.cockroachlabs.com/docs/v20.2/changefeed-for.html
  - rad_gruchalski 5y ago
    It's not complete (no rewrites) and does not provide RBAC component out of the box. It has the pieces but one has to put stuff together themselves. I have written more about it: https://gruchalski.com/posts/2021-05-15-rbac-with-ory-keto/ https://gruchalski.com/posts/2021-05-15-rbac-with-ory-keto/.
- jakemoshenko 5y ago
  We're working on just that at Authzed! After being bitten by inflexible authZ in the past, we decided to build a company to solve it. Take a look, we would love any feedback: critical or otherwise.
- FunnyLookinHat 5y ago
  System76 released one recently - https://github.com/system76/recognizer https://github.com/system76/recognizer You're right though - we have a home rolled one where I work that is very robust and could easily be generalized, but it's also fairly simple (to build). I suspect most engineers jump at the chance to get to build a project like this rather than vendoring in something external (whether FOSS or commercial).
softveda 5y ago
I am not clear how Himeji works if attribute based rules are required. Like Users in country X cannot see listing in country Y unless they are platinum users.
eatonphil 5y ago
> Himeji has been serving checks in production for about a year and its throughput has scaled up from 0 in March 2020 to 850k entities / sec in March 2021, while maintaining its availability and latency targets over the last year The architecture seems pretty complex but I guess it's warranted for this level of performance / uptime.
jeswin 5y ago
These write-ups are not very useful or interesting if they aren't publishing the code, or any research papers related to it. Not sure who the target audience is; very little to learn.
- jitl 5y ago
  I see this as an implementation report of Zanzibar, which is a Google paper referenced by the article. https://research.google/pubs/pub48190/ https://research.google/pubs/pub48190/ I find this useful because I’ve thought about building a Zanzibar-lite to solve permissions in my product, so it’s nice to hear about others walking the same path. (Personal bias note: I worked at Airbnb, but before this project started)
brodock 5y ago
The hidden costs of migrating away from a monolith... The promise is you will be able to better allocate resources as they are divided into small pieces. The reality is over engineering everywhere just to the what you have been doing before, but with added latency
- dboreham 5y ago
  The only reason to migrate away is Conway.
alephnan 5y ago
I've noticed people in the West love reaching for words in other cultures to name their project, whether it's Japan or Greek mythology. One practical reason for this is that the string is more unique on Google search. Meanwhile, one of the most popular projects to come out of Japan is Ruby the programming language. The tech demographic probably did a few more rounds of Google/Wiki (re)searching than a person who gets a random Chinese character tattooed, but I wonder how much is lost in translation and whether the conceptualization / romanticization / admiration that English-speaking Westerners hold of a word the same as in the original culture. Himeji probably refers to Himeji castle in Japan, and I wonder if developers outside the West consider naming their project The Alamo or Buckingham. It feels more authentic with Apple of California naming their recent Mac OS X releases Mojave, Catalina, Big Sur. Now that I live in Japan, that inclination to name my projects after Japanese things have gone away

34 points by kiyanwang 5y ago

I don't know why a packaged solution (OSS or commercial) doesn't exists for this kind of Authorization. Many companies suffer in silence and cobble together some home grow stuff. The last place I worked and looked at this exact problem the only standard that was available was XACML and it was too complex for our need within the project timeframe so we went with some simple DB based solution

jitl 5y ago
Here’s an open source Zanzibar implementation: https://github.com/ory/keto https://github.com/ory/keto Ory appears to offer a complete stack for the entire AuthN/AuthZ space: https://www.ory.sh https://www.ory.sh (I haven’t used or evaluated this software)
- bradleyjg 5y ago
  I remember reading the Zanzibar paper (highly recommended) and it depended heavily on spanner—-especially its consistency latency guarantees—-for its nice properties. This looks more like an (in progress) api compatible competitor than a open source clone.
  - jakemoshenko 5y ago
    At Authzed, our implementation builds on top of CockroachDB (inspired by spanner) to get those same latency and consistency guarantees. As you are someone else who has read the paper, we would love to get your feedback on how well we've captured the spirit.
    - bradleyjg 5y ago
      Do you have a white paper or something? I saw some good blog posts, but I’m interested particularly in the differences with Zanzibar especially those that are the necessary result of the differences between cockroach and spanner.
      - jakemoshenko 5y ago
        We don't have a white paper. I wrote about my particular interpretation of the Zanzibar paper[1]. As far as the differences due to cockroach vs spanner. There are only a few minor ones. Without TrueTime cockroachdb relies on their hybrid logical clocks[2] which at the very least require more storage. The Zanzibar paper talks about separating tuples into separate databases by namespace, and shards [3§3.1.1]. We haven't had to do that yet, but we are far from operating at Google's scale. Finally, cockroachdb's CHANGEFEED API[4] combined with our single tuple table have allowed us to skip storing a separate changelog table (for now). [1]: https://authzed.com/blog/what-is-zanzibar/ https://authzed.com/blog/what-is-zanzibar/ [2]: https://www.cockroachlabs.com/blog/living-without-atomic-clocks/ https://www.cockroachlabs.com/blog/living-without-atomic-clo... [3]: https://research.google/pubs/pub48190/ https://research.google/pubs/pub48190/ [4]: https://www.cockroachlabs.com/docs/v20.2/changefeed-for.html https://www.cockroachlabs.com/docs/v20.2/changefeed-for.html
- rad_gruchalski 5y ago
  It's not complete (no rewrites) and does not provide RBAC component out of the box. It has the pieces but one has to put stuff together themselves. I have written more about it: https://gruchalski.com/posts/2021-05-15-rbac-with-ory-keto/ https://gruchalski.com/posts/2021-05-15-rbac-with-ory-keto/.
jakemoshenko 5y ago
We're working on just that at Authzed! After being bitten by inflexible authZ in the past, we decided to build a company to solve it. Take a look, we would love any feedback: critical or otherwise.
FunnyLookinHat 5y ago
System76 released one recently - https://github.com/system76/recognizer https://github.com/system76/recognizer You're right though - we have a home rolled one where I work that is very robust and could easily be generalized, but it's also fairly simple (to build). I suspect most engineers jump at the chance to get to build a project like this rather than vendoring in something external (whether FOSS or commercial).

softveda 5y ago

I am not clear how Himeji works if attribute based rules are required. Like Users in country X cannot see listing in country Y unless they are platinum users.

eatonphil 5y ago

> Himeji has been serving checks in production for about a year and its throughput has scaled up from 0 in March 2020 to 850k entities / sec in March 2021, while maintaining its availability and latency targets over the last year The architecture seems pretty complex but I guess it's warranted for this level of performance / uptime.

jeswin 5y ago

These write-ups are not very useful or interesting if they aren't publishing the code, or any research papers related to it. Not sure who the target audience is; very little to learn.

jitl 5y ago
I see this as an implementation report of Zanzibar, which is a Google paper referenced by the article. https://research.google/pubs/pub48190/ https://research.google/pubs/pub48190/ I find this useful because I’ve thought about building a Zanzibar-lite to solve permissions in my product, so it’s nice to hear about others walking the same path. (Personal bias note: I worked at Airbnb, but before this project started)

brodock 5y ago

The hidden costs of migrating away from a monolith... The promise is you will be able to better allocate resources as they are divided into small pieces. The reality is over engineering everywhere just to the what you have been doing before, but with added latency

dboreham 5y ago
The only reason to migrate away is Conway.

alephnan 5y ago

I've noticed people in the West love reaching for words in other cultures to name their project, whether it's Japan or Greek mythology. One practical reason for this is that the string is more unique on Google search. Meanwhile, one of the most popular projects to come out of Japan is Ruby the programming language. The tech demographic probably did a few more rounds of Google/Wiki (re)searching than a person who gets a random Chinese character tattooed, but I wonder how much is lost in translation and whether the conceptualization / romanticization / admiration that English-speaking Westerners hold of a word the same as in the original culture. Himeji probably refers to Himeji castle in Japan, and I wonder if developers outside the West consider naming their project The Alamo or Buckingham. It feels more authentic with Apple of California naming their recent Mac OS X releases Mojave, Catalina, Big Sur. Now that I live in Japan, that inclination to name my projects after Japanese things have gone away