9 ms·

AuthZ: Scalable permissions system

108 points by jbredeche 5y ago

adamfeldman 5y ago
The article discusses Concord, a visualization tool they built. From the screenshot [1], it looks like it's hosted within some larger dashboard system – does anyone know if that's OSS or something custom? [1]: https://miro.medium.com/max/700/0*JATQRwLPT_MgrMHT https://miro.medium.com/max/700/0*JATQRwLPT_MgrMHT EDIT: Ah, looks like Backstage – https://backstage.io/demos https://backstage.io/demos
- deleted 5y ago
  [deleted]
buro9 5y ago
> But we had a problem: There weren’t any open source implementations available. We decided to build our own system, adding several of our own modifications to make it an even better fit for Carta Was this problem solved? Is AuthZ open source? (I couldn't see a link to a repo anywhere - but am reading on mobile and maybe I missed it)
- random_kris 5y ago
  At my workplace we are using OPA that the op writes about in the post
bingo_cannon 5y ago
How does this compare to Apache Shiro?
- supermatt 5y ago
  Shiro is a framework. This is a service that checks (static) permissions.
- gneray 5y ago
  I'm not an expert on Shiro, but there are a number of high level differences, e.g., - Shiro covers authorization, plus authentication, session mgt, etc. The Carta project is authorization only - Shiro relies on external data to make authorization decisions. The Carta project stores that data internally - Shiro is run as part of your app. The Carta project is run as a separate service - Related: Shiro is specific to Java.
supermatt 5y ago
What aspect of this makes it "highly scalable"? It seems that the scalability limitation is the same as that of every graph - the efficiency of scaling a graph beyond a single node. Not bashing what you are saying, just want to understand what makes it "highly scalable", as it isn't clear from the article. Also, I was wondering: Are ALL permissions stored as an Object::relation pair? i.e. do you need to register permissions for all new entities relationships, or do you have some way of storing more dynamic permissions?
- samjs 5y ago
  > What aspect of this makes it "highly scalable"? The idea is to put all data in one place, and then aggressively optimise for answering queries about that data. For authorization you're pretty much always asking specific reachability questions. That's where the design of the index comes into it. > Also, I was wondering: Are ALL permissions stored as an Object::relation pair? i.e. do you need to register permissions for all new entities relationships, or do you have some way of storing more dynamic permissions? In Zanzibar, there is a configuration format for computing relationships dynamically. E.g. "anyone who is an editor of a document is a viewer of a document". I'm not sure if they implemented something like that. If you're interested, I wrote a guide on relationship-based access control that includes a section on how Zanzibar(-like) systems fit in: https://www.osohq.com/academy/authorization-academy-chapter-4-modeling-relationships https://www.osohq.com/academy/authorization-academy-chapter-...
- jzelinskie 5y ago
  Full disclosure: I'm a founder of Authzed (YC W21)[0], which is a productized Zanzibar system. > just want to understand what makes it "highly scalable", as it isn't clear from the article. While this post might not cover in depth why their system is scalable, the Zanzibar paper[1] and this post[2] about Zanzibar covers why similar systems are scalable. A oversimplification would be to say that finding a path between two nodes in a graph can be recursively broken down and performed in parallel -- the Authzed implementation looks roughly similar to a map reduce. >Are ALL permissions stored as an Object::relation pair? i.e. do you need to register permissions for all new entities relationships, or do you have some way of storing more dynamic permissions? Zanzibar implementations do have graph schemas (called Namespace Configs in the paper) which offer set operations for computing dynamic relationships. When you create a new user or object in your app, you typically also write a few relationships into Zanzibar and the rest of their permissions are implicitly granted through the computed definitions in your schema. If you're interested in learning more about Zanzibar, we're giving a PapersWeLove talk that will be streamed on Twitch next week: https://www.meetup.com/papers-we-love/events/278148236/ https://www.meetup.com/papers-we-love/events/278148236/ [0]: https://authzed.com https://authzed.com [1]: https://research.google/pubs/pub48190/ https://research.google/pubs/pub48190/ [2]: https://authzed.com/blog/what-is-zanzibar/ https://authzed.com/blog/what-is-zanzibar/
debarshri 5y ago
Maybe this is a naive question, at what point does an organisation decide to build a framework or a product internally than buying? Sure there was no opensource solution available but why not buy a product from other security companies. Just wondering.
- spockz 5y ago
  Integration into your own systems for example. Or specific requirements for which it is not clear they can be met by the product. Or unawareness. Or a combination of the above.
- gneray 5y ago
  It's a fair question, but in fairness to Carta, until recently, there have been ~zero commercial or open source solutions to building application authorization (beyond limited helper libraries or things like Active Directory, which solve only a portion of the problem). Developers have typically figured out the models on their own and implemented them in application code with IF statements, etc. and/or by building separate authorization services. Other publicly documented examples include teams at Slack (https://slack.engineering/role-management-at-slack/ https://slack.engineering/role-management-at-slack/) and Gusto (https://engineering.gusto.com/layering-authorization-into-a-web-application/ https://engineering.gusto.com/layering-authorization-into-a-...) OPA is one solution that's emerged in recent years. The author describes his team's point of view after evaluating it for their purposes. Oso (https://www.osohq.com/ https://www.osohq.com/) is a framework for application authorization. Disclosure: I'm a cofounder. Based on the post, the Carta team started their work in 2019, but we only open sourced Oso mid-2020, so they wouldn't have known about it. And in the last few months, a few other companies have announced offerings to address this problem too. So it's understandable why Carta chose to build. Now/soon other teams like them should have more options to choose from when they encounter similar challenges too.
- simonw 5y ago
  Build v.s. buy is genuinely one of the harder problems in software engineering strategy. You'll often hear people say "you should never, ever build something if you can buy a solution instead" (where buy also includes using an open source package) - but there are actually a bunch of other things to consider: - Is the solution you're considering well maintained and likely to keep working? Open source projects sometimes stop being updated. Vendors go out of business. - What's your escape strategy if the vendor DOES go out of business, or massively increases their pricing? - Integrating with a vendor / package is often a non-trivial amount of development work. Have you considered this relative to building your own, custom solution? - Do you have specific requirements that go beyond what's available off-the-shelf? It's absolutely true that your engineering efforts should be spent on the things that make your organization unique: time spent on your customer's problems is more valuable than time spent reinventing a wheel. But sometimes, the available wheels really aren't fit for your purposes. I've built when I should have bought, and I've bought when I should have built. Getting it right is hard! Often it doesn't become clear if you made the right decision until several years after you've made it.
deleted 5y ago
[deleted]
IndignantNerd 5y ago
When a startup I work for was borderline manipulated into onboarding onto the Carta platform [0], I did some brief security review. I noticed they’re loading third party JS with at least one postMessage handler with a wildcard origin. They were also sending the current page title (which can include things like company or investor name) to this provider. It appeared they were also using a version of Knockout.js that is 2+ years old. [1] This did not inspire a lot of confidence. The linked blog post at least takes security seriously, but it’s clear they’re patching over a lot of legacy cruft. There’s a hell of a lot of sensitive data on Carta, which I’m sure they’re already selling to their “partners” anyway, but IMO it’s only a matter of time before they (or one of their “partners”) are hacked and a whole bunch of companies suffer from information breach. IMO the main reason they haven’t been hacked yet is simply because you need to pay $2k just to get access to their questionably secure app. [0] Your lawyer tells you that you need a 409a and Carta is the best. But if you only want the 409A, and don’t want to get growth hacked onto Carta’s platform, you’re out of luck. They won’t do the 409A for you unless you complete the full onboarding process, which includes appointing Carta as your company’s transfer agent, and uploading the email addresses of your investors and allowing Carta to spam them with tasks like re-signing legal documents that you issued off-platform. [1] This is not a technical analysis and I’m just going by what I remember from watching my local chrome dev tools. Don’t @ me.
- steevenwee 5y ago
  I sense a lot of bitterness from this post. I don't think you have a full context on history of 409a valuations industry. Considering the fact that 409a valuations pre carta would cost $10k, $2k doesnt sound like a such an unfair deal. You could always go the legacy lawyer companies route and pay way more for a single 409a valuation. As for onboarding process - you do need a hell of a lot information to conclude said valuation.
  - deleted 5y ago
    [deleted]
simonw 5y ago
I was pleased to see a section in here on what they call "flattening" - the problem of "efficiently provide a list of resources my user has access to". I've made this mistake more than once when designing an authorization system. Sure, you can build a fast mechanism for answering the question "does actor A have access to object B" - but can you efficiently answer that for 10,000 objects at once? Say for example you're building a search engine, and you want to search across all public data but ALSO all of the data your user has access to that's private. You probably can't run the search, get back 10,000 results, then apply your "can X access Y" logic to all 10,000 of them. (I'd love to know how GitHub code search implements this - it shows me results from my private repositories mixed in with results from their public index).
- jzelinskie 5y ago
  "Flattening", the equivalent of denormalizing a graph, is one way to implement performant ACL Filtering[0]. This is the same strategy used in the Zanzibar paper to gain the desired performance out of their Leopard index system. I tend to consider this largely a performance optimization strategy and not a necessarily the "solution" to this problem. We actually just this week wrote about how Authzed[1] actually reverse walks the graph in order to implement this API. Full disclosure: I'm a founder of Authzed (YC W21), which is a productized Zanzibar system. [0]: https://authzed.com/blog/six-month-profile-page/ https://authzed.com/blog/six-month-profile-page/ [1]: https://authzed.com/blog/acl-filtering-in-authzed/ https://authzed.com/blog/acl-filtering-in-authzed/
- Diggsey 5y ago
  The permissions system I designed at work solves the search problem, although it is not as powerful as what's described in the article. We have three main concepts in the sytem: - Capabilities (eg. "read profile") - Selectors (eg. "profiles belonging to customer X) - Groups (eg. "profile") All queries are of the form "Do I have capability X on resource Y" (where resource Y belongs to a particular group). A "permission set" is a mapping from a capability to a set of selectors. Selectors can be composed (eg. you could OR two selectors together) but are predefined. Since they are predefined, our search implementation can understand a subset of the selectors, and translate them into an ElasticSearch-compatible query. We then filter the results by evaluating the permissions normally: this catches any cases where there was no way to construct a query, or if the constructed query could not be completely precise. If this occurs, then we show a message on the page like "some profiles were hidden because you do not have permission to view them" and then we might show 19 profiles on a page instead of the full 20, so that our pagination can still work efficiently.
philsnow 5y ago
I hope "AuthZ" is just a temporary codename, that will be changed if/when they open source their work. authz/authn are very generic terms that already mean something and it's distasteful when companies squat on such terms. Imagine if a company came around and created a networking-related project and called it "TcP". > During our discovery process, we evaluated both open source and vendor products. > But we had a problem: There weren’t any open source implementations available. We decided to build our own system, adding several of our own modifications to make it an even better fit for Carta. There's no mention or implication that they'll release it as open source, this seems to be an engineering blog post (for hiring lead gen or building the engineering brand of the company or whatever).
- fomine3 5y ago
  This. Nowadays products tend to named too generic name.
lmeyerov 5y ago
Love seeing the visuals here :) We work on scaling graph UIs, and I've been noticing the auth use case more and more! The discussion on Flattening is super important, though for most teams, I'd also add this dances around the costs of an extra piece of intertwined DB infra. For our internals, we ended up having to reject OPA and friends, including what Zanzibar does, because we decided our customers wouldn't want us to spend our infra innovation tokens on an extra DB and figuring out how it should intertwine on the hotpath for queries to the main DBs. For bigger/heavier sites like Carta (with especially complex rules) and especially high-traffic ones like Google, as hinted by the percentile charts, I can see their decision making way more sense. For most consumer & b2b sites though, I'm not as convinced, and especially long-term. We ended up going with Casbin as it can do ARBAC policies + reuses our SQL DB. So it's more like adding a lib than infra, even though it does come with policy features like an analysis-friendly DSL. Likewise, that means we get a few ways to tune bulk queries: most are automatically fast by smart default roles & logic, some via special-cased app-level opts (so out-of-DB), and if we ever need to, we can always do optimized in-DB queries. The options for policy engines for b2b apps is surprisingly slim. The whole experience has made me think that a SQL DB with a strong multitenant+arbac RLS story for Django/Rails/etc. users and a cloud autoscale-to-zero capability will make a killing for b2b SaaS's. Amazon Aurora and friends are close, but feels like it's still another couple years in practice.
tristanz 5y ago
Does anybody have any paid or open source solutions that they actually like? All the open sources solutions I've tried don't actually work for real world use cases with highly dynamic policies, users, groups, and objects (e.g. Google Docs). Often the issue is the inability to easily push down filters into the database or search index. Aserto looks very promising (https://www.aserto.com/ https://www.aserto.com/), but it still in private beta. It's crazy this still is part of the stack where there are no great solutions.
- samjs 5y ago
  > It's crazy this still is part of the stack where there are no great solutions. Seems like a few others have come to the same conclusion :) We're working on this at Oso (https://osohq.com https://osohq.com) - I'm the CTO. Oso is an open source framework for authorization. Policies can reference application directly, so any authorization decisions can be made dynamically based on the data. And your data doesn't need to leave your application. > Often the issue is the inability to easily push down filters into the database or search index. We actually support pushing down filters into the database directly through a feature we call... data filtering. We support Django and SQLAlchemy today (https://docs.osohq.com/python/guides/data_access.html https://docs.osohq.com/python/guides/data_access.html) but will be making this available for non-ORM/non-Python users in the next few months.
- Bombthecat 5y ago
  Doesn't opa do something like this?
- jbotz 5y ago
  There is Ory Keto (https://www.ory.sh/keto https://www.ory.sh/keto).
  - parkerhiggins 5y ago
    I was surprised not to see a comparable evaluation to Ory Keto at all in this article. Ory Keto continued to evolve out of the shortcomings of OPA https://github.com/ory/keto/issues/47 https://github.com/ory/keto/issues/47 . It's also based on Zanzibar, and it's used in production applications today. Cloud version coming soon (using cockroachdb) or use the open source version (postgres).
twiddling 5y ago
Having XACML flashbacks
- ogazitt 5y ago
  LOL :) XACML had some great ideas (my team worked on it back in 2004-2005), but god, all those angle brackets <shudder/>
netvarun 5y ago
There is also Keto - an open source project based on the Zanzibar model [https://github.com/ory/keto https://github.com/ory/keto]
- denysvitali 5y ago
  This! Also, the full Ory suite is very well developed and worth looking into. Fully open source!
spankalee 5y ago
It's wild that an equity management startup has to do this level of engineering. Goes to show how much more there is to build out in cloud and OSS platform offerings.
SahAssar 5y ago
- A: But how do you solve authz? - B: authz - A: Yeah, how do you do it? What system are you using? - B: authz - A: ... Why are you not answering me?
orf 5y ago
Tl;dr - a simple-sounding CRUD app for managing your options invests a huge amount of resources building a bespoke permission system inspired by Google to handle less than 15 RPS.
- p_l 5y ago
  But what's the latency requirements and security requirements of the domain?
0xbadcafebee 5y ago
Please don't build software just because Google published a paper. First off, it's not like it's a great design just because Google created it, they've churned out some real stinkers. And it's not like the code is guaranteed to be good; because you're not a Google engineer, you can't possibly be a God-like super-coder who designs nothing but perfect systems. (Or so you'd think from the way people fetishize FAANG papers) The most frequent design mistake I see is an engineer with a "hammers before nails" complex. They want a perfect design that will work for everything (that they have so far thought of) in some beautiful and hidden way. So they build the thing, and they think it's great. And then they go to hook it up to products... and it doesn't work well. The user story, the context diagram, and the mock-up all need to come first, not the code.
- p_l 5y ago
  Well then, maybe you should have read the article. It describes how they iterated until finally their requirements made them think of Zanzibar and decided to test it out, then how it evolved based on internal feedback.
jon-whit 5y ago
I've been working on a scalable open-source version of Google Zanzibar called the Authorizer platform. The implementation varies from that of Authzed by horizontally scaling the query space across more than one node. It uses consistent hashing with bounded loads to assign a specific node to serve the queries for the objects that it's responsible for. It also supports the namespace rewrite specification of the paper and the continuous namespace config monitoring. Much more progress is coming as well! The upcoming v0.2.0 release will have distributed caching to reduce latency on both the client and server side. Take a look, I'd love feedback and to build it into something valuable for the open-source community. https://github.com/authorizer-tech/access-controller https://github.com/authorizer-tech/access-controller https://authorizer-tech.github.io/ https://authorizer-tech.github.io/

108 points by jbredeche 5y ago

The article discusses Concord, a visualization tool they built. From the screenshot [1], it looks like it's hosted within some larger dashboard system – does anyone know if that's OSS or something custom? [1]: https://miro.medium.com/max/700/0*JATQRwLPT_MgrMHT https://miro.medium.com/max/700/0*JATQRwLPT_MgrMHT EDIT: Ah, looks like Backstage – https://backstage.io/demos https://backstage.io/demos

deleted 5y ago
[deleted]

buro9 5y ago

> But we had a problem: There weren’t any open source implementations available. We decided to build our own system, adding several of our own modifications to make it an even better fit for Carta Was this problem solved? Is AuthZ open source? (I couldn't see a link to a repo anywhere - but am reading on mobile and maybe I missed it)

random_kris 5y ago
At my workplace we are using OPA that the op writes about in the post

bingo_cannon 5y ago

How does this compare to Apache Shiro?

supermatt 5y ago
Shiro is a framework. This is a service that checks (static) permissions.
gneray 5y ago
I'm not an expert on Shiro, but there are a number of high level differences, e.g., - Shiro covers authorization, plus authentication, session mgt, etc. The Carta project is authorization only - Shiro relies on external data to make authorization decisions. The Carta project stores that data internally - Shiro is run as part of your app. The Carta project is run as a separate service - Related: Shiro is specific to Java.

supermatt 5y ago

What aspect of this makes it "highly scalable"? It seems that the scalability limitation is the same as that of every graph - the efficiency of scaling a graph beyond a single node. Not bashing what you are saying, just want to understand what makes it "highly scalable", as it isn't clear from the article. Also, I was wondering: Are ALL permissions stored as an Object::relation pair? i.e. do you need to register permissions for all new entities relationships, or do you have some way of storing more dynamic permissions?

samjs 5y ago
> What aspect of this makes it "highly scalable"? The idea is to put all data in one place, and then aggressively optimise for answering queries about that data. For authorization you're pretty much always asking specific reachability questions. That's where the design of the index comes into it. > Also, I was wondering: Are ALL permissions stored as an Object::relation pair? i.e. do you need to register permissions for all new entities relationships, or do you have some way of storing more dynamic permissions? In Zanzibar, there is a configuration format for computing relationships dynamically. E.g. "anyone who is an editor of a document is a viewer of a document". I'm not sure if they implemented something like that. If you're interested, I wrote a guide on relationship-based access control that includes a section on how Zanzibar(-like) systems fit in: https://www.osohq.com/academy/authorization-academy-chapter-4-modeling-relationships https://www.osohq.com/academy/authorization-academy-chapter-...
jzelinskie 5y ago
Full disclosure: I'm a founder of Authzed (YC W21)[0], which is a productized Zanzibar system. > just want to understand what makes it "highly scalable", as it isn't clear from the article. While this post might not cover in depth why their system is scalable, the Zanzibar paper[1] and this post[2] about Zanzibar covers why similar systems are scalable. A oversimplification would be to say that finding a path between two nodes in a graph can be recursively broken down and performed in parallel -- the Authzed implementation looks roughly similar to a map reduce. >Are ALL permissions stored as an Object::relation pair? i.e. do you need to register permissions for all new entities relationships, or do you have some way of storing more dynamic permissions? Zanzibar implementations do have graph schemas (called Namespace Configs in the paper) which offer set operations for computing dynamic relationships. When you create a new user or object in your app, you typically also write a few relationships into Zanzibar and the rest of their permissions are implicitly granted through the computed definitions in your schema. If you're interested in learning more about Zanzibar, we're giving a PapersWeLove talk that will be streamed on Twitch next week: https://www.meetup.com/papers-we-love/events/278148236/ https://www.meetup.com/papers-we-love/events/278148236/ [0]: https://authzed.com https://authzed.com [1]: https://research.google/pubs/pub48190/ https://research.google/pubs/pub48190/ [2]: https://authzed.com/blog/what-is-zanzibar/ https://authzed.com/blog/what-is-zanzibar/

debarshri 5y ago

Maybe this is a naive question, at what point does an organisation decide to build a framework or a product internally than buying? Sure there was no opensource solution available but why not buy a product from other security companies. Just wondering.

spockz 5y ago
Integration into your own systems for example. Or specific requirements for which it is not clear they can be met by the product. Or unawareness. Or a combination of the above.
gneray 5y ago
It's a fair question, but in fairness to Carta, until recently, there have been ~zero commercial or open source solutions to building application authorization (beyond limited helper libraries or things like Active Directory, which solve only a portion of the problem). Developers have typically figured out the models on their own and implemented them in application code with IF statements, etc. and/or by building separate authorization services. Other publicly documented examples include teams at Slack (https://slack.engineering/role-management-at-slack/ https://slack.engineering/role-management-at-slack/) and Gusto (https://engineering.gusto.com/layering-authorization-into-a-web-application/ https://engineering.gusto.com/layering-authorization-into-a-...) OPA is one solution that's emerged in recent years. The author describes his team's point of view after evaluating it for their purposes. Oso (https://www.osohq.com/ https://www.osohq.com/) is a framework for application authorization. Disclosure: I'm a cofounder. Based on the post, the Carta team started their work in 2019, but we only open sourced Oso mid-2020, so they wouldn't have known about it. And in the last few months, a few other companies have announced offerings to address this problem too. So it's understandable why Carta chose to build. Now/soon other teams like them should have more options to choose from when they encounter similar challenges too.
simonw 5y ago
Build v.s. buy is genuinely one of the harder problems in software engineering strategy. You'll often hear people say "you should never, ever build something if you can buy a solution instead" (where buy also includes using an open source package) - but there are actually a bunch of other things to consider: - Is the solution you're considering well maintained and likely to keep working? Open source projects sometimes stop being updated. Vendors go out of business. - What's your escape strategy if the vendor DOES go out of business, or massively increases their pricing? - Integrating with a vendor / package is often a non-trivial amount of development work. Have you considered this relative to building your own, custom solution? - Do you have specific requirements that go beyond what's available off-the-shelf? It's absolutely true that your engineering efforts should be spent on the things that make your organization unique: time spent on your customer's problems is more valuable than time spent reinventing a wheel. But sometimes, the available wheels really aren't fit for your purposes. I've built when I should have bought, and I've bought when I should have built. Getting it right is hard! Often it doesn't become clear if you made the right decision until several years after you've made it.

deleted 5y ago

[deleted]

IndignantNerd 5y ago

When a startup I work for was borderline manipulated into onboarding onto the Carta platform [0], I did some brief security review. I noticed they’re loading third party JS with at least one postMessage handler with a wildcard origin. They were also sending the current page title (which can include things like company or investor name) to this provider. It appeared they were also using a version of Knockout.js that is 2+ years old. [1] This did not inspire a lot of confidence. The linked blog post at least takes security seriously, but it’s clear they’re patching over a lot of legacy cruft. There’s a hell of a lot of sensitive data on Carta, which I’m sure they’re already selling to their “partners” anyway, but IMO it’s only a matter of time before they (or one of their “partners”) are hacked and a whole bunch of companies suffer from information breach. IMO the main reason they haven’t been hacked yet is simply because you need to pay $2k just to get access to their questionably secure app. [0] Your lawyer tells you that you need a 409a and Carta is the best. But if you only want the 409A, and don’t want to get growth hacked onto Carta’s platform, you’re out of luck. They won’t do the 409A for you unless you complete the full onboarding process, which includes appointing Carta as your company’s transfer agent, and uploading the email addresses of your investors and allowing Carta to spam them with tasks like re-signing legal documents that you issued off-platform. [1] This is not a technical analysis and I’m just going by what I remember from watching my local chrome dev tools. Don’t @ me.

steevenwee 5y ago
I sense a lot of bitterness from this post. I don't think you have a full context on history of 409a valuations industry. Considering the fact that 409a valuations pre carta would cost $10k, $2k doesnt sound like a such an unfair deal. You could always go the legacy lawyer companies route and pay way more for a single 409a valuation. As for onboarding process - you do need a hell of a lot information to conclude said valuation.
- deleted 5y ago
  [deleted]

simonw 5y ago

I was pleased to see a section in here on what they call "flattening" - the problem of "efficiently provide a list of resources my user has access to". I've made this mistake more than once when designing an authorization system. Sure, you can build a fast mechanism for answering the question "does actor A have access to object B" - but can you efficiently answer that for 10,000 objects at once? Say for example you're building a search engine, and you want to search across all public data but ALSO all of the data your user has access to that's private. You probably can't run the search, get back 10,000 results, then apply your "can X access Y" logic to all 10,000 of them. (I'd love to know how GitHub code search implements this - it shows me results from my private repositories mixed in with results from their public index).

jzelinskie 5y ago
"Flattening", the equivalent of denormalizing a graph, is one way to implement performant ACL Filtering[0]. This is the same strategy used in the Zanzibar paper to gain the desired performance out of their Leopard index system. I tend to consider this largely a performance optimization strategy and not a necessarily the "solution" to this problem. We actually just this week wrote about how Authzed[1] actually reverse walks the graph in order to implement this API. Full disclosure: I'm a founder of Authzed (YC W21), which is a productized Zanzibar system. [0]: https://authzed.com/blog/six-month-profile-page/ https://authzed.com/blog/six-month-profile-page/ [1]: https://authzed.com/blog/acl-filtering-in-authzed/ https://authzed.com/blog/acl-filtering-in-authzed/
Diggsey 5y ago
The permissions system I designed at work solves the search problem, although it is not as powerful as what's described in the article. We have three main concepts in the sytem: - Capabilities (eg. "read profile") - Selectors (eg. "profiles belonging to customer X) - Groups (eg. "profile") All queries are of the form "Do I have capability X on resource Y" (where resource Y belongs to a particular group). A "permission set" is a mapping from a capability to a set of selectors. Selectors can be composed (eg. you could OR two selectors together) but are predefined. Since they are predefined, our search implementation can understand a subset of the selectors, and translate them into an ElasticSearch-compatible query. We then filter the results by evaluating the permissions normally: this catches any cases where there was no way to construct a query, or if the constructed query could not be completely precise. If this occurs, then we show a message on the page like "some profiles were hidden because you do not have permission to view them" and then we might show 19 profiles on a page instead of the full 20, so that our pagination can still work efficiently.

philsnow 5y ago

I hope "AuthZ" is just a temporary codename, that will be changed if/when they open source their work. authz/authn are very generic terms that already mean something and it's distasteful when companies squat on such terms. Imagine if a company came around and created a networking-related project and called it "TcP". > During our discovery process, we evaluated both open source and vendor products. > But we had a problem: There weren’t any open source implementations available. We decided to build our own system, adding several of our own modifications to make it an even better fit for Carta. There's no mention or implication that they'll release it as open source, this seems to be an engineering blog post (for hiring lead gen or building the engineering brand of the company or whatever).

fomine3 5y ago
This. Nowadays products tend to named too generic name.

lmeyerov 5y ago

Love seeing the visuals here :) We work on scaling graph UIs, and I've been noticing the auth use case more and more! The discussion on Flattening is super important, though for most teams, I'd also add this dances around the costs of an extra piece of intertwined DB infra. For our internals, we ended up having to reject OPA and friends, including what Zanzibar does, because we decided our customers wouldn't want us to spend our infra innovation tokens on an extra DB and figuring out how it should intertwine on the hotpath for queries to the main DBs. For bigger/heavier sites like Carta (with especially complex rules) and especially high-traffic ones like Google, as hinted by the percentile charts, I can see their decision making way more sense. For most consumer & b2b sites though, I'm not as convinced, and especially long-term. We ended up going with Casbin as it can do ARBAC policies + reuses our SQL DB. So it's more like adding a lib than infra, even though it does come with policy features like an analysis-friendly DSL. Likewise, that means we get a few ways to tune bulk queries: most are automatically fast by smart default roles & logic, some via special-cased app-level opts (so out-of-DB), and if we ever need to, we can always do optimized in-DB queries. The options for policy engines for b2b apps is surprisingly slim. The whole experience has made me think that a SQL DB with a strong multitenant+arbac RLS story for Django/Rails/etc. users and a cloud autoscale-to-zero capability will make a killing for b2b SaaS's. Amazon Aurora and friends are close, but feels like it's still another couple years in practice.

tristanz 5y ago

Does anybody have any paid or open source solutions that they actually like? All the open sources solutions I've tried don't actually work for real world use cases with highly dynamic policies, users, groups, and objects (e.g. Google Docs). Often the issue is the inability to easily push down filters into the database or search index. Aserto looks very promising (https://www.aserto.com/ https://www.aserto.com/), but it still in private beta. It's crazy this still is part of the stack where there are no great solutions.

samjs 5y ago
> It's crazy this still is part of the stack where there are no great solutions. Seems like a few others have come to the same conclusion :) We're working on this at Oso (https://osohq.com https://osohq.com) - I'm the CTO. Oso is an open source framework for authorization. Policies can reference application directly, so any authorization decisions can be made dynamically based on the data. And your data doesn't need to leave your application. > Often the issue is the inability to easily push down filters into the database or search index. We actually support pushing down filters into the database directly through a feature we call... data filtering. We support Django and SQLAlchemy today (https://docs.osohq.com/python/guides/data_access.html https://docs.osohq.com/python/guides/data_access.html) but will be making this available for non-ORM/non-Python users in the next few months.
Bombthecat 5y ago
Doesn't opa do something like this?
jbotz 5y ago
There is Ory Keto (https://www.ory.sh/keto https://www.ory.sh/keto).
- parkerhiggins 5y ago
  I was surprised not to see a comparable evaluation to Ory Keto at all in this article. Ory Keto continued to evolve out of the shortcomings of OPA https://github.com/ory/keto/issues/47 https://github.com/ory/keto/issues/47 . It's also based on Zanzibar, and it's used in production applications today. Cloud version coming soon (using cockroachdb) or use the open source version (postgres).

twiddling 5y ago

Having XACML flashbacks

ogazitt 5y ago
LOL :) XACML had some great ideas (my team worked on it back in 2004-2005), but god, all those angle brackets <shudder/>

netvarun 5y ago

There is also Keto - an open source project based on the Zanzibar model [https://github.com/ory/keto https://github.com/ory/keto]

denysvitali 5y ago
This! Also, the full Ory suite is very well developed and worth looking into. Fully open source!

spankalee 5y ago

It's wild that an equity management startup has to do this level of engineering. Goes to show how much more there is to build out in cloud and OSS platform offerings.

SahAssar 5y ago

- A: But how do you solve authz? - B: authz - A: Yeah, how do you do it? What system are you using? - B: authz - A: ... Why are you not answering me?

orf 5y ago

Tl;dr - a simple-sounding CRUD app for managing your options invests a huge amount of resources building a bespoke permission system inspired by Google to handle less than 15 RPS.

p_l 5y ago
But what's the latency requirements and security requirements of the domain?

0xbadcafebee 5y ago

Please don't build software just because Google published a paper. First off, it's not like it's a great design just because Google created it, they've churned out some real stinkers. And it's not like the code is guaranteed to be good; because you're not a Google engineer, you can't possibly be a God-like super-coder who designs nothing but perfect systems. (Or so you'd think from the way people fetishize FAANG papers) The most frequent design mistake I see is an engineer with a "hammers before nails" complex. They want a perfect design that will work for everything (that they have so far thought of) in some beautiful and hidden way. So they build the thing, and they think it's great. And then they go to hook it up to products... and it doesn't work well. The user story, the context diagram, and the mock-up all need to come first, not the code.

p_l 5y ago
Well then, maybe you should have read the article. It describes how they iterated until finally their requirements made them think of Zanzibar and decided to test it out, then how it evolved based on internal feedback.

jon-whit 5y ago

I've been working on a scalable open-source version of Google Zanzibar called the Authorizer platform. The implementation varies from that of Authzed by horizontally scaling the query space across more than one node. It uses consistent hashing with bounded loads to assign a specific node to serve the queries for the objects that it's responsible for. It also supports the namespace rewrite specification of the paper and the continuous namespace config monitoring. Much more progress is coming as well! The upcoming v0.2.0 release will have distributed caching to reduce latency on both the client and server side. Take a look, I'd love feedback and to build it into something valuable for the open-source community. https://github.com/authorizer-tech/access-controller https://github.com/authorizer-tech/access-controller https://authorizer-tech.github.io/ https://authorizer-tech.github.io/