7 ms·

Tao: Facebook’s distributed data store for the social graph

170 points by mlerner 5y ago

techbio 5y ago
Later in the article the name is explained in that TAO is used as short for "The associations and objects" and while it's clever and I'm not really offended it really should be TAAO. Regardless, I'm surprised not to see anything about caching objects as application state using[1] or similar. [1]https://www.php.net/manual/en/book.apcu.php https://www.php.net/manual/en/book.apcu.php
techbio 5y ago
Yet taoism is unchanged.
deleted 5y ago
[deleted]
sizediterable 5y ago
I'm surprised that they published the more recent paper discussed in part 2 [0] since, IIRC, TAO has supported atomic transactions since at least when I was there a few years ago, but better late than never I suppose. I wonder if that means there's still hope that someone at the company will write a blog post about frameworks like: - EntSchema/EntPattern - EntQL - EntAction - PrivacyPolicy - Ent Deletion framework - Ent consistency checkers/fixers (I forget the name of the framework/tool) It's sad to think that people's only glimpse into these tools is through https://entgo.io https://entgo.io which is a really crude approximation of the real thing due to Go's lack of generics and expressiveness. Imagine if such amazing tools could be used at actually ethical companies. [0] https://www.micahlerner.com/2021/10/23/ramp-tao-layering-atomic-transactions-on-facebooks-online-tao-data-store.html https://www.micahlerner.com/2021/10/23/ramp-tao-layering-ato...
- colesantiago 5y ago
  why do tools like this and others always get invented at unethical companies?
  - baby 5y ago
    Unethical from your point of view
    - colesantiago 5y ago
      still unethical no matter how you slice it.
      - polio 5y ago
        Pack it in, guys, colesantiago has solved morality.
  - koolba 5y ago
    Developing tools costs money and unethical companies will always be more profitable than ethical ones as they have access to both ethical and unethical revenue streams. More profits means more money to throw at NIH projects.
    - nefitty 5y ago
      The less barriers the people in an organization perceive, the more likely they are to innovate, whether for the greater good or not.
      - int0x2e 5y ago
        I agree with your point, but I've personally also experienced the reverse as well - having very difficult constraints while being extremely motivated and well funded has often resulted in extremely innovative solutions. All of my examples are classified, sadly.
        nefitty 5y ago
        I can see how more constraints can lead to more focused problem solving for sure.
  - 5y ago
- cletus 5y ago
  Technically those are unrelated to TAO. Those are all part of layers that sit on top of TAO. The TAO part is optional too. For example, you can put the Ent data access layer on top of MySQL directly.
  - sizediterable 5y ago
    Right, I wasn't trying to imply a relationship. I only meant to say that if they're willing to write about old news for one piece of tech, then maybe they'd do it for another.
    - tehlike 5y ago
      Isn't ent pretty much an orm like n/hibernate
      - sizediterable 5y ago
        It is an ORM, but I wouldn't call it just an ORM. I don't have experience with NHibernate, but most ORMs I have experience with feel like just a way progammatically build queries. I found EntMutator to provide nice abstractions to couple data changes with business logic like mutation triggers and observers. Again, all the additional tooling built around the Ent framework adds to the value.
        tehlike 5y ago
        Things like observers are fairly available in the orm land too. Same with mutators and unit of work and so on.
- mhlakhani 5y ago
  There’s now a paper on the deletion framework at least. [1] is a high level summary that links to the paper. [1] https://engineering.fb.com/2020/08/12/security/delf/ https://engineering.fb.com/2020/08/12/security/delf/
  - sizediterable 5y ago
    Oh cool, I didn't know about this. I guess they tend to publish more infrastructure-oriented papers, which is understandable, but I wish there was more in general about the developer APIs that they enable and how they benefit productivity from a product engineer's point of view.
    - katrielalex 5y ago
      Lead author on the deletion paper here. Thanks for the feedback! I agree it would be cool to talk about the Ent APIs some more; they have a lot of obvious and less-obvious benefits. (For example, we are only able to add deletion constraints to Ents as described in the paper because the Ent specification language is expressive enough to do it.)
- numair 5y ago
  > I was there a few years ago > Imagine if such amazing tools could be used at actually ethical companies. Congratulations. How long did it take you to realize things are, in fact, as bad as the "haters" (to use a favorite term of the FB inner circle) said they were? On a more on-topic note, is entGo actually worth implementing as an ORM for Go? I've always been suspicious of most ORMs for all of the various reasons we all know about letting other people manage your database connections, in terms of performance/etc...
  - sizediterable 5y ago
    I didn't leave due to the realization that the company was unethical, but there is a correlation there. I left because I got burnt out by the culture built around performance incentives. I didn't encounter anyone at the company who had inherently malicious or exploitative intent, but the culture drives the individual behaviors that in aggregate leads to all the harm caused by the company. That being said, everyone there, including myself should feel somewhat accountable, but it's really mostly on the leadership for having the power to change the culture but neglecting to do so. > is entGo actually worth implementing as an ORM for Go I can't speak to how valuable it would be when judged on its own. I just think that it currently does not and probably will not come close to delivering the same level of value as the FB Ent framework.
    - numair 5y ago
      I appreciate this reply. The sentiment you’ve shared is the same that a lot of current/former Facebook employees have privately emailed / called / etc and told me. The interesting thing is, it’s a lot of the seriously talented engineers and designers who care about these things the most, so there is hope for our future in what all of you will do based on these bad experiences. EntGo seems to be like it would be useful if used in combination with a bunch of other stuff that was built to take advantage of it, but that’s just my quick impression after looking through the docs. If entity modeling is your thing, I would be interested to know what you thought of Ecto in the Phoenix/Elixir ecosystem.
- southerntofu 5y ago
  > ethical companies What a cute oxymoron!
- ahupp 5y ago
  > IIRC, TAO has supported atomic transactions since at least when I was there a few years ago TAO has supported single-shard atomic updates forever but this work is about cross-shared transactions. I left a few years ago as well and this didn't exist at the time. Yeah I think a lot of people would enjoy writeups of those frameworks. There's also this very nice Ent framework, written by ex-FB engineer: https://ent.dev/ https://ent.dev/
  - sizediterable 5y ago
    Ah, that clears things up. That's sounds huge! > There's also this very nice Ent framework, written by ex-FB engineer: https://ent.dev/ https://ent.dev/ Awesome, hope it succeeds!
DaiPlusPlus 5y ago
Looking at TAO just as a data-model (instead of as a database system) and I'm really liking its simplicity. But when you have a huge system like this, there's always engine-level rules (e.g. referential integrity) and business-rules that need to apply: for example, a Location-object can't "follow" a Comment and a PM cannot "check-in" to a User, and Human Users can't "friend" a declared bot User (I think?), given their overall design I assume the rules about what objects and associations can and cannot exist are enforced by the Association APIs when an association is created - so rejecting an invalid create-association request is straightforward, but how does it handle ever-changing business-requirements that can retroactively apply to existing objects and associations (e.g. supposing that FB decides to allow Human users to befriend bot users, so suddenly Mark Zuckerberg suddenly now has real Friend associations (shots fired), and then they decide to undo that policy change, how is the rule-change applied? What happens to the existing objects? What happens if two different physical FB servers running different versions of the software want to apply conflicting business rules? Another thing I don't yet understand about using MySQL (or any SQL-first RDBMS for that matter) as a storage layer for an ostensible object/document system (hierarchical structured data) is that now you run into the object-relational impedance mismatch problem - which has many unpleasant solutions (e.g. using the EAV anti-pattern, which defeats the point of using an RDBMS in the first place - or going all-in with Codd's normalization and ending up with a rigid, strict data model schema design which is very difficult to update when business requirements change (i.e. using a table for each Codd relation (tuple) type to for each TAO object type and having to run ALTER TABLE ADD/DROP COLUMN every time you want to update the TAO object schema - which obviously does not scale. I assume each TAO object can be treated as a keyed blob with type-name (i.e. a 3-tuple of (id, type, blob) ) - in which case using an RDMBS is overkill and introduces many overheads, so why use an RDMBS? Especially as I understand that pretty much every major web-service will disable things like referential integrity constraints for the sake of performance (and you can't have a constraint w.r.t data in a blob column anyway). ...so I assume I'm missing something, but what?
- bcherny 5y ago
  > how does it handle ever-changing business-requirements that can retroactively apply to existing objects and associations If the change is backwards-compatible: 1. Update the schema definition 2. Run codegen 3. Done If it’s not: 1. Create a new field/assoc 2. Double-write data to the new and old fields/assoc 3. Backfill data from the old one to the new one 4. Delete the old one 5. Done > I assume each TAO object can be treated as a keyed blob with type-name (i.e. a 3-tuple of (id, type, blob) ) - in which case using an RDMBS is overkill and introduces many overheads, so why use an RDMBS? I believe data is stored in tuples, same as most other graph DBs. I can’t speak to why MySQL in particular.
forgingahead 5y ago
> I was there a few years ago > Imagine if such amazing tools could be used at actually ethical companies. Glad you stood by your principles and left when you discovered they stood against your values. Honest question: did you return all the salary/equity you received while you were there?
ehnto 5y ago
Is there a philosophical or historical reason behind the world Tao being used for things that are distributed?
- travisd 5y ago
  In the post, they give the reason for the name in this case: The Associations and Objects.
peterburkimsher 5y ago
Tangentially related: On Facebook, "The average person is connected to every other person by an average of 3.57 steps." Three and a half degrees of separation https://research.fb.com/blog/2016/02/three-and-a-half-degrees-of-separation/ https://research.fb.com/blog/2016/02/three-and-a-half-degree... When the article was published, there was a tool that helped to visualise that, and I was surprised but excited to see my estimate at 2.91, presumably helped by extensive travelling and diverse social circles. I believe that if Facebook's success metric is connectedness, rather than profit, the world will benefit. It's easy to look the other way, and claim that big problems (plastic in the ocean, climate change) aren't my personal responsibility. And for 3 degrees of separation, that's reasonable: Switzerland has no ocean, so can't fix the plastic problem. But here in New Zealand, I bike past the ocean every day. To "love your neighbour" and stop to pick up litter while biking home from work will have a direct impact on the lives of birds and fish right here, and a butterfly effect impact on the whole ecosystem. By creating software tools like Tao to map the social graph, it's possible to measure the connectedness, and report these metrics. Links are also bidirectional (there's a "thank you" feedback loop) like Xanadu. For comparison, Wikipedia's connectedness is 3.019 degrees of separation. https://www.sixdegreesofwikipedia.com/blog/search-results-analysis https://www.sixdegreesofwikipedia.com/blog/search-results-an...

170 points by mlerner 5y ago

Later in the article the name is explained in that TAO is used as short for "The associations and objects" and while it's clever and I'm not really offended it really should be TAAO. Regardless, I'm surprised not to see anything about caching objects as application state using[1] or similar. [1]https://www.php.net/manual/en/book.apcu.php https://www.php.net/manual/en/book.apcu.php

techbio 5y ago

Yet taoism is unchanged.

deleted 5y ago

[deleted]

sizediterable 5y ago

I'm surprised that they published the more recent paper discussed in part 2 [0] since, IIRC, TAO has supported atomic transactions since at least when I was there a few years ago, but better late than never I suppose. I wonder if that means there's still hope that someone at the company will write a blog post about frameworks like: - EntSchema/EntPattern - EntQL - EntAction - PrivacyPolicy - Ent Deletion framework - Ent consistency checkers/fixers (I forget the name of the framework/tool) It's sad to think that people's only glimpse into these tools is through https://entgo.io https://entgo.io which is a really crude approximation of the real thing due to Go's lack of generics and expressiveness. Imagine if such amazing tools could be used at actually ethical companies. [0] https://www.micahlerner.com/2021/10/23/ramp-tao-layering-atomic-transactions-on-facebooks-online-tao-data-store.html https://www.micahlerner.com/2021/10/23/ramp-tao-layering-ato...

colesantiago 5y ago
why do tools like this and others always get invented at unethical companies?
- baby 5y ago
  Unethical from your point of view
  - colesantiago 5y ago
    still unethical no matter how you slice it.
    - polio 5y ago
      Pack it in, guys, colesantiago has solved morality.
- koolba 5y ago
  Developing tools costs money and unethical companies will always be more profitable than ethical ones as they have access to both ethical and unethical revenue streams. More profits means more money to throw at NIH projects.
  - nefitty 5y ago
    The less barriers the people in an organization perceive, the more likely they are to innovate, whether for the greater good or not.
    - int0x2e 5y ago
      I agree with your point, but I've personally also experienced the reverse as well - having very difficult constraints while being extremely motivated and well funded has often resulted in extremely innovative solutions. All of my examples are classified, sadly.
      - nefitty 5y ago
        I can see how more constraints can lead to more focused problem solving for sure.
- 5y ago
cletus 5y ago
Technically those are unrelated to TAO. Those are all part of layers that sit on top of TAO. The TAO part is optional too. For example, you can put the Ent data access layer on top of MySQL directly.
- sizediterable 5y ago
  Right, I wasn't trying to imply a relationship. I only meant to say that if they're willing to write about old news for one piece of tech, then maybe they'd do it for another.
  - tehlike 5y ago
    Isn't ent pretty much an orm like n/hibernate
    - sizediterable 5y ago
      It is an ORM, but I wouldn't call it just an ORM. I don't have experience with NHibernate, but most ORMs I have experience with feel like just a way progammatically build queries. I found EntMutator to provide nice abstractions to couple data changes with business logic like mutation triggers and observers. Again, all the additional tooling built around the Ent framework adds to the value.
      - tehlike 5y ago
        Things like observers are fairly available in the orm land too. Same with mutators and unit of work and so on.
mhlakhani 5y ago
There’s now a paper on the deletion framework at least. [1] is a high level summary that links to the paper. [1] https://engineering.fb.com/2020/08/12/security/delf/ https://engineering.fb.com/2020/08/12/security/delf/
- sizediterable 5y ago
  Oh cool, I didn't know about this. I guess they tend to publish more infrastructure-oriented papers, which is understandable, but I wish there was more in general about the developer APIs that they enable and how they benefit productivity from a product engineer's point of view.
  - katrielalex 5y ago
    Lead author on the deletion paper here. Thanks for the feedback! I agree it would be cool to talk about the Ent APIs some more; they have a lot of obvious and less-obvious benefits. (For example, we are only able to add deletion constraints to Ents as described in the paper because the Ent specification language is expressive enough to do it.)
numair 5y ago
> I was there a few years ago > Imagine if such amazing tools could be used at actually ethical companies. Congratulations. How long did it take you to realize things are, in fact, as bad as the "haters" (to use a favorite term of the FB inner circle) said they were? On a more on-topic note, is entGo actually worth implementing as an ORM for Go? I've always been suspicious of most ORMs for all of the various reasons we all know about letting other people manage your database connections, in terms of performance/etc...
- sizediterable 5y ago
  I didn't leave due to the realization that the company was unethical, but there is a correlation there. I left because I got burnt out by the culture built around performance incentives. I didn't encounter anyone at the company who had inherently malicious or exploitative intent, but the culture drives the individual behaviors that in aggregate leads to all the harm caused by the company. That being said, everyone there, including myself should feel somewhat accountable, but it's really mostly on the leadership for having the power to change the culture but neglecting to do so. > is entGo actually worth implementing as an ORM for Go I can't speak to how valuable it would be when judged on its own. I just think that it currently does not and probably will not come close to delivering the same level of value as the FB Ent framework.
  - numair 5y ago
    I appreciate this reply. The sentiment you’ve shared is the same that a lot of current/former Facebook employees have privately emailed / called / etc and told me. The interesting thing is, it’s a lot of the seriously talented engineers and designers who care about these things the most, so there is hope for our future in what all of you will do based on these bad experiences. EntGo seems to be like it would be useful if used in combination with a bunch of other stuff that was built to take advantage of it, but that’s just my quick impression after looking through the docs. If entity modeling is your thing, I would be interested to know what you thought of Ecto in the Phoenix/Elixir ecosystem.
southerntofu 5y ago
> ethical companies What a cute oxymoron!
ahupp 5y ago
> IIRC, TAO has supported atomic transactions since at least when I was there a few years ago TAO has supported single-shard atomic updates forever but this work is about cross-shared transactions. I left a few years ago as well and this didn't exist at the time. Yeah I think a lot of people would enjoy writeups of those frameworks. There's also this very nice Ent framework, written by ex-FB engineer: https://ent.dev/ https://ent.dev/
- sizediterable 5y ago
  Ah, that clears things up. That's sounds huge! > There's also this very nice Ent framework, written by ex-FB engineer: https://ent.dev/ https://ent.dev/ Awesome, hope it succeeds!

DaiPlusPlus 5y ago

Looking at TAO just as a data-model (instead of as a database system) and I'm really liking its simplicity. But when you have a huge system like this, there's always engine-level rules (e.g. referential integrity) and business-rules that need to apply: for example, a Location-object can't "follow" a Comment and a PM cannot "check-in" to a User, and Human Users can't "friend" a declared bot User (I think?), given their overall design I assume the rules about what objects and associations can and cannot exist are enforced by the Association APIs when an association is created - so rejecting an invalid create-association request is straightforward, but how does it handle ever-changing business-requirements that can retroactively apply to existing objects and associations (e.g. supposing that FB decides to allow Human users to befriend bot users, so suddenly Mark Zuckerberg suddenly now has real Friend associations (shots fired), and then they decide to undo that policy change, how is the rule-change applied? What happens to the existing objects? What happens if two different physical FB servers running different versions of the software want to apply conflicting business rules? Another thing I don't yet understand about using MySQL (or any SQL-first RDBMS for that matter) as a storage layer for an ostensible object/document system (hierarchical structured data) is that now you run into the object-relational impedance mismatch problem - which has many unpleasant solutions (e.g. using the EAV anti-pattern, which defeats the point of using an RDBMS in the first place - or going all-in with Codd's normalization and ending up with a rigid, strict data model schema design which is very difficult to update when business requirements change (i.e. using a table for each Codd relation (tuple) type to for each TAO object type and having to run ALTER TABLE ADD/DROP COLUMN every time you want to update the TAO object schema - which obviously does not scale. I assume each TAO object can be treated as a keyed blob with type-name (i.e. a 3-tuple of (id, type, blob) ) - in which case using an RDMBS is overkill and introduces many overheads, so why use an RDMBS? Especially as I understand that pretty much every major web-service will disable things like referential integrity constraints for the sake of performance (and you can't have a constraint w.r.t data in a blob column anyway). ...so I assume I'm missing something, but what?

bcherny 5y ago
> how does it handle ever-changing business-requirements that can retroactively apply to existing objects and associations If the change is backwards-compatible: 1. Update the schema definition 2. Run codegen 3. Done If it’s not: 1. Create a new field/assoc 2. Double-write data to the new and old fields/assoc 3. Backfill data from the old one to the new one 4. Delete the old one 5. Done > I assume each TAO object can be treated as a keyed blob with type-name (i.e. a 3-tuple of (id, type, blob) ) - in which case using an RDMBS is overkill and introduces many overheads, so why use an RDMBS? I believe data is stored in tuples, same as most other graph DBs. I can’t speak to why MySQL in particular.

forgingahead 5y ago

> I was there a few years ago > Imagine if such amazing tools could be used at actually ethical companies. Glad you stood by your principles and left when you discovered they stood against your values. Honest question: did you return all the salary/equity you received while you were there?

ehnto 5y ago

Is there a philosophical or historical reason behind the world Tao being used for things that are distributed?

travisd 5y ago
In the post, they give the reason for the name in this case: The Associations and Objects.

peterburkimsher 5y ago

Tangentially related: On Facebook, "The average person is connected to every other person by an average of 3.57 steps." Three and a half degrees of separation https://research.fb.com/blog/2016/02/three-and-a-half-degrees-of-separation/ https://research.fb.com/blog/2016/02/three-and-a-half-degree... When the article was published, there was a tool that helped to visualise that, and I was surprised but excited to see my estimate at 2.91, presumably helped by extensive travelling and diverse social circles. I believe that if Facebook's success metric is connectedness, rather than profit, the world will benefit. It's easy to look the other way, and claim that big problems (plastic in the ocean, climate change) aren't my personal responsibility. And for 3 degrees of separation, that's reasonable: Switzerland has no ocean, so can't fix the plastic problem. But here in New Zealand, I bike past the ocean every day. To "love your neighbour" and stop to pick up litter while biking home from work will have a direct impact on the lives of birds and fish right here, and a butterfly effect impact on the whole ecosystem. By creating software tools like Tao to map the social graph, it's possible to measure the connectedness, and report these metrics. Links are also bidirectional (there's a "thank you" feedback loop) like Xanadu. For comparison, Wikipedia's connectedness is 3.019 degrees of separation. https://www.sixdegreesofwikipedia.com/blog/search-results-analysis https://www.sixdegreesofwikipedia.com/blog/search-results-an...