3 ms·

Kinda disappointed by the simulation, where are all the viral posts? I've been digging around for a while and haven't found any posts with more than 20 faves.

Kinda disappointed by the simulation, where are all the viral posts? I've been digging around for a while and haven't found any posts with more than 20 faves. The accounts I've found with ~1 million followers have little to no engagement. I want to see how a post with a million faves holds up to the promises of "fast constant time". I'm especially curious about these queries — fave-count and has-user-faved — since a couple years ago Twitter stopped checking has-user-faved when rendering posts more than a month or so old, so I imagine it was expensive at scale.

nathanmarz 3y ago
The load generator generates boosts/favorites for a subset of posts that are randomly picked to be "popular". However, since the rate of posts is so high even individual posts picked to be "popular" are only getting ~70 reactions. Tracking reactions is considerably easier than timeline fanout though, as a favorite does a small handful of things (updates set of users favoriting a status and sending a notification), while fanout has to do an operation on every follower (403 operations on average, sometimes up to 22M). The code getting the favorite count for a status looks like: .localSelect("$$statusIdToFavoriters", Path.key("*statusId").view(Ops.SIZE)).out("*numFavorites") Because the nested set is subindexed, that's an extremely fast operation (looking at our telemetry, about 0.05ms). Determining "has-user-faved" looks like: .localSelect("$$statusIdToFavoriters", Path.key("*statusId").view(Ops.CONTAINS, "*accountId")).out("\*hasFavorited") The API server doesn't do these queries individually, which would be two roundtrips. It does them together in a query topology along with fetching other needed information (like number of boosts, number of replies, "has boosted", "has muted this status", etc.).
- kyle-rb 3y ago
  Thanks for the response, I'm still curious about the details of the subindexing and how that scales. I'll be keeping an eye out for the release!
  - nathanmarz 3y ago
    A lot of detail on subindexing and everything else in Rama is coming next week when we release the docs.

nathanmarz 3y ago

The load generator generates boosts/favorites for a subset of posts that are randomly picked to be "popular". However, since the rate of posts is so high even individual posts picked to be "popular" are only getting ~70 reactions. Tracking reactions is considerably easier than timeline fanout though, as a favorite does a small handful of things (updates set of users favoriting a status and sending a notification), while fanout has to do an operation on every follower (403 operations on average, sometimes up to 22M). The code getting the favorite count for a status looks like: .localSelect("$$statusIdToFavoriters", Path.key("*statusId").view(Ops.SIZE)).out("*numFavorites") Because the nested set is subindexed, that's an extremely fast operation (looking at our telemetry, about 0.05ms). Determining "has-user-faved" looks like: .localSelect("$$statusIdToFavoriters", Path.key("*statusId").view(Ops.CONTAINS, "*accountId")).out("\*hasFavorited") The API server doesn't do these queries individually, which would be two roundtrips. It does them together in a query topology along with fetching other needed information (like number of boosts, number of replies, "has boosted", "has muted this status", etc.).

kyle-rb 3y ago
Thanks for the response, I'm still curious about the details of the subindexing and how that scales. I'll be keeping an eye out for the release!
- nathanmarz 3y ago
  A lot of detail on subindexing and everything else in Rama is coming next week when we release the docs.