4 ms·

"What have I missed here?" They don't have to check 1.7 million users to see if they have @replies on, if they maintain a list with "@reply subscribers", there

by nop 17y ago

"What have I missed here?" They don't have to check 1.7 million users to see if they have @replies on, if they maintain a list with "@reply subscribers", there's little overhead as far as I can see. If a something rarely changes but is checked a lot, it's a pretty good idea to have it pre-calculated. Edited: "No overhead", was a bit to absolute thanks jpwagner. Re-Edited: Oh look there was a commenting feature on the blog, was pretty hard to find.

jpwagner 17y ago
agreed. actually that's not NO overhead, but it's trivial and the right solution. instead of just one flag link on an HN post, there should be flag as inappropriate vs. flag as stupid *edit: make that "flag as waste of time"
Carnage4Life 17y ago
That solution doesn't address my question. Caching @reply subscribers only addresses the 3% of users who have opted-in to receiving all @replies. For the remaining 97% whether they receive @replies from someone they are following or not is a function of which user the reply is directed to and whether that user is also their friend. You can't cache that, at best you can optimize how you calculate who should receive replies. In fact, what you've pointed out is that building an implementation to address the 3% case is straightforward which was the point of my post. PS: My blog does have a commenting feature. In fact there are 3 comments in response to the post.
- joshwa 17y ago
  those comments are very well hidden! Why not just display them by default they way the other 99% of blogs do? Or if you're going to hide them, at least put the view comments link at the bottom of the post where I'd expect the comments to be.
- jpwagner 17y ago
  Person X member of 97% Person Y member of 97% Person Z member of 3% Person W member of 3% ________________________ Person_A_@_List: X, Y, Z Person_B_@_List: X, W ________________________ Person_A writes: @Person_B you be cool! Received by X and Z...not Y or W
  - Carnage4Life 17y ago
    I've updated my post based on your comment above. Thanks, it was rather helpful.
- ck113 17y ago
  If you're interested in wild speculation (I have no idea how Twitter's database works), here's how I interpreted Biz's explanation: With the new system, say user Foo writes "@Bar lol me too!". Then Twitter can take Foo's follower list, join it with Bar's follower list, and send the message to everyone in the resulting list. Relational databases are very good at joins. On the other hand, with the old system, they'd have to do a deep inspection of the record for each of Foo's followers to know if they should send the message to that follower. Relational databases are much less good at this. But, as you and others have pointed out, if the number of users that use the "all @-replies" feature is really so small, it would be fairly inexpensive to cache the list of all of Foo's followers who use that feature, and join them in as well. I don't know why they don't do that -- maybe it adds up (like, if only 3% of users use the feature, but those users follow a lot of other users, they'll each end up in a lot of other users' caches).
- dws 17y ago
  The 3% (of users affected) number has been trotted out a few times. Has twitter ever backed that up? It feels like a way of marginalizing the people who're complaining about the change, by painting them (us) as a vocal minority. (Following up: A trusted source confirms 3%)
  - ubernostrum 17y ago
    Taking into consideration the fact that the percentage of people likely to put in the time to explore documentation and settings and discover the original configurability is probably quite small, and a reasonable assumption that not everyone who discovers the functionality will use it, yeah, the number sounds all right. Note, btw, that the number they seem to be going for is the above -- e.g., "according to the accounts database, only 3% of users ever changed this setting", _not_ "only 3% of users are claiming to care about this now that it's been removed".
gaius 17y ago
The thing you're missing is that everyone at Twitter slept through CS 101.
- dasil003 17y ago
  Give me a break, scaling Twitter is not easy. CS principles do not magically allow you to construct an optimal solution for real world engineering problems. At Twitter's volume, the tiniest bit of extra latency in some middleware or network hardware can be the bottleneck. It's so easy to sit here and play armchair engineer and diagnose Twitter's (mostly past, btw) problems with a nauseating mix of hubris and ignorance.
  - serhei 17y ago
    > ... with a nauseating mix of hubris and ignorance. That, and the pointing out of quite natural questions which Twitter has failed to address in its explanation of how it is acting. I mean, once you know what you are doing, being able to satisfy people asking ignorant and hubristic questions is a pretty sweet bonus.
  - gaius 17y ago
    nauseating mix of hubris and ignorance I'm afraid it's a nauseating mix of having operated real pub-sub systems such as TIBCO Rendezvous on major stock exchanges. What Twitter does is not difficult and they do it badly.
    - ubernostrum 17y ago
      So now you bait and switch? You started out with a condescending post which implied that anyone who'd stayed awake in CS 101 would find Twitter trivially easy to scale. Then you switched course to claiming that you know how to do this because of experience working a gigantic installation. Or are you asserting that the average CS 101 project consists of running the production messaging systems for major stock exchanges?
      - gaius 17y ago
cedsav 17y ago
What if they have a fast way to get the intersect between followers of 2 different users? In this case, if a message is a @reply they could ignore the follower lists of each users and use the intersect list only (which can be cached). Supporting the 'get all replies' option means that they also need to consider the full follower lists and then deduplicate. .. just guessing here

by nop 17y ago

jpwagner 17y ago

agreed. actually that's not NO overhead, but it's trivial and the right solution. instead of just one flag link on an HN post, there should be flag as inappropriate vs. flag as stupid *edit: make that "flag as waste of time"

Carnage4Life 17y ago

That solution doesn't address my question. Caching @reply subscribers only addresses the 3% of users who have opted-in to receiving all @replies. For the remaining 97% whether they receive @replies from someone they are following or not is a function of which user the reply is directed to and whether that user is also their friend. You can't cache that, at best you can optimize how you calculate who should receive replies. In fact, what you've pointed out is that building an implementation to address the 3% case is straightforward which was the point of my post. PS: My blog does have a commenting feature. In fact there are 3 comments in response to the post.

joshwa 17y ago
those comments are very well hidden! Why not just display them by default they way the other 99% of blogs do? Or if you're going to hide them, at least put the view comments link at the bottom of the post where I'd expect the comments to be.
jpwagner 17y ago
Person X member of 97% Person Y member of 97% Person Z member of 3% Person W member of 3% ________________________ Person_A_@_List: X, Y, Z Person_B_@_List: X, W ________________________ Person_A writes: @Person_B you be cool! Received by X and Z...not Y or W
- Carnage4Life 17y ago
  I've updated my post based on your comment above. Thanks, it was rather helpful.
ck113 17y ago
If you're interested in wild speculation (I have no idea how Twitter's database works), here's how I interpreted Biz's explanation: With the new system, say user Foo writes "@Bar lol me too!". Then Twitter can take Foo's follower list, join it with Bar's follower list, and send the message to everyone in the resulting list. Relational databases are very good at joins. On the other hand, with the old system, they'd have to do a deep inspection of the record for each of Foo's followers to know if they should send the message to that follower. Relational databases are much less good at this. But, as you and others have pointed out, if the number of users that use the "all @-replies" feature is really so small, it would be fairly inexpensive to cache the list of all of Foo's followers who use that feature, and join them in as well. I don't know why they don't do that -- maybe it adds up (like, if only 3% of users use the feature, but those users follow a lot of other users, they'll each end up in a lot of other users' caches).
dws 17y ago
The 3% (of users affected) number has been trotted out a few times. Has twitter ever backed that up? It feels like a way of marginalizing the people who're complaining about the change, by painting them (us) as a vocal minority. (Following up: A trusted source confirms 3%)
- ubernostrum 17y ago
  Taking into consideration the fact that the percentage of people likely to put in the time to explore documentation and settings and discover the original configurability is probably quite small, and a reasonable assumption that not everyone who discovers the functionality will use it, yeah, the number sounds all right. Note, btw, that the number they seem to be going for is the above -- e.g., "according to the accounts database, only 3% of users ever changed this setting", _not_ "only 3% of users are claiming to care about this now that it's been removed".

gaius 17y ago

The thing you're missing is that everyone at Twitter slept through CS 101.

dasil003 17y ago
Give me a break, scaling Twitter is not easy. CS principles do not magically allow you to construct an optimal solution for real world engineering problems. At Twitter's volume, the tiniest bit of extra latency in some middleware or network hardware can be the bottleneck. It's so easy to sit here and play armchair engineer and diagnose Twitter's (mostly past, btw) problems with a nauseating mix of hubris and ignorance.
- serhei 17y ago
  > ... with a nauseating mix of hubris and ignorance. That, and the pointing out of quite natural questions which Twitter has failed to address in its explanation of how it is acting. I mean, once you know what you are doing, being able to satisfy people asking ignorant and hubristic questions is a pretty sweet bonus.
- gaius 17y ago
  nauseating mix of hubris and ignorance I'm afraid it's a nauseating mix of having operated real pub-sub systems such as TIBCO Rendezvous on major stock exchanges. What Twitter does is not difficult and they do it badly.
  - ubernostrum 17y ago
    So now you bait and switch? You started out with a condescending post which implied that anyone who'd stayed awake in CS 101 would find Twitter trivially easy to scale. Then you switched course to claiming that you know how to do this because of experience working a gigantic installation. Or are you asserting that the average CS 101 project consists of running the production messaging systems for major stock exchanges?
    - gaius 17y ago

cedsav 17y ago

What if they have a fast way to get the intersect between followers of 2 different users? In this case, if a message is a @reply they could ignore the follower lists of each users and use the intersect list only (which can be cached). Supporting the 'get all replies' option means that they also need to consider the full follower lists and then deduplicate. .. just guessing here