3 ms·

> Even within a "typical request-response deadline" > blasting through some thousands of "events" does sound > kind of trivial? If they were all on t

> Even within a "typical request-response deadline" > blasting through some thousands of "events" does sound > kind of trivial? If they were all on the same physical machine, sure. But the events are spread across multiple nodes, so a single stream read can potentially touch every Redis instance. > I suppose it's great to be able to say that these > guarantees will hold, but I'm not sure I see them as > needed for (all of) soundcloud's events? I'm all about cheating, if it nets me a big win. How would you cheat, here? Honest question; it's not apparent to me what corners we could cut, given the whole point, more-or-less, is to avoid pre-materializing views.

peterwwillis 12y ago
Why do you want to avoid pre-materializing views? Storage issues? Concerns about not-live-enough information? Both?
- erichocean 12y ago
  Both, due to the Power Law property/problem inherent in natural networks (of which social networks are an obvious example). For example, (I'm told that) Twitter has effectively two separate systems for distributing updates for users, one for normal users (you and I), and one for celebrities. Propagating Lady Gaga's tweets to all 30 million followers can take something like five minutes to accomplish. It's even more fun when these celebrities start tweeting back and forth with each other. Maybe SoundCloud's CRDT approach would work better for natural networks? At some point, network communication could start to be an issue with a compile-the-list-at-read design, but I could be wrong. I personally don't see why a CRDT is necessary if you used something like Twitter's snowflake approach to assign ids. Pull in events is sorted order from each machine, merge, then push to the user. It's not clear to me what adding the CRDT abstraction adds to that besides overhead. Maybe there are other benefits I'm not grasping?
  - sagichmal 12y ago
    > At some point, network communication could start to be > an issue with a compile-the-list-at-read design, but I > could be wrong. You're absolutely right. Optimizing for that bottleneck occupied most of the development time.
e12e 12y ago
I suppose it just wasn't immediately obvious just how bad the read amplification problem was.

peterwwillis 12y ago

Why do you want to avoid pre-materializing views? Storage issues? Concerns about not-live-enough information? Both?

erichocean 12y ago
Both, due to the Power Law property/problem inherent in natural networks (of which social networks are an obvious example). For example, (I'm told that) Twitter has effectively two separate systems for distributing updates for users, one for normal users (you and I), and one for celebrities. Propagating Lady Gaga's tweets to all 30 million followers can take something like five minutes to accomplish. It's even more fun when these celebrities start tweeting back and forth with each other. Maybe SoundCloud's CRDT approach would work better for natural networks? At some point, network communication could start to be an issue with a compile-the-list-at-read design, but I could be wrong. I personally don't see why a CRDT is necessary if you used something like Twitter's snowflake approach to assign ids. Pull in events is sorted order from each machine, merge, then push to the user. It's not clear to me what adding the CRDT abstraction adds to that besides overhead. Maybe there are other benefits I'm not grasping?
- sagichmal 12y ago
  > At some point, network communication could start to be > an issue with a compile-the-list-at-read design, but I > could be wrong. You're absolutely right. Optimizing for that bottleneck occupied most of the development time.

e12e 12y ago

I suppose it just wasn't immediately obvious just how bad the read amplification problem was.