3 ms·
Interesting, I would have thought that the unidirectional, read-only nature of the publisher-subscriber relationship would have made this simple for a tradition
by abalone 13y ago
Interesting, I would have thought that the unidirectional, read-only nature of the publisher-subscriber relationship would have made this simple for a traditional SQL database with read replicas and a very basic partitioning scheme. You assign workers to monitor feeds for updates, they update the DB, and.. done.
Looks like they may have added some complexity with their feed parser implementation, what they refer to as "supernoders". Looks like they don't lock ownership of feeds during parsing, thus allowing concurrent supernoders to get into race conditions while parsing the same feed.
And so it turns into another NoSQL example of employing conflict resolution to fix things.
I wonder if they could just use a simple locking scheme to prevent more than 1 parser from parsing the same feed at the same time. This sounds simpler than conflict resolution, to me.
- bonzoesc 13y agoI wonder if they could just use a simple locking scheme to prevent more than 1 parser from parsing the same feed at the same time. This sounds simpler than conflict resolution, to me. You may want to check out Aphyr's "Call Me Maybe" series of posts about distributed databases: http://aphyr.com/tags/jepsen http://aphyr.com/tags/jepsen The short version is that convergent conflict resolution seems intimidating but works better than locking and synchronization.
- abalone 13y agoOk, I did. I actually read the whole series, including the postgres 2-phase commit I/O error case. I don't see where it draws that conclusion at all. The postgres post shows that even ACID databases have network error cases which can leave your client in an indeterminate state. Fair enough. However the solution for this is... to restart your client once the network's back up. All it needs to do is requery the DB to determine the truth. Compare that to writing conflict resolution logic for all your data because there is no single source of truth. This is considerably more complicated. The series actually ends up recommending "the right design for the right problem space." I am not making a general SQL vs. NoSQL argument here, but I think in this case they may have taken a more complicated approach than necessary.
- harryh 13y agoThat's a very strange interpretation of Aphyr's posts.
- julien 13y agoLocking is a no go has we get 2 things: high frequency updates feeds (up to a couple dozen entries per second), which we need to constantly update. Locking them for even a fraction of time could turn into a nightmare and a huge backlog if for any reason a single entry takes a little more time to be written.
- abalone 13y agoFor RSS feeds? Or are you talking about something else? RSS is not a streaming protocol. The .rss/.atom file would explode under the conditions you describe. Sounds like you are designing for a more complicated case?