4 ms·

Thought experiment: imagine you are going to port Bugzilla to a noSQL solution. How would you do it? There are some pretty obvious relationships between bugs, a

by wanderr 16y ago

Thought experiment: imagine you are going to port Bugzilla to a noSQL solution. How would you do it? There are some pretty obvious relationships between bugs, authors, commenters and comments. Imagine a bug that has 100 comments from as many commenters. A normalized soluion with no joins is going to require at least 200 round trips to look up all comment and commenter information, so the need to denormalize is pretty obvious. So let's say you enormalize all the way and save all relevant information in he bug document. Each bug has comments embedded in it and each comment has all of the commenter's information in it. Now each bug requires one lookup, which is awesome, but what about the fact that commenter information can change? Sure it's a somewhat rare event, but users can have thousands and thousands of comments. When a user changes their email address, do you tell them, this might take a few hours, in the meantime I hope you still have access to your old address? What kind of load do those massive updates place on your system? Unlike reads which are fairly steady and regular, these high load queries will be sporadic. How do you plan for and accommodate the spikes?

viraptor 16y ago
Even with millions of records where the user's email needs to be updated - that's not going to take longer than a couple of minutes. Then again - another solution would be to keep an "invalidated" entities list. It will never be very long - just keep all not-finished updates there. Bug report page goes up to 2 queries (bug + invalidated info list) + any number of entries you need to request by their actual ids.
artsrc 16y ago
One idea is that if an email address is updated the bugs are not updated straight away (inconsistency). They are updated as load on the system allows. Then you control the spikes as you see fit, to match your performance requirements. Another ideas is you keep a list of un-applied changes and post process with those (fix the inconsistency). Banks used to do those when I was at university (balance = overnight balance + todays transactions, plus an overnight batch to get the balance up to date.) If you original system was normalized, and did not fit in memory, then the joins cost (disk_access_time * items), which could have been 2 seconds of disk io. The notion that a db round trip is worse than a disk io is just wrong.
- wanderr 16y ago
  There's a solution for when data doesn't fit in memory on one box called sharding. Anyway, not trying to make the case that one round trip is worse, but certainly hundreds or thousands can be.
  - artsrc 16y ago
    Sharding does not allow arbitrary joins to work efficiently. This is a really big issue with the relation model for data, and flocks of cheap hardware for storage.
wanderr 16y ago
Update: people are offering suggestions of how to work around the problems I brought up. My point isn't that the problems are insurmountable, only that it adds (often unnecessary) complexity and opportunities for inconsistency to the system, and the suggested solutions certainly help to illustrate that. Is's also easy to look at this from an individualist perspective and thin "well, I won't screw that up" and maybe you're right. But what about every single developer in your organization? Are they all going to consider the complexities involved in this system? Are they even guaranteed to be aware of them?

by wanderr 16y ago

viraptor 16y ago

Even with millions of records where the user's email needs to be updated - that's not going to take longer than a couple of minutes. Then again - another solution would be to keep an "invalidated" entities list. It will never be very long - just keep all not-finished updates there. Bug report page goes up to 2 queries (bug + invalidated info list) + any number of entries you need to request by their actual ids.

artsrc 16y ago

One idea is that if an email address is updated the bugs are not updated straight away (inconsistency). They are updated as load on the system allows. Then you control the spikes as you see fit, to match your performance requirements. Another ideas is you keep a list of un-applied changes and post process with those (fix the inconsistency). Banks used to do those when I was at university (balance = overnight balance + todays transactions, plus an overnight batch to get the balance up to date.) If you original system was normalized, and did not fit in memory, then the joins cost (disk_access_time * items), which could have been 2 seconds of disk io. The notion that a db round trip is worse than a disk io is just wrong.

wanderr 16y ago
There's a solution for when data doesn't fit in memory on one box called sharding. Anyway, not trying to make the case that one round trip is worse, but certainly hundreds or thousands can be.
- artsrc 16y ago
  Sharding does not allow arbitrary joins to work efficiently. This is a really big issue with the relation model for data, and flocks of cheap hardware for storage.

wanderr 16y ago

Update: people are offering suggestions of how to work around the problems I brought up. My point isn't that the problems are insurmountable, only that it adds (often unnecessary) complexity and opportunities for inconsistency to the system, and the suggested solutions certainly help to illustrate that. Is's also easy to look at this from an individualist perspective and thin "well, I won't screw that up" and maybe you're right. But what about every single developer in your organization? Are they all going to consider the complexities involved in this system? Are they even guaranteed to be aware of them?