4 ms·

...non-relational data that updates very quickly, doesn't need joins, needs high availability and can be processed later... What kinds of problems fit this mod

...non-relational data that updates very quickly, doesn't need joins, needs high availability and can be processed later... What kinds of problems fit this model? Specifically, why would I want to defer processing if I can process instantaneously? NoSQL has its place for sure, but is always augmented by a relational database somewhere. The Lambda architecture that's so popular at the moment is cool and all, but I find a simpler scenario where you index data into a relational database before writing it to a document database is good enough. That way I can process immediately, and if needed get the document out of a NoSL database (usually only to display to a user). A relational database that points at corresponding locations in a document database is useful in ways that document databases on their own are not. Relational databases on their own are useful in ways that document databases are not. I cannot say whether the same is true for document databases, as I've yet to encounter a real problem that can only be solved with a document database.

belluchan 13y ago
The problem I am describing is a situation in which you have and want to maintain high amounts of granular data that can be queried for specific ranges with varying amounts of detail. For example look at something like Google Analytics. As simple as page views with timestamps. If you have high traffic data, you can be writing to your database quite a bit. You could bucket that data per second per unique viewer. Then in your YUI you can select to see a line graph with varying degrees of detail. Per hour, per day, per month, per year. No way would I want to store this in an SQL database. The processing that happens later are the per time period rollups. Creating buckets of minutes, days, etc per user. This can be done when asked for and cached (slow experience for the user), or scheduled and persisted (expensive - consumes more space). Again I would not want to store this in SQL. All the while I maintain my raw data for further processing later. These kinds of solutions are great for monitoring data, tracking, any kind of high volume, real time statistical data.
- Spearchucker 13y ago
  The example you use is Google Analytics. I've no idea how big that dataset is, but I imagine bigger than anything I've encountered before. That said, I'd do everything you mention in a relational database. SELECT COUNT(fieldname) FROM {tablename} and all that. Are you ok with eventual consistency as a price for partition tolerance and availability? The data sizes I typically work with allow me to go with availability and consistency at the expense of partition tolerance. And that's ok because my data replicates, and my response time objectives and recovery point objectives are set to something appropriate. This is worth it to me if the benefit is an operational data store that I can query in real time without impacting writes. Still haven't heard of a scenario which would compel me to go a NoSQL only route (a scenario that isn't Google or Facebook or similar).

belluchan 13y ago

The problem I am describing is a situation in which you have and want to maintain high amounts of granular data that can be queried for specific ranges with varying amounts of detail. For example look at something like Google Analytics. As simple as page views with timestamps. If you have high traffic data, you can be writing to your database quite a bit. You could bucket that data per second per unique viewer. Then in your YUI you can select to see a line graph with varying degrees of detail. Per hour, per day, per month, per year. No way would I want to store this in an SQL database. The processing that happens later are the per time period rollups. Creating buckets of minutes, days, etc per user. This can be done when asked for and cached (slow experience for the user), or scheduled and persisted (expensive - consumes more space). Again I would not want to store this in SQL. All the while I maintain my raw data for further processing later. These kinds of solutions are great for monitoring data, tracking, any kind of high volume, real time statistical data.

Spearchucker 13y ago
The example you use is Google Analytics. I've no idea how big that dataset is, but I imagine bigger than anything I've encountered before. That said, I'd do everything you mention in a relational database. SELECT COUNT(fieldname) FROM {tablename} and all that. Are you ok with eventual consistency as a price for partition tolerance and availability? The data sizes I typically work with allow me to go with availability and consistency at the expense of partition tolerance. And that's ok because my data replicates, and my response time objectives and recovery point objectives are set to something appropriate. This is worth it to me if the benefit is an operational data store that I can query in real time without impacting writes. Still haven't heard of a scenario which would compel me to go a NoSQL only route (a scenario that isn't Google or Facebook or similar).