3 ms·

I feel like developers who haven't worked with medium-sized data don't know where to draw the "big data" line. You see this with companies overspending for comp

I feel like developers who haven't worked with medium-sized data don't know where to draw the "big data" line. You see this with companies overspending for complicated "big data" solutions when the only have something on the order of billions of rows and nothing more. I deal with many billion row tables in the products, and SQL Server still works wonderfully there. Throw 1TB of RAM and a stack of Fusion IO drives on a pair of servers and I'm confident the pair (for mirroring obviously) and a DBA can easily handle the needs of any company that thinks they have "big data", save the current big data pioneers (e.g. Google, Facebook) who are only spending time on these problems because throwing more money (read: hardware) was hitting diminishing returns. In my specific case, we have billions of rows store in SQL tables whose schemas (including PK and indices) and queries were built by server engineers without any explicit DBA experience. If we can manage to store our data quite naively in SQL server by just putting it on a pair of superboxes, I find it hard to believe this solution wouldn't apply to all but a handful of tech companies.

belluchan 13y ago
Big data doesn't have to be "big". I'm not going to store statistical information, information that has at most a value and a date and maybe a key or two, and is updated multiple times per second per user in an SQL database. It doesn't fit in the table row column paradigm. For me at least, big data is more about a form of non-relational data that updates very quickly, doesn't need joins, needs high availability and can be processed later to derive value out of it.
- gaius 13y ago
  An observation: people who know about databases of all kinds say "relational database". People who don't say "SQL database". Relational databases on commodity hardware can easily handle hundreds if not thousands of updates per second these days.
  - belluchan 13y ago
    Non-SQL databases can have relational features, like rethinkdb for example. http://www.rethinkdb.com/docs/table-joins/ http://www.rethinkdb.com/docs/table-joins/
  - endianswap 13y ago
    Although I don't think you were commenting directly on my post (your grandparent), I'd like to point out when I say a "SQL database" or "SQL server" I'm talking about Microsoft SQL Server.
- endianswap 13y ago
  We do exactly this, with billions of rows each day of statistical data right into MSSQL server instances. For data like this, the only difference is that we partition the data into daily tables and provide views into the data as necessary. What experience or expertise can you share as to why this doesn't work?
  - belluchan 13y ago
    I mean if it works for you I cannot say it doesn't work. It doesn't seem like a palatable solution however. Does your SQL database also have other types of information? Like account data? Does having billions of rows of statistical data make it more difficult to manage, replicate and secure non-statistical data? Like your logs must just be useless. You can bring down trees with dynamite, but just because that "works" doesn't mean it's the optimal solution.
  - jules 13y ago
    What's the reason for partitioning into daily tables? Why not add a day column and index it on that column?
    - endianswap 13y ago
      Easier to move old historical data onto larger, slower drives if it's stored independently.
    - RyanZAG 13y ago
      Index would become bigger than ram. Removing old data would be more difficult. They likely only use todays data. This is probably a trading data system of some kind.
- Spearchucker 13y ago
  ...non-relational data that updates very quickly, doesn't need joins, needs high availability and can be processed later... What kinds of problems fit this model? Specifically, why would I want to defer processing if I can process instantaneously? NoSQL has its place for sure, but is always augmented by a relational database somewhere. The Lambda architecture that's so popular at the moment is cool and all, but I find a simpler scenario where you index data into a relational database before writing it to a document database is good enough. That way I can process immediately, and if needed get the document out of a NoSL database (usually only to display to a user). A relational database that points at corresponding locations in a document database is useful in ways that document databases on their own are not. Relational databases on their own are useful in ways that document databases are not. I cannot say whether the same is true for document databases, as I've yet to encounter a real problem that can only be solved with a document database.
  - belluchan 13y ago
    The problem I am describing is a situation in which you have and want to maintain high amounts of granular data that can be queried for specific ranges with varying amounts of detail. For example look at something like Google Analytics. As simple as page views with timestamps. If you have high traffic data, you can be writing to your database quite a bit. You could bucket that data per second per unique viewer. Then in your YUI you can select to see a line graph with varying degrees of detail. Per hour, per day, per month, per year. No way would I want to store this in an SQL database. The processing that happens later are the per time period rollups. Creating buckets of minutes, days, etc per user. This can be done when asked for and cached (slow experience for the user), or scheduled and persisted (expensive - consumes more space). Again I would not want to store this in SQL. All the while I maintain my raw data for further processing later. These kinds of solutions are great for monitoring data, tracking, any kind of high volume, real time statistical data.
    - Spearchucker 13y ago
      The example you use is Google Analytics. I've no idea how big that dataset is, but I imagine bigger than anything I've encountered before. That said, I'd do everything you mention in a relational database. SELECT COUNT(fieldname) FROM {tablename} and all that. Are you ok with eventual consistency as a price for partition tolerance and availability? The data sizes I typically work with allow me to go with availability and consistency at the expense of partition tolerance. And that's ok because my data replicates, and my response time objectives and recovery point objectives are set to something appropriate. This is worth it to me if the benefit is an operational data store that I can query in real time without impacting writes. Still haven't heard of a scenario which would compel me to go a NoSQL only route (a scenario that isn't Google or Facebook or similar).
- mbesto 13y ago
  > big data is more about a form of non-relational data that updates very quickly, doesn't need joins, needs high availability You're talking about NoSQL actually. Interestingly, I find that big data is normally synonymous with the unstructured data that updates very frequently, doesn't need joins, etc that you speak of. Structured data in large businesses, such as general ledgers, sales orders, purchase orders, etc. is generally quite fast. The only problem in that area today is getting that data out into complex reporting. It requires a lot of processing (either routine or manual) to get it into an optimized format. Big data came about because companies like FB, Google, Twitter, pretty much any online analytics company, AirBnb, etc. are crunching large sets of behavioral events. (user clicked this button) Outside of those use cases, data just doesn't grow as fast as the marketeers of Big Data claim it is.
rossjudson 13y ago
And...you don't need to denormalize your data set if the underlying system is fast enough to deliver timely results. And because it's not denormalized, it's smaller, and you can fit more data...etc etc. Folks who think they have big data problems really need to define the size of their data set -- not the size of the resulting BigDataDB, or the number of servers they have. The two are not the same.
bane 13y ago
Something I've noticed is that there's a real trend in easing developer effort by trade-off with computational effort. In some cases it's worth it, but I think the pendulum has swung too far in most. What I mean are these efforts to build huge distributed compute clusters that do X by distributing the problem easily, but at huge overhead cost for each individual actually useful computation. The distribution, tracking of the compute tasks, the coordination of it all and the frameworks to make it easy insert such a tremendous overhead that I suspect most of the problems that are approached this way would be better tackled with a single machine and some smart old-fashioned algorithm development and developer elbow grease. Of course simple economics might decide if 20 machines doing something are cheaper than 1 developer working an extra 3 or 4 months so 1 machine can do the same thing resolves it. But I suspect that level of analysis just isn't done and the result is to just throw hardware at the problem. I've had to remind my developers in the past that we're working with machines that are literally capable of doing billions of operations per second and can do something like this https://www.youtube.com/watch?v=MJfceF0syK8 https://www.youtube.com/watch?v=MJfceF0syK8 in realtime. Performance on individual machines is way more than sufficient to handle even pretty large datasets in the TBs, but we hamstring them with layers of garbage way too often. One unfair benchmark I've heard before is "if 1 machine can do X in k time using this distributed framework, then 100 machines can do X in q<<k time using the same framework" but it's like also true that the same 1 machine could do it w time with w within spitting distance of q.

belluchan 13y ago

Big data doesn't have to be "big". I'm not going to store statistical information, information that has at most a value and a date and maybe a key or two, and is updated multiple times per second per user in an SQL database. It doesn't fit in the table row column paradigm. For me at least, big data is more about a form of non-relational data that updates very quickly, doesn't need joins, needs high availability and can be processed later to derive value out of it.

gaius 13y ago
An observation: people who know about databases of all kinds say "relational database". People who don't say "SQL database". Relational databases on commodity hardware can easily handle hundreds if not thousands of updates per second these days.
- belluchan 13y ago
  Non-SQL databases can have relational features, like rethinkdb for example. http://www.rethinkdb.com/docs/table-joins/ http://www.rethinkdb.com/docs/table-joins/
- endianswap 13y ago
  Although I don't think you were commenting directly on my post (your grandparent), I'd like to point out when I say a "SQL database" or "SQL server" I'm talking about Microsoft SQL Server.
endianswap 13y ago
We do exactly this, with billions of rows each day of statistical data right into MSSQL server instances. For data like this, the only difference is that we partition the data into daily tables and provide views into the data as necessary. What experience or expertise can you share as to why this doesn't work?
- belluchan 13y ago
  I mean if it works for you I cannot say it doesn't work. It doesn't seem like a palatable solution however. Does your SQL database also have other types of information? Like account data? Does having billions of rows of statistical data make it more difficult to manage, replicate and secure non-statistical data? Like your logs must just be useless. You can bring down trees with dynamite, but just because that "works" doesn't mean it's the optimal solution.
- jules 13y ago
  What's the reason for partitioning into daily tables? Why not add a day column and index it on that column?
  - endianswap 13y ago
    Easier to move old historical data onto larger, slower drives if it's stored independently.
  - RyanZAG 13y ago
    Index would become bigger than ram. Removing old data would be more difficult. They likely only use todays data. This is probably a trading data system of some kind.
Spearchucker 13y ago
...non-relational data that updates very quickly, doesn't need joins, needs high availability and can be processed later... What kinds of problems fit this model? Specifically, why would I want to defer processing if I can process instantaneously? NoSQL has its place for sure, but is always augmented by a relational database somewhere. The Lambda architecture that's so popular at the moment is cool and all, but I find a simpler scenario where you index data into a relational database before writing it to a document database is good enough. That way I can process immediately, and if needed get the document out of a NoSL database (usually only to display to a user). A relational database that points at corresponding locations in a document database is useful in ways that document databases on their own are not. Relational databases on their own are useful in ways that document databases are not. I cannot say whether the same is true for document databases, as I've yet to encounter a real problem that can only be solved with a document database.
- belluchan 13y ago
  The problem I am describing is a situation in which you have and want to maintain high amounts of granular data that can be queried for specific ranges with varying amounts of detail. For example look at something like Google Analytics. As simple as page views with timestamps. If you have high traffic data, you can be writing to your database quite a bit. You could bucket that data per second per unique viewer. Then in your YUI you can select to see a line graph with varying degrees of detail. Per hour, per day, per month, per year. No way would I want to store this in an SQL database. The processing that happens later are the per time period rollups. Creating buckets of minutes, days, etc per user. This can be done when asked for and cached (slow experience for the user), or scheduled and persisted (expensive - consumes more space). Again I would not want to store this in SQL. All the while I maintain my raw data for further processing later. These kinds of solutions are great for monitoring data, tracking, any kind of high volume, real time statistical data.
  - Spearchucker 13y ago
    The example you use is Google Analytics. I've no idea how big that dataset is, but I imagine bigger than anything I've encountered before. That said, I'd do everything you mention in a relational database. SELECT COUNT(fieldname) FROM {tablename} and all that. Are you ok with eventual consistency as a price for partition tolerance and availability? The data sizes I typically work with allow me to go with availability and consistency at the expense of partition tolerance. And that's ok because my data replicates, and my response time objectives and recovery point objectives are set to something appropriate. This is worth it to me if the benefit is an operational data store that I can query in real time without impacting writes. Still haven't heard of a scenario which would compel me to go a NoSQL only route (a scenario that isn't Google or Facebook or similar).
mbesto 13y ago
> big data is more about a form of non-relational data that updates very quickly, doesn't need joins, needs high availability You're talking about NoSQL actually. Interestingly, I find that big data is normally synonymous with the unstructured data that updates very frequently, doesn't need joins, etc that you speak of. Structured data in large businesses, such as general ledgers, sales orders, purchase orders, etc. is generally quite fast. The only problem in that area today is getting that data out into complex reporting. It requires a lot of processing (either routine or manual) to get it into an optimized format. Big data came about because companies like FB, Google, Twitter, pretty much any online analytics company, AirBnb, etc. are crunching large sets of behavioral events. (user clicked this button) Outside of those use cases, data just doesn't grow as fast as the marketeers of Big Data claim it is.

rossjudson 13y ago

And...you don't need to denormalize your data set if the underlying system is fast enough to deliver timely results. And because it's not denormalized, it's smaller, and you can fit more data...etc etc. Folks who think they have big data problems really need to define the size of their data set -- not the size of the resulting BigDataDB, or the number of servers they have. The two are not the same.

bane 13y ago

Something I've noticed is that there's a real trend in easing developer effort by trade-off with computational effort. In some cases it's worth it, but I think the pendulum has swung too far in most. What I mean are these efforts to build huge distributed compute clusters that do X by distributing the problem easily, but at huge overhead cost for each individual actually useful computation. The distribution, tracking of the compute tasks, the coordination of it all and the frameworks to make it easy insert such a tremendous overhead that I suspect most of the problems that are approached this way would be better tackled with a single machine and some smart old-fashioned algorithm development and developer elbow grease. Of course simple economics might decide if 20 machines doing something are cheaper than 1 developer working an extra 3 or 4 months so 1 machine can do the same thing resolves it. But I suspect that level of analysis just isn't done and the result is to just throw hardware at the problem. I've had to remind my developers in the past that we're working with machines that are literally capable of doing billions of operations per second and can do something like this https://www.youtube.com/watch?v=MJfceF0syK8 https://www.youtube.com/watch?v=MJfceF0syK8 in realtime. Performance on individual machines is way more than sufficient to handle even pretty large datasets in the TBs, but we hamstring them with layers of garbage way too often. One unfair benchmark I've heard before is "if 1 machine can do X in k time using this distributed framework, then 100 machines can do X in q<<k time using the same framework" but it's like also true that the same 1 machine could do it w time with w within spitting distance of q.