3 ms·

So the author is suggesting. I think that the two are different, though. The difference: scale. Data warehousing solutions simply can't scale to "big data". Ma

So the author is suggesting. I think that the two are different, though. The difference: scale. Data warehousing solutions simply can't scale to "big data". Many companies have a "data warehouse" that is usually a single (albeit very beefy) server running SQL Server. But "big data" is the kind of data that can't be supported by a single server or by a traditional database. It's the difference between warehousing activity in a network of a 10k employees vs. warehousing activity on a website with a million daily uniques. That's my imrpession, anwyay -- "data warehousing" is dominated by the RDBMS, but "big data" is dominated by highly scalable, distributed databases.

aaronblohowiak 13y ago
Most people who use the term "big data" are wannabe big data people, but will never have data too large to fit/process on a single server. In this use, it is really just OLTP vs OLAP all over again, which leads us naturally to ETL our data into a DW.
ironchef 13y ago
I'm going to somewhat disagree. Traditional data warehouses (teradata for example) can handle well into the petabytes. I think Ebay's main TD cluster was around 15 or 20 PB last I knew. I think you're more along the proper direction in the single server vs N server. Every data warehouse I've worked on in the past decade or so has been multi server.
- delluminatus 13y ago
  Well, I think Teradata probably falls under the big data umbrella, given that its basically a distributed database, right? I was thinking more about just plain Oracle or SQL Server or whathaveyou, which can scale pretty big, big enough for 99% of companies, but not "big", and are notoriously difficult to cluster/distribute.
  - ironchef 13y ago
    Fair enough. I think the reason people struggle with classifying this stuff is because there are a lot of gray areas and it's difficult to verbalize all the details meaningfully. One could say "Oh...RDBMSes!"...but teradata is still relational...it's just distributed. "Oh! OLAP vs OLTP!"...but it's not really that either. One _could_ use teradata for a couple gig data mart just fine (and argue that it's not big data). That's why i tend to fall back to the 3 / 4 Vs that have tended to typify the 4 aspects of "big data" (volume, velocity, variety, veracity).

aaronblohowiak 13y ago

Most people who use the term "big data" are wannabe big data people, but will never have data too large to fit/process on a single server. In this use, it is really just OLTP vs OLAP all over again, which leads us naturally to ETL our data into a DW.

ironchef 13y ago

I'm going to somewhat disagree. Traditional data warehouses (teradata for example) can handle well into the petabytes. I think Ebay's main TD cluster was around 15 or 20 PB last I knew. I think you're more along the proper direction in the single server vs N server. Every data warehouse I've worked on in the past decade or so has been multi server.

delluminatus 13y ago
Well, I think Teradata probably falls under the big data umbrella, given that its basically a distributed database, right? I was thinking more about just plain Oracle or SQL Server or whathaveyou, which can scale pretty big, big enough for 99% of companies, but not "big", and are notoriously difficult to cluster/distribute.
- ironchef 13y ago
  Fair enough. I think the reason people struggle with classifying this stuff is because there are a lot of gray areas and it's difficult to verbalize all the details meaningfully. One could say "Oh...RDBMSes!"...but teradata is still relational...it's just distributed. "Oh! OLAP vs OLTP!"...but it's not really that either. One _could_ use teradata for a couple gig data mart just fine (and argue that it's not big data). That's why i tend to fall back to the 3 / 4 Vs that have tended to typify the 4 aspects of "big data" (volume, velocity, variety, veracity).