3 ms·

"The paper concludes by advising analysts to not go through the Hadoop hoops until your data size passes standard hard drive limits (currently around 1 Terabyte

• thingsilearned 14y ago
  Its hard to put a handle on what is 'Big' and what isn't. I think a decent place to draw the line is on data that can't fit on one disk, and 1 TB seems to be a pretty standard disk size right now.
• kodablah 14y ago
  I would say big enough to start thinking about alternate solutions than a monolithic RDBMS. Of course the solutions may include sharding and not necessarily a non-relational store.
• juiceandjuice 14y ago
  Unless you've got a ridiculous RAID array, there's plenty of times where processing anything over 100GB without a cluster is going to suck. Sequentially reading 1TB of data is going to take a few hours even with a RAID array, or be very expensive with SSDs. I maintain a database that's over 1TB and Oracle handles it very well. The trick there is understanding that, under absolutely no circumstance should you ever need to do a full table scan, because the table isn't designed for that. So, I'd argue that a monolithic database is okay even up to 10TB even with slower disks as long as you never need to touch more than 10% of it. If you need to touch 100% of the data 100% of the time, I'd say anything over 100GB is too big for one machine. The reality is that it just depends. There's times where you're going to want a hadoop cluster even for only 16GB of data, and there's going to be times where a database is going to be fine with 10TB of data.