4 ms·
Absolutely true, and the scale keeps getting larger > 12 billion rows, and comes up with even harder challenges. Worked on a 1.3 trillion row (for 1 table, oth
by AndyNemmity 10y ago
Absolutely true, and the scale keeps getting larger > 12 billion rows, and comes up with even harder challenges.
Worked on a 1.3 trillion row (for 1 table, others varying below this) database this year for predictive analytics, and it was mind boggling the hoops to jump through to try to get that thing to run anything in a manageable time frame.
Any POC level show of a database would be about meaningless at that scale. You have to do a real test to view the challenges around it.
- pmontra 10y agoWhich database did you use and what would you recommend?
- AndyNemmity 10y agoWe used SAP's HANA for a portion of the data, and then used an option on it called Dynamic Tiering which is essentially connecting Sybase IQ to HANA for a disk data store as well. Keeping 1.3 trillion in memory is pretty expensive, so we were trying to cut some costs by using that to funnel data in and out. Not a bad solution overall.
- askingqs 10y agobut do you need to access all 1.3 trillion rows to do predictive analytics? why not just use a very small sample of the data?
- AndyNemmity 10y agoYes, you need access to all 1.3 trillion rows (plus more) for the use case I worked on. Ideally, you'd actually use more than that, however diminishing returns leaves you with that as what we were going with.