3 ms·
Actually you can normally get away without resorting to "big data" techniques at 1PB scale, it's probably around 10PB that you're really forced to start thinkin
by jleahy 12y ago
Actually you can normally get away without resorting to "big data" techniques at 1PB scale, it's probably around 10PB that you're really forced to start thinking about things like hadoop style map-reduce/rack-locality.
- Demiurge 12y agoProbably depends on what you're doing with the data?
- jleahy 12y agoRight, I'm thinking of the "embarrassingly parallel" sort of problems, where you can shard the data (for example by city, country, company or some other obvious classifier) and give one shard to one server. I'm you were trying to do anything that's O(n.log n) or O(n^2) (think graph processing) then you'll run into trouble at much smaller scales.
- deleted 12y ago[deleted]