4 ms·
I agree with the sentiment that many solutions are over-engineered, but when you need to process billions of records a day, you do need more complex systems. B
by angarg12 8y ago
I agree with the sentiment that many solutions are over-engineered, but when you need to process billions of records a day, you do need more complex systems.
Bottom line: when facing an engineering problem, start with the simplest, fastest to implement solution, and build complexity as necessary. The simple solution suffices most of the time.
- lstodd 8y agoYou know, "billions a day" is only on the order of 10K per second. A single machine can handle that.
- monsieurbanana 8y agoThat would be amazing, wouldn't it? It's not true though, the problem with dealing with billion of operations a day is the spikes, most of the time you don't get a nice homogene rate for 24 hours straight.
- fwip 8y agoDepends on how much latency matters. A lot of big data is batch processing, for which if the data is 3 hours old that's more than good enough.
- deleted 8y ago[deleted]
- pjc50 8y ago> billions of records a day A good GPU can do >1bn calculations per frame.
- dahart 8y agoA good GPU today can do > 1 Tn calculations per frame.