4 ms·
> I wonder if one day we will find that sending all data to a data center for processing doesn't scale. Of course it scales. Some problems demand more scale. T
by duaneb 11y ago
> I wonder if one day we will find that sending all data to a data center for processing doesn't scale.
Of course it scales. Some problems demand more scale. Through more datacenters at it. Tweak the algorithms to distribute data more conservatively.
- fauigerzigerk 11y agoWhat could happen is that both processing capacities and sensor capacities grow faster than networking capacities for years. That could make latencies untenable for large classes of applications.
- duaneb 11y agoLatency is bounded by the speed of energy transfer (i.e. speed of light under superconducting situations). Even if bandwidth IS a big problem—as it is—latency will always be a sticking point for distributed systems. As someone else pointed, we need to start analyzing distributed algorithms with a) geographical locality and b) complexity on round trips, which have a constant cost determined by latency.
- fauigerzigerk 11y agoSpeed of light determines the lower bound of roundtrip latency, but the sticking point is the upper bound, and that depends on how congested networks are. If sensors produce more data and data centers can process more data but network capacities can't keep up, then the analysis you are suggesting will show that we need more decentralised processing, not more data centers.
- duaneb 11y agoAhh, I see. I don't think we were disagreeing; I was just pointing out that data centers scale very well for what data centers were built for. It's not that surprising that we are outgrowing them.