4 ms·
Have you considered having one box with 4TB of RAM to do the computation, then sending it around to all the other nodes?
by dsr_ 1y ago
Have you considered having one box with 4TB of RAM to do the computation, then sending it around to all the other nodes?
- jitl 1y agoEach node handles an independent ~4TB shard of data in horizontal scale-out fashion. Perhaps we could try some complex shenanigans where we rent 4TB RAM nodes, compute, send to 512GB RAM nodes then terminate the 4TB nodes but that’s a bunch of extra complexity for not much of a win.