4 ms·
Thanks for the reply. This is mostly a thought experiment in can you 'grep the web' (without indexing etc). If you split 24TB across ~128 servers ( so filling
by freediver 7y ago
Thanks for the reply. This is mostly a thought experiment in can you 'grep the web' (without indexing etc).
If you split 24TB across ~128 servers ( so filling up 256 GB RAM per server) you would have to search 256 GB of RAM in 100ms, or ~2.5TB/sec. Unfrotunately most RAM will give you only 20GB/s (DDR4 3200:25.6 GB/s) . Multi-core setups do not help much as they share memory channels (for example AMD's latest 64 core 'monster' has only 4 shared memmory channels). So it seems you would need insane number of servers (15,000 with ~2GB RAM) to do a 'grep of the web' in 100ms. Missing something?
- burntsushi 7y agoI don't think you're missing anything. As I said, it's a middle ground. :-)
- freediver 7y agoThanks for jumping in! What do you use to alert you?
- burntsushi 7y agoNothing. I just search HN from time to time and check the threads I'm participating in. :-)