5 ms·
> All the machines are highly utilized, and they have fast Infiniband/OmniPath networks that you simply cannot get in the cloud. It's weird that these networki
by zozbot234 7y ago
> All the machines are highly utilized, and they have fast Infiniband/OmniPath networks that you simply cannot get in the cloud.
It's weird that these networking technologies are not used more in "plain" datacentre settings, since networking latency and throughput has to be a significant challenge to scaling up non-trivial workloads and achieving true datacentre-scale computing. We hear a lot about how to "scale out", but that's only really feasible for relatively simple workloads where you just seek to do away with the whole issue of keeping different nodes in sync on a real-time basis, and accept the resulting compromises. In many cases, that's just not going to be enough.
- alexpotato 7y agoThere are a lot of people from the National Lab super computer world who end up in High Frequency Trading for just the reason you describe. Specifically, how do you optimize a large cluster of computers to operate at the lowest possible latency. For the National Labs, those computers could be in the lab or with other labs around the world. For the HFT folks, the machines could be in an exchange or spread across multiple exchanges around the world. Source: I used to be head of Global Latency Monitoring for a HFT.
- funfunfunction 7y agoHey I’d love to pick your brain about what it takes to do this kind of work coming from a web/systems engineering background. Shoot me an email at samheutmaker at gmail if you’re open to it.
- j88439h84 7y agoI'm curious why you moved to LLNL from HFT?
- eyegor 7y agoMoney is a safe guess. Research pay scales aren't even close to private sector, especially not finance.
- dwohnitmok 7y agoIt sounds like the opposite direction happened here.
- jeremyjh 7y agoMost business workloads simply aren't CPU bound, nor amenable to parallel computation. The public clouds would offer it if they had demand for it.
- zozbot234 7y agoSurely network latency (and perhaps throughput) is just as relevant to IO-bound and transactional workloads though? At least if they need to be served by a multiple-node cluster.
- eyegor 7y agoUmm, most workloads aren't so compute intense that cpu interconnect speed would have a huge impact. The vast majority of rented computer tasks will be for things that need to communicate with external services (payments, region synchronizing, general api state comms, etc). In these cases, I doubt that your code is tuned tightly enough to reveal interconnect latency (as opposed to cache misses and swaps). Besides, hooking up interconnect runs gets ludicrously complicated and expensive fast. Like, $50k in cables and a week of experimenting with numa zones and kernel settings for a small cluster. I can only imagine the headaches of orchestration on national lab clusters. Source: I have set up, and currently manage a small cluster which is always pegged. I also collaborate with folks at a couple of the national labs for scientific research.
- hexaga 7y agoInfiniband and friends aren't cpu interconnects, they're alternatives to ethernet. Network latency is important and it's not that hard to write code whose performance is bounded by io latency/throughput.
- zozbot234 7y agoWell they can be both, so I can see how parent might have gotten confused.
- hexaga 7y agoTrue enough.
- twic 7y ago> it's not that hard to write code whose performance is bounded by io latency/throughput I suspect that most people don't, though. How many applications are built in Ruby, Python, or PHP, using some very productive but inefficient framework? Those are all very fine tools in their own way, and are the right tool for the job in many contexts. But they're unlikely to saturate IO.
- deleted 7y ago
- shaklee3 7y agoThose technologies, or mostly IB, were more important when Ethernet speeds lagged behind. IB was first to 100G. Now that Ethernet has caught up, IB has almost no advantage besides a tiny latency decrease. Even mellanox makes more Ethernet than IB products now.