5 ms·

Real scientific computing needs high-bandwidth low-latency networks.

by jff 12y ago

kolencherry 12y ago
Microsoft Azure actually offers InfiniBand for connectivity, which is interesting. I'm not sure how many people are using their HPC-oriented offering.
collyw 12y ago
Go on then, let me know what "real scientific computing" is. I work with bioinformaticians, which is an area with lots of research going on. I don't see much that is complex there. Split up your jobs and send them to a cluster. I take it that's not "real scientific computing" though?
- sp332 12y ago
  From the article: planet-wide weather forecasting, seismic measuring for oil exploration, financial analysis or crash simulations. You can't split up a physical simulation.
  - jff 12y ago
    You actually can split up a physical simulation. Imagine you're trying to model heat diffusion through a piece of metal. You split the chunk of metal up into many small pieces and model temperature in each one. Each node in your cluster is simulating the behavior of some portion of the model; at each timestep, nodes simulating chunks that border each other must swap the state of their "edges", sharing the outer edges of their own state with their neighbors. Edit: unless you meant that you can't split up a physical simulation on loosely-coupled nodes and still get good performance. Then, well, yeah.
    - sp332 12y ago
      Well I guess the simulation would have to be broken up to run on multiple GPUs at once anyway. But even if you had a good interconnect, the GPUs in this box (K40) have almost 2x as many processors and 3x as much RAM. (Edit: compared to AWS GPU instances.) If you're doing a 3D sim, each node needs to communicate with 26 others, so just having 1/3 as many nodes is going to help a lot.
- jff 12y ago
  Split up your jobs how? Is the job embarrassingly parallel? Because if it isn't, and most things aren't, it's going to require significant programmer overhead to simply "split up your jobs". And then once you've split up the job, the different portions are going to need to exchange their in-progress results, because most real scientific computing programs involve iterating until either a certain number of timesteps have passed or a certain residual is reached. I mean, I only write HPC programming frameworks for scientific computing as my job, what do I know. But if you're doing anything more complex than map-reduce, a high-speed interconnect is essential, and even in mapreduce it's pretty handy.

Real scientific computing needs high-bandwidth low-latency networks.

by jff 12y ago

Real scientific computing needs high-bandwidth low-latency networks.

Microsoft Azure actually offers InfiniBand for connectivity, which is interesting. I'm not sure how many people are using their HPC-oriented offering.

collyw 12y ago

Go on then, let me know what "real scientific computing" is. I work with bioinformaticians, which is an area with lots of research going on. I don't see much that is complex there. Split up your jobs and send them to a cluster. I take it that's not "real scientific computing" though?

sp332 12y ago
From the article: planet-wide weather forecasting, seismic measuring for oil exploration, financial analysis or crash simulations. You can't split up a physical simulation.
- jff 12y ago
  You actually can split up a physical simulation. Imagine you're trying to model heat diffusion through a piece of metal. You split the chunk of metal up into many small pieces and model temperature in each one. Each node in your cluster is simulating the behavior of some portion of the model; at each timestep, nodes simulating chunks that border each other must swap the state of their "edges", sharing the outer edges of their own state with their neighbors. Edit: unless you meant that you can't split up a physical simulation on loosely-coupled nodes and still get good performance. Then, well, yeah.
  - sp332 12y ago
    Well I guess the simulation would have to be broken up to run on multiple GPUs at once anyway. But even if you had a good interconnect, the GPUs in this box (K40) have almost 2x as many processors and 3x as much RAM. (Edit: compared to AWS GPU instances.) If you're doing a 3D sim, each node needs to communicate with 26 others, so just having 1/3 as many nodes is going to help a lot.
jff 12y ago
Split up your jobs how? Is the job embarrassingly parallel? Because if it isn't, and most things aren't, it's going to require significant programmer overhead to simply "split up your jobs". And then once you've split up the job, the different portions are going to need to exchange their in-progress results, because most real scientific computing programs involve iterating until either a certain number of timesteps have passed or a certain residual is reached. I mean, I only write HPC programming frameworks for scientific computing as my job, what do I know. But if you're doing anything more complex than map-reduce, a high-speed interconnect is essential, and even in mapreduce it's pretty handy.