5 ms·

Yep. I'm part of a significant project which is being shoehorned into a Cray because the government has already spent the money for it. If it's not an MPI job,

• gbrown_ 10y ago
  > If it's not an MPI job, it's going to be painful. What are you using for communication between nodes? To make the tone clear I'm genuinely interested as I'm not sure what running on a Cray prevents you from doing?
  • batbomb 10y ago
    That's the problem: We don't need to communicate between the nodes; We don't need MPI. We need distributed databases and we might need Spark. We need machine learning. We need servers. We would like availability and reliability. We get full batch nodes that are relatively anemic in memory for our workload (4GB/core) for a maximum of 24 hours at a time.
    • deleted 10y ago
      [deleted]
    • gbrown_ 10y ago
      Aha thanks! Yes I agree with you points here, not an ideal workload for a Cray system.
    • ams6110 10y ago
      Basically it's hard to use a Cray node as a stand-alone server. Yes it runs linux, but it has no storage, no external network. There is the so-called cluster compatibility mode but bottom line everything is a scheduled batch job in a very walled garden.
      • crznp 10y ago
        Cray invests a lot in their interconnect; if you aren't using it, there is little reason to be on a Cray. Which just emphasizes the point that most applications would be better served by cheaper, more resilient hardware.
        • deleted 10y ago
          [deleted]
        • ams6110 10y ago
          All the stranger that they sold their interconnect technology, arguably their "secret sauce" to Intel. Without the interconnect, they are just a bunch of (very high quality) diskless x86 systems. http://www.eetimes.com/document.asp?doc_id=1261622 http://www.eetimes.com/document.asp?doc_id=1261622
    • nickbauman 10y ago
      > We don't need to communicate between the nodes Huh. At the core of almost every problem I've ever encountered in my 20+ year career programming that I would consider "interesting" is the need to allocate work across many compute nodes that have some amount of shared state. You could just as well take A* as a baseline for these types of problems. Your typical cloud computing infra/design is not a good match for these.
      • batbomb 10y ago
        Well I guess computing in astrophysics and HEP isn't that interesting then :) It's true that we need shared data, but we don't need shared state (memory) for any of our workloads, and shared data (e.g. disk/db) is a much easier problem to solve than shared state. There is still coordination and orchestration, but that's an extremely coarse amount of communication compared to the cost of Cray interconnects. That being said, there are cases where we might benefit from the Cray networking, but that comes at the cost of other tradeoffs (no local disk, low memory per core). So what do we do? Well, we use a handful of Ferraris to get the job done because they are available when cheap bus would have suited us fine. The double whammy is that the Ferraris end up in the shop all the time and occasionally somebody else gets exclusivity to them when they want to get in a Bell prize submission.
        • cinquemb 10y ago
          Well this is where you talk to the network admins at your university and get them in on a top secret program, code name: "It Was China", to hack into all the detectable ps4's on campus, call all those students to a hall for a random lottery social for a couple of hours while you commandeer there equipment, for science of course. Or you can send get the network admins to allow you to send them a email, but the first suggestion sounds more fun :P
          • nickbauman 10y ago
            That, sir, is hilariously awesome.
        • 10y ago
    • siculars 10y ago
      Talk to me about Riak. /I work for Basho/