4 ms·

” Our implementation, shown in the figure above, runs on 12 nodes in the Atlas Cloud, each equipped with 8 H100 GPUs.” Maybe the cost of renting?

34679 1y ago
I'm confused because I wouldn't consider a cloud implementation to be local.
- deleted 1y ago
  [deleted]
- randomjoe2 1y ago
  Local doesn't refer to "on metal" anymore to many people
  - mwcz 1y ago
    "On metal" is muddied too. I've heard people refer to web apps running in an OCI container as being "bare metal" deployment, as opposed to AWS or whatever hosting platform. That's silly, but the idea that "local" is not the opposite of remote is even sillier.
    - ffsm8 1y ago
      You can run an OCI container on bare metal though. It doesn't stop being run on bare metal just because you're running in kernel namespaces, aka docker container Lots of people were advocating for running their k8s on bare metal servers to maximize the performance of their containers Now wherever that's applied to your conversation... I've no clue, too little context ( ｡ ŏ ﹏ ŏ )
      - okasaki 1y ago
        In my opinion, if you're running k8s on bare metal, that's "k8s on bare metal" but still "<your app> on kubernetes", not "<your app> on bare metal".
        ffsm8 1y ago
        Sorry, but then your opinion is just plain wrong Bare metal in the context of running software is a technical term with a clear meaning that hasn't become contested like "AI" or "Crypto" - and that meaning is that the software is running directly on the hardware. As k8s isn't virtualization, processes spawned by its orchestrator are still running on bare metal. It's the whole reason why containers are more efficient compared to virtual machines
        bee_rider 1y ago
        Bare metal as in, no operating system? Does Linux really get in the way of these LLM inference engines?
        ffsm8 1y ago
        No, as I said in my previous comment: bare metal as in not a virtual machine https://en.m.wikipedia.org/wiki/Bare-metal_server https://en.m.wikipedia.org/wiki/Bare-metal_server
        pessimizer 1y ago
        Note that this is a term whose meaning has been expanded to refer to non-VPS servers very recently. Bare-metal has traditionally meant "without an operating system." It did not mean "a server that is an actual server," because that was the default. It also does not always "clearly" have this new meaning. Somebody who is used to running programs directly (with no intermediate OS) on hardware might not understand what you're saying, or might ask you to clarify, and you probably shouldn't feel put upon by a totally understandable misinterpretation. edit: Especially when you keep repeating "directly on hardware" when you mean "not on a VM." VMs also run on hardware. You're saying that you're only running on one OS instead an OS in your OS.
    - dtech 1y ago
      If you do bare metal as not being under a VM it fits. OCI on linux is cgroup so that counts as not a VM I'd say. Or at least it's a layer closer to the metal than a typical VM running OCI images. I a Java app running on Linux bare metal?
  - monsieurbanana 1y ago
    I missed that train
    - vFunct 1y ago
      My basement server really confused by all this...
      - demodulation 1y ago
        The one down in your Gaza tunnels?
  - bee_rider 1y ago
    Local doesn’t need to be “on metal,” but I’m still confused as to what they are saying. Are they running some local cloud system?
- DSingularity 1y ago
  I guess local for him is independent/private.
ollybee 1y ago
H100's can be $2 and hour, so $192 an hour for the full cluster. They report 22k tokens per second, so ~ 80 million an hour, thats $16 an hour at $0.2 per million. Maybe a bit more for input tokens, but it seems a long way off.
- zipy124 1y ago
  I think you mis-read. Thats 22k tokens per second per node, so per 8 h100's. With 12 nodes they get 264k tokens per second, or 950 million an hour. This get's you to roughly $0.2021 per million at $2 an hour.

34679 1y ago

I'm confused because I wouldn't consider a cloud implementation to be local.

deleted 1y ago
[deleted]
randomjoe2 1y ago
Local doesn't refer to "on metal" anymore to many people
- mwcz 1y ago
  "On metal" is muddied too. I've heard people refer to web apps running in an OCI container as being "bare metal" deployment, as opposed to AWS or whatever hosting platform. That's silly, but the idea that "local" is not the opposite of remote is even sillier.
  - ffsm8 1y ago
    You can run an OCI container on bare metal though. It doesn't stop being run on bare metal just because you're running in kernel namespaces, aka docker container Lots of people were advocating for running their k8s on bare metal servers to maximize the performance of their containers Now wherever that's applied to your conversation... I've no clue, too little context ( ｡ ŏ ﹏ ŏ )
    - okasaki 1y ago
      In my opinion, if you're running k8s on bare metal, that's "k8s on bare metal" but still "<your app> on kubernetes", not "<your app> on bare metal".
      - ffsm8 1y ago
        Sorry, but then your opinion is just plain wrong Bare metal in the context of running software is a technical term with a clear meaning that hasn't become contested like "AI" or "Crypto" - and that meaning is that the software is running directly on the hardware. As k8s isn't virtualization, processes spawned by its orchestrator are still running on bare metal. It's the whole reason why containers are more efficient compared to virtual machines
        bee_rider 1y ago
        Bare metal as in, no operating system? Does Linux really get in the way of these LLM inference engines?
        ffsm8 1y ago
        No, as I said in my previous comment: bare metal as in not a virtual machine https://en.m.wikipedia.org/wiki/Bare-metal_server https://en.m.wikipedia.org/wiki/Bare-metal_server
        pessimizer 1y ago
        Note that this is a term whose meaning has been expanded to refer to non-VPS servers very recently. Bare-metal has traditionally meant "without an operating system." It did not mean "a server that is an actual server," because that was the default. It also does not always "clearly" have this new meaning. Somebody who is used to running programs directly (with no intermediate OS) on hardware might not understand what you're saying, or might ask you to clarify, and you probably shouldn't feel put upon by a totally understandable misinterpretation. edit: Especially when you keep repeating "directly on hardware" when you mean "not on a VM." VMs also run on hardware. You're saying that you're only running on one OS instead an OS in your OS.
  - dtech 1y ago
    If you do bare metal as not being under a VM it fits. OCI on linux is cgroup so that counts as not a VM I'd say. Or at least it's a layer closer to the metal than a typical VM running OCI images. I a Java app running on Linux bare metal?
- monsieurbanana 1y ago
  I missed that train
  - vFunct 1y ago
    My basement server really confused by all this...
    - demodulation 1y ago
      The one down in your Gaza tunnels?
- bee_rider 1y ago
  Local doesn’t need to be “on metal,” but I’m still confused as to what they are saying. Are they running some local cloud system?
DSingularity 1y ago
I guess local for him is independent/private.

ollybee 1y ago

H100's can be $2 and hour, so $192 an hour for the full cluster. They report 22k tokens per second, so ~ 80 million an hour, thats $16 an hour at $0.2 per million. Maybe a bit more for input tokens, but it seems a long way off.

zipy124 1y ago
I think you mis-read. Thats 22k tokens per second per node, so per 8 h100's. With 12 nodes they get 264k tokens per second, or 950 million an hour. This get's you to roughly $0.2021 per million at $2 an hour.