5 ms·

What exactly does the Xeon do in this situation, is there a reason you couldn't use any other x86 processor?

VladVladikoff 2y ago
I think it’s that most non Xeon motherboards don’t have the memory channels to have this much memory with any sort of commercially viable dimms.
- genewitch 2y ago
  Pcie lanes
  - hedora 2y ago
    I was about to correct you because this doesn't use PCIe for anything, and then I realized Arc was a GPU (and they support up to 8 per machine). Any idea how many Arc's it takes to match an H100?
    - npodbielski 2y ago
      I am reading from time to time about multi GPU solution and last time I found some real life information about this (it was two 7900 xtx) the result was that performance is the same at best often it is slower. So even if you manage to slap like 8 cheap cards onto motherboard, even if you would somehow make it work (people have problems with such setups), even if this would work continuously without much problems (crashes, power consumption) performance would be just OK. I am not sure if spending 10k on such setup would be better than buying 10k card with 40gbs of RAM.
      - pshirshov 2y ago
        Ollama works fine with multi-gpu setups. Since rocm 6.3 everything is stable and you can mix different GPU generations. The performance is good enough for the models to be useful. The only thing which doesn't work well is running on iGPUs. It might work but it's very unstable.
        npodbielski 2y ago
        Good to know. Still, is it viable option? Buying i.e. AMD Threadripper for 2.5k$, motherboard and ram for 2k$ and i.e. 4 GPUs for 4k$ to have total of 96GB of VRAM? Total should be around 10k$ which is roughly price of Intel GPU specifically for AI if I am not mistaken? Which option would be better performance wise. I did never saw a comparison anywhere and this is too much money to make fun experiment over the weekend.
        zargon 2y ago
        > 10k$ which is roughly price of Intel GPU specifically for AI Huh? The largest vram card that Intel has is the A770 which is around $350. What exactly are you trying to compare against? Are you doing inference only or training?
        genewitch 2y ago
        It can be read as a typo for "Nvidia" as their cards are about that price (or more, i haven't looked)
walrus01 2y ago
There's not much else (other than Epyc) in the way of affordably priced motherboards that have enough cumulative RAM. You can buy a used Dell dual socket older xeon CPU server with 512GB of RAM for test/development purposes for not very much money. Under $1500 (before adding video cards or your own SSD), easily, with what I just found in a few minutes of searching. I'm also seeing things with 1024GB of RAM for under $2000. You also want to have the capability for more than one full speed at minimum PCI-Express x16 3.0 card, which means you need enough PCI-E lanes, which you aren't going to find on a single socket Intel workstation motherboard. Here's a couple of somewhat randomly chosen examples with 512GB of RAM and affordably priced. they'll be power hungry, and noisy. Same general idea from other x86-64 hardware such as from hp, supermicro, etc. These are fairly common in quantity so I'm using them as a baseline for specification vs price. Configurations will be something with 16 x 32GB DDR4 DIMMs. https://www.ebay.com/itm/186991103256?_skw=dell+poweredge+t630+512GB&itmmeta=01JNMMXN9KZYW18BXTS3FJ4P78&hash=item2b8989cd18:g:DUkAAOSwTd5nvh7A&itmprp=enc%3AAQAKAAABAFkggFvd1GGDu0w3yXCmi1cDq8JYEdwMA0hCESi8LOpm%2BTjrKqMmORBN7JFMXm73nxNogCCkoNK7mIilY5m5nhExssyrqPElZCR%2BV0fcwgekHTmp5mbcdSFW4GWbgTegh0qzcXlKe%2BOHQK%2FlVT1IY7iQHn35%2Fdrr3%2Fjj3xcAT42QQloifXpmA%2BUXkCdEAkeG6dxQ2Yjoba7RX0PVXnNDeB32GdVdvfCccltPkxGYgiX5pIf6GBEwAac6e5uxLYSoEeu6xxahh%2BP9x2PGmslRMvA--mWocWZH4yZE53GlEa9zoR7I4MmTYzq79RH9T%2FkWY1y5a1FLb11CjKBr5Ak5NV4%3D%7Ctkp%3ABk9SR_rU9pStZQ https://www.ebay.com/itm/186991103256?_skw=dell+poweredge+t6... https://www.ebay.com/itm/235978320621?_skw=dell+poweredge+r730+512GB&itmmeta=01JNMN0TSK98S5CR6MNXNK0V1B&hash=item36f1676aed:g:Oo4AAeSwNLVnvx4p&itmprp=enc%3AAQAKAAAA0FkggFvd1GGDu0w3yXCmi1f1RaGZaXc0qaT27Jp2wWgP%2BeLlJxoqY7RKJvyzvXiKcnHBcbdm8vUEZGWFkZ6EcuM6gL%2BWcOxltSOfauG1rU8jdfEuEFLQ%2FK6xzFo2khmE90CQ%2Fvt0cLdCGBZq%2BDlzBi3bkEBOGP08Xub8gvckcIQDgx8dqMFkliHm3B%2FcUhPT2qC8%2BKapKrpaSJ4swbpG14PZBpqWN4bOWhi%2BT%2BvWrZ5ZEE9tNKi9aZ%2BAAkuFj4mGudg2crAeu1HVVPRxVGoA%2BJ4%3D%7Ctkp%3ABk9SR56tg5WtZQ https://www.ebay.com/itm/235978320621?_skw=dell+poweredge+r7... https://www.ebay.com/itm/115819389940?_skw=dell+poweredge+r730+512GB&itmmeta=01JNMN0TSK5PH9QM5C04KXP61W&hash=item1af75fa3f4:g:O2MAAOSwRtNkh97Y&itmprp=enc%3AAQAKAAAA0FkggFvd1GGDu0w3yXCmi1fVZjjmHsBiQ8qEYVz5I4ZnPDcIqGMowe5PZuSmUhbqWY8zUYOBURPCEKnFDBuZ8KVE5sw3DsMEdhJOHeD94MrLAzUin0oreQdhkAmOhxvitKBKA6ykXKlQQhdVPb4TRfqj%2Br2uYF%2BObtCgbVcEJ8LASVSG4PSzeochGZNA3eWVpGd0TuLt%2BZd789g4i3zJuXG9e%2Bc0vR%2BTIwGNPzRs8l9biTG0YHwdXvme9Jkl4BKkGaTTMbHGOKFFOrRizjBMOW0%3D%7Ctkp%3ABk9SR56tg5WtZQ https://www.ebay.com/itm/115819389940?_skw=dell+poweredge+r7...
- numpad0 2y ago
numpad0 2y ago
DDR4 UDIMM is up to 32GB/module DDR5 UDIMM is up to 64GB/module[0] non-Xeon M/B has up to 4 UDIMM slots -> non-Xeon is up to 128GB/256GB per node Server motherboards have as many as 16 DIMM slots per socket with RDIMM/LRDIMM support, which allows more modules as well as higher capacity modules to be installed. 0: there has been a 128GB UDIMM launch at peak COVID

I think it’s that most non Xeon motherboards don’t have the memory channels to have this much memory with any sort of commercially viable dimms.

genewitch 2y ago
Pcie lanes
- hedora 2y ago
  I was about to correct you because this doesn't use PCIe for anything, and then I realized Arc was a GPU (and they support up to 8 per machine). Any idea how many Arc's it takes to match an H100?
  - npodbielski 2y ago
    I am reading from time to time about multi GPU solution and last time I found some real life information about this (it was two 7900 xtx) the result was that performance is the same at best often it is slower. So even if you manage to slap like 8 cheap cards onto motherboard, even if you would somehow make it work (people have problems with such setups), even if this would work continuously without much problems (crashes, power consumption) performance would be just OK. I am not sure if spending 10k on such setup would be better than buying 10k card with 40gbs of RAM.
    - pshirshov 2y ago
      Ollama works fine with multi-gpu setups. Since rocm 6.3 everything is stable and you can mix different GPU generations. The performance is good enough for the models to be useful. The only thing which doesn't work well is running on iGPUs. It might work but it's very unstable.
      - npodbielski 2y ago
        Good to know. Still, is it viable option? Buying i.e. AMD Threadripper for 2.5k$, motherboard and ram for 2k$ and i.e. 4 GPUs for 4k$ to have total of 96GB of VRAM? Total should be around 10k$ which is roughly price of Intel GPU specifically for AI if I am not mistaken? Which option would be better performance wise. I did never saw a comparison anywhere and this is too much money to make fun experiment over the weekend.
        zargon 2y ago
        > 10k$ which is roughly price of Intel GPU specifically for AI Huh? The largest vram card that Intel has is the A770 which is around $350. What exactly are you trying to compare against? Are you doing inference only or training?
        genewitch 2y ago
        It can be read as a typo for "Nvidia" as their cards are about that price (or more, i haven't looked)

walrus01 2y ago

There's not much else (other than Epyc) in the way of affordably priced motherboards that have enough cumulative RAM. You can buy a used Dell dual socket older xeon CPU server with 512GB of RAM for test/development purposes for not very much money. Under $1500 (before adding video cards or your own SSD), easily, with what I just found in a few minutes of searching. I'm also seeing things with 1024GB of RAM for under $2000. You also want to have the capability for more than one full speed at minimum PCI-Express x16 3.0 card, which means you need enough PCI-E lanes, which you aren't going to find on a single socket Intel workstation motherboard. Here's a couple of somewhat randomly chosen examples with 512GB of RAM and affordably priced. they'll be power hungry, and noisy. Same general idea from other x86-64 hardware such as from hp, supermicro, etc. These are fairly common in quantity so I'm using them as a baseline for specification vs price. Configurations will be something with 16 x 32GB DDR4 DIMMs. https://www.ebay.com/itm/186991103256?_skw=dell+poweredge+t630+512GB&itmmeta=01JNMMXN9KZYW18BXTS3FJ4P78&hash=item2b8989cd18:g:DUkAAOSwTd5nvh7A&itmprp=enc%3AAQAKAAABAFkggFvd1GGDu0w3yXCmi1cDq8JYEdwMA0hCESi8LOpm%2BTjrKqMmORBN7JFMXm73nxNogCCkoNK7mIilY5m5nhExssyrqPElZCR%2BV0fcwgekHTmp5mbcdSFW4GWbgTegh0qzcXlKe%2BOHQK%2FlVT1IY7iQHn35%2Fdrr3%2Fjj3xcAT42QQloifXpmA%2BUXkCdEAkeG6dxQ2Yjoba7RX0PVXnNDeB32GdVdvfCccltPkxGYgiX5pIf6GBEwAac6e5uxLYSoEeu6xxahh%2BP9x2PGmslRMvA--mWocWZH4yZE53GlEa9zoR7I4MmTYzq79RH9T%2FkWY1y5a1FLb11CjKBr5Ak5NV4%3D%7Ctkp%3ABk9SR_rU9pStZQ https://www.ebay.com/itm/186991103256?_skw=dell+poweredge+t6... https://www.ebay.com/itm/235978320621?_skw=dell+poweredge+r730+512GB&itmmeta=01JNMN0TSK98S5CR6MNXNK0V1B&hash=item36f1676aed:g:Oo4AAeSwNLVnvx4p&itmprp=enc%3AAQAKAAAA0FkggFvd1GGDu0w3yXCmi1f1RaGZaXc0qaT27Jp2wWgP%2BeLlJxoqY7RKJvyzvXiKcnHBcbdm8vUEZGWFkZ6EcuM6gL%2BWcOxltSOfauG1rU8jdfEuEFLQ%2FK6xzFo2khmE90CQ%2Fvt0cLdCGBZq%2BDlzBi3bkEBOGP08Xub8gvckcIQDgx8dqMFkliHm3B%2FcUhPT2qC8%2BKapKrpaSJ4swbpG14PZBpqWN4bOWhi%2BT%2BvWrZ5ZEE9tNKi9aZ%2BAAkuFj4mGudg2crAeu1HVVPRxVGoA%2BJ4%3D%7Ctkp%3ABk9SR56tg5WtZQ https://www.ebay.com/itm/235978320621?_skw=dell+poweredge+r7... https://www.ebay.com/itm/115819389940?_skw=dell+poweredge+r730+512GB&itmmeta=01JNMN0TSK5PH9QM5C04KXP61W&hash=item1af75fa3f4:g:O2MAAOSwRtNkh97Y&itmprp=enc%3AAQAKAAAA0FkggFvd1GGDu0w3yXCmi1fVZjjmHsBiQ8qEYVz5I4ZnPDcIqGMowe5PZuSmUhbqWY8zUYOBURPCEKnFDBuZ8KVE5sw3DsMEdhJOHeD94MrLAzUin0oreQdhkAmOhxvitKBKA6ykXKlQQhdVPb4TRfqj%2Br2uYF%2BObtCgbVcEJ8LASVSG4PSzeochGZNA3eWVpGd0TuLt%2BZd789g4i3zJuXG9e%2Bc0vR%2BTIwGNPzRs8l9biTG0YHwdXvme9Jkl4BKkGaTTMbHGOKFFOrRizjBMOW0%3D%7Ctkp%3ABk9SR56tg5WtZQ https://www.ebay.com/itm/115819389940?_skw=dell+poweredge+r7...

numpad0 2y ago

numpad0 2y ago

DDR4 UDIMM is up to 32GB/module DDR5 UDIMM is up to 64GB/module[0] non-Xeon M/B has up to 4 UDIMM slots -> non-Xeon is up to 128GB/256GB per node Server motherboards have as many as 16 DIMM slots per socket with RDIMM/LRDIMM support, which allows more modules as well as higher capacity modules to be installed. 0: there has been a 128GB UDIMM launch at peak COVID