3 ms·

I hope that open models will dominate. The difficult part to reconcile for me is the amount of compute that's required to create and run such models. Small mode

by tadasv 3mo ago

I hope that open models will dominate. The difficult part to reconcile for me is the amount of compute that's required to create and run such models. Small models are fine, I run local llms 27b param on a gpu, but it's not even close to frontier in capability. Who wants to drop $40k+ on hardware to run these things. Companies, maybe/perhapts. On the other hand, to run a DB I can get a server for $3k and handle tons of traffic on it and other things too.

Avicebron 3mo ago
Has anyone tried to run a data center as a Co-Op?
- rzerowan 3mo ago
  Would actually be a good biz model for the Colo facilities that keep shutting down as everyone moves to the big cloud providers.Now if they can get their hands on enough GPUs and RAM.
  - eru 3mo ago
    Why would it be a good business model?
    - danielheath 3mo ago
      I think it aligns incentives way better (but is almost impossible to set up).
      - eru 3mo ago
        Incentives are pretty easy to align, if you have repeat customers and repetition. McDonald's sells me burgers I expect at a reasonable price, because they want me to come back tomorrow and not go across the street to Burger King.
  - orbifold 3mo ago
    can you give examples of colo facilities in SF / New York that are shutting down?
- orbifold 3mo ago
  I was thinking that would be great, too. What would be the equivalent for the property developer: one gpu server is 450k.
- rasur 3mo ago
  coopcloud.tech ?
- fc417fc802 3mo ago
  A data center or a cloud? It's not difficult to find a good data center to colo at. The problem is then you have to bring your own hardware, technicians, and sysadmins. However, if you don't trust cloud providers or inference providers for whatever reason then you probably aren't going to be excited to enter a co-op model where you're still effectively renting access to hardware that you don't directly own. There are already reasonably priced options to rent bare metal from a cloud provider. The only way I see it working is if it's a bunch of medium to large sized businesses getting together to be able to rent out the spare capacity on hardware that they physically control. So an AWS equivalent where each rack is owned by a different company and retail VMs migrate between them transparently. But I question the overall economics of such an arrangement.
  - embedding-shape 3mo ago
    > enter a co-op model where you're still effectively renting access to hardware that you don't directly own If we're imagining a co-op, then the participants should all be equal owners in an organisation that owns the hardware itself, otherwise it's not much of a co-op really.
    - fc417fc802 3mo ago
      But at that point you don't have the same sort of security guarantees about the hardware. This isn't like a farming co-op where a single expensive piece of equipment gets passed around to each participant one week at a time. There needs to be either an economic, security, or other advantage to entering into this arrangement for a medium to large sized player that would otherwise be colocating multiple racks in a privately owned datacenter or else renting bare metal instances in bulk from one of the hyperscalers.
      - mattlutze 3mo ago
        Participants as equal owners doesn't preclude security guarantees. The advantage is having a voice in setting direction, investment and ultimate shared incremental profit recovery over the rent capture from hyperscalers. It's an interesting idea really. But it also comes with cooperation, which means trust, and a lot of for-profit enterprises are bad at operating cooperatively outside of 2- or 3- way engagements.
rzerowan 3mo ago
I believe until the hardware designs catch up to be more commodized ala cryto mining evolution from GPUs to ASICS for specfic algos. Designs (like Google TPUs equivalent) would also need to evolve to be more memory dense to be able to handle them. Untill then it seems will be system time shares for the larger models , probably with a bring your own model and pay as you go.
- Dylan16807 3mo ago
  > ala cryto mining evolution from GPUs to ASICS for specfic algos I don't see it happening. A current gen GPU with a huge and fast block of memory isn't a perfect fit for these algorithms but it's relatively close. With cryptocurrency, mass small sha256 hashing was a totally different kind of computation.
  - fc417fc802 3mo ago
    > isn't a perfect fit for these algorithms but it's relatively close I don't think that's true. The best fit out of what's presently available perhaps. Inference is almost entirely memory bandwidth bound at present, to the extent that GPUs with HBM have a massive advantage over those with GDDR. TPUs appear to be a much better overall design. I expect that a hypothetical advance in fabrication enabling processing elements to be placed directly adjacent to dense RAM on the same silicon (not merely in the same package) would be superior in all regards.
    - Dylan16807 3mo ago
      > I expect that a hypothetical advance in fabrication enabling processing elements to be placed directly adjacent to dense RAM on the same silicon (not merely in the same package) would be superior in all regards. Processing scales better than DRAM does. I think an HBM-like stack where the bottom layer has the math units is probably the ultimate form of that. And it's possible that flash instead of DRAM is actually the better play, as long as you can hook up enough in parallel. RIP Optane.
  - nobrains 3mo ago
bluecalm 3mo ago
There might be a community effort at some point. This happened in chess where the community recreated and then improved on Alpha Zero. You could run small training chunks on your machine. Some people donated thousands of hours of server time.
tsss 3mo ago
They never will. The only reason China releases open weights is because they can't compete on frontier models. Whoever has the frontier model has no incentive to give it away for free.

by tadasv 3mo ago

Avicebron 3mo ago

Has anyone tried to run a data center as a Co-Op?

rzerowan 3mo ago
Would actually be a good biz model for the Colo facilities that keep shutting down as everyone moves to the big cloud providers.Now if they can get their hands on enough GPUs and RAM.
- eru 3mo ago
  Why would it be a good business model?
  - danielheath 3mo ago
    I think it aligns incentives way better (but is almost impossible to set up).
    - eru 3mo ago
      Incentives are pretty easy to align, if you have repeat customers and repetition. McDonald's sells me burgers I expect at a reasonable price, because they want me to come back tomorrow and not go across the street to Burger King.
- orbifold 3mo ago
  can you give examples of colo facilities in SF / New York that are shutting down?
orbifold 3mo ago
I was thinking that would be great, too. What would be the equivalent for the property developer: one gpu server is 450k.
rasur 3mo ago
coopcloud.tech ?
fc417fc802 3mo ago
A data center or a cloud? It's not difficult to find a good data center to colo at. The problem is then you have to bring your own hardware, technicians, and sysadmins. However, if you don't trust cloud providers or inference providers for whatever reason then you probably aren't going to be excited to enter a co-op model where you're still effectively renting access to hardware that you don't directly own. There are already reasonably priced options to rent bare metal from a cloud provider. The only way I see it working is if it's a bunch of medium to large sized businesses getting together to be able to rent out the spare capacity on hardware that they physically control. So an AWS equivalent where each rack is owned by a different company and retail VMs migrate between them transparently. But I question the overall economics of such an arrangement.
- embedding-shape 3mo ago
  > enter a co-op model where you're still effectively renting access to hardware that you don't directly own If we're imagining a co-op, then the participants should all be equal owners in an organisation that owns the hardware itself, otherwise it's not much of a co-op really.
  - fc417fc802 3mo ago
    But at that point you don't have the same sort of security guarantees about the hardware. This isn't like a farming co-op where a single expensive piece of equipment gets passed around to each participant one week at a time. There needs to be either an economic, security, or other advantage to entering into this arrangement for a medium to large sized player that would otherwise be colocating multiple racks in a privately owned datacenter or else renting bare metal instances in bulk from one of the hyperscalers.
    - mattlutze 3mo ago
      Participants as equal owners doesn't preclude security guarantees. The advantage is having a voice in setting direction, investment and ultimate shared incremental profit recovery over the rent capture from hyperscalers. It's an interesting idea really. But it also comes with cooperation, which means trust, and a lot of for-profit enterprises are bad at operating cooperatively outside of 2- or 3- way engagements.

rzerowan 3mo ago

I believe until the hardware designs catch up to be more commodized ala cryto mining evolution from GPUs to ASICS for specfic algos. Designs (like Google TPUs equivalent) would also need to evolve to be more memory dense to be able to handle them. Untill then it seems will be system time shares for the larger models , probably with a bring your own model and pay as you go.

Dylan16807 3mo ago
> ala cryto mining evolution from GPUs to ASICS for specfic algos I don't see it happening. A current gen GPU with a huge and fast block of memory isn't a perfect fit for these algorithms but it's relatively close. With cryptocurrency, mass small sha256 hashing was a totally different kind of computation.
- fc417fc802 3mo ago
  > isn't a perfect fit for these algorithms but it's relatively close I don't think that's true. The best fit out of what's presently available perhaps. Inference is almost entirely memory bandwidth bound at present, to the extent that GPUs with HBM have a massive advantage over those with GDDR. TPUs appear to be a much better overall design. I expect that a hypothetical advance in fabrication enabling processing elements to be placed directly adjacent to dense RAM on the same silicon (not merely in the same package) would be superior in all regards.
  - Dylan16807 3mo ago
    > I expect that a hypothetical advance in fabrication enabling processing elements to be placed directly adjacent to dense RAM on the same silicon (not merely in the same package) would be superior in all regards. Processing scales better than DRAM does. I think an HBM-like stack where the bottom layer has the math units is probably the ultimate form of that. And it's possible that flash instead of DRAM is actually the better play, as long as you can hook up enough in parallel. RIP Optane.
- nobrains 3mo ago

bluecalm 3mo ago

There might be a community effort at some point. This happened in chess where the community recreated and then improved on Alpha Zero. You could run small training chunks on your machine. Some people donated thousands of hours of server time.

tsss 3mo ago

They never will. The only reason China releases open weights is because they can't compete on frontier models. Whoever has the frontier model has no incentive to give it away for free.