4 ms·

I guess in every world I've worked in, deployment involved deploying a small executable which would run millions of times on thousands of servers, each instance

by dekhn 3y ago

I guess in every world I've worked in, deployment involved deploying a small executable which would run millions of times on thousands of servers, each instance loading a different model (or models) over its lifetime, and the weights are stored in a large, fast filesystem with much higher aggregate bandwidth than a typical local storage device. The executable itself doesn't even contain the final model- just a description of the model which is compiled only after the executable starts (so the compilation has all the runtime info on the machine it will run on). But, I think llama plus obese binaries must be targeting a very, very different community- one that doesn't build its own binaries, runs in any number of different locations, and focuses on getting the model to run with the least friction.

csdvrx 3y ago
> a large, fast filesystem with much higher aggregate bandwidth than a typical local storage device that assumption gets wrong very fast with nvme storage, even before you add herding effects
- dekhn 3y ago
  Until you compare a single machine with nvme to a cluster of storage servers with nvme, and each machine has 800Gbit connectivity and you use smart replication for herding. but yes, nvme definitely has amazing transfer rates.
  - csdvrx 3y ago
    > Until you compare a single machine with nvme to a cluster of storage servers with nvme No, only as long as you compare against a very low number of machines with local nvme. The sum of the bandwith available on typical storage device (even cheap and low end) will be at most times greater than what you have of your expansive top of the line cluster If you have a single local storage, you don't have scale, so you won't have money for an expansive top of the line cluster either. But if you are wasting money on it, yes you will have more bandwidth, but that's a degenerate case. If you have a few local storage machines, the assumption gets very wrong and very fast: 1 low end tier nvme=1 G/s at worst, one top of the line WD 990: 8G/s at best, so we're talking about a ratio of ~ 8 in the most favorable scenario.
fullspectrumdev 3y ago
> But, I think llama plus obese binaries must be targeting a very, very different community- one that doesn't build its own binaries, runs in any number of different locations, and focuses on getting the model to run with the least friction. Yes, the average user.

by dekhn 3y ago

csdvrx 3y ago

> a large, fast filesystem with much higher aggregate bandwidth than a typical local storage device that assumption gets wrong very fast with nvme storage, even before you add herding effects

dekhn 3y ago
Until you compare a single machine with nvme to a cluster of storage servers with nvme, and each machine has 800Gbit connectivity and you use smart replication for herding. but yes, nvme definitely has amazing transfer rates.
- csdvrx 3y ago
  > Until you compare a single machine with nvme to a cluster of storage servers with nvme No, only as long as you compare against a very low number of machines with local nvme. The sum of the bandwith available on typical storage device (even cheap and low end) will be at most times greater than what you have of your expansive top of the line cluster If you have a single local storage, you don't have scale, so you won't have money for an expansive top of the line cluster either. But if you are wasting money on it, yes you will have more bandwidth, but that's a degenerate case. If you have a few local storage machines, the assumption gets very wrong and very fast: 1 low end tier nvme=1 G/s at worst, one top of the line WD 990: 8G/s at best, so we're talking about a ratio of ~ 8 in the most favorable scenario.

fullspectrumdev 3y ago

> But, I think llama plus obese binaries must be targeting a very, very different community- one that doesn't build its own binaries, runs in any number of different locations, and focuses on getting the model to run with the least friction. Yes, the average user.