4 ms·

I've honestly lost all interest in anything integrating with OpenAI at this point. Llama 2 is giving completions at ChatGPT levels with a single GPU. I've repla

I've honestly lost all interest in anything integrating with OpenAI at this point. Llama 2 is giving completions at ChatGPT levels with a single GPU. I've replaced all of my LLM usage with it. Open local models are the future.

starik36 3y ago
Which GPU is capable of all that? And which do you have a write up for this?
- ramesh31 3y ago
  > Which GPU is capable of all that? And which do you have a write up for this? Anything down to an RTX3090. You need 16GB VRAM to comfortably fit llama2-7b-chat with 8 bit quantization and a 2048 context length with llama.cpp. I'm getting completions (in real time) that are easily on par with ChatGPT on a VM with a single V100. People are even getting decent performance on Mac silicon with Metal.
Jeff_Brown 3y ago
Open local models are literally the past, but the very recent past. I therefore agree -- as long as they remain only on the order of a year older than the cutting edge, their future looks extremely bright. I just hope for-profit enterprises continue wanting to push that frontier as hard as they have been. (Wait, actually also if they stop doing that it might be for the best ...)
- ramesh31 3y ago
  >Open local models are literally the past, but the very recent past. I therefore agree -- as long as they remain only on the order of a year older than the cutting edge, their future looks extremely bright. I think the qualitative difference between Llama 2 and any previous open LLM is sufficient to the point that we can call this a new epoch. It took OpenAI spending millions on free compute to show the world what these things are capable of. And because it's something you can really only believe when you see it, that's what set things off. But the cat's out of the bag now, and it's never going back. I think OpenAI would do well to return to their roots of pure research rather than bothering with the product side of things. Come up with the latest and greatest new models, then chuck 'em over the fence for Microsoft to monetize. >I just hope for-profit enterprises continue wanting to push that frontier as hard as they have been. (Wait, actually also if they stop doing that it might be for the best ...) My bet is that Meta is pivoting hard right now. Llama is probably their most successful project/product/whatever since Instagram. They have the talent, the money, and (crucially right now) the hardware to do it. And this plays directly into Zuck's desire for a platform. It seems pretty obvious their play is to build an ecosystem around these things and start gradually introducing licensing fees (and/or hosted models) for big commercial users.

starik36 3y ago

Which GPU is capable of all that? And which do you have a write up for this?

ramesh31 3y ago
> Which GPU is capable of all that? And which do you have a write up for this? Anything down to an RTX3090. You need 16GB VRAM to comfortably fit llama2-7b-chat with 8 bit quantization and a 2048 context length with llama.cpp. I'm getting completions (in real time) that are easily on par with ChatGPT on a VM with a single V100. People are even getting decent performance on Mac silicon with Metal.

Jeff_Brown 3y ago

Open local models are literally the past, but the very recent past. I therefore agree -- as long as they remain only on the order of a year older than the cutting edge, their future looks extremely bright. I just hope for-profit enterprises continue wanting to push that frontier as hard as they have been. (Wait, actually also if they stop doing that it might be for the best ...)

ramesh31 3y ago
>Open local models are literally the past, but the very recent past. I therefore agree -- as long as they remain only on the order of a year older than the cutting edge, their future looks extremely bright. I think the qualitative difference between Llama 2 and any previous open LLM is sufficient to the point that we can call this a new epoch. It took OpenAI spending millions on free compute to show the world what these things are capable of. And because it's something you can really only believe when you see it, that's what set things off. But the cat's out of the bag now, and it's never going back. I think OpenAI would do well to return to their roots of pure research rather than bothering with the product side of things. Come up with the latest and greatest new models, then chuck 'em over the fence for Microsoft to monetize. >I just hope for-profit enterprises continue wanting to push that frontier as hard as they have been. (Wait, actually also if they stop doing that it might be for the best ...) My bet is that Meta is pivoting hard right now. Llama is probably their most successful project/product/whatever since Instagram. They have the talent, the money, and (crucially right now) the hardware to do it. And this plays directly into Zuck's desire for a platform. It seems pretty obvious their play is to build an ecosystem around these things and start gradually introducing licensing fees (and/or hosted models) for big commercial users.