13 ms·

I've spent an embarassing amount of time since the llamas leaked playing with them, the tools to run them, and writing wrappers for them. They are technically a

I've spent an embarassing amount of time since the llamas leaked playing with them, the tools to run them, and writing wrappers for them. They are technically alternatives in the sense that they're incomparably better chat bots than anything in the past. But at least for the 30B and under versions (65B is too big for me to run), no matter what fine tuning is done (alpaca, gpt4all, vicuna, etc), the llamas themselves are incomparably worse at doing useful tasks than openai's gpt3.5 models like text-davinci-003, or even the gimped gpt3.5-turbo. I wish it wasn't so, but the llamas are toys. Amazing toys, but toys. What openai is getting out of gpt3.5 (and presumbably 4, though I have no access) are actually useful responses for getting work done.

cubefox 4y ago
Your conclusion seems not to be warranted since you haven't tried out the 65B model.
- superkuh 4y ago
  I agree, but I think my experience is representative. So far most human people don't have the resources to be able to use 65B. And most small companies / university groups don't have the resources to fine-tune a 65B. I've talked to a couple dozen people in real time who've played with up to 30B but no one I know has the resources to run the 65B at all or fast enough to actually use and get an opinion of. None of the open source llama projects out there are using 65B in practice (despite support for it) so I think my 30B and under conclusions are applicable to the topic the article covers. I'd love to be wrong and I'm excited for this to change in the future.
  - lolinder 4y ago
    llama.cpp has great support for 65B, and I've been using it on a Linux box (I upgraded my RAM for that purpose). 64GB of RAM for a desktop is like $160 today, so it's not out of reach for most people here if they cared to. Admittedly, it's quite slow and therefore not useful for chatting or real-time applications, and it's unreliable enough in its quality that I'd like to be able to iterate faster. Definitely more of a toy at this point, at least when run on CPU.
    - Scene_Cast2 4y ago
      Could you quantify "quite slow"?
      - lolinder 4y ago
        A token per second-ish with a Ryzen 7 5800X. If I run it for too long it gets slower as heat throttling kicks in, I need a better cooling system if I'm going to run it non-stop.
        dandongus 4y ago
        For a bit of comparison, if you've tested, how fast are 13B or 7B on the same setup?
        lolinder 4y ago
        Really fast. I didn't bother timing, but they're faster than ChatGPT by a long shot. I didn't spend very long with them because the quality is so much worse than the 65B. I should probably go back and try again to see if it's worth it for the extra speed, now that I've played with 65B for a while.
        morrbo 4y ago
        i've had the same experience tbh, 7/13/30 on ryzen (local) and intel (server) both on rhel/centos. It's a shame really
        ImprobableTruth 4y ago
        Is this with full or empty context?
        lolinder 4y ago
        Good question--I'm counting it from empty and around empty. By the time it gets to full I'm also getting heat throttling (I can tell looking at the temp), so it's hard to know the degree to which the slowdown is one or the other.
  - qeternity 4y ago
    I agree with your premise: I have used 65b variants and of course they’re not as good as OpenAI. GPT3 has 175b parameters, and OpenAI has done more RLHF than anyone else. Why would we expect to get comparable performance with models a fraction of the size and a pittance of the fine tuning? That said, it’s clear that replicating GPT4+ performance is within the resources of a number of large tech orgs. And the smaller models can definitely still be useful for tasks.
    - bitL 4y ago
      > Why would we expect to get comparable performance with models a fraction of the size and a pittance of the fine tuning? LLaMA incorporated new techniques that make 65B perform way better than GPT-3's 175B so the model size argument is not very strong.
    - lhl 4y ago
      It's worth pointing out that size isn't everything. From Meta's benchmarking [1] LLaMA 33B outperforms GPT-3 175B, Gopher 280B, Chinchilla 70B and even matches PaLM 540B on a bunch of common evals. Those interested in doing more comparisons can look at https://crfm.stanford.edu/helm/latest/?group=core_scenarios https://crfm.stanford.edu/helm/latest/?group=core_scenarios and https://paperswithcode.com/paper/llama-open-and-efficient-foundation-language-1 https://paperswithcode.com/paper/llama-open-and-efficient-fo... to see where it sits (with some GPT 3.5 and 4 numbers here: https://paperswithcode.com/paper/gpt-4-technical-report-1 https://paperswithcode.com/paper/gpt-4-technical-report-1) I'd agree the secret sauce for how great the newest services perform is probably in the fine-tuning. We're seeing almost daily releases of fine-tuning data sets, training methods and models (at lower and lower costs) so I'm personally pretty optimistic that we'll be seeing some big improvement in self-hosted LLM performance pretty quickly. [1] https://ar5iv.labs.arxiv.org/html/2302.13971#:~:text=Table%203%3A,Sense%20Reasoning%20tasks https://ar5iv.labs.arxiv.org/html/2302.13971#:~:text=Table%2....
  - danShumway 4y ago
    This is a good point. Even if ordinary people did have the resources to run the 65B well on their existing devices, the speed would limit its usefulness quite a bit. In practice, 30B is what most people are going to interact with (if even, I've seen a lot of projects use 13B). My experience here is pretty similar. I'm heavily (emotionally at least) invested in models running locally, I refuse to build something around a remote AI that I can only interact with through an API. But I'm not going to pretend that LLaMA has been amazing locally. I really couldn't figure out what to build with it that would be useful. I'm vaguely hoping that compression actually gets better and that targeted reinforcement/alignment training might change that. GPT can handle a wide range of tasks, but for a smaller AI it wouldn't be too much of a problem to have a much more targeted domain, and at that point maybe the 30B model is actually good enough if it's been refined around a very specific problem domain. For that to happen, training needs to get more accessible though. Or communities need to start getting together and deciding to build very targeted models and then distributing the weights as "plug-and-play" models you can swap out for different tasks. And if there's a way to get 65B more accessible, that would be great too.
    - lhl 4y ago
      I'm pretty confident that the landscape is going to look very different by the end of the year, as there are so many people poking around now. I think that significantly smaller models will definitely be good enough for specialized tasks, but an equivalently tuned larger model will always be better, the question is by how much. On Meta's benchmarks [1], there's only a tiny gap between 30B and 65B, for example. For 65B, GPTQ 4-bit should fit LLaMA 65B into 40GiB of memory. Currently the cheapest way to run that at an acceptable speed would be to use 2 x RTX 3090/4090s (~$2500-3000) or maybe a Jetson Orin 64GB (~$2000). I've seen people trying to run it on an M1 Max and it's just a bit too slow to comfortably use (I get a similar speed to when I try it on my 5950X - about 1-2 tokens/s), but it seems like it's within a factor or two of being fast enough, so not out of the question that it might get there just through software optimizations. I'd definitely upgrade to a 7950X/X3D or a Threadripper (w/ 96GB of DDR5-5200) if I could get 65B running at a comfortable speed all the time. I think training is also advancing at a pretty good clip. LLaMA-adapter [2] is doing fine tuning of LLaMA 13B on a single 8xA100 system in 1h (so for ~$12 for a spot instance) and was already over 3X faster than Alpaca's training. To me, the biggest thing limiting easy plug-and-play distribution is actually LLaMA's licensing issues, so maybe someone will offer a better open foundational model soon and the community can standardize on that. It'd be nice to have a larger context window (Flash Attention?) as well. [1] https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md#quantitative-analysis https://github.com/facebookresearch/llama/blob/main/MODEL_CA... [2] https://github.com/ZrrSkywalker/LLaMA-Adapter https://github.com/ZrrSkywalker/LLaMA-Adapter
  - matthewdgreen 4y ago
    Is it hard to spin up an appropriate EC2 instance with 64GB+ of additional RAM? The m6a.8xlarge seems to have 128GB and costs $1.38 per hour. Was going to try this myself, but now I'm wondering if even that won't be worth the trouble. (I know this is not "your own devices" but would give a feel for what you're missing with the smaller models.)
    - pmoriarty 4y ago
      AWS is notoriously pricy. You should be able to find a much better deal elsewhere.
- muyuu 4y ago
  his conclusion is simultaneously not warranted and correct a like-for-like comparison would be GPT-4 against the larger models like LLaMA 65B, but those cannot be run on consumer-grade hardware so one ends up comparing the stuff one can run... against the top stuff from OpenAI running on high-end GPU farms, and this technology clearly benefits a lot still from much larger scale than most people can afford the great revelation this year is how much does it get better as it get much, much bigger without a clear horizon on where will diminishing returns be hit but at the same time, some useful stuff can be done on consumer hardware - just not the most impressive stuff
  - Szpadel 4y ago
    if "leaks" are true then even a 65B model is a toy when GPT-4 is supposed to be 1000B I would really like to have an open source model that could compete with even gpt3, but basically all lammas I tried failed to create correct email validation regex, where gpt3 is able to even refer to what rfc it is
    - muyuu 4y ago
      13B on my computer does produce useful output, it's pretty hit and miss though GPT-3.5 OTOH is much better, but it's also much better at producing convincing-sounding but completely incorrect answers
  - wokwokwok 4y ago
    The 65B model runs fine on a Mac Studio with 64GB of memory. The output is unremarkable; it’s not significantly better than the 13B model for most uses. GPT 3.5 is an order of magnitude better at least.
    - muyuu 4y ago
      You cannot heavily quantise models and assume they didn't degrade significantly. To run it properly you need a lot more than a Mac Studio, and then comparisons need to be done more or less seriously, not just a few random prompts, because anything in a black box will "cheat" and will be fine tuned to do well at popular benchmarks.
Closi 4y ago
I’ve got access to 4 and it’s a huge leap up from 3.5 - much more subtlety in the response, less hallucinations, less hitting a brick wall, but all of it adding up to a giant leap.
- WXLCKNO 4y ago
  It's funny how big the difference feels between 3.5 and 4 but also feels subtle. Like it's just doing what I expect it to versus seeing the limitations more clearly in 3.5.
  - novok 4y ago
    It's also an order of magnitude larger in parameter count...
    - buryat 4y ago
      unconfirmed, there’s a theory that it maybe similarly sized as gpt3.5 but with additional RLHF and because of other things it outperforms
      - Closi 4y ago
        Apparently the RLHF actually reduces it's capability, it just improves alignment/safety.
- icelancer 4y ago
  I agree. I think it depends on the context, though. I still use 3.5-turbo for text generation, content, ideas, etc, where it's completely suitable, fine, and even quite good. However for any coding, complex analysis, or problems requiring calculation, there's no substitute for GPT-4. It blows 3 and 3.5 out of the water for code analysis, generation, debugging, and self-healing.
hrpnk 4y ago
What this also shows is the degree of control that a single company has over the market. At a time where GPT-4 is integrated into products while others are on the waitlist, competition can move far ahead of a company that's just trying to gain access to the technology w/o clear insights on the prioritization approach. As said, there just are no alternatives at the moment, so to a degree competition is skewed.
Al-Khwarizmi 4y ago
In my particular tests (YMMV), even 13B vicuna beats Bard, though... tough times for Google.
- morrbo 4y ago
  massive YMMV moment for me. my particular usecase was "extract the following attributes from a load of unstructured text, format the results as JSON". ChatGPT was the best (but only on 4 and Davinci), Vicuna just didn't perform at all (nor other variants of llama 7/13/33). Bard smashed it, relatively speaking, in terms of speed. I gave up pretty quickly though because of no information on pricing and/or API. It's funny how all-or-nothing these things seem to be
  - avereveard 4y ago
    On the smaller models you may want to split the task in smaller chunk either in parallel one value at a time or in sequence like extract the attributes, then take the output and ask to format it into a json
    - Tostino 4y ago
      In relation to this, when using GPT4, I have added the addendum to my prompts: "This seems like a lot of work, please split the work into two chunks, and let's start on the first chunk now." It will generally segment the problem in some logical way and work just fine, with vastly improved reasoning abilities due to not trying to do as much at once.
      - avereveard 4y ago
        Ah, clever, thanks for sharing.
jonplackett 4y ago
The difference between 3.5 and 4 is gigantic even in my fairly limited experience. I gave them both some common sense tests and this one stuck out to me. Q: A glass door has ‘push’ written on it in mirror writing. Should you push or pull it GPT-3.5: If the word "push" is written in mirror writing on a glass door, you should push the door to open it GPT-4: Since the word "push" is written in mirror writing, it suggests that the instruction is intended for people on the other side of the door. Therefore, you should pull the door to open it.
- amrb 4y ago
  Would like to see a yearly benchmark's for models like this!
- two_in_one 4y ago
  [dead]
- PoignardAzur 4y ago
  Wait, how does GPT-4 even... Does it benefit from its visual attention, or is it a case of "the question wasn't in GPT-3's training set but it was in GPT-4's"?
  - hutzlibu 4y ago
    I would suspect, that this is one of the manual fine tuned questions. Meaning in before versions people used this question to show flaws and now this specific flaw is fixed. Otherwise it would be indeed reasoning in my understanding.
    - steveBK123 4y ago
      The evolution of answers from version to version makes it clear there are insane amounts of manual fine tunings happening. I think this is largely overlooked by the "its learning" crowd.
      - 4y ago
seydor 4y ago
> (just a hobby, won't be big and professional like gnu) Llamas are creating the linux of AI and the ecosystem around it. Even though openAI has a head start, this whole thing is just starting. Llammas are showing the world that it doesn't take monopoly-level hardware to run those things. And because it's fun, like, video-game-fun there is going to be a lot of attention on them. Running a fully-owned, uncensored chat is the kind of thing that gets people creative
- FooBarWidget 4y ago
  But what about the training data? You can't rely on weights keep being leaked (assuming that that even raises no legal issues) in order for open source AI to advance.
- danShumway 4y ago
  This is my hope as well. It would be disastrous if the future of AI is one where only megacorps can run it and where they control all access to it. In that sense, LLaMA is really encouraging and I'm seriously rooting for it to improve. It's just not there yet. I tend to be kind of bearish on LLMs in general, I think there's a lot more hype than is warranted, and people are overlooking some pretty significant downsides like prompt-injection that are going to end up making them a lot harder to use in ubiquitous contexts in practice, but... I mean, the big LLMs (even GPT-3.5) are definitely still in a class above LLaMA. I understand why they're hyped. I look at GPT and think, "I'm not sure this is worth the trouble of using." But I look at LLaMA and I'm not sure how/where to use it at all. It's a whole different level of output. But that doesn't mean I'm not rooting for the "hobbyists" to succeed. And it doesn't mean LLaMA can't succeed, it doesn't necessarily need to be better than GPT-4, it just needs to be good enough at a lot of the stuff GPT-4 does to be usable, and to have the accessibility and access outweigh everything else. It's just not there yet.
  - amrb 4y ago
    LoRa has been pretty popular and untill the llama leak was not aware of it, maybe will see something cool out of the open assistant project, we have a lot of English and Spanish prompts and was crazy to see people doing an massive open source project for ML.
iamflimflam1 4y ago
GPT-4 is pretty mind blowing. It can follow very complex prompts that 3.5 struggles with.
avereveard 4y ago
What version are you running? Despite good benchmark figures, the coherence and ability to keep on task for large question and structured answers seems to drop significantly using int4 and int8, also depending on the frontend you may get silent corruption from many of those that had been hastily thrown togheter (I.e. In one you get garbage out if you go over the token limit in the inputs and no warning at all)
bradleyjg 4y ago
I haven’t seen to much discussion of what’s possible at various sizes for an early stage start up, which is a discussion I’d expect to see on yc. Clearly a company with $5-5MM in the bank can’t train a competitive LLM from scratch but what would it cost to fine tune and/or run a 65B parameter model or a hypothetical future open source 165B parameter model?
- Robotbeat 4y ago
  > Clearly a company with $5-5MM in the bank can’t train a competitive LLM from scratch... Wait, are we sure? I'm going to make the massive mistake of assuming we're compute bound instead of memory bound, and assume we can train at FP16 (which is a bad assumption because, of course, you're doing calculus where the little pieces you're adding up could get rounded to zero at FP16 pretty easily... although mixed precision FP32/FP16 training is possible). Consumer GPUs like GeForce RTX 4090 can do 3e14 flop/s under certain conditions with fp16. They retailed for about $1600. It took reportedly 3e23 flop to train GPT-3. A year is 3e7 seconds. So the upfront cost of retail GPUs doing 3e23 fp16 operations in a single year is potentially as low as ~$50k (and about $20k worth of electricity). (FP32 peak is about a factor of 4 worse, so ~$200k.) So it's not actually impossible to imagine a particularly clever approach to training that could maybe achieve competitive LLM training for less than $5 million in hardware costs. (except for the fact that compute isn't really the bottleneck, memory really is.)
  - bradleyjg 4y ago
    It would be pretty funny if a start up took its first round and put a huge chunk of it into a crypto mining style ad hoc data center.
    - Robotbeat 4y ago
      Exactly. I think we haven’t optimized for hardware cost much with LLM training. Crypto is kind of like that but taken to an extreme. There are some reasons why it’s not comparable (crypto mining is embarrassingly parallel, doesn’t need a lot of memory, tends to burn at least as much cost in electricity as on hardware, whereas LLM training is memory-bound, etc), but it shows kind of roughly what scrappy, hardcore computer hardware cost-optimization looks like at scale.
valine 4y ago
Are you using quantization? I’ve gotten very good results from the float16 13B vicuna model.
- tarruda 4y ago
  Did you use GPU inference? If so, how much memory is required?
  - valine 4y ago
    Yeah I'm using GPU inference. The vicuna 13B model uses 26.3GB of VRAM with my setup. I'm running it split between two rtx4090s which gives me about 20 token/s.
samstave 4y ago
>>useful responses for getting work done. So what is this WORK where invest? Where di-vest?
novok 4y ago
Well you are comparing 165B and 1000B parameter models to 30B and 60B parameter models. That is about 3 to 33x larger, not an apples to apples comparison.
- crakenzak 4y ago
  I agree with your general statement, but if one thing the recent DeepMind Chinchilla research paper showed us is that the size of the model (number of parameters) is much less of the determinant of model performance & quality than the amount of high-quality data (number of high-quality tokens). Their 70B chinchilla model significantly outperforms the 175B GPT3 model. Possibly where OpenAI has a leg up is their high-quality data sourcing & curating infrastructure and their RLHF mechanisms. Paper: https://arxiv.org/abs/2203.15556 https://arxiv.org/abs/2203.15556
lhl 4y ago
I was playing around w/ a lot of these models as well, and was surprised by how bad LLaMA performed vs it's benchmark scores [1][2][3]. However, recently @tyfon mentioned he had great success w/ LLaMA and shared his prompt [4] (based off of more recent work by llama.cpp contributors and it performed much better in my own personal testing. There's basically a new fine tune a day and while some I don't like (Alpaca, Vicuna, Baize, Koala are all fine-tuned to be too limiting IMO), I'm interested in what gpt4-x-alpaca and OA (Open Assistant) are doing, and the various un-filtered fine tunes (especially w/ lighter weight adapter/LoRA training which would let you personalize/specialize). GPTQ-for-LLaMa let's me load the 4-bit quantized 30B model (~17GiB) onto my GPU in about 5 seconds (and I know llama.cpp's mmap improvements have also made it quite a lot quicker) so I think it's perfectly reasonable to switch between tuned models for tasks in code assistance, correspondence, etc. I have access to ChatGPT 4, and agree it's signficantly better than what's out there atm, and it can basically do anything I've thrown at it (here's it helping me with my WM yak shaving: https://sharegpt.com/c/Xv73Vwl https://sharegpt.com/c/Xv73Vwl or discussing MAPS/psychedelics for clinical applications https://sharegpt.com/c/N3VXFxS https://sharegpt.com/c/N3VXFxS - it's amazing what it can pull from memory and it hallucinates much less than 3.5). That being said, I've found the Browsing 3.5 model to be quite useful for doing things like catching up on the last few years of LLM advancements: https://sharegpt.com/c/JFexqvm https://sharegpt.com/c/JFexqvm [1] https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md https://github.com/facebookresearch/llama/blob/main/MODEL_CA... [2] https://github.com/ggerganov/llama.cpp/discussions/406 https://github.com/ggerganov/llama.cpp/discussions/406 [3] https://paperswithcode.com/sota/language-modelling-on-wikitext-2 https://paperswithcode.com/sota/language-modelling-on-wikite... [4] https://news.ycombinator.com/item?id=35484341 https://news.ycombinator.com/item?id=35484341
- Tostino 4y ago
  I think part of the benefits of LoRA is that you can load the base model once, and then just swap out the vastly smaller LoRA fine-tune to fit the specific task it is working on.

cubefox 4y ago

Your conclusion seems not to be warranted since you haven't tried out the 65B model.

superkuh 4y ago
I agree, but I think my experience is representative. So far most human people don't have the resources to be able to use 65B. And most small companies / university groups don't have the resources to fine-tune a 65B. I've talked to a couple dozen people in real time who've played with up to 30B but no one I know has the resources to run the 65B at all or fast enough to actually use and get an opinion of. None of the open source llama projects out there are using 65B in practice (despite support for it) so I think my 30B and under conclusions are applicable to the topic the article covers. I'd love to be wrong and I'm excited for this to change in the future.
- lolinder 4y ago
  llama.cpp has great support for 65B, and I've been using it on a Linux box (I upgraded my RAM for that purpose). 64GB of RAM for a desktop is like $160 today, so it's not out of reach for most people here if they cared to. Admittedly, it's quite slow and therefore not useful for chatting or real-time applications, and it's unreliable enough in its quality that I'd like to be able to iterate faster. Definitely more of a toy at this point, at least when run on CPU.
  - Scene_Cast2 4y ago
    Could you quantify "quite slow"?
    - lolinder 4y ago
      A token per second-ish with a Ryzen 7 5800X. If I run it for too long it gets slower as heat throttling kicks in, I need a better cooling system if I'm going to run it non-stop.
      - dandongus 4y ago
        For a bit of comparison, if you've tested, how fast are 13B or 7B on the same setup?
        lolinder 4y ago
        Really fast. I didn't bother timing, but they're faster than ChatGPT by a long shot. I didn't spend very long with them because the quality is so much worse than the 65B. I should probably go back and try again to see if it's worth it for the extra speed, now that I've played with 65B for a while.
      - morrbo 4y ago
        i've had the same experience tbh, 7/13/30 on ryzen (local) and intel (server) both on rhel/centos. It's a shame really
      - ImprobableTruth 4y ago
        Is this with full or empty context?
        lolinder 4y ago
        Good question--I'm counting it from empty and around empty. By the time it gets to full I'm also getting heat throttling (I can tell looking at the temp), so it's hard to know the degree to which the slowdown is one or the other.
- qeternity 4y ago
  I agree with your premise: I have used 65b variants and of course they’re not as good as OpenAI. GPT3 has 175b parameters, and OpenAI has done more RLHF than anyone else. Why would we expect to get comparable performance with models a fraction of the size and a pittance of the fine tuning? That said, it’s clear that replicating GPT4+ performance is within the resources of a number of large tech orgs. And the smaller models can definitely still be useful for tasks.
  - bitL 4y ago
    > Why would we expect to get comparable performance with models a fraction of the size and a pittance of the fine tuning? LLaMA incorporated new techniques that make 65B perform way better than GPT-3's 175B so the model size argument is not very strong.
  - lhl 4y ago
    It's worth pointing out that size isn't everything. From Meta's benchmarking [1] LLaMA 33B outperforms GPT-3 175B, Gopher 280B, Chinchilla 70B and even matches PaLM 540B on a bunch of common evals. Those interested in doing more comparisons can look at https://crfm.stanford.edu/helm/latest/?group=core_scenarios https://crfm.stanford.edu/helm/latest/?group=core_scenarios and https://paperswithcode.com/paper/llama-open-and-efficient-foundation-language-1 https://paperswithcode.com/paper/llama-open-and-efficient-fo... to see where it sits (with some GPT 3.5 and 4 numbers here: https://paperswithcode.com/paper/gpt-4-technical-report-1 https://paperswithcode.com/paper/gpt-4-technical-report-1) I'd agree the secret sauce for how great the newest services perform is probably in the fine-tuning. We're seeing almost daily releases of fine-tuning data sets, training methods and models (at lower and lower costs) so I'm personally pretty optimistic that we'll be seeing some big improvement in self-hosted LLM performance pretty quickly. [1] https://ar5iv.labs.arxiv.org/html/2302.13971#:~:text=Table%203%3A,Sense%20Reasoning%20tasks https://ar5iv.labs.arxiv.org/html/2302.13971#:~:text=Table%2....
- danShumway 4y ago
  This is a good point. Even if ordinary people did have the resources to run the 65B well on their existing devices, the speed would limit its usefulness quite a bit. In practice, 30B is what most people are going to interact with (if even, I've seen a lot of projects use 13B). My experience here is pretty similar. I'm heavily (emotionally at least) invested in models running locally, I refuse to build something around a remote AI that I can only interact with through an API. But I'm not going to pretend that LLaMA has been amazing locally. I really couldn't figure out what to build with it that would be useful. I'm vaguely hoping that compression actually gets better and that targeted reinforcement/alignment training might change that. GPT can handle a wide range of tasks, but for a smaller AI it wouldn't be too much of a problem to have a much more targeted domain, and at that point maybe the 30B model is actually good enough if it's been refined around a very specific problem domain. For that to happen, training needs to get more accessible though. Or communities need to start getting together and deciding to build very targeted models and then distributing the weights as "plug-and-play" models you can swap out for different tasks. And if there's a way to get 65B more accessible, that would be great too.
  - lhl 4y ago
    I'm pretty confident that the landscape is going to look very different by the end of the year, as there are so many people poking around now. I think that significantly smaller models will definitely be good enough for specialized tasks, but an equivalently tuned larger model will always be better, the question is by how much. On Meta's benchmarks [1], there's only a tiny gap between 30B and 65B, for example. For 65B, GPTQ 4-bit should fit LLaMA 65B into 40GiB of memory. Currently the cheapest way to run that at an acceptable speed would be to use 2 x RTX 3090/4090s (~$2500-3000) or maybe a Jetson Orin 64GB (~$2000). I've seen people trying to run it on an M1 Max and it's just a bit too slow to comfortably use (I get a similar speed to when I try it on my 5950X - about 1-2 tokens/s), but it seems like it's within a factor or two of being fast enough, so not out of the question that it might get there just through software optimizations. I'd definitely upgrade to a 7950X/X3D or a Threadripper (w/ 96GB of DDR5-5200) if I could get 65B running at a comfortable speed all the time. I think training is also advancing at a pretty good clip. LLaMA-adapter [2] is doing fine tuning of LLaMA 13B on a single 8xA100 system in 1h (so for ~$12 for a spot instance) and was already over 3X faster than Alpaca's training. To me, the biggest thing limiting easy plug-and-play distribution is actually LLaMA's licensing issues, so maybe someone will offer a better open foundational model soon and the community can standardize on that. It'd be nice to have a larger context window (Flash Attention?) as well. [1] https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md#quantitative-analysis https://github.com/facebookresearch/llama/blob/main/MODEL_CA... [2] https://github.com/ZrrSkywalker/LLaMA-Adapter https://github.com/ZrrSkywalker/LLaMA-Adapter
- matthewdgreen 4y ago
  Is it hard to spin up an appropriate EC2 instance with 64GB+ of additional RAM? The m6a.8xlarge seems to have 128GB and costs $1.38 per hour. Was going to try this myself, but now I'm wondering if even that won't be worth the trouble. (I know this is not "your own devices" but would give a feel for what you're missing with the smaller models.)
  - pmoriarty 4y ago
    AWS is notoriously pricy. You should be able to find a much better deal elsewhere.
muyuu 4y ago
his conclusion is simultaneously not warranted and correct a like-for-like comparison would be GPT-4 against the larger models like LLaMA 65B, but those cannot be run on consumer-grade hardware so one ends up comparing the stuff one can run... against the top stuff from OpenAI running on high-end GPU farms, and this technology clearly benefits a lot still from much larger scale than most people can afford the great revelation this year is how much does it get better as it get much, much bigger without a clear horizon on where will diminishing returns be hit but at the same time, some useful stuff can be done on consumer hardware - just not the most impressive stuff
- Szpadel 4y ago
  if "leaks" are true then even a 65B model is a toy when GPT-4 is supposed to be 1000B I would really like to have an open source model that could compete with even gpt3, but basically all lammas I tried failed to create correct email validation regex, where gpt3 is able to even refer to what rfc it is
  - muyuu 4y ago
    13B on my computer does produce useful output, it's pretty hit and miss though GPT-3.5 OTOH is much better, but it's also much better at producing convincing-sounding but completely incorrect answers
- wokwokwok 4y ago
  The 65B model runs fine on a Mac Studio with 64GB of memory. The output is unremarkable; it’s not significantly better than the 13B model for most uses. GPT 3.5 is an order of magnitude better at least.
  - muyuu 4y ago
    You cannot heavily quantise models and assume they didn't degrade significantly. To run it properly you need a lot more than a Mac Studio, and then comparisons need to be done more or less seriously, not just a few random prompts, because anything in a black box will "cheat" and will be fine tuned to do well at popular benchmarks.

Closi 4y ago

I’ve got access to 4 and it’s a huge leap up from 3.5 - much more subtlety in the response, less hallucinations, less hitting a brick wall, but all of it adding up to a giant leap.

WXLCKNO 4y ago
It's funny how big the difference feels between 3.5 and 4 but also feels subtle. Like it's just doing what I expect it to versus seeing the limitations more clearly in 3.5.
- novok 4y ago
  It's also an order of magnitude larger in parameter count...
  - buryat 4y ago
    unconfirmed, there’s a theory that it maybe similarly sized as gpt3.5 but with additional RLHF and because of other things it outperforms
    - Closi 4y ago
      Apparently the RLHF actually reduces it's capability, it just improves alignment/safety.
icelancer 4y ago
I agree. I think it depends on the context, though. I still use 3.5-turbo for text generation, content, ideas, etc, where it's completely suitable, fine, and even quite good. However for any coding, complex analysis, or problems requiring calculation, there's no substitute for GPT-4. It blows 3 and 3.5 out of the water for code analysis, generation, debugging, and self-healing.

hrpnk 4y ago

What this also shows is the degree of control that a single company has over the market. At a time where GPT-4 is integrated into products while others are on the waitlist, competition can move far ahead of a company that's just trying to gain access to the technology w/o clear insights on the prioritization approach. As said, there just are no alternatives at the moment, so to a degree competition is skewed.

Al-Khwarizmi 4y ago

In my particular tests (YMMV), even 13B vicuna beats Bard, though... tough times for Google.

morrbo 4y ago
massive YMMV moment for me. my particular usecase was "extract the following attributes from a load of unstructured text, format the results as JSON". ChatGPT was the best (but only on 4 and Davinci), Vicuna just didn't perform at all (nor other variants of llama 7/13/33). Bard smashed it, relatively speaking, in terms of speed. I gave up pretty quickly though because of no information on pricing and/or API. It's funny how all-or-nothing these things seem to be
- avereveard 4y ago
  On the smaller models you may want to split the task in smaller chunk either in parallel one value at a time or in sequence like extract the attributes, then take the output and ask to format it into a json
  - Tostino 4y ago
    In relation to this, when using GPT4, I have added the addendum to my prompts: "This seems like a lot of work, please split the work into two chunks, and let's start on the first chunk now." It will generally segment the problem in some logical way and work just fine, with vastly improved reasoning abilities due to not trying to do as much at once.
    - avereveard 4y ago
      Ah, clever, thanks for sharing.

jonplackett 4y ago

The difference between 3.5 and 4 is gigantic even in my fairly limited experience. I gave them both some common sense tests and this one stuck out to me. Q: A glass door has ‘push’ written on it in mirror writing. Should you push or pull it GPT-3.5: If the word "push" is written in mirror writing on a glass door, you should push the door to open it GPT-4: Since the word "push" is written in mirror writing, it suggests that the instruction is intended for people on the other side of the door. Therefore, you should pull the door to open it.

amrb 4y ago
Would like to see a yearly benchmark's for models like this!
two_in_one 4y ago
[dead]
PoignardAzur 4y ago
Wait, how does GPT-4 even... Does it benefit from its visual attention, or is it a case of "the question wasn't in GPT-3's training set but it was in GPT-4's"?
- hutzlibu 4y ago
  I would suspect, that this is one of the manual fine tuned questions. Meaning in before versions people used this question to show flaws and now this specific flaw is fixed. Otherwise it would be indeed reasoning in my understanding.
  - steveBK123 4y ago
    The evolution of answers from version to version makes it clear there are insane amounts of manual fine tunings happening. I think this is largely overlooked by the "its learning" crowd.
    - 4y ago

seydor 4y ago

> (just a hobby, won't be big and professional like gnu) Llamas are creating the linux of AI and the ecosystem around it. Even though openAI has a head start, this whole thing is just starting. Llammas are showing the world that it doesn't take monopoly-level hardware to run those things. And because it's fun, like, video-game-fun there is going to be a lot of attention on them. Running a fully-owned, uncensored chat is the kind of thing that gets people creative

FooBarWidget 4y ago
But what about the training data? You can't rely on weights keep being leaked (assuming that that even raises no legal issues) in order for open source AI to advance.
danShumway 4y ago
This is my hope as well. It would be disastrous if the future of AI is one where only megacorps can run it and where they control all access to it. In that sense, LLaMA is really encouraging and I'm seriously rooting for it to improve. It's just not there yet. I tend to be kind of bearish on LLMs in general, I think there's a lot more hype than is warranted, and people are overlooking some pretty significant downsides like prompt-injection that are going to end up making them a lot harder to use in ubiquitous contexts in practice, but... I mean, the big LLMs (even GPT-3.5) are definitely still in a class above LLaMA. I understand why they're hyped. I look at GPT and think, "I'm not sure this is worth the trouble of using." But I look at LLaMA and I'm not sure how/where to use it at all. It's a whole different level of output. But that doesn't mean I'm not rooting for the "hobbyists" to succeed. And it doesn't mean LLaMA can't succeed, it doesn't necessarily need to be better than GPT-4, it just needs to be good enough at a lot of the stuff GPT-4 does to be usable, and to have the accessibility and access outweigh everything else. It's just not there yet.
- amrb 4y ago
  LoRa has been pretty popular and untill the llama leak was not aware of it, maybe will see something cool out of the open assistant project, we have a lot of English and Spanish prompts and was crazy to see people doing an massive open source project for ML.

iamflimflam1 4y ago

GPT-4 is pretty mind blowing. It can follow very complex prompts that 3.5 struggles with.

avereveard 4y ago

What version are you running? Despite good benchmark figures, the coherence and ability to keep on task for large question and structured answers seems to drop significantly using int4 and int8, also depending on the frontend you may get silent corruption from many of those that had been hastily thrown togheter (I.e. In one you get garbage out if you go over the token limit in the inputs and no warning at all)

bradleyjg 4y ago

I haven’t seen to much discussion of what’s possible at various sizes for an early stage start up, which is a discussion I’d expect to see on yc. Clearly a company with $5-5MM in the bank can’t train a competitive LLM from scratch but what would it cost to fine tune and/or run a 65B parameter model or a hypothetical future open source 165B parameter model?

Robotbeat 4y ago
> Clearly a company with $5-5MM in the bank can’t train a competitive LLM from scratch... Wait, are we sure? I'm going to make the massive mistake of assuming we're compute bound instead of memory bound, and assume we can train at FP16 (which is a bad assumption because, of course, you're doing calculus where the little pieces you're adding up could get rounded to zero at FP16 pretty easily... although mixed precision FP32/FP16 training is possible). Consumer GPUs like GeForce RTX 4090 can do 3e14 flop/s under certain conditions with fp16. They retailed for about $1600. It took reportedly 3e23 flop to train GPT-3. A year is 3e7 seconds. So the upfront cost of retail GPUs doing 3e23 fp16 operations in a single year is potentially as low as ~$50k (and about $20k worth of electricity). (FP32 peak is about a factor of 4 worse, so ~$200k.) So it's not actually impossible to imagine a particularly clever approach to training that could maybe achieve competitive LLM training for less than $5 million in hardware costs. (except for the fact that compute isn't really the bottleneck, memory really is.)
- bradleyjg 4y ago
  It would be pretty funny if a start up took its first round and put a huge chunk of it into a crypto mining style ad hoc data center.
  - Robotbeat 4y ago
    Exactly. I think we haven’t optimized for hardware cost much with LLM training. Crypto is kind of like that but taken to an extreme. There are some reasons why it’s not comparable (crypto mining is embarrassingly parallel, doesn’t need a lot of memory, tends to burn at least as much cost in electricity as on hardware, whereas LLM training is memory-bound, etc), but it shows kind of roughly what scrappy, hardcore computer hardware cost-optimization looks like at scale.

valine 4y ago

Are you using quantization? I’ve gotten very good results from the float16 13B vicuna model.

tarruda 4y ago
Did you use GPU inference? If so, how much memory is required?
- valine 4y ago
  Yeah I'm using GPU inference. The vicuna 13B model uses 26.3GB of VRAM with my setup. I'm running it split between two rtx4090s which gives me about 20 token/s.

samstave 4y ago

>>useful responses for getting work done. So what is this WORK where invest? Where di-vest?

novok 4y ago

Well you are comparing 165B and 1000B parameter models to 30B and 60B parameter models. That is about 3 to 33x larger, not an apples to apples comparison.

crakenzak 4y ago
I agree with your general statement, but if one thing the recent DeepMind Chinchilla research paper showed us is that the size of the model (number of parameters) is much less of the determinant of model performance & quality than the amount of high-quality data (number of high-quality tokens). Their 70B chinchilla model significantly outperforms the 175B GPT3 model. Possibly where OpenAI has a leg up is their high-quality data sourcing & curating infrastructure and their RLHF mechanisms. Paper: https://arxiv.org/abs/2203.15556 https://arxiv.org/abs/2203.15556

lhl 4y ago

I was playing around w/ a lot of these models as well, and was surprised by how bad LLaMA performed vs it's benchmark scores [1][2][3]. However, recently @tyfon mentioned he had great success w/ LLaMA and shared his prompt [4] (based off of more recent work by llama.cpp contributors and it performed much better in my own personal testing. There's basically a new fine tune a day and while some I don't like (Alpaca, Vicuna, Baize, Koala are all fine-tuned to be too limiting IMO), I'm interested in what gpt4-x-alpaca and OA (Open Assistant) are doing, and the various un-filtered fine tunes (especially w/ lighter weight adapter/LoRA training which would let you personalize/specialize). GPTQ-for-LLaMa let's me load the 4-bit quantized 30B model (~17GiB) onto my GPU in about 5 seconds (and I know llama.cpp's mmap improvements have also made it quite a lot quicker) so I think it's perfectly reasonable to switch between tuned models for tasks in code assistance, correspondence, etc. I have access to ChatGPT 4, and agree it's signficantly better than what's out there atm, and it can basically do anything I've thrown at it (here's it helping me with my WM yak shaving: https://sharegpt.com/c/Xv73Vwl https://sharegpt.com/c/Xv73Vwl or discussing MAPS/psychedelics for clinical applications https://sharegpt.com/c/N3VXFxS https://sharegpt.com/c/N3VXFxS - it's amazing what it can pull from memory and it hallucinates much less than 3.5). That being said, I've found the Browsing 3.5 model to be quite useful for doing things like catching up on the last few years of LLM advancements: https://sharegpt.com/c/JFexqvm https://sharegpt.com/c/JFexqvm [1] https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md https://github.com/facebookresearch/llama/blob/main/MODEL_CA... [2] https://github.com/ggerganov/llama.cpp/discussions/406 https://github.com/ggerganov/llama.cpp/discussions/406 [3] https://paperswithcode.com/sota/language-modelling-on-wikitext-2 https://paperswithcode.com/sota/language-modelling-on-wikite... [4] https://news.ycombinator.com/item?id=35484341 https://news.ycombinator.com/item?id=35484341

Tostino 4y ago
I think part of the benefits of LoRA is that you can load the base model once, and then just swap out the vastly smaller LoRA fine-tune to fit the specific task it is working on.