5 ms·

I tried this myself using GPT-2J and nearly 15 years of IRC logs trimmed down to only contexts that include me. Each training item would be my message and the

I tried this myself using GPT-2J and nearly 15 years of IRC logs trimmed down to only contexts that include me. Each training item would be my message and the preceding 20 messages. Still ended up being an 80 MB log file of over 1M lines. The results were...not good at all. I ran an IRC bot that would watch the channel, and any time someone sent a message, it would send the last 20 messages to the trained model to see what it would predict the next message to be. It generated mostly garbage. Generated chat messages were on-topic, but non-sensical. It's like the AI was like "uh-huh...yeah...I know some of these words!" and spat something incoherent that seemed related. At some point, I'll probably try again with a far better model. Likely once I do my next system overhaul so I can do it locally. The only thing I'm not sure of is how to properly train and generate. GPT3 and its predecessors are very much simple text-completion AIs, whereas it seems all the models today are designed for a conversation back-and-forth between the AI and a user.

jokethrowaway 3y ago
> It's like the AI was like "uh-huh...yeah...I know some of these words!" and spat something incoherent that seemed related. An accurate depiction of what's going on! I had terrible results with GPT2 but the newer models are much better. I recommend fine tuning mistral (I had better results than any llama based model)
- Sohcahtoa82 3y ago
  > I recommend fine tuning mistral What kind of hardware does that take? I've got an RTX 3080, i9-9900K, and 32 GB of RAM right now. I eventually plan on upgrading to an RTX 5090 when those come out, as well as i9-15900K (Or whatever they're gonna call it) and 128 GB of RAM once the next Intel CPU comes out. My understanding is that even with an RTX 5090 (Assuming 32 GB of VRAM, nothing has been announced yet), if I use fp32, I still won't have enough GPU memory to do inference on a 7B model, let alone training. Though if I quantize down to fp16, I could do inference, but still not training.
  - eightysixfour 3y ago
    Pretty sure an fp32 7b model uses just shy of 27gb of vram. The performance losses at int8 are minuscule, even down to int5 is still very usable. Why would you try and run it fp32 or fp16?
    - Sohcahtoa82 3y ago
      My naive intuition is that you'd want to train on the highest precision you have the horsepower for, and then after training, you might be able to get away with quantizing to a lower precision for inference. I would also expect that fp32 might be overkill, but that anything less than fp16 would start to show a noticeable loss in quality, especially in training. I can't imagine int8 being good at all. And I would think training with int5 would produce garbage. I didn't even know int5 was an option. Do you have any published benchmarks comparing the quality of results between the different precisions, going all the way down to int5?
      - eightysixfour 3y ago
        https://github.com/ggerganov/llama.cpp/pull/1684 https://github.com/ggerganov/llama.cpp/pull/1684 You are broadly better off running a larger, more quantized model, than a smaller model at a higher precision. A 65b parameter model with 2bit quantization still outperforms an fp16 30b parameter model in perplexity and qualitative testing. No one I know does fp32, there's no benefit. People train LoRAs at low precision regularly. The folks on /r/localllama are really pushing the boundaries of what can be done on consumer hardware, and the boundaries are further out than you'd think. Also, if you want to really blow-up your intuition, check out model merges...
        Sohcahtoa82 3y ago
        That's really interesting to know. Kind of mind-blowing, tbh. Does all that apply to training as well, or just inference?
        3y ago
krainboltgreene 3y ago
It seems really unethical to train a model on someone else's words without their permission.
- oasisaimlessly 3y ago
  Not to me.
- infoseek12 3y ago
  It may be just about as unethical as someone discussing something they overheard in a crowd.
  - krainboltgreene 3y ago
    Sure if you have a really warped understanding of context.
- altdataseller 3y ago
  Isn't that basically what OpenAI did?
  - krainboltgreene 3y ago
    It's what the original dataset did, and then they further compounded on that.
- renewiltord 3y ago
  Sounds horrifically unethical to try to censor people like you’re doing.
  - krainboltgreene 3y ago
    That's not what that word means and I'm not surprised you don't know that.
- valine 3y ago
  When I read your comment I trained my own mental model on your words. How is that any different? When a human reads words they apply a sophisticated theory of mind to contextualize the writing and the mental state of the author. If anything, LLM fine tuning is far less invasive than having a person read your writing.
visarga 3y ago
My guess is you'd better use your message + a bunch of other messages with RLHF, yours being the positive one and the others negative. If you don't have other messages, just generate some. You can also do some supervised fine tuning to have the model memorize your facts.

jokethrowaway 3y ago

> It's like the AI was like "uh-huh...yeah...I know some of these words!" and spat something incoherent that seemed related. An accurate depiction of what's going on! I had terrible results with GPT2 but the newer models are much better. I recommend fine tuning mistral (I had better results than any llama based model)

Sohcahtoa82 3y ago
> I recommend fine tuning mistral What kind of hardware does that take? I've got an RTX 3080, i9-9900K, and 32 GB of RAM right now. I eventually plan on upgrading to an RTX 5090 when those come out, as well as i9-15900K (Or whatever they're gonna call it) and 128 GB of RAM once the next Intel CPU comes out. My understanding is that even with an RTX 5090 (Assuming 32 GB of VRAM, nothing has been announced yet), if I use fp32, I still won't have enough GPU memory to do inference on a 7B model, let alone training. Though if I quantize down to fp16, I could do inference, but still not training.
- eightysixfour 3y ago
  Pretty sure an fp32 7b model uses just shy of 27gb of vram. The performance losses at int8 are minuscule, even down to int5 is still very usable. Why would you try and run it fp32 or fp16?
  - Sohcahtoa82 3y ago
    My naive intuition is that you'd want to train on the highest precision you have the horsepower for, and then after training, you might be able to get away with quantizing to a lower precision for inference. I would also expect that fp32 might be overkill, but that anything less than fp16 would start to show a noticeable loss in quality, especially in training. I can't imagine int8 being good at all. And I would think training with int5 would produce garbage. I didn't even know int5 was an option. Do you have any published benchmarks comparing the quality of results between the different precisions, going all the way down to int5?
    - eightysixfour 3y ago
      https://github.com/ggerganov/llama.cpp/pull/1684 https://github.com/ggerganov/llama.cpp/pull/1684 You are broadly better off running a larger, more quantized model, than a smaller model at a higher precision. A 65b parameter model with 2bit quantization still outperforms an fp16 30b parameter model in perplexity and qualitative testing. No one I know does fp32, there's no benefit. People train LoRAs at low precision regularly. The folks on /r/localllama are really pushing the boundaries of what can be done on consumer hardware, and the boundaries are further out than you'd think. Also, if you want to really blow-up your intuition, check out model merges...
      - Sohcahtoa82 3y ago
        That's really interesting to know. Kind of mind-blowing, tbh. Does all that apply to training as well, or just inference?
        3y ago

krainboltgreene 3y ago

It seems really unethical to train a model on someone else's words without their permission.

oasisaimlessly 3y ago
Not to me.
infoseek12 3y ago
It may be just about as unethical as someone discussing something they overheard in a crowd.
- krainboltgreene 3y ago
  Sure if you have a really warped understanding of context.
altdataseller 3y ago
Isn't that basically what OpenAI did?
- krainboltgreene 3y ago
  It's what the original dataset did, and then they further compounded on that.
renewiltord 3y ago
Sounds horrifically unethical to try to censor people like you’re doing.
- krainboltgreene 3y ago
  That's not what that word means and I'm not surprised you don't know that.
valine 3y ago
When I read your comment I trained my own mental model on your words. How is that any different? When a human reads words they apply a sophisticated theory of mind to contextualize the writing and the mental state of the author. If anything, LLM fine tuning is far less invasive than having a person read your writing.

visarga 3y ago

My guess is you'd better use your message + a bunch of other messages with RLHF, yours being the positive one and the others negative. If you don't have other messages, just generate some. You can also do some supervised fine tuning to have the model memorize your facts.