5 ms·

If you use the web interface (oobabooga), then training a LoRa is as easy as clicking the "training" tab, keeping all the defaults, and giving it a flat text fi

If you use the web interface (oobabooga), then training a LoRa is as easy as clicking the "training" tab, keeping all the defaults, and giving it a flat text file of your data. The defaults are sane enough to not begin undermining any instruction tuning too much. Takes 3-5 hours on a 3080 for 7B, 4bit model (and ~1KWh). So far I've trained 3: 2 on the entire text of ASOIAF (converted from e-books) and 1 on the Harry Potter series. I can ask questions like "tell me a story about a long winter in Westeros" and get something in the "voice" of GRRM and with real references to the text. It can write HP fanfics all day long. My favorite so far was the assistant self-inserting into a story with Jon Snow, complete with "The Assistant has much data for you. Please wait while it fetches it." and actually having a conversation with Jon. Asking specific questions is way more of a miss (e.x. "Who are Jon Snow's real parents?" returns total BS), but that may be because my 3080 is too weak to train anything other than 7B models in 4bit (which is only supported with hacked patches). I used Koala as my base model. I'm getting close to dropping $1600 on a 4090, but I should find employment first... but then I'll have less time to mess with it.

hotfixguru 3y ago
Which model did you use? I read your comment when this HN thread was alive and decided to try to feed it a few websites (such as PEP8[0] for Python), and it only took a few minutes for LoRa training to complete, and the answers were not good. I also have a 3080. [0]: https://peps.python.org/pep-0008/ https://peps.python.org/pep-0008/
seydor 3y ago
how much memory does the 7B training need?
- eulers_secret 3y ago
  ~7.5GB - it'll be the same as running inference with a full context. That's for 4-bit quantization, the 8-bit quantization uses more RAM than my 3080 has...
  - seydor 3y ago
    I wonder how much it would take to train the 4b 13B
    - MacsHeadroom 3y ago
      About 15GB training it in the webui. If you use https://github.com/johnsmith0031/alpaca_lora_4bit https://github.com/johnsmith0031/alpaca_lora_4bit then 30B only needs 24GB, and works on a single 3090 or $200 P40.
Tiktaalik 3y ago
That's really interesting. I guess it would do really well with world building lore type stuff, being able to go into great depth about the Brackens vs the Blackwoods but would struggle at the sort of subtext that even human readers may have missed (eg. who poisoned Tywin? and as you said, who are Jon Snow's parents?)
Kuinox 3y ago
Used 3090 are getting really cheaps on the second hand market. Then if you only need VRAM, the Tesla M40 are even cheaper at 100€ per unit, which has 24GB of VRAM.
- MacsHeadroom 3y ago
  The M40 does not support 4bit, so it's basically useless for LLMs. The P40 24GB is only $200, supports 4bit, and is about 80% the speed of a 3090 (surprisingly) for LLM purposes.
  - Kuinox 3y ago
    > The M40 does not support 4bit, so it's basically useless for LLMs. Good to know, thanks !
  - awestroke 3y ago
    What's the catch?
    - zirgs 3y ago
      No video output, because it's a data center card.
sbrother 3y ago
Will it distribute training across multiple GFX cards? I have a 4x 2080Ti box I would love to be able to use for this sort of thing.
- eulers_secret 3y ago
  Not for training with the webui: https://github.com/oobabooga/text-generation-webui/issues/1107 https://github.com/oobabooga/text-generation-webui/issues/11... It does seem to work using alpca-lora directly, though.
se4u 3y ago
I am surprised that people aren't using google colab pro/pro+ in this context. You basically get access to multiple A100 for $10/month and with some simple javascript tricks, you can get a session to last for 24hrs at least. Pro+ is more expensive at $50/mo but it allows for more simplified background execution. if you are only just getting started and don't expect to be training for multiple months, then colab or other cloud-notebook providers are really great to start.
- addandsubtract 3y ago
  Google has recently limited the pro plan to 100 compute units per month. Using an A100 on colab burns 13 units an hour. So you could be out of units within 8 hours. Not really the $10/month deal you're looking for.

hotfixguru 3y ago

Which model did you use? I read your comment when this HN thread was alive and decided to try to feed it a few websites (such as PEP8[0] for Python), and it only took a few minutes for LoRa training to complete, and the answers were not good. I also have a 3080. [0]: https://peps.python.org/pep-0008/ https://peps.python.org/pep-0008/

seydor 3y ago

how much memory does the 7B training need?

eulers_secret 3y ago
~7.5GB - it'll be the same as running inference with a full context. That's for 4-bit quantization, the 8-bit quantization uses more RAM than my 3080 has...
- seydor 3y ago
  I wonder how much it would take to train the 4b 13B
  - MacsHeadroom 3y ago
    About 15GB training it in the webui. If you use https://github.com/johnsmith0031/alpaca_lora_4bit https://github.com/johnsmith0031/alpaca_lora_4bit then 30B only needs 24GB, and works on a single 3090 or $200 P40.

Tiktaalik 3y ago

That's really interesting. I guess it would do really well with world building lore type stuff, being able to go into great depth about the Brackens vs the Blackwoods but would struggle at the sort of subtext that even human readers may have missed (eg. who poisoned Tywin? and as you said, who are Jon Snow's parents?)

Kuinox 3y ago

Used 3090 are getting really cheaps on the second hand market. Then if you only need VRAM, the Tesla M40 are even cheaper at 100€ per unit, which has 24GB of VRAM.

MacsHeadroom 3y ago
The M40 does not support 4bit, so it's basically useless for LLMs. The P40 24GB is only $200, supports 4bit, and is about 80% the speed of a 3090 (surprisingly) for LLM purposes.
- Kuinox 3y ago
  > The M40 does not support 4bit, so it's basically useless for LLMs. Good to know, thanks !
- awestroke 3y ago
  What's the catch?
  - zirgs 3y ago
    No video output, because it's a data center card.

sbrother 3y ago

Will it distribute training across multiple GFX cards? I have a 4x 2080Ti box I would love to be able to use for this sort of thing.

eulers_secret 3y ago
Not for training with the webui: https://github.com/oobabooga/text-generation-webui/issues/1107 https://github.com/oobabooga/text-generation-webui/issues/11... It does seem to work using alpca-lora directly, though.

se4u 3y ago

I am surprised that people aren't using google colab pro/pro+ in this context. You basically get access to multiple A100 for $10/month and with some simple javascript tricks, you can get a session to last for 24hrs at least. Pro+ is more expensive at $50/mo but it allows for more simplified background execution. if you are only just getting started and don't expect to be training for multiple months, then colab or other cloud-notebook providers are really great to start.

addandsubtract 3y ago
Google has recently limited the pro plan to 100 compute units per month. Using an A100 on colab burns 13 units an hour. So you could be out of units within 8 hours. Not really the $10/month deal you're looking for.