5 ms·

I’d keep an eye out on huggingface. I’ve yet to see alpaca 65B, but there are copies of 30B (and 7/13B) floating around already. I’m sure someone will eventua

I’d keep an eye out on huggingface. I’ve yet to see alpaca 65B, but there are copies of 30B (and 7/13B) floating around already. I’m sure someone will eventually pony up for the compute time to do 65B. I’ve found watching these models, https://huggingface.co/decapoda-research https://huggingface.co/decapoda-research and then looking at the “spaces using …” section is a good way to find what’s being worked on (for example, check the comments for ipfs/torrents links). Also just doing a search for “alpaca” could work. You won’t be able to do the finetuning on your MacBook, but should be able to run inference on a 4 bit quantized model.

nico 4y ago
> I’m sure someone will eventually pony up for the compute time to do 65B. How much would it cost? Can you give a breakdown of hardware/paas requirements and the costs?
- nullstyle 4y ago
  I was able to finetune the pretrained 13B model (using the same alpaca dataset, just running on my dime) in about 14 hours on an A6000. I think it will work out to between $50 and $60 when the bill hits me. Be warned everyone is slapping code together so fast that, if your experience is like mine, you'll spend most of your time working around assumptions made by prior authors or hand merging patches between forks to get your setup running well.
  - nico 4y ago
    Thank you! Very insightful. Crazy pace.
- gorbypark 4y ago
  Another datapoint, first hand report from a random internet person that 7B took 2.5-3hrs on a 8xA100 80Gb setup. If it ends up being a straight up linear thing, that means about ~27 hours for the 65B model. Depending on the host and if it's a preemptable instance or not, that could be about $12-30 per hour.
  - nullstyle 4y ago
    Have a link? I haven't seen any finetuning scripts in the wild that train a PEFT model on a multrigpu setup yet and would love to play around with one.
    - gorbypark 4y ago
      The original Alpaca repo has the training script. The readme has the torchrun command and arguments used for train.py. https://github.com/tatsu-lab/stanford_alpaca/blob/main/train.py https://github.com/tatsu-lab/stanford_alpaca/blob/main/train...
      - nico 4y ago
        Awesome, thank you!
reasonabl_human 4y ago
I’ve been sidetracked with work but planning on tuning llama 65B to produce alpaca 65B, will distribute via huggingface or torrent.. FWIW running the 30B alpaca-lora model quantized to 4-bit via llama.cpp has given me great results, and while I don’t expect much of an improvement from 65B at FP16, 65B will probably perform better than 30B when quantized The interesting next steps in my head are more focused around curating a better instruction-tuning dataset using GPT-4, then fine-tuning again, and integrating the LangChain project with the resulting agent
- gorbypark 4y ago
  Sounds great! It would be great if you could release your intermediate steps, so Alpaca-65B and then Alpaca-65B-LangChain, etc. I'd like to compare the model sizes directly when fine-tuned using the same method / dataset! I also just realized that I don't believe there's an "alpaca-native" 30B floating around, just the alpaca-lora one, so 30B would be pretty cool too (and the biggest I can run w/ llama.cpp on my MacBook).
Jonny_Cosmo 4y ago
Is the coding itself pretty trivial, cos I've got a decent pc and fancy giving it a go. Got threadripper 3960x with 128GB RAM and 2x quadro 8000s on nvlink. It's not an AI powerhouse by any means but should be better than a Macbook

nico 4y ago

> I’m sure someone will eventually pony up for the compute time to do 65B. How much would it cost? Can you give a breakdown of hardware/paas requirements and the costs?

nullstyle 4y ago
I was able to finetune the pretrained 13B model (using the same alpaca dataset, just running on my dime) in about 14 hours on an A6000. I think it will work out to between $50 and $60 when the bill hits me. Be warned everyone is slapping code together so fast that, if your experience is like mine, you'll spend most of your time working around assumptions made by prior authors or hand merging patches between forks to get your setup running well.
- nico 4y ago
  Thank you! Very insightful. Crazy pace.
gorbypark 4y ago
Another datapoint, first hand report from a random internet person that 7B took 2.5-3hrs on a 8xA100 80Gb setup. If it ends up being a straight up linear thing, that means about ~27 hours for the 65B model. Depending on the host and if it's a preemptable instance or not, that could be about $12-30 per hour.
- nullstyle 4y ago
  Have a link? I haven't seen any finetuning scripts in the wild that train a PEFT model on a multrigpu setup yet and would love to play around with one.
  - gorbypark 4y ago
    The original Alpaca repo has the training script. The readme has the torchrun command and arguments used for train.py. https://github.com/tatsu-lab/stanford_alpaca/blob/main/train.py https://github.com/tatsu-lab/stanford_alpaca/blob/main/train...
    - nico 4y ago
      Awesome, thank you!

reasonabl_human 4y ago

I’ve been sidetracked with work but planning on tuning llama 65B to produce alpaca 65B, will distribute via huggingface or torrent.. FWIW running the 30B alpaca-lora model quantized to 4-bit via llama.cpp has given me great results, and while I don’t expect much of an improvement from 65B at FP16, 65B will probably perform better than 30B when quantized The interesting next steps in my head are more focused around curating a better instruction-tuning dataset using GPT-4, then fine-tuning again, and integrating the LangChain project with the resulting agent

gorbypark 4y ago
Sounds great! It would be great if you could release your intermediate steps, so Alpaca-65B and then Alpaca-65B-LangChain, etc. I'd like to compare the model sizes directly when fine-tuned using the same method / dataset! I also just realized that I don't believe there's an "alpaca-native" 30B floating around, just the alpaca-lora one, so 30B would be pretty cool too (and the biggest I can run w/ llama.cpp on my MacBook).

Jonny_Cosmo 4y ago

Is the coding itself pretty trivial, cos I've got a decent pc and fancy giving it a go. Got threadripper 3960x with 128GB RAM and 2x quadro 8000s on nvlink. It's not an AI powerhouse by any means but should be better than a Macbook