5 ms·
What’s the use case here? The author notes doing inference from disk is too slow. Fine tune a model and do nothing with it? Fine tune and then upload it to a cl
by sbierwagen 3y ago
What’s the use case here? The author notes doing inference from disk is too slow. Fine tune a model and do nothing with it? Fine tune and then upload it to a cloud machine? Not sure how often I’d like to try uploading multi gigabyte files from a consumer internet connection with narrow upload bandwidth.
- turnsout 3y agoNot sure about 70b, but this is a really useful repo for fine-tuning the 7b models on M1 machines—and all the examples in the README seem to use 7b.
- visarga 3y agoOnly LLaMA2, or does it also work on Mistral-7B?
- miloignis 3y agoDepending on how much ram you have, fine-tune then quanitize & run with llama.cpp, which works quite well.
- thelastparadise 3y ago> Not sure how often I’d like to try uploading multi gigabyte files from a consumer internet connection with narrow upload bandwidth It's 2023... this is a fairly standard use case (e.g. YouTubers uploading videos, etc).
- askiiart 3y agoI'm on the mid-tier plan available to me. 500 Mbps down, 20 Mbps up. It would take me, bare minimum, about 6.67 minutes to upload 1 gigabyte. I'll be the first to admit that I don't know much about AI/ML, so I could be wrong, but assuming a fine-tuned model is the same size as the base model, it would take me 90 minutes to upload the llama2 7B model (13.5 GB). If it were standard practice for many people, we might finally get symmetric cable. But most people aren't YouTubers. In my experience, people who aren't into tech usually have no idea their upload speeds are any slower than their download speeds, much less 25 times slower. They usually just let their phone upload their photos in the background, and it just works. Hell, I can't even find my upload speed on the consumer section of my ISP's site, only in a pdf for their business plans. Not even in legalese.
- mlyle 3y agoSpectrum? Lots of cable internet right now has upstream limitations, but fiber is making it to more homes and upstream limitations are (slowly) improving on cable. As a society we have something like a 20% CAGR on upstream speeds over the past few years. It may get better even faster, because DOCSIS 4 is coming soon and is full duplex over the same channels. But even at 20mbps, uploading a 70B model might take you 12 hours... after you spent many hours fine-tuning it. It's annoying but not an unmanageable problem.
- obscur 3y agoIt uses LoRA which is much smaller than full model
- manmal 3y agoIn my country there’s a tendency to provide max 50MBit/s up on consumer connections, but a few do offer more. I‘m on a FTTH plan with 130 up now, and think this would suffice for most current use cases. There’s also the possibility to bundle multiple connections with enterprise grade hardware - last time I looked that was a 1k one time investment. I‘m dubious as to whether that would actually boost uploading speeds though.
- stavros 3y agoSame here, I'm on 600 Mbps up/down and I enjoy the up more than the down.
- j245 3y ago> But most people aren't YouTubers. In my experience, people who aren't into tech usually have no idea their upload speeds are any slower than their download speeds, much less 25 times slower. People who aren’t into tech aren’t trying to fine tune Llama2 and then upload it to a cloud machine
- kaffekaka 3y agoBe a lot cooler if they did.
- moneywoes 3y agofine tune for local use?