4 ms·
Ask HN: How can I experiment with LLMs with a old machine?
Dear all,
Recently I purchased "[Build a Large Language Model (From Scratch)](https://www.manning.com/books/build-a-large-language-model-from-scratch)" by Sebastian Raschka, so that I could learn more about how to build and/or fine-tune a LLM, and even developing some applications with them. I have also been skimming and reading on this sub for several months, and have witnessed many interesting developments that I would like to follow and experiment with.
However, there is a problem: The machine I have is a very old Macbook Pro from 2011 and I probably would not be able to afford a new one until I'm in graduate school next year. So I was wondering that, other than getting a new machine, what are the other (online/cloud) alternatives and/or options that I could use, to experiments with LLMs?
Many thanks!
- tarun_anand 2y agoI am on a 2019 mac and finding it difficult too. Best bet would be to start with a small language model?
- hedgehog0 2y agoMay I ask how small? From what I see on r/localllama, it seems that at least 3B and 7B ones are starting to be useful? Also I cannot use CUDA or modern AMD cards..
- LargoLasskhyfv 2y agoMake yourself comfortable with https://blogs.oracle.com/database/post/freedom-to-build-announcing-oracle-cloud-free-tier-with-new-always-free-services-and-always-free-oracle-autonomous-database https://blogs.oracle.com/database/post/freedom-to-build-anno... https://gist.github.com/rssnyder/51e3cfedd730e7dd5f4a816143b25dbd https://gist.github.com/rssnyder/51e3cfedd730e7dd5f4a816143b... https://www.reddit.com/r/oraclecloud/ https://www.reddit.com/r/oraclecloud/ or any other offer. Deploy some minimal Linux on them, or use what's offered. Plus optionally, if you don't want to instantly start coding from first principles/scratch, make use of established and excellent solutions, like https://future.mozilla.org/builders/news_insights/introducing-llamafile/ https://future.mozilla.org/builders/news_insights/introducin... https://ai-guide.future.mozilla.org/content/running-llms-locally/ https://ai-guide.future.mozilla.org/content/running-llms-loc... https://github.com/mozilla-Ocho/llamafile https://github.com/mozilla-Ocho/llamafile https://justine.lol/matmul/ https://justine.lol/matmul/ and parallelize them with https://github.com/b4rtaz/distributed-llama https://github.com/b4rtaz/distributed-llama Obviously this needs some knowledge of the command line, so get a good terminal emulator like https://iterm2.com/ https://iterm2.com/ Mend, bend, rend that stuff and see what works how and why, and what not. Edit: Optionally, if you really want to go low-level, with some debugger like https://justine.lol/blinkenlights/ https://justine.lol/blinkenlights/ for 'toy-installations' of smallest models. 'Toy' because that doesn't fully support the CPU-instructions which are used in production. Could still help conceptually.
- hedgehog0 2y agoThank you for the Mozilla AI guide, it's really useful! I knew some of Justine's work and they are really cool.
- roosgit 2y agoI've never used it, but I think Google Colab has a free plan. As another option, you can rent a machine with a decent GPU on vast.ai. An Nvidia 3090 can be rented for about $0.20/hr.