I think it's fair to implement complex preprocessing and external integrations (such as memory). But you can actually prototype llms at a tiny scale! A cluster

I think it's fair to implement complex preprocessing and external integrations (such as memory). But you can actually prototype llms at a tiny scale! A cluster of 3090's (SLI) or 4090's with some kind of splicing (4-8 input(s) are spread across 4 gpus and then merged together). The cost to entry is roughly around $15k. While you won't be making anything even close to llama1 it is possible to develop technologies that can be used to train llama4. I've personally been using gh copilot and it's an amazing tool for the 'braindead' parts of my workflow. I usually let it generate a rough template and then fill in the parts by entirely replacing them, but it at times manages to do exactly what I wanted when working in larger files which already have much of the functionality defined and it's rather trivial to figure out the rest. Copilot does what it says, it's a copilot. It feels like I am iterating over a problem with a 2nd developer suggesting ideas as I code.