You can feed in a lot of context, but the memory requirements go up and up. It was crashing on me in some experiments, but I saw a pull request that let's you s

You can feed in a lot of context, but the memory requirements go up and up. It was crashing on me in some experiments, but I saw a pull request that let's you set the context size on the command line. I was using the 30B model and with 500 words of context it was using over 100GB of RAM. You probably would be better off fine-tuning the LLaMA model like they did with Alpaca though.