> Are you using a normal training script i.e. "continued pretraining" on ALL parameters with just document fragments rather than input output pairs? Yes, this

> Are you using a normal training script i.e. "continued pretraining" on ALL parameters with just document fragments rather than input output pairs? Yes, this one. > do you make a custom dataset that has qa pairs about that particular knowledgebase? This one. Once you have a checkpoint w knowledge, it makes sense to finetune. You can use either LORA or PEFT. We do it depending on the case. (some orgs have like millions of tokens and i am not that confident that PEFT). LoRA with raw document text may not work, haven't tried that. Google has a good example of training scripts here: https://github.com/google-research/t5x https://github.com/google-research/t5x (under training. and then finetuning). I like this one. Facebook Research also has a few on their repo. If you are just looking to scrape by, I would suggest just do what they tell you to do. You can offer suggestions, but better let them take the call. A lot of fluff, a lot of chatter online, so everyone is figuring out stuff.