RE: How much is the training dataset. It's probably over 50GB so it's very unlikely I will be able to overfit the model, the best part is this means the model

RE: How much is the training dataset. It's probably over 50GB so it's very unlikely I will be able to overfit the model, the best part is this means the model is always learning new things and showing new abilities. I noticed that straight after giving it ONLY a coding dataset it became a bit better at logical puzzles so I think there's a side-effect of training an LLM on sequential information like code. The LLM seems more inclined to learn structure for example: Structure of language Structure of poems Structure of music lyrics etc etc.