3 ms·
I didn't know 2 thirds of the training data would be source code.
by speedgoose 2mo ago
I didn't know 2 thirds of the training data would be source code.
- jerrygenser 2mo agothat is the the "data used to improve the model" when signing up for the subscription plans
- leothetechguy 2mo agothis is the rl run, not the pretraining run
- ahmadyan 2mo agoeven in pre-training, usually 30%-50% is code these days.
- leothetechguy 2mo agoThat would be far too high in my opinion. But happy if anybody can give insights from their own experience with pretraining runs.