4 ms·
There's at least an argument to be made that this is because all the models are heavily trained on GPT-4 outputs (or whatever the SOTA happens to be during trai
by n2d4 3y ago
There's at least an argument to be made that this is because all the models are heavily trained on GPT-4 outputs (or whatever the SOTA happens to be during training). All those models are, in a way, a product of inbreeding.
- pram 3y agoConsider the bulldog: https://youtube.com/watch?v=hUgmkCgMWbg https://youtube.com/watch?v=hUgmkCgMWbg
- sumo43 3y agoMaybe true for instruct, but pretraining datasets do not usually contain GPT-4 outputs. So the base model does not rely on GPT-4 in any way.
- fragmede 3y agoBut is it the kind of inbreeding that gets you Downs, or the kwisatz haderach?
- batshit_beaver 3y agoYes