3 ms·
I don't think anyone is throwing raw datasets into LLMs and hoping for high quality weights anymore. Nowadays most of the datasets are filtered one way or anoth
by CaptainOfCoit 1y ago
I don't think anyone is throwing raw datasets into LLMs and hoping for high quality weights anymore. Nowadays most of the datasets are filtered one way or another, and some of them highly curated even.
- BoredPositron 1y agoI doubt they are highly created you would need experts in every field to do so. Which gives me more performance anxiety for LLMs because one of the most curated fields should be code...
- nradov 1y agoOpenAI has been literally hiring human experts in certain targeted subject areas to write custom proprietary training content.
- BoredPositron 1y agoI bet the dataset is mostly comprised of certain areas™.
- groby_b 1y agoThe major labs are hiring experts. They carefully build & curate synthetic data. The market for labelled non-synthetic data is currently ~$3B/year. The idea that LLMs are just trained on a pile of raw Internet is severely outdated. (Not sure it was ever fully true, but it's far away from that by now). Coding's one of the easier datasets to curate, because we have a number of ways to actually (somewhat) assess code quality. (Does it work? Does it come with a set of tests and pass it? Does it have stylistic integrity? How many issues get flagged by various analysis tools? Etc, etc)
- satellite2 1y agoIs that right? Isn't the current way of doing thing to throw "everything" at it then fine tune?