5 ms·

Ethics and restrictive terms aside, it doesn't seem like GPT 4 was necessary for what the poster did. How much worse or more difficult would it have been to us

Ethics and restrictive terms aside, it doesn't seem like GPT 4 was necessary for what the poster did. How much worse or more difficult would it have been to use Mixtral or 3.5 to generate the first 100 good prompt-response pairs and then manually tweak them as the poster did?

thrwayaistartup 3y ago
Or just... write 100 good prompt-repsonse pairs yourself. 2024 will be the year of synthetic data. 2025 will be the year of "you know you can use your own brain and type out 100 datapoints faster and cheaper than generating and filtering assloads of synthetic data, right?" Maybe we can even skip 2024 :)
- sp332 3y ago
  Databricks had their employees write up 15,000 of them. https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm https://www.databricks.com/blog/2023/04/12/dolly-first-open-...
  - code_runner 3y ago
    I’ve worked at plenty of places where we did a ton of labeling by hand. People concerned with data quality from LLMs should really see the inconsistencies we came up with!
  - icyfox 3y ago
    Favorite part of this piece: > We were initially skeptical whether we would get to 10,000 results. But with nightly leaderboard gamification, we managed to break 15,000 results within a week. Out of fear of eating into our productivity, we closed the contest. I've hosted a few of these corporate data labeling events. If sufficiently gamified / there's a good enough UX, they can be surprisingly engaging. It helps a lot if you have a large employee base though. Distributing results over 5000 employees is exponentially easier than even 50 - in practicality, even larger than the orders of magnitude.
  - ThrowawayTestr 3y ago
    Anybody have this downloaded and can paste a few examples?
    - sp332 3y ago
      You can browse them here: https://huggingface.co/datasets/databricks/databricks-dolly-15k https://huggingface.co/datasets/databricks/databricks-dolly-...
- schreiaj 3y ago
  Yes and no, for text type stuff? Yes you're right. But I think in the vision space synthetic data will remain useful for a lot of things. I'm currently working on building a pipeline for personal projects to go from CAD models of environment to segmented training data. So far it looks almost as useful as real world data at a fraction of the cost of manual labeling.
cjbprime 3y ago
Using 3.5 seems like it would have been okay (technically but not legally given OpenAI ToS), but Mixtral is already somewhere around that level, so it's not achieving a goal of improving towards GPT-4 level responses without having to write those yourself, which seems to have been the goal OP wanted to get to. I've heard that training e.g. Mixtral on Mixtral's own outputs is a really bad idea, don't know full details.
gumballindie 3y ago
I am sorry what ethics? Openai is built on stolen content and your worry is people using openai’s output is the unethical issue? Whoa.
atleta 3y ago
Well, it seems that initially started with GPT4 but his costs were becoming high so he had to do something and had to do it quickly. Technically he could have written a few hundred responses himself while the site was still using GPT4 using the prompts from the users but that could have been slow (expensive)/boring, etc.

thrwayaistartup 3y ago

Or just... write 100 good prompt-repsonse pairs yourself. 2024 will be the year of synthetic data. 2025 will be the year of "you know you can use your own brain and type out 100 datapoints faster and cheaper than generating and filtering assloads of synthetic data, right?" Maybe we can even skip 2024 :)

sp332 3y ago
Databricks had their employees write up 15,000 of them. https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm https://www.databricks.com/blog/2023/04/12/dolly-first-open-...
- code_runner 3y ago
  I’ve worked at plenty of places where we did a ton of labeling by hand. People concerned with data quality from LLMs should really see the inconsistencies we came up with!
- icyfox 3y ago
  Favorite part of this piece: > We were initially skeptical whether we would get to 10,000 results. But with nightly leaderboard gamification, we managed to break 15,000 results within a week. Out of fear of eating into our productivity, we closed the contest. I've hosted a few of these corporate data labeling events. If sufficiently gamified / there's a good enough UX, they can be surprisingly engaging. It helps a lot if you have a large employee base though. Distributing results over 5000 employees is exponentially easier than even 50 - in practicality, even larger than the orders of magnitude.
- ThrowawayTestr 3y ago
  Anybody have this downloaded and can paste a few examples?
  - sp332 3y ago
    You can browse them here: https://huggingface.co/datasets/databricks/databricks-dolly-15k https://huggingface.co/datasets/databricks/databricks-dolly-...
schreiaj 3y ago
Yes and no, for text type stuff? Yes you're right. But I think in the vision space synthetic data will remain useful for a lot of things. I'm currently working on building a pipeline for personal projects to go from CAD models of environment to segmented training data. So far it looks almost as useful as real world data at a fraction of the cost of manual labeling.

cjbprime 3y ago

Using 3.5 seems like it would have been okay (technically but not legally given OpenAI ToS), but Mixtral is already somewhere around that level, so it's not achieving a goal of improving towards GPT-4 level responses without having to write those yourself, which seems to have been the goal OP wanted to get to. I've heard that training e.g. Mixtral on Mixtral's own outputs is a really bad idea, don't know full details.

gumballindie 3y ago

I am sorry what ethics? Openai is built on stolen content and your worry is people using openai’s output is the unethical issue? Whoa.

atleta 3y ago

Well, it seems that initially started with GPT4 but his costs were becoming high so he had to do something and had to do it quickly. Technically he could have written a few hundred responses himself while the site was still using GPT4 using the prompts from the users but that could have been slow (expensive)/boring, etc.