3 ms·

That was very much evident even from back ehwn the first GPT's came out. The moment you started introducing synthetic data, the quality plummeted. But there is

That was very much evident even from back ehwn the first GPT's came out. The moment you started introducing synthetic data, the quality plummeted. But there is another use case where LLM's can truly help with synthetic data: the more classical classification and regression problems - specifically gathering training data. I had this exact case at work two days ago: A large dataset with a small subset of labeled data. For a binary classifier, there was a huge imbalance in the data - the ratio was roughly 75-25%. I did not have the desire to do all this manually so I used an LLM to get a list that would even out the numbers(and get a 50-50 ratio). And using the data I had, plus the additional synthetic data, the accuracy of my small classifier ended up picture-perfect(given that my actual target was "85-90%" accuracy and the actual result was just shy of 99%).

kerkeslager 2y ago
I'd argue that the case you give isn't an example of using a computer to generate data, it's a case of a human adding data (the data being the fact that the binary classifier should have a 50/50 balance). This sort of massaging of data has its drawbacks as well--obviously this only works if the balance of that binary classifier actually is 50/50 in reality: I don't know enough about your case to say you were wrong, but I can imagine a lot of scenarios where a binary classifier should not be represented 50/50 in the data.
- axegon_ 2y ago
  This is a question of definition. It is synthetic in that I just passed a prompt, asking for N amount of examples of X. And I did not go over the entire list I got and blindly trusted it. In this context, I needed an even(or nearly even distribution) of samples in the training data and it worked way better than I was hoping. Mind you, I have to face a similar issue next week and I'm not sure this approach would cut it - I need way more training data and way more classes to work with. 249 classes if I'm not mistaking.
  - kerkeslager 2y ago
    I question what "it worked way better than I was hoping" means in this context. If you're saying that filtering the input data to create a uniform distribution created a uniform distribution in the output, I'm not sure why you'd hope for any less--that's exactly what I'd expect to happen. But that's a poor measure of success, because you don't know what side effects that had: the removed data ostensibly contained other variables besides your binary variable, and you don't know if those variables were sampled in any useful way, so I'd be hesitant to say this worked well without at least an attempt to measure those other variables.
low_tech_love 2y ago
Just curious, but did you compute that 99% using purely real test data, or does your test set also include artificial data?
- axegon_ 2y ago
  Apart from using the results from the training/testing/validation sets? Several people manually went over several thousand random samples.
bobsyourbuncle 2y ago
Can you clarify what you mean by using an LLM to “get a list that would even out the numbers”? If you’re doing binary classification, you need datapoints for features as well as the target class so how does an LLM synthetically create that without causing problems?

kerkeslager 2y ago

I'd argue that the case you give isn't an example of using a computer to generate data, it's a case of a human adding data (the data being the fact that the binary classifier should have a 50/50 balance). This sort of massaging of data has its drawbacks as well--obviously this only works if the balance of that binary classifier actually is 50/50 in reality: I don't know enough about your case to say you were wrong, but I can imagine a lot of scenarios where a binary classifier should not be represented 50/50 in the data.

axegon_ 2y ago
This is a question of definition. It is synthetic in that I just passed a prompt, asking for N amount of examples of X. And I did not go over the entire list I got and blindly trusted it. In this context, I needed an even(or nearly even distribution) of samples in the training data and it worked way better than I was hoping. Mind you, I have to face a similar issue next week and I'm not sure this approach would cut it - I need way more training data and way more classes to work with. 249 classes if I'm not mistaking.
- kerkeslager 2y ago
  I question what "it worked way better than I was hoping" means in this context. If you're saying that filtering the input data to create a uniform distribution created a uniform distribution in the output, I'm not sure why you'd hope for any less--that's exactly what I'd expect to happen. But that's a poor measure of success, because you don't know what side effects that had: the removed data ostensibly contained other variables besides your binary variable, and you don't know if those variables were sampled in any useful way, so I'd be hesitant to say this worked well without at least an attempt to measure those other variables.

low_tech_love 2y ago

Just curious, but did you compute that 99% using purely real test data, or does your test set also include artificial data?

axegon_ 2y ago
Apart from using the results from the training/testing/validation sets? Several people manually went over several thousand random samples.

bobsyourbuncle 2y ago

Can you clarify what you mean by using an LLM to “get a list that would even out the numbers”? If you’re doing binary classification, you need datapoints for features as well as the target class so how does an LLM synthetically create that without causing problems?