5 ms·

Big fan of this write up as it presents a really easy to understand and at the same time brutally honest example of a domain in which a) you would expect LLMs t

Big fan of this write up as it presents a really easy to understand and at the same time brutally honest example of a domain in which a) you would expect LLMs to perform very well, b) they don't and c) the solution is to make the use of ML more targeted, a complement to human reasoning rather than a replacement for it. Over and over again we see businesses sinking money into "AI" where they are effectively doing a) and then calling it a day, blithely expecting profit to roll in. The day cannot come too soon when these businesses all lose their money and the hype finally dies - and we can go back to using ML the way this write up does (ie the way it is meant to be used). Let's hope no critical systems (eg healthcare or law enforcement) make the same mistake businesses are before that time.

infecto 2y ago
On the flip side I thought the write up was weak on details and while "brutally honest" it did not touch on how they even tried to implement an LLM in the workflow and for all we know they were using an outdated model or a bad implementation. Your bias seems to follow it though, you have jumped so quickly into a camp that its easy to enjoy an article that supports your worldview.
- jerf 2y ago
  To be honest, I exited the article thinking the answer is "no", or at least, perilously close to "no". The same amount of work put into a conventional solution probably would have been better. That cross-product "solution" is a generalized fix for data generation from a weak data source and as near as I can tell is what is actually doing most of the lifting, not the LLM. That said, I'm not convinced there isn't something to the idea, I just don't know that that is the correct use of LLMs. I find myself wondering if from-scratch training, of a much, much smaller model trained on the original data, using LLM technology but not using one of the current monsters, might not work better. I also wonder if this might be a case where prompt engineering isn't the way to go but directly sampling the resulting model might be a better way to go. Or maybe start with GPT-2 and ask it for lists of things; in a weird sort of way, GPT-2's "spaciness" and inaccuracy is sort of advantageous for this. Asking "give me a list of names" and getting "Johongle X. Boodlesmith" would be disastrous from a modern model, but for this task is actually a win. (And I wouldn't ask GPT-2 to try to format the data, I'd probably go for just getting a list of nicely randomized-but-plausible data, and solve all the issues like "tying the references together" conventionally.)
- krainboltgreene 2y ago
  Is this the new normal for comments? Incredibly bad faith.
  - infecto 2y ago
    How so? Their implementation was interesting but I think it missed the whole setup on what did and did not work on the LLM side. Have just a few of those details would have made it very interesting. As it stands its really hard to decide if LLM is or is not the way. If you have such an opinion why not share how I could communicate it better?
    - gopher_space 2y ago
      Your line about the parent commenter's bias was weird and rude. You've never met the person and are accusing them of something you're in the process of doing yourself. https://www.youtube.com/watch?v=_cJO7pkx2jQ https://www.youtube.com/watch?v=_cJO7pkx2jQ
      - infecto 2y ago
        Darn I hate being weird. Thanks!
        hluska 2y ago
        It was very rude.
        emptiestplace 2y ago
        Wow!
    - bcoates 2y ago
      "If it didn't work you didn't believe hard enough" also known as "Real Communism has never been tried" or "Conservatism never fails, it can only be failed" is a sort of... information-free stock position. Basically, if thing is good it needs to still be good when tried in the real world by flawed humans, so if someone says "I tried thing and it didn't work" replying with "well maybe thing is good but you suck" isn't productive.

infecto 2y ago

On the flip side I thought the write up was weak on details and while "brutally honest" it did not touch on how they even tried to implement an LLM in the workflow and for all we know they were using an outdated model or a bad implementation. Your bias seems to follow it though, you have jumped so quickly into a camp that its easy to enjoy an article that supports your worldview.

jerf 2y ago
To be honest, I exited the article thinking the answer is "no", or at least, perilously close to "no". The same amount of work put into a conventional solution probably would have been better. That cross-product "solution" is a generalized fix for data generation from a weak data source and as near as I can tell is what is actually doing most of the lifting, not the LLM. That said, I'm not convinced there isn't something to the idea, I just don't know that that is the correct use of LLMs. I find myself wondering if from-scratch training, of a much, much smaller model trained on the original data, using LLM technology but not using one of the current monsters, might not work better. I also wonder if this might be a case where prompt engineering isn't the way to go but directly sampling the resulting model might be a better way to go. Or maybe start with GPT-2 and ask it for lists of things; in a weird sort of way, GPT-2's "spaciness" and inaccuracy is sort of advantageous for this. Asking "give me a list of names" and getting "Johongle X. Boodlesmith" would be disastrous from a modern model, but for this task is actually a win. (And I wouldn't ask GPT-2 to try to format the data, I'd probably go for just getting a list of nicely randomized-but-plausible data, and solve all the issues like "tying the references together" conventionally.)
krainboltgreene 2y ago
Is this the new normal for comments? Incredibly bad faith.
- infecto 2y ago
  How so? Their implementation was interesting but I think it missed the whole setup on what did and did not work on the LLM side. Have just a few of those details would have made it very interesting. As it stands its really hard to decide if LLM is or is not the way. If you have such an opinion why not share how I could communicate it better?
  - gopher_space 2y ago
    Your line about the parent commenter's bias was weird and rude. You've never met the person and are accusing them of something you're in the process of doing yourself. https://www.youtube.com/watch?v=_cJO7pkx2jQ https://www.youtube.com/watch?v=_cJO7pkx2jQ
    - infecto 2y ago
      Darn I hate being weird. Thanks!
      - hluska 2y ago
        It was very rude.
        emptiestplace 2y ago
        Wow!
  - bcoates 2y ago
    "If it didn't work you didn't believe hard enough" also known as "Real Communism has never been tried" or "Conservatism never fails, it can only be failed" is a sort of... information-free stock position. Basically, if thing is good it needs to still be good when tried in the real world by flawed humans, so if someone says "I tried thing and it didn't work" replying with "well maybe thing is good but you suck" isn't productive.