4 ms·

>> GPT-Migrate is currently in development alpha and is not yet ready for production use. For instance, on the relatively simple benchmarks, it gets through "ea

>> GPT-Migrate is currently in development alpha and is not yet ready for production use. For instance, on the relatively simple benchmarks, it gets through "easy" languages like python or javascript without a hitch ~50% of the time, and cannot get through more complex languages like C++ or Rust without some human assistance. In other words, it doesn't really work. The current wave of LLM applications still seems to me like someone just invented homeopathy and a whole bunch of people are convinced it's real and are trying to use it to create a cure for cancer. It's just people waving their hands about and intoning magick formulae, that don't work and don't produce anything useful at all. I am curious to see where all this is going to end up. Is someone going figure out a way to make LLMs work for real-world er work? Are we all waiting patiently the next big LLM version to see if it can do the things that the current best-of-the-best can't?

golergka 3y ago
A prototype that works 50% of the time is not homeopathy. A new cure for some types of cancer that would succeed 50% of time would be an achievement.
- groestl 3y ago
  Unless it straight up kills the other 50%.
  - golergka 3y ago
    Than it's certainly not homeopathy. It doesn't kill anyone.
- YeGoblynQueenne 3y ago
  The readme says "~50%" of the time, which seems to be a number pulled out of a hat (to be nice) rather than any serious attempt at quantifying performance (there are no metrics of any kind and the authors are asking for relevant benchmarks). It's more like the authors' feeling, than anything they have ascertained in any reliable way. That's pretty much the same way homeopaths test their "remedies" (i.e. water).
padolsey 3y ago
I don’t understand why this talking point keeps surfacing. LLMs are in significant usage already in many production environments and are already improving peoples’ lives and productivity. I can have a dig for more examples but I’d urge you to do a bit of research and try out the gpt-4 API for yourself. Or make some more qualified statements with specific bugbears. Equating it all to homeopathy is so confusing. - Image descriptions eg. Be My Eyes: https://www.bemyeyes.com/ https://www.bemyeyes.com/ - Summarisation and content distillation, question answering Etc: literally everywhere. This is now a solved problem (to the point of it being dull) thanks to LLMs. - Customer service chat triage. - helping students learn- eg khan academy, tutor Lily - Helping create and debug software. If you don’t think is happening then you’re either living under a rock or just close to believe people are lying about how they’ve used it?
- skepticATX 3y ago
  I'd push back on summarization being solved. I have extensively used GPT 3.5 and 4 for this in production, and it works very well but it still often just ignores critical aspects of the original text, and it will rarely even completely hallucinate details. And this is with a lot of iteration on the prompt. We still have to throw out > 50% of its output because a human can summarize the text much better.
- dns_snek 3y ago
  > - Summarisation and content distillation, question answering Etc: literally everywhere. This is now a solved problem (to the point of it being dull) thanks to LLMs. Calling this problem domain "solved" only undermines whatever point you're trying to make, unless you really believe that current LLMs which fabricate information, flip flop between answers with follow up questions, and even gaslight the user, can be called "solutions". LLMs have proven themselves to be such untrustworthy "sources" of information and knowledge that I'm struggling to understand why anyone would even try to make this particular claim. It's trivial to refute by anyone who's used them and has been thoroughly refuted by StackOverflow Developer survey which reported that only 2.8% of developers "highly trust" the output of AI tools. https://survey.stackoverflow.co/2023/#section-developer-tools-accuracy-of-ai-tools https://survey.stackoverflow.co/2023/#section-developer-tool...
famouswaffles 3y ago
[flagged]
- deleted 3y ago
  [deleted]
klyrs 3y ago
I came here looking for a rust->forth pipleline, are you telling me that isn't available? /s I've long believed that interlingual glue should occur within llvm at the IR level. Reversing each layer of a compiler feels plausible if complex.

golergka 3y ago

A prototype that works 50% of the time is not homeopathy. A new cure for some types of cancer that would succeed 50% of time would be an achievement.

groestl 3y ago
Unless it straight up kills the other 50%.
- golergka 3y ago
  Than it's certainly not homeopathy. It doesn't kill anyone.
YeGoblynQueenne 3y ago
The readme says "~50%" of the time, which seems to be a number pulled out of a hat (to be nice) rather than any serious attempt at quantifying performance (there are no metrics of any kind and the authors are asking for relevant benchmarks). It's more like the authors' feeling, than anything they have ascertained in any reliable way. That's pretty much the same way homeopaths test their "remedies" (i.e. water).

padolsey 3y ago

I don’t understand why this talking point keeps surfacing. LLMs are in significant usage already in many production environments and are already improving peoples’ lives and productivity. I can have a dig for more examples but I’d urge you to do a bit of research and try out the gpt-4 API for yourself. Or make some more qualified statements with specific bugbears. Equating it all to homeopathy is so confusing. - Image descriptions eg. Be My Eyes: https://www.bemyeyes.com/ https://www.bemyeyes.com/ - Summarisation and content distillation, question answering Etc: literally everywhere. This is now a solved problem (to the point of it being dull) thanks to LLMs. - Customer service chat triage. - helping students learn- eg khan academy, tutor Lily - Helping create and debug software. If you don’t think is happening then you’re either living under a rock or just close to believe people are lying about how they’ve used it?

skepticATX 3y ago
I'd push back on summarization being solved. I have extensively used GPT 3.5 and 4 for this in production, and it works very well but it still often just ignores critical aspects of the original text, and it will rarely even completely hallucinate details. And this is with a lot of iteration on the prompt. We still have to throw out > 50% of its output because a human can summarize the text much better.
dns_snek 3y ago
> - Summarisation and content distillation, question answering Etc: literally everywhere. This is now a solved problem (to the point of it being dull) thanks to LLMs. Calling this problem domain "solved" only undermines whatever point you're trying to make, unless you really believe that current LLMs which fabricate information, flip flop between answers with follow up questions, and even gaslight the user, can be called "solutions". LLMs have proven themselves to be such untrustworthy "sources" of information and knowledge that I'm struggling to understand why anyone would even try to make this particular claim. It's trivial to refute by anyone who's used them and has been thoroughly refuted by StackOverflow Developer survey which reported that only 2.8% of developers "highly trust" the output of AI tools. https://survey.stackoverflow.co/2023/#section-developer-tools-accuracy-of-ai-tools https://survey.stackoverflow.co/2023/#section-developer-tool...

famouswaffles 3y ago

[flagged]

deleted 3y ago
[deleted]

klyrs 3y ago

I came here looking for a rust->forth pipleline, are you telling me that isn't available? /s I've long believed that interlingual glue should occur within llvm at the IR level. Reversing each layer of a compiler feels plausible if complex.