3 ms·

Can you provide a specific example of where an LLM failed? If you show us your prompt, your "want/need", and the result, we can better judge your situation. M

by throwaway2037 2y ago

Can you provide a specific example of where an LLM failed? If you show us your prompt, your "want/need", and the result, we can better judge your situation. My guess: Your domain is weird/rare, so LLMs are terrible because their training data is very limited.

thesz 2y ago
> Your domain is weird/rare, so LLMs are terrible because their training data is very limited. And this is how knowledge collapse [1] shows its' head. [1] https://arxiv.org/abs/2404.03502 https://arxiv.org/abs/2404.03502 I am not a big fan of LLMs so I try them once in a while, asking to "implement blocked clause decomposition in Haskell." They can recite a paper (and several other papers, with references) almost verbatim, but they do not posess enough comprehension of what is going on in these papers. As time passes by, with each new LLM, the level of comprehension drops. Last time LLM tried to persuade me to write code instead of providing me with code.
- jcheng 2y ago
  I gave Claude 3.5 Sonnet your prompt and it generated this: https://claude.site/artifacts/7aa41881-937e-4863-a407-c999ea5cc4c6 https://claude.site/artifacts/7aa41881-937e-4863-a407-c999ea... With this example usage: -- Example usage: let clause1 = Set.fromList [1, 2] -- represents (x1 ∨ x2) let clause2 = Set.fromList [-1, 3] -- represents (¬x1 ∨ x3) let formula = Set.fromList [clause1, clause2] -- Decompose the formula let (nonBlocked, blocked) = decompose formula How did it do?
  - thesz 2y ago
    This is very good. Did it do just from the prompt or you had to nudge it? Can you share full chat history?
    - jcheng 2y ago
      No nudging. This was the whole interaction: https://gist.github.com/jcheng5/c6f15f4c3dc31bf15ab44683ad6a7ed3 https://gist.github.com/jcheng5/c6f15f4c3dc31bf15ab44683ad6a...
      - thesz 2y ago
        This is a gist, not an interaction with Claude AI. Can you share actual interaction on their site?
        jcheng 2y ago
        No, ChatGPT has that feature but not Claude (that I could find) so I pasted my input and its response into a gist. It’s verbatim, I promise.
        thesz 2y ago
        https://www.reddit.com/r/ClaudeAI/comments/156j5st/shareclaude_easily_share_your_claude/ https://www.reddit.com/r/ClaudeAI/comments/156j5st/shareclau... https://support.anthropic.com/en/articles/9450526-how-can-i-export-my-claude-ai-data https://support.anthropic.com/en/articles/9450526-how-can-i-... > It’s verbatim, I promise. Of course, I believe you. It is verbatim.
guappa 2y ago
> My guess: Your domain is weird/rare, so LLMs are terrible because their training data is very limited. The training data is always limited, because if there was an existing software that already did what I'm doing, I'd be using it instead of writing it :) So by definition, unless I'm learning how to program and doing exercises that thousands have done before me, I'm doing something for which there is no training data.
- danielbln 2y ago
  It's great that you work on truly novel, unseen problems. The rest of us peasants tends to recombine previously solved problems into to novel solutions to solve business needs, and that works rather well with LLMs.
  - throwaway2037 2y ago
    > The rest of us peasants tends to recombine previously solved problems into to novel solutions to solve business needs, and that works rather well with LLMs. You took the words right out of my mouth! I wrote a draft reply to OP (but discarded it) with similar thoughts -- roughly: I work on CRUD apps, and so do most other devs; LLMs are a terrific fit for this subject matter.
deleted 2y ago
[deleted]

thesz 2y ago

> Your domain is weird/rare, so LLMs are terrible because their training data is very limited. And this is how knowledge collapse [1] shows its' head. [1] https://arxiv.org/abs/2404.03502 https://arxiv.org/abs/2404.03502 I am not a big fan of LLMs so I try them once in a while, asking to "implement blocked clause decomposition in Haskell." They can recite a paper (and several other papers, with references) almost verbatim, but they do not posess enough comprehension of what is going on in these papers. As time passes by, with each new LLM, the level of comprehension drops. Last time LLM tried to persuade me to write code instead of providing me with code.

jcheng 2y ago
I gave Claude 3.5 Sonnet your prompt and it generated this: https://claude.site/artifacts/7aa41881-937e-4863-a407-c999ea5cc4c6 https://claude.site/artifacts/7aa41881-937e-4863-a407-c999ea... With this example usage: -- Example usage: let clause1 = Set.fromList [1, 2] -- represents (x1 ∨ x2) let clause2 = Set.fromList [-1, 3] -- represents (¬x1 ∨ x3) let formula = Set.fromList [clause1, clause2] -- Decompose the formula let (nonBlocked, blocked) = decompose formula How did it do?
- thesz 2y ago
  This is very good. Did it do just from the prompt or you had to nudge it? Can you share full chat history?
  - jcheng 2y ago
    No nudging. This was the whole interaction: https://gist.github.com/jcheng5/c6f15f4c3dc31bf15ab44683ad6a7ed3 https://gist.github.com/jcheng5/c6f15f4c3dc31bf15ab44683ad6a...
    - thesz 2y ago
      This is a gist, not an interaction with Claude AI. Can you share actual interaction on their site?
      - jcheng 2y ago
        No, ChatGPT has that feature but not Claude (that I could find) so I pasted my input and its response into a gist. It’s verbatim, I promise.
        thesz 2y ago
        https://www.reddit.com/r/ClaudeAI/comments/156j5st/shareclaude_easily_share_your_claude/ https://www.reddit.com/r/ClaudeAI/comments/156j5st/shareclau... https://support.anthropic.com/en/articles/9450526-how-can-i-export-my-claude-ai-data https://support.anthropic.com/en/articles/9450526-how-can-i-... > It’s verbatim, I promise. Of course, I believe you. It is verbatim.

guappa 2y ago

> My guess: Your domain is weird/rare, so LLMs are terrible because their training data is very limited. The training data is always limited, because if there was an existing software that already did what I'm doing, I'd be using it instead of writing it :) So by definition, unless I'm learning how to program and doing exercises that thousands have done before me, I'm doing something for which there is no training data.

danielbln 2y ago
It's great that you work on truly novel, unseen problems. The rest of us peasants tends to recombine previously solved problems into to novel solutions to solve business needs, and that works rather well with LLMs.
- throwaway2037 2y ago
  > The rest of us peasants tends to recombine previously solved problems into to novel solutions to solve business needs, and that works rather well with LLMs. You took the words right out of my mouth! I wrote a draft reply to OP (but discarded it) with similar thoughts -- roughly: I work on CRUD apps, and so do most other devs; LLMs are a terrific fit for this subject matter.

deleted 2y ago

[deleted]