3 ms·

The general rule seems to be, the more layers you automate with LLMs, the worse each successive layer gets. Piping LLM output as input into new LLM calls, you'r

The general rule seems to be, the more layers you automate with LLMs, the worse each successive layer gets. Piping LLM output as input into new LLM calls, you're already starting to notice how things fall apart and get lost quickly. If you have the idea, more or less the implementation plan, let the LLM do the coding, you can end up with something maintainable and nice, it's basically up to you. Strip away one layer, so you have the idea, but let the LLM come up with the implementation plan, then also the implementation, and things end up a lot less than ideal. Remove another layer, let the LLM do it all, and it's all a mess.

quotemstr 9mo ago
I think this principle applies only if you lack feedbacks. Yes, when you go through multiple layers of open loop control, you're going to get less precise at each layer. It's less clear that the situation is as dire if each level has metrics and can self-adjust to optimize its performance.
- embedding-shape 9mo ago
  But these are inherently subjective things, what the "right idea" is, or the "right implementation" is all up in our head that we can try to verbalize, but I don't think you can come up with an objective score for it, ask 100 programmers you'll get 100 different answers what "clean design" is.
  - quotemstr 9mo ago
    And that's why my whole schtick when it comes to agent design is that agents need to learn online, continuously, and in adapter space via some PEFT mechanism (I like soft prompts and prefix tuning), because it's really hard to ascend gradients in discrete domains like tokens.
    - embedding-shape 9mo ago
      > The model knows damn well when it's written ugly code. You can just ask it. That's not been my experience at all, what model and prompt would you use for that? Every single one I've tried is oblivious to if a design makes sense or not unless explicitly prompted for it with constraints, future ideas and so on.
      - CuriouslyC 9mo ago
        The problem is that the model doesn't know what you mean by "bad code" a priori. If you list specific issues you care about (e.g. separation of concerns, don't repeat yourself, single responsibility, prefer pure functions, etc) it's pretty good at picking them out. Humans have this problem as well, we're just more opinionated.
        embedding-shape 9mo ago
        Yes, that's exactly what I mentioned earlier, if you describe the implementation, you can get something you can work with long-term. But if you just describe an idea, and let the LLM do both the design of the implementation and the implementation itself, eventually it seems to fall over itself and changes takes longer and longer time.
godelski 9mo ago
People like to make the comparison between zip file compressions, where you can degrade something by continually compressing. Same with using jpeg or mp3. But I like to use the analogy of the game "Telephone" (also called "Chinese Whispers"). I think it also highlights how fraught natural language is and just how quickly it can degrade. I think a lot of people are insufficiently impressed with how good we are at communicating at all.
- sweetjuly 9mo ago
  I suggest you find a new DEFLATE library if you're losing data when you compress things with it :)
  - godelski 9mo ago
    You do realize there is both lossy and lossless compression, right? Or did you hyperfixate on the colloquial usage of zip
    - ethmarks 9mo ago
      > You do realize there is both lossy and lossless compression, right? This is simply wrong. There is no lossy compression for ZIP. DEFLATE, by far the most common ZIP compression method, uses LZ77 and Huffman Coding, both of which are lossless. There are other methods compatible with ZIP containers as specified by PKWARE (e.g. BZIP2, LZMA, Zstandard, PPMd, etc), but all of them are lossless. According to both the official ZIP specification and every ZIP implementation on Earth, you cannot have a lossy ZIP unless it is corrupted. There do exist lossy data formats (e.g. JPEG), but if you put those in a ZIP file it'll still encode and decode it losslessly.
      - defrost 9mo ago
        > There is no lossy compression for ZIP. Did you hyperfixate on the colloquial usage of zip?
        9mo ago
nimonian 9mo ago
It's like those sequences of images where we ask the LLM to reproduce the same image exactly, and we just get some kind of grotesque collapse after a few dozen iterations. The same happens with text and code. I call this "semantic collapse". I conjecture that after some years of LLMs reading a SharePoint site, producing summaries, then summaries of those summaries, etc... We will end up with a grotesque slurry. At some point, fresh human input is needed to inject something meaningful into the process.
stitched2gethr 9mo ago
It's all about how full the context is, right? For a task that can be completed in 20% of the context it doesn't matter, but you don't want to fill your context with exploration before you do the hard part. I have actually found something close to the opposite. I work on a large codebase and I often use the LLM to generate artifacts before performing the task (for complex tasks). I use a prompt to say "go explore this area if the code and write about it". It documents concepts and has pointers to specific code. Then a fresh session can use that without reading the stuff that doesn't matter. It uses more tokens overall, but includes important details that can get totally missed when you just let it go.
- embedding-shape 9mo ago
  > It's all about how full the context is, right? No, even when you restart the context from scratch, which I do for each change really, seeing that same effect.
energy123 9mo ago
> Piping LLM output as input into new LLM calls Google's Aletheia works like this, and instead of degrading it keeps getting better. I get what you're trying to say, though. The less world knowledge you provide the LLM, which it otherwise lacks, the worse its outputs will be.
- embedding-shape 9mo ago
  > I get what you're trying to say, though. The less world knowledge you provide the LLM, which it otherwise lacks, the worse its outputs will be ... No, wasn't trying to say that at all, I'm saying that it seems like the tokens a LLM produce works much worse as inputs than the tokens a human would produce, regardless of what it actually seems to say.

quotemstr 9mo ago

I think this principle applies only if you lack feedbacks. Yes, when you go through multiple layers of open loop control, you're going to get less precise at each layer. It's less clear that the situation is as dire if each level has metrics and can self-adjust to optimize its performance.

embedding-shape 9mo ago
But these are inherently subjective things, what the "right idea" is, or the "right implementation" is all up in our head that we can try to verbalize, but I don't think you can come up with an objective score for it, ask 100 programmers you'll get 100 different answers what "clean design" is.
- quotemstr 9mo ago
  And that's why my whole schtick when it comes to agent design is that agents need to learn online, continuously, and in adapter space via some PEFT mechanism (I like soft prompts and prefix tuning), because it's really hard to ascend gradients in discrete domains like tokens.
  - embedding-shape 9mo ago
    > The model knows damn well when it's written ugly code. You can just ask it. That's not been my experience at all, what model and prompt would you use for that? Every single one I've tried is oblivious to if a design makes sense or not unless explicitly prompted for it with constraints, future ideas and so on.
    - CuriouslyC 9mo ago
      The problem is that the model doesn't know what you mean by "bad code" a priori. If you list specific issues you care about (e.g. separation of concerns, don't repeat yourself, single responsibility, prefer pure functions, etc) it's pretty good at picking them out. Humans have this problem as well, we're just more opinionated.
      - embedding-shape 9mo ago
        Yes, that's exactly what I mentioned earlier, if you describe the implementation, you can get something you can work with long-term. But if you just describe an idea, and let the LLM do both the design of the implementation and the implementation itself, eventually it seems to fall over itself and changes takes longer and longer time.

godelski 9mo ago

People like to make the comparison between zip file compressions, where you can degrade something by continually compressing. Same with using jpeg or mp3. But I like to use the analogy of the game "Telephone" (also called "Chinese Whispers"). I think it also highlights how fraught natural language is and just how quickly it can degrade. I think a lot of people are insufficiently impressed with how good we are at communicating at all.

sweetjuly 9mo ago
I suggest you find a new DEFLATE library if you're losing data when you compress things with it :)
- godelski 9mo ago
  You do realize there is both lossy and lossless compression, right? Or did you hyperfixate on the colloquial usage of zip
  - ethmarks 9mo ago
    > You do realize there is both lossy and lossless compression, right? This is simply wrong. There is no lossy compression for ZIP. DEFLATE, by far the most common ZIP compression method, uses LZ77 and Huffman Coding, both of which are lossless. There are other methods compatible with ZIP containers as specified by PKWARE (e.g. BZIP2, LZMA, Zstandard, PPMd, etc), but all of them are lossless. According to both the official ZIP specification and every ZIP implementation on Earth, you cannot have a lossy ZIP unless it is corrupted. There do exist lossy data formats (e.g. JPEG), but if you put those in a ZIP file it'll still encode and decode it losslessly.
    - defrost 9mo ago
      > There is no lossy compression for ZIP. Did you hyperfixate on the colloquial usage of zip?
      - 9mo ago

nimonian 9mo ago

It's like those sequences of images where we ask the LLM to reproduce the same image exactly, and we just get some kind of grotesque collapse after a few dozen iterations. The same happens with text and code. I call this "semantic collapse". I conjecture that after some years of LLMs reading a SharePoint site, producing summaries, then summaries of those summaries, etc... We will end up with a grotesque slurry. At some point, fresh human input is needed to inject something meaningful into the process.

stitched2gethr 9mo ago

It's all about how full the context is, right? For a task that can be completed in 20% of the context it doesn't matter, but you don't want to fill your context with exploration before you do the hard part. I have actually found something close to the opposite. I work on a large codebase and I often use the LLM to generate artifacts before performing the task (for complex tasks). I use a prompt to say "go explore this area if the code and write about it". It documents concepts and has pointers to specific code. Then a fresh session can use that without reading the stuff that doesn't matter. It uses more tokens overall, but includes important details that can get totally missed when you just let it go.

embedding-shape 9mo ago
> It's all about how full the context is, right? No, even when you restart the context from scratch, which I do for each change really, seeing that same effect.

energy123 9mo ago

> Piping LLM output as input into new LLM calls Google's Aletheia works like this, and instead of degrading it keeps getting better. I get what you're trying to say, though. The less world knowledge you provide the LLM, which it otherwise lacks, the worse its outputs will be.

embedding-shape 9mo ago
> I get what you're trying to say, though. The less world knowledge you provide the LLM, which it otherwise lacks, the worse its outputs will be ... No, wasn't trying to say that at all, I'm saying that it seems like the tokens a LLM produce works much worse as inputs than the tokens a human would produce, regardless of what it actually seems to say.