6 ms·

Not sure if I would tradeoff speed for accuracy. Yes, it's incredible boring to wait for the AI Agents in IDEs to finish their job. I get distracted and open Y

• cedws 1y ago
  You left an LLM to code for two hours and then were surprised when you had to spend a significant amount of time more cleaning up after it? Is this really what people are doing these days?
  • deleted 1y ago
    [deleted]
• kadushka 1y ago
  AI field desperately needs smarter models - not faster models.
  • Wazako 1y ago
    Yet deepseek has shown that more dialogue increases quality. Increasing speed is therefore important if you need thinking models.
  • geysersam 1y ago
    Definitely needs faster and cheaper models. Fast and cheap models could replace software in tons of situations. Imagine a vending machine or a mobile game or a word processor where basically all logic is implemented as a prompt to an llm. It would serve as the ultimate high level programming language.
    • janalsncm 1y ago
      I think natural language to code is the right abstraction. Easy enough barrier to entry but still debuggable. Debugging why an LLM randomly gives you Mountain Dew instead of Sprite if you have a southern accent sounds like a nightmare.
      • geysersam 1y ago
        I'm not sure it would be that hard to debug. Make sure you can reproduce the llm state (by storing the random seed for the session, or something like that) and then ask it "why did you just now give that customer mountain dew when they ordered sprite?"
        • otabdeveloper4 1y ago
          > and then ask it "why did you just now give that customer mountain dew when they ordered sprite?" Worse than useless for debugging. An LLM can't think and doesn't have capabilities for self-reflection. It will just generate a plausible stream of tokens in reply that may or may not correspond to the real reason why.
          • geysersam 1y ago
            Of course a llm can't think. But that doesn't mean it can't answer simple questions about the output that was produced. Just try it out with chatgpt when you have time. Even if it's not perfectly accurate it's still useful for debugging. Just think about it as a human employee. Can they always say why they did what they did? Often, but not always. Sometimes you will have to work to figure out the misunderstanding.
  • otabdeveloper4 1y ago
    LLM's can't think, so "smarter" is not possible.
    • IshKebab 1y ago
      They can by the normal English definitions of "think" and "smart". You're just redefining those words to exclude AI because you feel threatened by it. It's tedious.
      • otabdeveloper4 1y ago
        Incorrect. LLM's have no self-reflection capability. That's a key prerequisite for "thinking". ("I think, therefore I am.") They are simple calculators that answer with whatever tokens are most likely given the context. If you want reasonable or correct answers (rather than the most likely) then you're out of luck.
        • IshKebab 1y ago
          It is not a key prerequisite for "thinking". It's "I think therefore I am" not "I am self-aware therefore I think". In the 90s if your cursor turned into an hourglass and someone said "it's thinking" would you have pedantically said "NO! It is merely calculating!" Maybe you would... but normal people with normal English would not.
          • otabdeveloper4 1y ago
            Self-reflection is not the same thing as self-awareness. Computers have self-reflection to a degree - e.g., they react to malfunctions and can evaluate their own behavior. LLMs can't do this, in this respect they are even less of a thinking machine than plain old dumb software.
    • baq 1y ago
      Technically correct and completely besides the point.
    • 1y ago
  • guiriduro 1y ago
    If you have much more speed in the available time, for an activity like coding, you could use that for iteration, writing more tests and satisfying them, especially if you can pair that with a concurrent test runner to provide feedback. I'm not sure the end result would be lower scoring/smartness than an LLM could achieve in the same duration.
    • kadushka 1y ago
      I'm not sure the end result would be lower scoring/smartness than an LLM could achieve in the same duration. It probably wouldn’t with current models. That’s exactly why I said we need smarter models - not more speed. Unless you want to “use that for iteration, writing more tests and satisfying them, especially if you can pair that with a concurrent test runner to provide feedback.” - I personally don’t.
• tyre 1y ago
  Check out RooCode if you haven’t. There’s an orchestrator mode that can start with a big model to come up with a plan and break down, then spin out small tasks to smaller models for scoped implementation.
  • danenania 1y ago
    If you’re open to a terminal-based approach, this is exactly what my project Plandex[1] focuses on—breaking up and completing large tasks step by step. 1 - https://github.com/plandex-ai/plandex https://github.com/plandex-ai/plandex
• amelius 1y ago
  Wouldn't it be possible to trade speed back for accuracy, e.g. by asking the model to look at a problem from different angles, let it criticize its own output, etc.?
  • nowittyusername 1y ago
    Just have it sample for longer or create a simple workflow that uses a monte carlo tree search approach. Don't see why this wont improve accuracy. I would love to see someone run tests to see how accurate the model is compared to similar parameter models in a per time block benchmark. Like if it can get same accuracy as a similar parameter autoregressive model but with half the speed, you already have a winner, besides other advantages of a diffusion based model.
• XenophileJKO 1y ago
  So my personal belief is that diffusion models will enable higher degrees of accuracy. This is because unlike an auto-regressive model it can adjust a whole block of tokens when it encounters some kind of disjunction. Think of the old example where an auto regressive model would output: "There are 2 possibilities.." before it really enumerated them. Often the model has trouble overcoming the bias and will hallucinate a response to fit the proceeding tokens. Chain of thought and other approaches help overcome this and other issues by incentivizing validation, etc. With diffusion however it is easier for the other generated answer to change that set of tokens to match the actual number of possibilities enumerated. This is why I think you'll see diffusion models be able to do some more advanced problem solving with a smaller number of "thinking" tokens.
  • AlexCoventry 1y ago
    > it can adjust a whole block of tokens when it encounters some kind of disjunction. This is true in principle for general diffusion models, but I don't think it's true for the noise model they use in Mercury (at least, going by a couple of academic papers authored by the Inception co-founders.) Their model generates noise by masking a token, and once it's masked, it stays masked. So the reverse-diffusion gets to decide on the contents of a masked token once, and after that it's fixed.
    • macleginn 1y ago
      The exact types of path dependencies in inference on text-diffusion models look like an interesting research project.
      • AlexCoventry 1y ago
        Yes, the problem is coming up with a noise model where reverse diffusion is tractable.
    • XenophileJKO 1y ago
      Thank you, I'll have to read the papers. I don't think I have read theirs.
• g-mork 1y ago
  The excitement for me is the implications for lower energy models. Tech like this could thoroughly break the Nvidia stranglehold at least for some segments
• kazinator 1y ago
  > Not sure if I would tradeoff speed for accuracy. Are you, though? There are obvious examples of obtaining speed without losing accuracy, like using a faster processor with bigger caches, or more processors. Or optimizing something without changing semantics, or the safety profile. Slow can be unreliable; a 10 gigabit ethernet can be more reliable than a 110 baud acoustically-coupled modem in mean time between accidental bit flips. Here, the technique is different, so it is apples to oranges. Could you tune the LLM paradigm so that it gets the same speed, and how accurate would it be?
• kittikitti 1y ago
  I don't use the best available models for prototyping because it can be expensive or more time consuming. This innovation makes prototyping faster and practicing prompts on slightly lower accuracy models can provide more realistic expectations.
• otabdeveloper4 1y ago
  > I now realize I should have done things incrementally even when I have a pretty good idea of the final picture. Or just save yourself the time and money and code it yourself like it's 2020. (Unless it's your employer paying for this waste, in which case go for it, I guess.)
• jillesvangurp 1y ago
  I think speed and convenience are essential. I use chat gpt desktop for coding. Not because it's the best but because it's fast and easy and doesn't interrupt my flow too much. I mostly stick to the 4o model. I only use the o3 model when I really have to. Because at that point getting an answer is slooooow. 4o is more than good enough most of the time. And more importantly it's a simple option+shift+1 away. I simply type something like "fix that" and it has all the context it needs to do its thing. Because it connects to my IDE and sees my open editor and the highlighted line of code that is bothering me. If I don't like the answer, I might escalate to o3 sometimes. Other models might be better but they don't have the same UX. Claude desktop is pretty terrible, for example. I'm sure the model is great. But if I have to spoon feed it everything it's going to annoy me. What I'd love is for smaller faster models to be used by default and for them to escalate to slower more capable models on a need to have basis only. Using something like o3 by default makes no sense. I don't want to have to think about which model is optimal for what question. The problem of figuring out what model is best to use is a much simpler one than answering my questions. And automating that decision opens the doors to having a multitude of specialized models.
  • matznerd 1y ago
    You're missing that Claude desktop has MCP servers, which can extend it to do a lot more, including much better real life "out of the box" uses. You can do things like use Obsidian as a filesystem or connect to local databases to really extend the abilities. You can also read and write to github directly and bring in all sorts of other tools.
• thomastjeffery 1y ago
  Accuracy is a myth. These models do not reason. They do not calculate. They perform no objectivity whatsoever. Instead, these models show us what is most statistically familiar. The result is usually objectively sound, or at least close enough that we can rewrite it as something that is.