4 ms·

Are there any resources anyone could share that explain how LLMs can do things like design functioning circuits from next token prediction? I am totally baffled

• abletonlive 1mo ago
  You fell for the stochastic parrot meme and next token over simplification. That's the explanation.
  • sethaurus 1mo ago
    That's just derision, not an explanation. And it's a bad way to treat someone humbly trying to learn.
    • IshKebab 1mo ago
      It kind of is an explanation though - the explanation is that they believed the stochastic parrot / "just" next token prediction nonsense, and that those are actually not true. You can ask for a deeper explanation of why they aren't true I guess.
    • abletonlive 29d ago
      I'm just surprised that so many intellectuals on HN hang on to false models of reality for so long after that reality has been demonstrably destroyed. It's not humble at all. In fact it's the opposite, completely arrogant and stubborn. It's been obvious and demonstrated at least since the end of 2025 for anybody that used LLMs at any capacity without dismissing them. If you are still surprised that your model of reality doesn't hold up, then someone needs to bluntly tell you what's wrong at the core of your being. Notice the original comment is asking people to validate their false premise about next token prediction. The deeper subtext of the original comment is that they are surprised that there's dissonance from observed reality and this false premise that they have convinced themself is true. I'm explaining that dissonance because it doesn't matter what the actual mechanism is if they are still working with their false premise. The dissonance exists because they, without evidence and a very weak understanding of how LLMs work, believed an oversimplification and meme about them being stochastic parrots. Here's a tip: Just because you hear something repeated over and over on social media, doesn't mean it's true, or at the very least: you don't need to take it literally to the point where it conflicts with demonstrated reality. It is deeply disturbing that such a large cohort of HN writers and redditors exemplify such stubbornness, because I must imagine that some of this cohort hold real positions of responsibility within society. If you can't get this simple thing right about reality, I firmly believe much of your model of reality is wrong and you should have no business shaping society. Another comment to the original comment frames it perfectly: "At what point do you challenge your own assumptions?" The author of the original comment has demonstrated no progress towards making this trivial act of self reflection. It's straight up intellectual dishonesty, the opposite of how you're framing it. Their judgment in all other matters must be questioned as well. I am alarmed that I have to participate in the same reality and be affected by such people that can't seem to get it together. So yes, it is derision and sometimes that's called for.
      • dboreham 29d ago
        There are so many of these buried ostrich head posts that I suspect some sort of bot farming. Why such posts make money for someone, I'm not sure.
• lukan 1mo ago
  "my brain hurts trying to understand how that can be" Well, we all are, some are just more used to it by now and take the magic for granted. My simple explanation, those neural networks save lot's of patterns of data, and that pattern can represent an image, a code snippet, a poem, or well ... description of a circuit board. And especially the text variant, LLM's - did copy all from us - so obviously they sound like humans, when they internally debate how to do something as this is what is in their trainings data how humans sound, when doing similar tasks. But really understanding it? Not sure if there is a single person on earth who does.
• therealdrag0 1mo ago
  Now go read Blindsight and enjoy the mental crisis.
• mpodeley 1mo ago
  “Next-token prediction” describes the output format, not the computation required to choose each token. During training, models develop internal representations of concepts, constraints, possible futures, and algorithms. The PCB agent also writes circuit code, runs simulations, reads failures, and revises the design. It isn’t one-shot autocomplete. Astra and Fable are already hard to square with “mere autocomplete.” We may be (really) close to AGI, and token-by-token generation certainly doesn’t rule out subjective experience (I think we should at least treat that as an open question). Great videos: https://www.youtube.com/watch?v=D8GOeCFFby4 https://www.youtube.com/watch?v=D8GOeCFFby4 https://www.youtube.com/watch?v=Bj9BD2D3DzA https://www.youtube.com/watch?v=Bj9BD2D3DzA https://www.youtube.com/watch?v=l6DKRf-fAAM https://www.youtube.com/watch?v=l6DKRf-fAAM https://www.youtube.com/watch?v=GlYgs6v2YfU https://www.youtube.com/watch?v=GlYgs6v2YfU
  • kneyed 1mo ago
    right on! I like to say "token prediction is a task, not a limitation"
  • Zambyte 1mo ago
    How can you define "general" and "intelligence" in a way that has existed for years now?
    • Zambyte 29d ago
      hasn't*
  • pineaux 1mo ago
    Yeah most of us are so fucked. With almost no way of protecting ourselves. No real amount of assets that will give enough power to save ourselves from the people in a position that can maximally leverage AI and lock others out. I see a future where these capabilities will be locked behind super high price pay walls. Why wouldn't they? How recoup investments if the price doesnt go up?
    • brandnewideas 29d ago
• akiselev 1mo ago
  > Yet, it's all the most likely token and my brain hurts trying to understand how that can be. You and everyone else. That's the great mystery of transformer architectures as applied to language. To be clear though, they're only good at schematic capture, which is very much a textual representation. Most of the data basically boils down to netlists, which are a text based format mapping connections between abstract pins that only later map to physical copper. The actual schematic portion is for human consumption and LLMs don't need to produce those to be useful. Where LLMs completely break down is the next step, PCB routing. That's an NP-complete research problem that's been ongoing for decades without much progress. I've had some fun playing with using LLMs to better specify DRC rules in Altium so that the "classical" algorithms are more usable, but at the end of the day their geometric intuition is nonexistent.
  • mapontosevenths 1mo ago
    They actually can route just fine. I used Sol to design and route mine from start to finish. Sent it to PCBWay and had a working prototype in a few weeks. It was a pretty simple rp2040 based thing, similar to Adadfruits USB feather.I just gave it kicad and it wrote python to route it. The board was probably larger than it had to be, and two of the silkscreens were swapped, but it worked on the first go. FWIW - Computer vision is also NP complete, but we do that all the time now.
    • akiselev 1mo ago
      I'd love to see that chat log, and the final board. To be fair I've only been testing on nontrivial PCBs with 6+ layers and I haven't had the luck you have. > FWIW - Computer vision is also NP complete, but we do that all the time now. I have no idea what you mean by this. What's your definition of NP complete?
      • cwmoore 29d ago
        Not OP but “actual exponential complexity” should work, what is really your issue with that comment?
      • 29d ago
• hackinthebochs 1mo ago
  >without an actual understanding of what they're doing. At what point do you start to question your assumptions that are causing you so much cognitive dissonance? But to answer your question: to predict the next token really well you just have to model the world. Think of it like this, a simple statistical model might say "when token A is seen respond with token B". The next step will add conditions, "...respond with token B unless X has been seen, then respond with Y". Add a few billion more of these contexual clauses and you have a sequence of logical rules that indirectly model the relevant processes in the world.
  • deleted 1mo ago
    [deleted]
• cmrx64 1mo ago
  type “shai next-token” and then “transformers learn shortcuts to automata” into arxiv and prepare to be blown away
• lampiaio 1mo ago
  humans when a machine better than them at spotting patterns appears:
• ninkendo 1mo ago
  My 2¢: When google trained a neural net on Go moves, using some text notation for them, with no other vocabulary of any kind, just predict the next go move, they noticed a representation of a Go board had essentially formed in the network, all on its own. It had never “seen” a go board, or had one explained, but they could map neuron states to go board squares pretty much 1:1. I truly think that LLM’s with hundreds of billions of parameters in their neural networks have all kinds of hidden “models” of things that arise from the simple act of predicting tokens. We’ve seen that the hidden layers in their networks model all sorts of program execution state for instance, when they’re working on coding tasks. “Predict the next token” is a way of shaping/reshaping the neural network until it actually develops models of the things you’re giving it. Like the go board example. And I would wager that it has a compounding effect: once you have some useful models in the network, they can unlock the creation of other models, and so on.
  • smokel 29d ago
    > It had never “seen” a go board, or had one explained I assume that you are referring to AlphaGo or AlphaZero. In either case, this statement is not correct. Both algorithms most certainly know exactly what a go board looks like, and what the rules are. In the case of AlphaZero, it initially did not know how to best play the game, or what strategy or tactics would work. But the connections between the neural network and the go board are hardcoded, by humans.
    • mariebks 29d ago
      Incredibly, Muzero didn’t even know the rules, it figured them out from starting with random moves: https://deepmind.google/research/alphazero-and-muzero/ https://deepmind.google/research/alphazero-and-muzero/
• mrshadowgoose 1mo ago
  Look up "mechanistic interpretability" in the context of LLMs. The next token prediction machinery is just a foundation for a higher order learned structure that appears to encode specific concepts, regardless of input language. The analogy to humans is that the human brain is "just atoms bouncing around", but there's unquestionably something "more" going on that just that.
  • Brian_K_White 29d ago
    Until ais get better from feeding on their own output the way humans do there is nothing to question. That will probably be the fundamental indicator that something other than repeating things some human previously created is going on. So far, ais only get worse from feeding on their own output. Meanging/including the output of other ais not a single ai feeding on it's own output. Also bear in mind that so far even the output of ais is 100% the downstream of a human command. No ai has persued it's own curiosity that didn't result from a human asking a question or giving a command. That is input which is different from a human taking in their environment even though our limited language can call those both the same word input. The fact that humans also repeat and remix things, and humans also produce essentially procedurally generated empty output like corporate-speak etc, is an irrelevant distraction in the same way that both a human and an electric motor can both perform the same simple mechanical task.
• IshKebab 1mo ago
  Humans essentially do "next token prediction" too - there's always a choice between the next actions to take and they pick a good one based on what has happened in the past. That doesn't really limit how clever we can get internally when picking the next action.
• zarzavat 29d ago
  Humans are evolved to survive in the wild. We are not evolved for circuit design. Yet we can design circuits because evolution found it easier to develop a general problem solving nervous system than a nervous system which is adapted for every single specific problem a human might encounter.
  • catlifeonmars 29d ago
    Circuit design might be different if discovered by mollusks. That is to say, while we are not evolved for circuit design per se, circuit design has evolved for humans, by humans (so far).
• throwaway219450 29d ago
  You can express a circuit as a graph, and many schematic formats are plaintext. Same goes for the Gerbers which are an ASCII format used to describe the masks that are used to define the PCB traces. The models are trained on a lot of academic information about how circuits work, most component datasheets are public and they've sucked up all sorts of niche greybeard advice from internet forums. A huge advantage of electrical design is that the connectivity is testable with Design and Electrical Rule Checks (DRC/ERC). I suspect you could even tell the models to run physics checks on the traces that are important for things like crosstalk.
• Marha01 29d ago
  > next token prediction Saying that LLMs just produce the next token is like saying that human brains just produce the next electrical impulse. If the algorithm that produces the next token (or electrical impulse) is complex enough, it can do anything that is in principle computable.