14 ms·
Testing Generative AI for Circuit Board Design
- deleted 2y ago[deleted]
- bottlepalm 2y agoIt'd be interesting to see how Sonnet 3.5 does at this. I've found Sonnet a step change better than Opus, and for a fraction of the cost. Opus for me is already far better than GPT-4. And same as the poster found, GPT-4o is plain worse at reasoning. Edit: Better at chain of thought, long running agentic tasks, following rigid directions.
- stavros 2y agoOpus is better than GPT-4? I've heard mixed experiences.
- imperio59 2y agoThat's because the sample size is probably small and for niche prompts or topics. It's very hard to evaluate whether a model is better than another, especially doing it in a scientifically sound way is time consuming and hard. This is why I find these types of comments like "model X is so much better than model Y" to be about as useful as "chocolate ice cream is so much better than vanilla"
- r2_pilot 2y agoAnd both flavors have a base flavor of excrement... Still, since I started using Claude 3 Opus (and now 3.5 Sonnet) a couple of months back, I don't see myself switching from them nor stopping use of LLM-based AI tech; it's just made me feel like the computer is actually working for and with me and even that alone can be enough to get me motivated and accomplish what I set out to do.
- skapadia 2y ago"it's just made me feel like the computer is actually working for and with me and even that alone can be enough to get me motivated and accomplish what I set out to do." This is a great way to describe what I've been feeling / experiencing as well.
- r2_pilot 2y agoJust an update on my initial impressions of Claude 3.5 Sonnet. It's a better programmer than I am in Python; that's not saying much, but this is now two nights in a row I've been impressed with what I've created with it.
- stavros 2y agoTrue, I just tried it for generating a book summary, and Sonnet 3.5 was very bad. GPT-4o is equally bad at that , gpt-4-turbo is great.
- netsec_burn 2y agoThis more likely has to do with context length?
- stavros 2y agoNo, all the information is there, but gpt-4o tends to produce bullet points (https://www.thesummarist.net/summary/the-making-of-a-manager/your-first-three-months/ https://www.thesummarist.net/summary/the-making-of-a-manager...), whereas gpt-4-turbo tends to produce much more readable prose (https://www.thesummarist.net/summary/supercommunicators/the-matching-principle-how-to-fail-at-recruiting-spies/ https://www.thesummarist.net/summary/supercommunicators/the-...).
- Obscurity4340 2y agoHow is prose more readable than bullets?
- stavros 2y ago* Clearer narrative * Connection between points * Flows better * Eyes don't start-stop as much
- Obscurity4340 2y agoI think I was thinking more along the lines of if I'm looking for specific information or to get a condensed understanding of something Different readable than the more flowing, conjunct readable than yours (which is the more typical use of it I concede)
- DHaldane 2y agoIt really depends on the type of question, but generally I'm between Gemini and Claude these days for most things.
- anticensor 2y agoOpus 3.5 is not yet released.
- stavros 2y agoI assume the GP was talking about 3.0.
- DHaldane 2y agoThat's an interesting question - I'll take a few pokes at it now to see if there's improvement.
- DHaldane 2y agoUpdate: Sonnet 3.5 is better than any other model for the circuit design and part finding tasks. Going to iterate a bit on the prompts to see how much I can push the new model on performance. Figures that any article written on LLM limits is immediately out of date. I'll write an update piece to summarize new findings.
- CamperBob2 2y agoThat name threw me for a loop. 'Sonnet' already means something to EEs ( https://www.sonnetsoftware.com/ https://www.sonnetsoftware.com/ ).
- deleted 2y ago[deleted]
- RF_Savage 2y agoYeah same here. Thought Sonnet had added some ML stuff into their EM simulator.
- cjk2 2y agoEx EE here > The AI generated circuit was three times the cost and size of the design created by that expert engineer at TI. It is also missing many of the necessary connections. Exactly what I expected. Edit: to clarify this is even below the expectations of a junior EE who had a heavy weekend on the vodka.
- shrimp_emoji 2y agoIt's like a generated image with an eye missing but for circuits. :D
- cjk2 2y agoAI proceeds to use 2n3904 as a thyristor. AI happy as it worked the first 10ns of the cycle.
- jeffreygoesto 2y agoEvery natural Intelligence knows that you need to reach out to a 2N3055 for heavy duty. ;)
- FourierEnvy 2y agoWhy do people think inserting an LLM into the mix will make it better than just an evolutionary or reinforcement model applied? Who cares if you can talk to it like a human?
- Terr_ 2y agoYeah, when the author was writing about that initial query about delay-per-unit-length, I'm thinking: "This doesn't tell us whether an LLM can apply the concepts, only whether relevant text was included in its training data." It's a distinction I fear many people will have trouble keeping in-mind, faced with the misleading eloquence of LLM output.
- Kuinox 2y ago
- dindobre 2y agoUsing neural networks to solve combinatorial or discrete problems is a waste of time imo, but I'd be more than happy if somebody could convince me of the opposite.
- utkuumur 2y agoThere are recent papers based on diffusion that perform quite well. Here's an example of a recent paper https://arxiv.org/pdf/2406.01661 https://arxiv.org/pdf/2406.01661. I am also working on ML-based CO. My approach has a close 1% gap on hard instances with 800-1200 nodes and less than 0.1% for 200-300 nodes on Maximum Cut, Minimum Independent Set, and Maximum Clique problems. I think these are very promising times for neural network-based discrete optimization.
- dindobre 2y agoThanks, will try to give it a read this weekend. Would you say that diffusion is the architectural change that opened up CO for neural nets? Haven't followed this particular niche in a while
- utkuumur 2y agoI believe it helps but not the sole reason. Because there are also autoregressive models that perform slightly worse. Unsupervised learning + Diffusion + Neural Search is the way to go in my opinion. However, currently, the literature lacks efficient Neural search space exploration. The diffusion process is a good starting point for neural search space exploration, especially when it is used not just to create a solution from scratch but also as a local search method. Still, there is no clear exploration and exploration control in current papers. We need to incorporate more ideas from heuristic search paradigms to neural network CO pipelines to take it to the next step.
- HanClinto 2y agoThis feels like an excellent demonstration of the limitation of zero-shot LLMs. It feels like the wrong way to approach this. I'm no expert in the matter, but for "holistic" things (where there are a lot of cross-connections and inter-dependencies) it feels like a diffusion-based generative structure would be better-suited than next-token-prediction. I've felt this way about poetry-generation, and I feel like it might apply in these sorts of cases as well. Additionally, this is a highly-specialized field. From the conclusion of the article: > Overall we have some promising directions. Using LLMs for circuit board design looks a lot like using them for other complex tasks. They work well for pulling concrete data out of human-shaped data sources, they can do slightly more difficult tasks if they can solve that task by writing code, but eventually their capabilities break down in domains too far out of the training distribution. > We only tested the frontier models in this work, but I predict similar results from the open-source Llama or Mistral models. Some fine tuning on netlist creation would likely make the generation capabilities more useful. I agree with the authors here. While it's nice to imagine that AGI would be able to generalize skills to work competently in domain-specific tasks, I think this shows very clearly that we're not there yet, and if one wants to use LLMs in such an area, one would need to fine-tune for it. Would like to see round 2 of this made using a fine-tuning approach.
- DHaldane 2y agoMy gut agrees with you that LLMs shouldn't do this well on a specialty domain. But I think there's also the bitter lesson to be learned here: many times people say LLMs won't do well on a task, they are often surprised either immediately or a few months later. Overall not sure what to expect, but fine tuning experiments would be interesting regardless.
- cjk2 2y agoI doubt it'd work any better. Most of EE time I have spent is swearing at stuff that looked like it'd work on paper but didn't due to various nuances. I have my own library of nuances but how would you even fine tune anything to understand the black box abstraction of an IC to work out if a nuance applies or not between it and a load or what a transmission line or edge would look like between the IC and the load? This is where understanding trumps generative AI instantly.
- guidoism 2y agoThis reminds me of my professor's (probably very poor) description of NP-complete problems where the computer would provide an answer that may or may not be correct and you just had to check that it was correct and you do test for correctness in polynomial time. It kind of grosses me out that we are entering a world where programming could be just testing (to me) random permutations of programs for correctness.
- moffkalast 2y agoWell we had to keep increasing inefficiency somehow, right? Otherwise how would Wirth's law continue to hold?
- thechao 2y agoMost of the HW engineers I work with consider the webstack to be far more efficient than the HW-synthesis stack; ie, there's more room for improvement in HW implementation than in SW optimization.
- cushychicken 2y agoI'm terrified that JITX will get into the LLM / Generative AI for boards business. (Don't make me homeless, Duncan!) They are already far ahead of many others with respect to next generation EE CAD. Judicious application of AI would be a big win for them. Edit: adding "TL;DRN'T" to my vocabulary XD
- DHaldane 2y agoI promise that we want to stay a software company that helps people design things! Adding Skynetn't to company charter...
- AdamH12113 2y agoThe conclusions are very optimistic given the results. The LLMs: * Failed to properly understand and respond to the requirements for component selection, which were already pretty generic. * Succeeded in parsing the pinout for an IC but produced an incomplete footprint with incorrect dimensions. * Added extra components to a parsed reference schematic. * Produced very basic errors in a description of filter topologies and chose the wrong one given the requirements. * Generated utterly broken schematics for several simple circuits, with missing connections and aggressively-incorrect placement of decoupling capacitors. Any one of these failures, individually, would break the entire design. The article's conclusion for this section buries the lede slightly: > The AI generated circuit was three times the cost and size of the design created by that expert engineer at TI. It is also missing many of the necessary connections. Cost and size are irrelevant if the design doesn't work. LLMs aren't a third as good as a human at this task, they just fail. The LLMs do much better converting high-level requirements into (very) high-level source code. This make sense (it's fundamentally a language task), but also isn't very useful. Turning "I need an inverting amplifier with a gain of 20" into "amp = inverting_amplifier('amp1', gain=-20.0)" is pretty trivial. The fact that LLMs apparently perform better if you literally offer them a cookie is, uh... something.
- richie-guix 2y ago[flagged]
- lemonlime0x3C33 2y agothank you for summarizing the results, I feel much better about my job security. Now if AI could make a competent auto router for fine pitch BGA components that would be really nice :)
- neltnerb 2y agoI think the only bit that looked handy in there would be if it could parse PDF datasheets and help you sort them by some hidden parameter. If I give it 100 datasheets for microphones it really should be able to sort them by mechanical height. Maybe I'm too optimistic. The number of times I've had to entirely redo a circuit because of one misplaced connection, yeah, none of those circuits worked for any price before I fixed every single error.
- sehugg 2y agoHow does this compare to Flux.ai? https://docs.flux.ai/tutorials/ai-for-hardware-design https://docs.flux.ai/tutorials/ai-for-hardware-design
- built_with_flux 2y agoflux.ai founder here Agree with OP that the raw models aren't that useful for schematic/pcb design. It's why we build flux from the ground up to provide the models with the right context. The models are great moderators but poor sources of great knowledge. Here are some great use cases: https://www.youtube.com/watch?v=XdH075ClrYk https://www.youtube.com/watch?v=XdH075ClrYk https://www.youtube.com/watch?v=J0CHG_fPxzw&t=276s https://www.youtube.com/watch?v=J0CHG_fPxzw&t=276s https://www.youtube.com/watch?v=iGJOzVf0o7o&t=2s https://www.youtube.com/watch?v=iGJOzVf0o7o&t=2s and here a great example of levering AI to go from idea to full design https://x.com/BuildWithFlux/status/1804219703264706578 https://x.com/BuildWithFlux/status/1804219703264706578
- shrubble 2y agoReminds me of this, an earlier expert-system method for CPU design, which was not used in subsequent designs for some reason: https://en.wikipedia.org/wiki/VAX_9000#SID_Scalar_and_Vector_Processor_Synthesis https://en.wikipedia.org/wiki/VAX_9000#SID_Scalar_and_Vector...
- surfingdino 2y agoLook! You can design thousands of shit appliances at scale! /s
- smmseller 2y ago[dead]
- Terr_ 2y agoTo recycle a rant, there's a whole bunch of hype and investor money riding on a very questionable idea here, namely: "If we make a really really good specialty text-prediction engine, it could be able to productively mimic an imaginary general AI, and if it can do that then it can productively mimic other specialty AIs, because it's all just intelligence, right?"
- ai4ever 2y agoinvestor money is seduced by the possibilities and many of the investors are in it for FOMO. few really understand what the limits of the tech are. and if it will even unlock the usecases for which it is being touted.
- seveibar 2y agoI work on generative AI for circuit board design with tscircuit, IMO it's definitely going to be the dominant form of bootstrapping or combining circuit designs in the near future (<5 years) Most people are wrong that AI won't be able to do this soon. The same way you can't expect an AI to generate a website in assembly, but you CAN expect it to generate a website with React/tailwind, you can't expect an AI to generate circuits without having strong functional blocks to work with. Great work from the author studying existing solutions/models- I'll post some of my findings soon as well! The more you play with it, the more inevitable it feels!
- HanClinto 2y agoI'd be interested in reading more of your findings! Are you able to accomplish this with prompt-engineering, or are you doing fine-tuning of LLMs / custom-trained models?
- seveibar 2y agoNo fine tuning needed, as long as the target language/DSL is fairly natural, just give eg a couple examples of tscircuit React, atopile JotX etc and it can generate compliant circuits. It can hallucinate imports, but if you give it an import list you can improve that a lot.
- DHaldane 2y agoI've found the same thing - a little syntax example, some counter examples and generative AI does well generating syntactically correct code for PCB design. A lot of the netlists are electrically nonsense when it's doing synthesis for me. Have you found otherwise?
- seveibar 2y agoNetlists, footprint diagrams, constraint diagrams etc. are mostly nonsense. I’m working on finetuning Phi3 and I’m hopeful it’ll get better. I’m also working on synthesized datasets and mini-DSLs to make that tuning possible eg https://text-to-footprint.tscircuit.com https://text-to-footprint.tscircuit.com My impression is that synthetic datasets and finetuning will basically completely solve the problem, but eventually it’ll be available in general purpose models- so it’s not clear if its worth it to build a dedicated model. Overall the article’s analysis is great. I’m very optimistic that this will be solved in the next 2 years.
- blueyes 2y agoSee Quilter: https://www.quilter.ai https://www.quilter.ai
- kristopolous 2y agoJust the other day I came up with an idea of doing a flatbed scan of a circuit board and then using machine learning and a bit of text promoting to get to a schematic I don't know how feasible it is. This would probably take low $millions or so of training, data collection and research to get not trash results. I'd certainly love it for trying to diagnose circuits. It's probably not really that possible even at higher end consumer grade 1200dpi.
- cmbuck 2y agoThis would be an interesting idea if you were able to solve the problem of inner layers. Currently to reverse engineer a board with more than 2 layers an x-ray machine is required to glean information about internal routing. Otherwise you're making inferences based on surface copper only.
- kristopolous 2y agoMaybe not. I scanned a bluetooth aux transceiver yesterday as a test of how well a flatbed can pick up details. There's a bunch of these on the market and the cheap ones, they are more or less equivalent. It's a CSR 8365 based device, which you can read from the scan. The industry is generally convergent on the major design decisions for some hardware purpose for some given time period. And the devices, in this case, bluetooth aux transceivers, they all do the same things. They've even more or less converged on all being 3 buttons. When optimizing for cost reduction with the commodity chips that everyone is using to do the same things, the manufacturer variation isn't that vast. In the same way you can get 3d models from 2d photos because you can identify the object based on a database of samples and then guess the 3d contours, the hypothesis to test is whether with enough scans and schematics, a sufficiently large statistical model will be good enough to make decent guesses. If you've got say 40 devices with 80% of the same chips doing the same things for the same purpose, a 41st device might have lots of guessable things that you can't necessarily capture on a cheap flatbed This will probably work but it's a couple million away from becoming a reality. There's shortcuts that might make this a couple $100,000s project (essentially data contracts with bespoke chip printers) but I'd have to make those connections. And even then, it's just a hobbyist product. The chances of recouping that investment is probably zero although the tech would certainly be cool and useful. Just not "I'll pay you money" level useful.
- amelius 2y agoCan we have an AI that reads datasheets and produces Spice circuits? With the goal of building a library of simulation components.
- klysm 2y agoThat's the kind of thing where verification is really hard, and things will look plausible even if incorrect.
- ncrmro 2y agoI had it generate some opencad but never looked into it further.
- rkagerer 2y agoAny discussion of evolved circuits would be incomplete without mentioning Dr. Adrian Thompson's pioneering work in the 90's: https://www.damninteresting.com/on-the-origin-of-circuits/ https://www.damninteresting.com/on-the-origin-of-circuits/
- djaouen 2y agoSure, this will end well lol
- al2o3cr 2y agoTBH the LLM seems worse than useless for a lot of these tasks - entering a netlist from a datasheet is tedious, but CHECKING a netlist that's mostly correct (except for some hallucinated resistors) seems even more tedious.
- teleforce 2y agoToo Lazy To Click (TLTC): TLDR: We test LLMs to figure out how helpful they are for designing a circuit board. We focus on utility of frontier models (GPT4o, Claude 3 Opus, Gemini 1.5) across a set of design tasks, to find where they are and are not useful. They look pretty good for building skills, writing code, and getting useful data out of datasheets. TLDRN'T: We do not explore any proprietary copilots, or how to apply a things like a diffusion model to the place and route problem.
- amelius 2y agoThe whole approach reminds me of: https://gpt-unicorn.adamkdean.co.uk/ https://gpt-unicorn.adamkdean.co.uk/
- roody15 2y agoIt makes me think of the saying “a jack of all trades a master of none”. I cannot help but think there are some similarities between large model generative AI and human reasoning abilities. For example if I ask a physician with a really high IQ some general questions about say anything like fixing shocks on my mini van … he may have some better ideas than me. However he may be wrong since he specialized in medicine, although he may have provided some good overall info. Let’s take a lower IQ mechanic who has worked as a mechanic for 15 years. Despite this human having less IQ, less overall knowledge on general topics … he gives a much better answer of fixing my shocks. So with LLM AI fine tuning looks to be key as it is with human beings. Large data sets that are filtered / summarized with specific fields as the focus.
- pylua 2y agoThat’s not really reasoning, right ? Maybe humans rely disproportionate on association in general.
- MOARDONGZPLZ 2y agoAuthor mentions prompting techniques to get better results, presumable “you are an expert EE” or “do this and you get a digital cookie” are among these. Can anyone point me to non-SEO article that outlines the latest and greatest in the promoting techniques domain?
- fagrobot 2y ago[dead]