10 ms·

Why It’s So Hard to Create New Processors

211 points by Lind5 6y ago

artemonster 6y ago
This article is spot on. Verifying hardware is a much more complex task than designing it. I've spent ~1.5 years on verifying custom cpu core re-implementation (next gen) that was almost binary compatible with previous gen, we had own gcc backend and LOTS of legacy programs written in assembly. And even then it was a nightmare. Even after tons of time spent on covering most obscure cases, our FPGA platform, running our multi-task FW, would often crash: cue in hours of back-stepping FPGA trace dumps, where you have a total recording of program counter, load-store IF, internal registers for aroung 100k clocks and you have to trace back the execution and see where the fault occurs. Usually it would turn out to be some weirdly obscure combination of recently-fired interrupt, some pipeline stall with execution of some weird instruction. fixed in 3 seconds when found. goddamn that was exciting. I wonder if there are any people here on HN that work on advanced HW verification techniques that are currently ongoing in the industy?
- zozbot234 6y ago
  > This article is spot on. Verifying hardware is a much more complex task than designing it. Are we sure about that? They start out talking about how hard it is to write correct state machines in Verilog, and they follow that with a lot of skepticism about RISC-V processor designs. But most RISC-V designs use Chisel, not Verilog. Chisel is much higher level than Verilog and using it should also make it quite a bit easier to prevent many design-level errors.
  - artemonster 6y ago
    Chisel is not a magic bullet and does not make you suddenly write bug-free hardware. Its the same logic applied when people compare coding in x vs Haskell. The latter does not suddenly grant you superpowers to write bug-free software. >Chisel is much higher level than Verilog have to disagree here. the level of abstraction is exactly the same in chisel as in verilog, its still RTL level. High-level abstraction in hardware is a long-chased holy grail that is revived in the industry every now-and-then and then immdediately dismissed and forgotten, when faced with real-world industry challenges.
    - rowanG077 6y ago
      No is claiming that Chisel code is always bug free. Just like no is claiming Haskell code is bug free. However the point they are making is that these languages are way better in statically catching a whole host of bugs. They also allow certain properties to be expressed in the type system which is essentially having a proof that that invariant holds now matter what.
  - sweden 6y ago
    I don't think that is the point of the article. Even a state machine that is very well written can be wrong if it's fed with the wrong input. And this is the core of the problem with hardware development. You want to make sure that the hardware you are designing never ends up in a situation that you didn't foresee. So you want to make sure that during hardware verification you go through every single possible combination of inputs and you want to make sure you exercise every single possible outcome. The quality of the hardware is not about the language you write your hardware in. It's about the tools, the methodology and the infrastructure you use to exhaustively test your hardware. And this is the point of the article, current processors are so complex and do so many things that the hardware verification suffers from an explosion of states to exercise.
    - petra 6y ago
      On the other hand it's a repeating problem. We've being doing processor verification for a long time. Why can't we reuse that knowledge, that code ?
  - aseipp 6y ago
    It's not the same thing. Chisel will help you do things like write more reusable RTL, because the language itself is more expressive -- so the potential level of abstraction is much higher than what you can achieve in Verilog. You can do more flexible kinds of parameterization, for example. And doing that can help avoid certain kinds of bugs, for sure. (And it also means you simply need less RTL to describe the same thing, because re-use is so much higher.) But Chisel will not help you when you have some errata like "the BRAMs on these devices take 67 cycles to initialize after coming out of reset" tucked away in a manual somewhere that you forgot to read. That's just an FPGA example -- you can run a hundred simulations and your design will still fail immediately in real hardware because of such things. (Even "cycle accurate simulations" from your vendor may not account for these things.) You're going to encounter a lot of problems like this, before you even move to ASIC level flows. Ultimately the hardware industry does need much better RTLs, because the current ones mostly suck from a language POV in a huge number of ways. But verification is still a much bigger problem in general no matter what RTL you've chosen.
    - 9q9 6y ago
      tucked away in a manual With a better eco-system, the (temporal) specification (such as "take 67 cycles to initialize after coming out of reset") such be given formally, so a tool can ensure that it's not forgotten. In principle this is possible. much better RTLs Have you got any concrete ideas and proposals? I'm asking because (A) I agree with you, and (B) I'm very much in a position where I can influence research in this direction.
      - _chris_ 6y ago
        > (B) I'm very much in a position where I can influence research in this direction. Could you elaborate on what you do? > Have you got any concrete ideas and proposals? My recommendation is: try to build something complex, or take something from open-source that is very complex and extend it (test it, verify it, improve performance, etc.). Notice the problems you encounter, and then try to solve those problems. I could give you a laundry list of pain points myself (why do I have to pay money for lint, and why does lint still suck???), but I think the best research is done by people trying to solve problem X, but along the way ended up having to solve Y and Z just to get to X.
- thechao 6y ago
  Yep — and I’ve already said too much. I get upset how secretive this industry is; I truly believe it’s at least 30 years behind the curve (and falling further) compared to SW. I also firmly believe it is the EDA vendors who sell this mindset because $$$.
  - artemonster 6y ago
    this industry is a Borland's wet dream - everyone uses crappy expensive software that reeks of 90s, the libraries are closed source with barely exposed interfaces, the language is a mess (SV), etc. Yes, you are right, as long as its making them huge money nothing will change. And nothing can change, since the field is a frickin mine field of patents, you cannot innovate. Also, why you've said too much? What have you said? :)
    - naringas 6y ago
      sounds like it's well functioning capitalist market. lots of money is made by a small group of very smart people in an entrenched position.
      - dang 6y ago
        Please do not take HN threads on generic ideological tangents. It leads to generic flamewar hell. Regardless of ideological preference, none of us want to end up there. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
      - rsynnott 6y ago
        If the low quality of the software impedes productivity generally (and, being honest, it MUST) then that’s not well-functioning. It’s more a common depressing failure of a capitalist market (and a lot of software verticals suffer from it). It’s a space with a high barrier to entry, which is probably the biggest problem. Software engineering tools got good, and affordable, due to a combination of lowish barrier to entry and open source.
- 9q9 6y ago
  I work in processor verification and my employer spends a great deal of money on it. Like "thechao" I am not allowed to talk about this, alas. A good place to look at some of what ARM do is: https://alastairreid.github.io/
  - carapace 6y ago
    The secrecy is pretty effed up. CPUs are arguably the most crucial mechanical invention in history and yet the practical art of making them actually work right is shrouded in secrecy in service of greed.
    - bgorman 6y ago
      On Windows/MacOS, the communication between the operating system and external devices (drivers) are completely shrouded in mystery. Theoretically the Nvidia graphics driver could be intercepting all your ethernet packets and sending them to the US government. (I have no evidence of such a attack, but it is theoretically possible) Drivers are extremely privileged, and are also a major gateway towards compromising computer systems)
      - tsimionescu 6y ago
        That is not really comparable. If someone murdered all of the GPU driver writers working or retired, we'd have new drivers in maybe 1 year? 2? 5? On the other hand, if someone were to murder all of the currently working or retired processor designers, we'd probably have new i7s in a few decades.
        StillBored 6y ago
        If you don't think the GPU drivers are complex, then look at the perf difference between nouveau and the nvidia proprietary driver. Those guys are also working on the bleeding edge of optimization. Sure you could write a working driver in short order, but its not going to perform the same.
- lisper 6y ago
  I've been working for the last four years as a consultant for a chip design company on one of their internal design tools. The complexity is absolutely mind-boggling. And seeing how the sausage is made doesn't make me any less queasy about it.
- carapace 6y ago
  It sounds like hubris, y'all are trying to bat above your league. Metaphorically, it's like the rush to build auto-autos (self-driving cars): the problem is too hard. If they had started by trying to make a self-driving golf cart Elaine Herzberg might still be alive. If "verifying hardware is a much more complex task than designing it" (and I don't doubt it) then that is the limiting factor. (Or should be IMO. The liberties taken by software cowboys are bad enough w/o the hardware getting all squirrelly too.)
  - DSingularity 6y ago
    Yeah I agree. In my opinion the consequences are showing now that all processor designs are suspect and likely incompatible with the very real threat models we face day to day when we browse the untrusted web.
- Florin_Andrei 6y ago
  So... Doing a complete verification of a CPU in a strict sense is impossible. These days we are relying more and more on computational proofs for math and science research. Therefore - what can we say about the solidity of our scientific knowledge going forward?
- drmpeg 6y ago
  we had own gcc backend Brings me back to 1995 when I was working on the C-Cube CL4010 MPEG-2 encoder processor. A fully custom 32-bit (with some 36-bit registers) RISC architecture. http://www.w6rz.net/cl4010.png http://www.w6rz.net/cl4010.png
- tails4e 6y ago
  This may be true in some cases, but from my experience (nearly 20 years in custom IC design) good design is still more difficult than verification. Verification seems to have an element of over complication. I've seen simple, totally independent sub components take many man weeks of verification, not because it needed to, but because the verification approach, even for simple things can be overly complex. So sometimes the reason verification can take more resources is not because its harder, its because the approach for that particular problem was not efficient. Of course like anything there are extremes on bot sides - but good design must be both correct, efficient, and meet timing/power goals, while verification must just be correct.
- jhallenworld 6y ago
  I have been there, it's horrible. Often traces are in binary, and I spent time making them more readable, to take off the intellectual load of just decoding what you see from the actual debugging. These days I design Verilog for FPGAs without a verification department. I've learned to code defensively, to use what I know is robust without creating complex corner cases. This often involves clear handshaking and dataflow. Anyway, it's more fun than fixing bugs deep in the weeds of a complex system.
bem94 6y ago
I try to keep a list of useful open source hardware verification tools: https://github.com/ben-marshall/awesome-open-hardware-verification https://github.com/ben-marshall/awesome-open-hardware-verifi...
seemslegit 6y ago
Genuine question - other than increasing the number of players, what is the horizon of value for new processors to begin with ? Is there anything more impactful than squeezing a bit more performance/watt ?
- seibelj 6y ago
  If Moore’s law is indeed over and quantum computing isn’t a silver bullet (even if they get it to work), then the future will be clever design rather than raw horsepower in order to increase the speed of computation.
- Havoc 6y ago
  It has the potential to spawn a new eco-system. Like if you look at what happened with the raspberry pis. Other players could have pulled that off technically but didn't attempt it.
  - rrss 6y ago
    raspberry pi didn't really have a new processor though. The original had an ARM11 core (designed and verified by ARM) that was almost a decade old when the raspberry pi launched.
    - ozim 6y ago
      They also leveraged existing Linux drivers and other hardware that was available to integrate without big problems. Spawning new ecosystem is not about making something completely from scratch like processor. One have to align a lot of stars in the sky to make that happen. They had a specific goal and niche where they planted the seed for RPi.
- artemonster 6y ago
  cost. it is stupidly expensive to include any sort of commercial MCU core(s) in your chip. Ancedata: ARM won't even talk with small ASIC fabless companies, even if they are willing to shell out big upfront costs and royalties per chip that ARM demands. Having free or affordable alternaties is a great driving force for the industry.
Havoc 6y ago
>Brute-force solutions to verification closure aren’t feasible. Perhaps a fuzzy/AI type approach? Yeah does seem like a intractable problem for sure
- bem94 6y ago
  Hardware verification engineers call it "constrained random verification", but it's basically fuzzing. This has been the backbone of most commercial hardware verification flows for a long time.
tyingq 6y ago
"Verification of a processor is different from the verification of other pieces of IP, or even an SoC." Not sure I understand this. An SoC is a processor, plus more stuff (memory, I/O), right? Is the idea that it might be easier because the "more stuff" abstracts away some inner details?
- rrss 6y ago
  The processor is usually already verified on its own before it is put into a SoC. Using the raspberry pi example from another thread: broadcom designed the BCM2835 SoC, which included an ARM1176 core. Broadcom probably didn't do a ton of verification for the ARM1176 core itself, since ARM already verified it.
devit 6y ago
Why is it supposed to be so complicated to do "processor verification"? Why can't you simply upload the design to an FPGA, and then check that it can: 1. Boot all available operating systems (Linux, *BSD, Windows, etc.) 2. Successfully compile and run the testsuites for a bunch of open-source software (several languages like Rust have a standardized repository and method to build and run tests, so this is very easy) 3. Correctly run stress testing software (Prime95, etc.) 4. Correctly run several software unit tests that you write to exercise instructions that may not be produced by LLVM/GCC 5. Correctly run tests you write to exercise specific processor/cache states 6. Properly handling fuzzed code without freezing the whole CPU (using afl-fuzz) Start with the simplest possible in-order core so that you get it working very easily, and then evolve to your desired end-state with a series of small commits, and if the verification fails use `git bisect` if needed to find the offending commit, insert any instrumentation you might need to detect the issue and fix it. I don't see why you would need a specialized tool for that, or even what a specialized tool could possibly do.
- rrss 6y ago
  I think you are glossing over a lot of complexity in > 4. Correctly run several software unit tests that you write to exercise instructions that may not be produced by LLVM/GCC and > 5. Correctly run tests you write to exercise specific processor/cache states These two alone seem like they could be really quite complicated. Also, "run the world" is pretty slow when you have to do it in simulation (or emulation if you wait to have a netlist to find out how broken it is). I suspect the coverage from your list is substantially lower than you might expect. Would this have caught F00F? FDIV? AMD Phenom's TLB bug?
- joosters 6y ago
  Have a look at the extensive errata Intel publish for their CPUs. There are hundreds of mistakes in the chips’ behaviour, and yet each buggy CPU would pass your set of tests with flying colours. While you could never release a CPU that didn’t pass the tests you describe, they don’t even begin to exercise all the corner cases for a chip. Multiplying two specific numbers together, while the instruction crosses two memory pages, when an interrupt arrives? How do you even test for that kind of thing?
tyingq 6y ago
Interesting when coupled with how many we are losing. It wasn't that long ago that PA-RISC, Sparc, Alpha, Power, MIPS, etc all credibly competed with one another and Intel. Now it's almost all x86-64 and ARM.
- gok 6y ago
  ISAs are consolidating sure, but the interesting parts of the chips are also consolidating. A few years ago several companies were designing new Arm cores; now it's pretty much just Arm and Apple.
  - rrss 6y ago
    There are still other companies doing new core designs: Huawei/HiSilicon (Taishan v110 in 2019), Marvell/Cavium (Thunder X2 in 2018), Samsung (M4 in 2019), Fujitsu (A64FX is 2019), Nvidia (carmel in 2018). If you count semi-custom cores derived from ARM designs, then add Ampere Computing and Qualcomm as well.
    - gok 6y ago
      In the server space that's true, Arm cores are proliferating. But Samsung shut their core group down and Qualcomm is moving that direction.
mhh__ 6y ago
Anyone working with this stuff: what are some textbooks on verification etc. Preferably a bit of theorem proving too but I'm not sure if that's in the same area of study? I'm coming from a physics background so I never really know where to start when I inevitably start looking this stuff up at 4am.
- vegetablepotpie 6y ago
  I’ve read the Universal Verification Methodology Primer by Ray Salemi. The goal of UVM is to try to catch all the edge cases in simulation before you fab. The basic idea is to make a model of your design and compare it to your implementation by making a set of random inputs and checking the outputs.
- Balgair 6y ago
  The super basics are here: https://www.amazon.com/Art-Electronics-Paul-Horowitz/dp/0521809266 https://www.amazon.com/Art-Electronics-Paul-Horowitz/dp/0521... You'll first have to become 'fluent' in EE, but for a physicist, it's just spending the time and getting used to things. Not terrible, long, but straightforward. As towards what the article is talking about, you need to be trained in it. Honestly, you have to apprentice with the Greybeards (they are mostly men, but not always). There are other ways, like reading through Intel docs or the manuals for ICs or digging through forum posts from 2003. But those guys in the basement with funny newspaper clippings from the 80s or old xkcd printouts are a much better return on your time. They have tons of knowledge about specific chips and machines, stuff that is nearly impossible to recite unless prompted. You just got to spend long lunches blabbering with them, despite their strange political and societal views. Just listen to them, then write down every little thing they said. They are gold in terms of hardware.
  - rrss 6y ago
    Art of Electronics is a nice book, but pretty irrelevant to verification.
  - dirtydroog 6y ago
    What a strange comment.
Someone 6y ago
I would say a modern CPU is a massively parallel program that has lots of shared global state. No wonder that it’s hard to design them. Or is that view to simple?
- qppo 6y ago
  It's more like you have a massively parallel program that you compile ten million times, but only five million complete the process usable afterwards and of those five million, they may or may not have all the parts of the program you intended in the binary. And you have very little insight into it while the compilation takes place. And you have to check/debug those 10 million compiler passes at various stages, and each design change may require developing a new debugger or disassembler from scratch to plug into the compiler at each stage of compilation. What I'm saying is that CPU designs aren't programs, because you can generally trust the compiler to be infallible (and compiler bugs are there, but they're rare). In a CPU process you have to consider the physical impact of the design on manufacturing, what yields you get, how the product is binned, and so on. There are feedback loops between the packaging, testing, and design teams to alter the silicon before production ramps up to go to market. There are tons of moving parts to the actual design process itself, let alone what is being designed.
StillBored 6y ago
Its probably better stated, as its not hard to create a new processor, anymore than it is to create a toy OS. The hard part is creating something that is competitive with top of the line commercial processors that have thousands of man years of R&D poured into them. Its not just verification, but the huge effort that goes into eaking out another couple percent on something like a branch predictor, or optimizing some "edge case" that turns out to be a significant portion of a benchmark if its not done correctly. Then there are all the general optimizations that give you a 10% uplift here and there. Worse, yet if you go with something that doesn't have a large installed software base (x86/arm/power?) because your going to be spending crazy amounts of effort doing compiler+application optimizations as well.
buzzert 6y ago
If anyone is interested in this sort of thing, I would highly recommend checking out the documentary Rise of the Centaur. It’s about a company that I previously hadn’t heard of who was making an x86 compatible CPU based in Austin TX. They show a lot of the verification process throughout the film, including an exciting moment when the chip boots Windows for the first time.
- kqr2 6y ago
  On Amazon prime video: https://smile.amazon.com/Rise-Centaur-Glenn-Henry/dp/B01FSZU6FK https://smile.amazon.com/Rise-Centaur-Glenn-Henry/dp/B01FSZU...
  - 7373737373 6y ago
    Couldn't watch it outside of the US but it's also on Vimeo: https://vimeo.com/ondemand/riseofthecentaur https://vimeo.com/ondemand/riseofthecentaur
ur-whale 6y ago
>“This brings a whole new set of challenges because they are speaking a completely different language, both technically and mentally.” Both true, but also an incomplete picture. It's not just the mental models and the language but also the culture that is very different. H/W guys are bred, born and raised in an environment that thrives on secrecy and where nothing is ever free. The way they transact with one another, the tools they use, and in the end, the very thing they produce all exude that culture. It is exactly the software industry 40 years ago.
- tsimionescu 6y ago
  It's not just HW, it's all big, niche industry, today. Networking for example is the same. If you want to test high scale network equipment OR virtualized network functions, you will buy hundreds of thousands of dollars worth of closed-source testing hardware, software and/or professional services from one of a few big vendors. You will not let anything about your algorithms and designs slip to the outside world, and neither will your test vendors. Edit: the same is true of most of the software world in general. Sure, you have Microsoft and Google and many others collaborating on Linux, or releasing Kubernetes, VS Code, Go and so on. But the core IP that is key to their business? That is staying in-house, fiercely guarded, developped and tested by an army of engineers. The main difference is that there are far fewer well-defined software classes that can be tested generally, so it doesn't make too much sense to look for a 'software testing' industry, like you can for hardware. There are some tool vendors, but they offer far fewer guarantees, since it's hard to imagine a product that could find a large proportion of the bugs in both the Haskell compiler and World of Warcraft.
bgorman 6y ago
Have there been any attempts to use generative testing (e.g. quickcheck) or dependent types to verify processors? I am not sure how quite how this would be integrated into the synthesis of the processor, but it seems to be in line with the general "declarative" approach to building RTL through VHDL I remember from undergrad.
- 9q9 6y ago
  There is a lot of random testing in processor design. To what extent you'd call it property-based testing can be argued. Intersting factoid: Koen Claessen, one of QuickCheck's inventors, also co-designed Lava, a circuit designer DSL in the Haskell eco-system. If by dependent types you mean theorem provers, then that is used, but rarely -- hand-verification doesn't scale to modern processors, usually you model check against some temporal logic formulas that the processor meets its specification. If OTOH you mean using HDLs (= hardware description languages) that use dependent types, then mostly not. Arm's ASL (= Architecture Specification Language) has a tiny bit of dependency build in to reason about length of bit vectors.
alain94040 6y ago
For our software friends wondering what’s so special about processors: they are the most parallel state machines designs out there. Most other state machines have fairly well defined and narrow inputs and outputs. For performance reasons, a cpu pipeline is the biggest collection of state machines interacting with each other directly. Therefore most formal methods blow up on cpu designs and random coverage is really hard to define and even harder to reach.
IshKebab 6y ago
I don't think this is quite right. Based on my (admittedly limited) experience, it takes a lot of work to design and verify a fast processor, but unless your processor is very similar to existing ones (in which case why bother?) it takes way way more work to write all the software needed to support it. I guess everyone underestimates how long it takes to write software - even hardware designers.
qwerty456127 6y ago
Another important question is why is it so hard to use the old ones.
deleted 6y ago
[deleted]

211 points by Lind5 6y ago

This article is spot on. Verifying hardware is a much more complex task than designing it. I've spent ~1.5 years on verifying custom cpu core re-implementation (next gen) that was almost binary compatible with previous gen, we had own gcc backend and LOTS of legacy programs written in assembly. And even then it was a nightmare. Even after tons of time spent on covering most obscure cases, our FPGA platform, running our multi-task FW, would often crash: cue in hours of back-stepping FPGA trace dumps, where you have a total recording of program counter, load-store IF, internal registers for aroung 100k clocks and you have to trace back the execution and see where the fault occurs. Usually it would turn out to be some weirdly obscure combination of recently-fired interrupt, some pipeline stall with execution of some weird instruction. fixed in 3 seconds when found. goddamn that was exciting. I wonder if there are any people here on HN that work on advanced HW verification techniques that are currently ongoing in the industy?

zozbot234 6y ago
> This article is spot on. Verifying hardware is a much more complex task than designing it. Are we sure about that? They start out talking about how hard it is to write correct state machines in Verilog, and they follow that with a lot of skepticism about RISC-V processor designs. But most RISC-V designs use Chisel, not Verilog. Chisel is much higher level than Verilog and using it should also make it quite a bit easier to prevent many design-level errors.
- artemonster 6y ago
  Chisel is not a magic bullet and does not make you suddenly write bug-free hardware. Its the same logic applied when people compare coding in x vs Haskell. The latter does not suddenly grant you superpowers to write bug-free software. >Chisel is much higher level than Verilog have to disagree here. the level of abstraction is exactly the same in chisel as in verilog, its still RTL level. High-level abstraction in hardware is a long-chased holy grail that is revived in the industry every now-and-then and then immdediately dismissed and forgotten, when faced with real-world industry challenges.
  - rowanG077 6y ago
    No is claiming that Chisel code is always bug free. Just like no is claiming Haskell code is bug free. However the point they are making is that these languages are way better in statically catching a whole host of bugs. They also allow certain properties to be expressed in the type system which is essentially having a proof that that invariant holds now matter what.
- sweden 6y ago
  I don't think that is the point of the article. Even a state machine that is very well written can be wrong if it's fed with the wrong input. And this is the core of the problem with hardware development. You want to make sure that the hardware you are designing never ends up in a situation that you didn't foresee. So you want to make sure that during hardware verification you go through every single possible combination of inputs and you want to make sure you exercise every single possible outcome. The quality of the hardware is not about the language you write your hardware in. It's about the tools, the methodology and the infrastructure you use to exhaustively test your hardware. And this is the point of the article, current processors are so complex and do so many things that the hardware verification suffers from an explosion of states to exercise.
  - petra 6y ago
    On the other hand it's a repeating problem. We've being doing processor verification for a long time. Why can't we reuse that knowledge, that code ?
- aseipp 6y ago
  It's not the same thing. Chisel will help you do things like write more reusable RTL, because the language itself is more expressive -- so the potential level of abstraction is much higher than what you can achieve in Verilog. You can do more flexible kinds of parameterization, for example. And doing that can help avoid certain kinds of bugs, for sure. (And it also means you simply need less RTL to describe the same thing, because re-use is so much higher.) But Chisel will not help you when you have some errata like "the BRAMs on these devices take 67 cycles to initialize after coming out of reset" tucked away in a manual somewhere that you forgot to read. That's just an FPGA example -- you can run a hundred simulations and your design will still fail immediately in real hardware because of such things. (Even "cycle accurate simulations" from your vendor may not account for these things.) You're going to encounter a lot of problems like this, before you even move to ASIC level flows. Ultimately the hardware industry does need much better RTLs, because the current ones mostly suck from a language POV in a huge number of ways. But verification is still a much bigger problem in general no matter what RTL you've chosen.
  - 9q9 6y ago
    tucked away in a manual With a better eco-system, the (temporal) specification (such as "take 67 cycles to initialize after coming out of reset") such be given formally, so a tool can ensure that it's not forgotten. In principle this is possible. much better RTLs Have you got any concrete ideas and proposals? I'm asking because (A) I agree with you, and (B) I'm very much in a position where I can influence research in this direction.
    - _chris_ 6y ago
      > (B) I'm very much in a position where I can influence research in this direction. Could you elaborate on what you do? > Have you got any concrete ideas and proposals? My recommendation is: try to build something complex, or take something from open-source that is very complex and extend it (test it, verify it, improve performance, etc.). Notice the problems you encounter, and then try to solve those problems. I could give you a laundry list of pain points myself (why do I have to pay money for lint, and why does lint still suck???), but I think the best research is done by people trying to solve problem X, but along the way ended up having to solve Y and Z just to get to X.
thechao 6y ago
Yep — and I’ve already said too much. I get upset how secretive this industry is; I truly believe it’s at least 30 years behind the curve (and falling further) compared to SW. I also firmly believe it is the EDA vendors who sell this mindset because $$$.
- artemonster 6y ago
  this industry is a Borland's wet dream - everyone uses crappy expensive software that reeks of 90s, the libraries are closed source with barely exposed interfaces, the language is a mess (SV), etc. Yes, you are right, as long as its making them huge money nothing will change. And nothing can change, since the field is a frickin mine field of patents, you cannot innovate. Also, why you've said too much? What have you said? :)
  - naringas 6y ago
    sounds like it's well functioning capitalist market. lots of money is made by a small group of very smart people in an entrenched position.
    - dang 6y ago
      Please do not take HN threads on generic ideological tangents. It leads to generic flamewar hell. Regardless of ideological preference, none of us want to end up there. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
    - rsynnott 6y ago
      If the low quality of the software impedes productivity generally (and, being honest, it MUST) then that’s not well-functioning. It’s more a common depressing failure of a capitalist market (and a lot of software verticals suffer from it). It’s a space with a high barrier to entry, which is probably the biggest problem. Software engineering tools got good, and affordable, due to a combination of lowish barrier to entry and open source.
9q9 6y ago
I work in processor verification and my employer spends a great deal of money on it. Like "thechao" I am not allowed to talk about this, alas. A good place to look at some of what ARM do is: https://alastairreid.github.io/
- carapace 6y ago
  The secrecy is pretty effed up. CPUs are arguably the most crucial mechanical invention in history and yet the practical art of making them actually work right is shrouded in secrecy in service of greed.
  - bgorman 6y ago
    On Windows/MacOS, the communication between the operating system and external devices (drivers) are completely shrouded in mystery. Theoretically the Nvidia graphics driver could be intercepting all your ethernet packets and sending them to the US government. (I have no evidence of such a attack, but it is theoretically possible) Drivers are extremely privileged, and are also a major gateway towards compromising computer systems)
    - tsimionescu 6y ago
      That is not really comparable. If someone murdered all of the GPU driver writers working or retired, we'd have new drivers in maybe 1 year? 2? 5? On the other hand, if someone were to murder all of the currently working or retired processor designers, we'd probably have new i7s in a few decades.
      - StillBored 6y ago
        If you don't think the GPU drivers are complex, then look at the perf difference between nouveau and the nvidia proprietary driver. Those guys are also working on the bleeding edge of optimization. Sure you could write a working driver in short order, but its not going to perform the same.
lisper 6y ago
I've been working for the last four years as a consultant for a chip design company on one of their internal design tools. The complexity is absolutely mind-boggling. And seeing how the sausage is made doesn't make me any less queasy about it.
carapace 6y ago
It sounds like hubris, y'all are trying to bat above your league. Metaphorically, it's like the rush to build auto-autos (self-driving cars): the problem is too hard. If they had started by trying to make a self-driving golf cart Elaine Herzberg might still be alive. If "verifying hardware is a much more complex task than designing it" (and I don't doubt it) then that is the limiting factor. (Or should be IMO. The liberties taken by software cowboys are bad enough w/o the hardware getting all squirrelly too.)
- DSingularity 6y ago
  Yeah I agree. In my opinion the consequences are showing now that all processor designs are suspect and likely incompatible with the very real threat models we face day to day when we browse the untrusted web.
Florin_Andrei 6y ago
So... Doing a complete verification of a CPU in a strict sense is impossible. These days we are relying more and more on computational proofs for math and science research. Therefore - what can we say about the solidity of our scientific knowledge going forward?
drmpeg 6y ago
we had own gcc backend Brings me back to 1995 when I was working on the C-Cube CL4010 MPEG-2 encoder processor. A fully custom 32-bit (with some 36-bit registers) RISC architecture. http://www.w6rz.net/cl4010.png http://www.w6rz.net/cl4010.png
tails4e 6y ago
This may be true in some cases, but from my experience (nearly 20 years in custom IC design) good design is still more difficult than verification. Verification seems to have an element of over complication. I've seen simple, totally independent sub components take many man weeks of verification, not because it needed to, but because the verification approach, even for simple things can be overly complex. So sometimes the reason verification can take more resources is not because its harder, its because the approach for that particular problem was not efficient. Of course like anything there are extremes on bot sides - but good design must be both correct, efficient, and meet timing/power goals, while verification must just be correct.
jhallenworld 6y ago
I have been there, it's horrible. Often traces are in binary, and I spent time making them more readable, to take off the intellectual load of just decoding what you see from the actual debugging. These days I design Verilog for FPGAs without a verification department. I've learned to code defensively, to use what I know is robust without creating complex corner cases. This often involves clear handshaking and dataflow. Anyway, it's more fun than fixing bugs deep in the weeds of a complex system.

bem94 6y ago

I try to keep a list of useful open source hardware verification tools: https://github.com/ben-marshall/awesome-open-hardware-verification https://github.com/ben-marshall/awesome-open-hardware-verifi...

seemslegit 6y ago

Genuine question - other than increasing the number of players, what is the horizon of value for new processors to begin with ? Is there anything more impactful than squeezing a bit more performance/watt ?

seibelj 6y ago
If Moore’s law is indeed over and quantum computing isn’t a silver bullet (even if they get it to work), then the future will be clever design rather than raw horsepower in order to increase the speed of computation.
Havoc 6y ago
It has the potential to spawn a new eco-system. Like if you look at what happened with the raspberry pis. Other players could have pulled that off technically but didn't attempt it.
- rrss 6y ago
  raspberry pi didn't really have a new processor though. The original had an ARM11 core (designed and verified by ARM) that was almost a decade old when the raspberry pi launched.
  - ozim 6y ago
    They also leveraged existing Linux drivers and other hardware that was available to integrate without big problems. Spawning new ecosystem is not about making something completely from scratch like processor. One have to align a lot of stars in the sky to make that happen. They had a specific goal and niche where they planted the seed for RPi.
artemonster 6y ago
cost. it is stupidly expensive to include any sort of commercial MCU core(s) in your chip. Ancedata: ARM won't even talk with small ASIC fabless companies, even if they are willing to shell out big upfront costs and royalties per chip that ARM demands. Having free or affordable alternaties is a great driving force for the industry.

Havoc 6y ago

>Brute-force solutions to verification closure aren’t feasible. Perhaps a fuzzy/AI type approach? Yeah does seem like a intractable problem for sure

bem94 6y ago
Hardware verification engineers call it "constrained random verification", but it's basically fuzzing. This has been the backbone of most commercial hardware verification flows for a long time.

tyingq 6y ago

"Verification of a processor is different from the verification of other pieces of IP, or even an SoC." Not sure I understand this. An SoC is a processor, plus more stuff (memory, I/O), right? Is the idea that it might be easier because the "more stuff" abstracts away some inner details?

rrss 6y ago
The processor is usually already verified on its own before it is put into a SoC. Using the raspberry pi example from another thread: broadcom designed the BCM2835 SoC, which included an ARM1176 core. Broadcom probably didn't do a ton of verification for the ARM1176 core itself, since ARM already verified it.

devit 6y ago

Why is it supposed to be so complicated to do "processor verification"? Why can't you simply upload the design to an FPGA, and then check that it can: 1. Boot all available operating systems (Linux, *BSD, Windows, etc.) 2. Successfully compile and run the testsuites for a bunch of open-source software (several languages like Rust have a standardized repository and method to build and run tests, so this is very easy) 3. Correctly run stress testing software (Prime95, etc.) 4. Correctly run several software unit tests that you write to exercise instructions that may not be produced by LLVM/GCC 5. Correctly run tests you write to exercise specific processor/cache states 6. Properly handling fuzzed code without freezing the whole CPU (using afl-fuzz) Start with the simplest possible in-order core so that you get it working very easily, and then evolve to your desired end-state with a series of small commits, and if the verification fails use `git bisect` if needed to find the offending commit, insert any instrumentation you might need to detect the issue and fix it. I don't see why you would need a specialized tool for that, or even what a specialized tool could possibly do.

rrss 6y ago
I think you are glossing over a lot of complexity in > 4. Correctly run several software unit tests that you write to exercise instructions that may not be produced by LLVM/GCC and > 5. Correctly run tests you write to exercise specific processor/cache states These two alone seem like they could be really quite complicated. Also, "run the world" is pretty slow when you have to do it in simulation (or emulation if you wait to have a netlist to find out how broken it is). I suspect the coverage from your list is substantially lower than you might expect. Would this have caught F00F? FDIV? AMD Phenom's TLB bug?
joosters 6y ago
Have a look at the extensive errata Intel publish for their CPUs. There are hundreds of mistakes in the chips’ behaviour, and yet each buggy CPU would pass your set of tests with flying colours. While you could never release a CPU that didn’t pass the tests you describe, they don’t even begin to exercise all the corner cases for a chip. Multiplying two specific numbers together, while the instruction crosses two memory pages, when an interrupt arrives? How do you even test for that kind of thing?

tyingq 6y ago

Interesting when coupled with how many we are losing. It wasn't that long ago that PA-RISC, Sparc, Alpha, Power, MIPS, etc all credibly competed with one another and Intel. Now it's almost all x86-64 and ARM.

gok 6y ago
ISAs are consolidating sure, but the interesting parts of the chips are also consolidating. A few years ago several companies were designing new Arm cores; now it's pretty much just Arm and Apple.
- rrss 6y ago
  There are still other companies doing new core designs: Huawei/HiSilicon (Taishan v110 in 2019), Marvell/Cavium (Thunder X2 in 2018), Samsung (M4 in 2019), Fujitsu (A64FX is 2019), Nvidia (carmel in 2018). If you count semi-custom cores derived from ARM designs, then add Ampere Computing and Qualcomm as well.
  - gok 6y ago
    In the server space that's true, Arm cores are proliferating. But Samsung shut their core group down and Qualcomm is moving that direction.

mhh__ 6y ago

Anyone working with this stuff: what are some textbooks on verification etc. Preferably a bit of theorem proving too but I'm not sure if that's in the same area of study? I'm coming from a physics background so I never really know where to start when I inevitably start looking this stuff up at 4am.

vegetablepotpie 6y ago
I’ve read the Universal Verification Methodology Primer by Ray Salemi. The goal of UVM is to try to catch all the edge cases in simulation before you fab. The basic idea is to make a model of your design and compare it to your implementation by making a set of random inputs and checking the outputs.
Balgair 6y ago
The super basics are here: https://www.amazon.com/Art-Electronics-Paul-Horowitz/dp/0521809266 https://www.amazon.com/Art-Electronics-Paul-Horowitz/dp/0521... You'll first have to become 'fluent' in EE, but for a physicist, it's just spending the time and getting used to things. Not terrible, long, but straightforward. As towards what the article is talking about, you need to be trained in it. Honestly, you have to apprentice with the Greybeards (they are mostly men, but not always). There are other ways, like reading through Intel docs or the manuals for ICs or digging through forum posts from 2003. But those guys in the basement with funny newspaper clippings from the 80s or old xkcd printouts are a much better return on your time. They have tons of knowledge about specific chips and machines, stuff that is nearly impossible to recite unless prompted. You just got to spend long lunches blabbering with them, despite their strange political and societal views. Just listen to them, then write down every little thing they said. They are gold in terms of hardware.
- rrss 6y ago
  Art of Electronics is a nice book, but pretty irrelevant to verification.
- dirtydroog 6y ago
  What a strange comment.

Someone 6y ago

I would say a modern CPU is a massively parallel program that has lots of shared global state. No wonder that it’s hard to design them. Or is that view to simple?

qppo 6y ago
It's more like you have a massively parallel program that you compile ten million times, but only five million complete the process usable afterwards and of those five million, they may or may not have all the parts of the program you intended in the binary. And you have very little insight into it while the compilation takes place. And you have to check/debug those 10 million compiler passes at various stages, and each design change may require developing a new debugger or disassembler from scratch to plug into the compiler at each stage of compilation. What I'm saying is that CPU designs aren't programs, because you can generally trust the compiler to be infallible (and compiler bugs are there, but they're rare). In a CPU process you have to consider the physical impact of the design on manufacturing, what yields you get, how the product is binned, and so on. There are feedback loops between the packaging, testing, and design teams to alter the silicon before production ramps up to go to market. There are tons of moving parts to the actual design process itself, let alone what is being designed.

StillBored 6y ago

Its probably better stated, as its not hard to create a new processor, anymore than it is to create a toy OS. The hard part is creating something that is competitive with top of the line commercial processors that have thousands of man years of R&D poured into them. Its not just verification, but the huge effort that goes into eaking out another couple percent on something like a branch predictor, or optimizing some "edge case" that turns out to be a significant portion of a benchmark if its not done correctly. Then there are all the general optimizations that give you a 10% uplift here and there. Worse, yet if you go with something that doesn't have a large installed software base (x86/arm/power?) because your going to be spending crazy amounts of effort doing compiler+application optimizations as well.

buzzert 6y ago

If anyone is interested in this sort of thing, I would highly recommend checking out the documentary Rise of the Centaur. It’s about a company that I previously hadn’t heard of who was making an x86 compatible CPU based in Austin TX. They show a lot of the verification process throughout the film, including an exciting moment when the chip boots Windows for the first time.

kqr2 6y ago
On Amazon prime video: https://smile.amazon.com/Rise-Centaur-Glenn-Henry/dp/B01FSZU6FK https://smile.amazon.com/Rise-Centaur-Glenn-Henry/dp/B01FSZU...
- 7373737373 6y ago
  Couldn't watch it outside of the US but it's also on Vimeo: https://vimeo.com/ondemand/riseofthecentaur https://vimeo.com/ondemand/riseofthecentaur

ur-whale 6y ago

>“This brings a whole new set of challenges because they are speaking a completely different language, both technically and mentally.” Both true, but also an incomplete picture. It's not just the mental models and the language but also the culture that is very different. H/W guys are bred, born and raised in an environment that thrives on secrecy and where nothing is ever free. The way they transact with one another, the tools they use, and in the end, the very thing they produce all exude that culture. It is exactly the software industry 40 years ago.

tsimionescu 6y ago
It's not just HW, it's all big, niche industry, today. Networking for example is the same. If you want to test high scale network equipment OR virtualized network functions, you will buy hundreds of thousands of dollars worth of closed-source testing hardware, software and/or professional services from one of a few big vendors. You will not let anything about your algorithms and designs slip to the outside world, and neither will your test vendors. Edit: the same is true of most of the software world in general. Sure, you have Microsoft and Google and many others collaborating on Linux, or releasing Kubernetes, VS Code, Go and so on. But the core IP that is key to their business? That is staying in-house, fiercely guarded, developped and tested by an army of engineers. The main difference is that there are far fewer well-defined software classes that can be tested generally, so it doesn't make too much sense to look for a 'software testing' industry, like you can for hardware. There are some tool vendors, but they offer far fewer guarantees, since it's hard to imagine a product that could find a large proportion of the bugs in both the Haskell compiler and World of Warcraft.

bgorman 6y ago

Have there been any attempts to use generative testing (e.g. quickcheck) or dependent types to verify processors? I am not sure how quite how this would be integrated into the synthesis of the processor, but it seems to be in line with the general "declarative" approach to building RTL through VHDL I remember from undergrad.

9q9 6y ago
There is a lot of random testing in processor design. To what extent you'd call it property-based testing can be argued. Intersting factoid: Koen Claessen, one of QuickCheck's inventors, also co-designed Lava, a circuit designer DSL in the Haskell eco-system. If by dependent types you mean theorem provers, then that is used, but rarely -- hand-verification doesn't scale to modern processors, usually you model check against some temporal logic formulas that the processor meets its specification. If OTOH you mean using HDLs (= hardware description languages) that use dependent types, then mostly not. Arm's ASL (= Architecture Specification Language) has a tiny bit of dependency build in to reason about length of bit vectors.

alain94040 6y ago

For our software friends wondering what’s so special about processors: they are the most parallel state machines designs out there. Most other state machines have fairly well defined and narrow inputs and outputs. For performance reasons, a cpu pipeline is the biggest collection of state machines interacting with each other directly. Therefore most formal methods blow up on cpu designs and random coverage is really hard to define and even harder to reach.

IshKebab 6y ago

I don't think this is quite right. Based on my (admittedly limited) experience, it takes a lot of work to design and verify a fast processor, but unless your processor is very similar to existing ones (in which case why bother?) it takes way way more work to write all the software needed to support it. I guess everyone underestimates how long it takes to write software - even hardware designers.

qwerty456127 6y ago

Another important question is why is it so hard to use the old ones.

deleted 6y ago

[deleted]