8 ms·

This article is spot on. Verifying hardware is a much more complex task than designing it. I've spent ~1.5 years on verifying custom cpu core re-implementation

This article is spot on. Verifying hardware is a much more complex task than designing it. I've spent ~1.5 years on verifying custom cpu core re-implementation (next gen) that was almost binary compatible with previous gen, we had own gcc backend and LOTS of legacy programs written in assembly. And even then it was a nightmare. Even after tons of time spent on covering most obscure cases, our FPGA platform, running our multi-task FW, would often crash: cue in hours of back-stepping FPGA trace dumps, where you have a total recording of program counter, load-store IF, internal registers for aroung 100k clocks and you have to trace back the execution and see where the fault occurs. Usually it would turn out to be some weirdly obscure combination of recently-fired interrupt, some pipeline stall with execution of some weird instruction. fixed in 3 seconds when found. goddamn that was exciting. I wonder if there are any people here on HN that work on advanced HW verification techniques that are currently ongoing in the industy?

zozbot234 7y ago
> This article is spot on. Verifying hardware is a much more complex task than designing it. Are we sure about that? They start out talking about how hard it is to write correct state machines in Verilog, and they follow that with a lot of skepticism about RISC-V processor designs. But most RISC-V designs use Chisel, not Verilog. Chisel is much higher level than Verilog and using it should also make it quite a bit easier to prevent many design-level errors.
- artemonster 7y ago
  Chisel is not a magic bullet and does not make you suddenly write bug-free hardware. Its the same logic applied when people compare coding in x vs Haskell. The latter does not suddenly grant you superpowers to write bug-free software. >Chisel is much higher level than Verilog have to disagree here. the level of abstraction is exactly the same in chisel as in verilog, its still RTL level. High-level abstraction in hardware is a long-chased holy grail that is revived in the industry every now-and-then and then immdediately dismissed and forgotten, when faced with real-world industry challenges.
  - rowanG077 7y ago
    No is claiming that Chisel code is always bug free. Just like no is claiming Haskell code is bug free. However the point they are making is that these languages are way better in statically catching a whole host of bugs. They also allow certain properties to be expressed in the type system which is essentially having a proof that that invariant holds now matter what.
- sweden 7y ago
  I don't think that is the point of the article. Even a state machine that is very well written can be wrong if it's fed with the wrong input. And this is the core of the problem with hardware development. You want to make sure that the hardware you are designing never ends up in a situation that you didn't foresee. So you want to make sure that during hardware verification you go through every single possible combination of inputs and you want to make sure you exercise every single possible outcome. The quality of the hardware is not about the language you write your hardware in. It's about the tools, the methodology and the infrastructure you use to exhaustively test your hardware. And this is the point of the article, current processors are so complex and do so many things that the hardware verification suffers from an explosion of states to exercise.
  - petra 7y ago
    On the other hand it's a repeating problem. We've being doing processor verification for a long time. Why can't we reuse that knowledge, that code ?
- aseipp 7y ago
  It's not the same thing. Chisel will help you do things like write more reusable RTL, because the language itself is more expressive -- so the potential level of abstraction is much higher than what you can achieve in Verilog. You can do more flexible kinds of parameterization, for example. And doing that can help avoid certain kinds of bugs, for sure. (And it also means you simply need less RTL to describe the same thing, because re-use is so much higher.) But Chisel will not help you when you have some errata like "the BRAMs on these devices take 67 cycles to initialize after coming out of reset" tucked away in a manual somewhere that you forgot to read. That's just an FPGA example -- you can run a hundred simulations and your design will still fail immediately in real hardware because of such things. (Even "cycle accurate simulations" from your vendor may not account for these things.) You're going to encounter a lot of problems like this, before you even move to ASIC level flows. Ultimately the hardware industry does need much better RTLs, because the current ones mostly suck from a language POV in a huge number of ways. But verification is still a much bigger problem in general no matter what RTL you've chosen.
  - 9q9 7y ago
    tucked away in a manual With a better eco-system, the (temporal) specification (such as "take 67 cycles to initialize after coming out of reset") such be given formally, so a tool can ensure that it's not forgotten. In principle this is possible. much better RTLs Have you got any concrete ideas and proposals? I'm asking because (A) I agree with you, and (B) I'm very much in a position where I can influence research in this direction.
    - _chris_ 7y ago
      > (B) I'm very much in a position where I can influence research in this direction. Could you elaborate on what you do? > Have you got any concrete ideas and proposals? My recommendation is: try to build something complex, or take something from open-source that is very complex and extend it (test it, verify it, improve performance, etc.). Notice the problems you encounter, and then try to solve those problems. I could give you a laundry list of pain points myself (why do I have to pay money for lint, and why does lint still suck???), but I think the best research is done by people trying to solve problem X, but along the way ended up having to solve Y and Z just to get to X.
thechao 7y ago
Yep — and I’ve already said too much. I get upset how secretive this industry is; I truly believe it’s at least 30 years behind the curve (and falling further) compared to SW. I also firmly believe it is the EDA vendors who sell this mindset because $$$.
- artemonster 7y ago
  this industry is a Borland's wet dream - everyone uses crappy expensive software that reeks of 90s, the libraries are closed source with barely exposed interfaces, the language is a mess (SV), etc. Yes, you are right, as long as its making them huge money nothing will change. And nothing can change, since the field is a frickin mine field of patents, you cannot innovate. Also, why you've said too much? What have you said? :)
  - naringas 7y ago
    sounds like it's well functioning capitalist market. lots of money is made by a small group of very smart people in an entrenched position.
    - dang 7y ago
      Please do not take HN threads on generic ideological tangents. It leads to generic flamewar hell. Regardless of ideological preference, none of us want to end up there. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
    - rsynnott 7y ago
      If the low quality of the software impedes productivity generally (and, being honest, it MUST) then that’s not well-functioning. It’s more a common depressing failure of a capitalist market (and a lot of software verticals suffer from it). It’s a space with a high barrier to entry, which is probably the biggest problem. Software engineering tools got good, and affordable, due to a combination of lowish barrier to entry and open source.
9q9 7y ago
I work in processor verification and my employer spends a great deal of money on it. Like "thechao" I am not allowed to talk about this, alas. A good place to look at some of what ARM do is: https://alastairreid.github.io/
- carapace 7y ago
  The secrecy is pretty effed up. CPUs are arguably the most crucial mechanical invention in history and yet the practical art of making them actually work right is shrouded in secrecy in service of greed.
  - bgorman 7y ago
    On Windows/MacOS, the communication between the operating system and external devices (drivers) are completely shrouded in mystery. Theoretically the Nvidia graphics driver could be intercepting all your ethernet packets and sending them to the US government. (I have no evidence of such a attack, but it is theoretically possible) Drivers are extremely privileged, and are also a major gateway towards compromising computer systems)
    - tsimionescu 7y ago
      That is not really comparable. If someone murdered all of the GPU driver writers working or retired, we'd have new drivers in maybe 1 year? 2? 5? On the other hand, if someone were to murder all of the currently working or retired processor designers, we'd probably have new i7s in a few decades.
      - StillBored 7y ago
        If you don't think the GPU drivers are complex, then look at the perf difference between nouveau and the nvidia proprietary driver. Those guys are also working on the bleeding edge of optimization. Sure you could write a working driver in short order, but its not going to perform the same.
lisper 7y ago
I've been working for the last four years as a consultant for a chip design company on one of their internal design tools. The complexity is absolutely mind-boggling. And seeing how the sausage is made doesn't make me any less queasy about it.
carapace 7y ago
It sounds like hubris, y'all are trying to bat above your league. Metaphorically, it's like the rush to build auto-autos (self-driving cars): the problem is too hard. If they had started by trying to make a self-driving golf cart Elaine Herzberg might still be alive. If "verifying hardware is a much more complex task than designing it" (and I don't doubt it) then that is the limiting factor. (Or should be IMO. The liberties taken by software cowboys are bad enough w/o the hardware getting all squirrelly too.)
- DSingularity 7y ago
  Yeah I agree. In my opinion the consequences are showing now that all processor designs are suspect and likely incompatible with the very real threat models we face day to day when we browse the untrusted web.
Florin_Andrei 7y ago
So... Doing a complete verification of a CPU in a strict sense is impossible. These days we are relying more and more on computational proofs for math and science research. Therefore - what can we say about the solidity of our scientific knowledge going forward?
drmpeg 7y ago
we had own gcc backend Brings me back to 1995 when I was working on the C-Cube CL4010 MPEG-2 encoder processor. A fully custom 32-bit (with some 36-bit registers) RISC architecture. http://www.w6rz.net/cl4010.png http://www.w6rz.net/cl4010.png
tails4e 7y ago
This may be true in some cases, but from my experience (nearly 20 years in custom IC design) good design is still more difficult than verification. Verification seems to have an element of over complication. I've seen simple, totally independent sub components take many man weeks of verification, not because it needed to, but because the verification approach, even for simple things can be overly complex. So sometimes the reason verification can take more resources is not because its harder, its because the approach for that particular problem was not efficient. Of course like anything there are extremes on bot sides - but good design must be both correct, efficient, and meet timing/power goals, while verification must just be correct.
jhallenworld 7y ago
I have been there, it's horrible. Often traces are in binary, and I spent time making them more readable, to take off the intellectual load of just decoding what you see from the actual debugging. These days I design Verilog for FPGAs without a verification department. I've learned to code defensively, to use what I know is robust without creating complex corner cases. This often involves clear handshaking and dataflow. Anyway, it's more fun than fixing bugs deep in the weeds of a complex system.

zozbot234 7y ago

> This article is spot on. Verifying hardware is a much more complex task than designing it. Are we sure about that? They start out talking about how hard it is to write correct state machines in Verilog, and they follow that with a lot of skepticism about RISC-V processor designs. But most RISC-V designs use Chisel, not Verilog. Chisel is much higher level than Verilog and using it should also make it quite a bit easier to prevent many design-level errors.

artemonster 7y ago
Chisel is not a magic bullet and does not make you suddenly write bug-free hardware. Its the same logic applied when people compare coding in x vs Haskell. The latter does not suddenly grant you superpowers to write bug-free software. >Chisel is much higher level than Verilog have to disagree here. the level of abstraction is exactly the same in chisel as in verilog, its still RTL level. High-level abstraction in hardware is a long-chased holy grail that is revived in the industry every now-and-then and then immdediately dismissed and forgotten, when faced with real-world industry challenges.
- rowanG077 7y ago
  No is claiming that Chisel code is always bug free. Just like no is claiming Haskell code is bug free. However the point they are making is that these languages are way better in statically catching a whole host of bugs. They also allow certain properties to be expressed in the type system which is essentially having a proof that that invariant holds now matter what.
sweden 7y ago
I don't think that is the point of the article. Even a state machine that is very well written can be wrong if it's fed with the wrong input. And this is the core of the problem with hardware development. You want to make sure that the hardware you are designing never ends up in a situation that you didn't foresee. So you want to make sure that during hardware verification you go through every single possible combination of inputs and you want to make sure you exercise every single possible outcome. The quality of the hardware is not about the language you write your hardware in. It's about the tools, the methodology and the infrastructure you use to exhaustively test your hardware. And this is the point of the article, current processors are so complex and do so many things that the hardware verification suffers from an explosion of states to exercise.
- petra 7y ago
  On the other hand it's a repeating problem. We've being doing processor verification for a long time. Why can't we reuse that knowledge, that code ?
aseipp 7y ago
It's not the same thing. Chisel will help you do things like write more reusable RTL, because the language itself is more expressive -- so the potential level of abstraction is much higher than what you can achieve in Verilog. You can do more flexible kinds of parameterization, for example. And doing that can help avoid certain kinds of bugs, for sure. (And it also means you simply need less RTL to describe the same thing, because re-use is so much higher.) But Chisel will not help you when you have some errata like "the BRAMs on these devices take 67 cycles to initialize after coming out of reset" tucked away in a manual somewhere that you forgot to read. That's just an FPGA example -- you can run a hundred simulations and your design will still fail immediately in real hardware because of such things. (Even "cycle accurate simulations" from your vendor may not account for these things.) You're going to encounter a lot of problems like this, before you even move to ASIC level flows. Ultimately the hardware industry does need much better RTLs, because the current ones mostly suck from a language POV in a huge number of ways. But verification is still a much bigger problem in general no matter what RTL you've chosen.
- 9q9 7y ago
  tucked away in a manual With a better eco-system, the (temporal) specification (such as "take 67 cycles to initialize after coming out of reset") such be given formally, so a tool can ensure that it's not forgotten. In principle this is possible. much better RTLs Have you got any concrete ideas and proposals? I'm asking because (A) I agree with you, and (B) I'm very much in a position where I can influence research in this direction.
  - _chris_ 7y ago
    > (B) I'm very much in a position where I can influence research in this direction. Could you elaborate on what you do? > Have you got any concrete ideas and proposals? My recommendation is: try to build something complex, or take something from open-source that is very complex and extend it (test it, verify it, improve performance, etc.). Notice the problems you encounter, and then try to solve those problems. I could give you a laundry list of pain points myself (why do I have to pay money for lint, and why does lint still suck???), but I think the best research is done by people trying to solve problem X, but along the way ended up having to solve Y and Z just to get to X.

thechao 7y ago

Yep — and I’ve already said too much. I get upset how secretive this industry is; I truly believe it’s at least 30 years behind the curve (and falling further) compared to SW. I also firmly believe it is the EDA vendors who sell this mindset because $$$.

artemonster 7y ago
this industry is a Borland's wet dream - everyone uses crappy expensive software that reeks of 90s, the libraries are closed source with barely exposed interfaces, the language is a mess (SV), etc. Yes, you are right, as long as its making them huge money nothing will change. And nothing can change, since the field is a frickin mine field of patents, you cannot innovate. Also, why you've said too much? What have you said? :)
- naringas 7y ago
  sounds like it's well functioning capitalist market. lots of money is made by a small group of very smart people in an entrenched position.
  - dang 7y ago
    Please do not take HN threads on generic ideological tangents. It leads to generic flamewar hell. Regardless of ideological preference, none of us want to end up there. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
  - rsynnott 7y ago
    If the low quality of the software impedes productivity generally (and, being honest, it MUST) then that’s not well-functioning. It’s more a common depressing failure of a capitalist market (and a lot of software verticals suffer from it). It’s a space with a high barrier to entry, which is probably the biggest problem. Software engineering tools got good, and affordable, due to a combination of lowish barrier to entry and open source.

9q9 7y ago

I work in processor verification and my employer spends a great deal of money on it. Like "thechao" I am not allowed to talk about this, alas. A good place to look at some of what ARM do is: https://alastairreid.github.io/

carapace 7y ago
The secrecy is pretty effed up. CPUs are arguably the most crucial mechanical invention in history and yet the practical art of making them actually work right is shrouded in secrecy in service of greed.
- bgorman 7y ago
  On Windows/MacOS, the communication between the operating system and external devices (drivers) are completely shrouded in mystery. Theoretically the Nvidia graphics driver could be intercepting all your ethernet packets and sending them to the US government. (I have no evidence of such a attack, but it is theoretically possible) Drivers are extremely privileged, and are also a major gateway towards compromising computer systems)
  - tsimionescu 7y ago
    That is not really comparable. If someone murdered all of the GPU driver writers working or retired, we'd have new drivers in maybe 1 year? 2? 5? On the other hand, if someone were to murder all of the currently working or retired processor designers, we'd probably have new i7s in a few decades.
    - StillBored 7y ago
      If you don't think the GPU drivers are complex, then look at the perf difference between nouveau and the nvidia proprietary driver. Those guys are also working on the bleeding edge of optimization. Sure you could write a working driver in short order, but its not going to perform the same.

lisper 7y ago

I've been working for the last four years as a consultant for a chip design company on one of their internal design tools. The complexity is absolutely mind-boggling. And seeing how the sausage is made doesn't make me any less queasy about it.

carapace 7y ago

It sounds like hubris, y'all are trying to bat above your league. Metaphorically, it's like the rush to build auto-autos (self-driving cars): the problem is too hard. If they had started by trying to make a self-driving golf cart Elaine Herzberg might still be alive. If "verifying hardware is a much more complex task than designing it" (and I don't doubt it) then that is the limiting factor. (Or should be IMO. The liberties taken by software cowboys are bad enough w/o the hardware getting all squirrelly too.)

DSingularity 7y ago
Yeah I agree. In my opinion the consequences are showing now that all processor designs are suspect and likely incompatible with the very real threat models we face day to day when we browse the untrusted web.

Florin_Andrei 7y ago

So... Doing a complete verification of a CPU in a strict sense is impossible. These days we are relying more and more on computational proofs for math and science research. Therefore - what can we say about the solidity of our scientific knowledge going forward?

drmpeg 7y ago

we had own gcc backend Brings me back to 1995 when I was working on the C-Cube CL4010 MPEG-2 encoder processor. A fully custom 32-bit (with some 36-bit registers) RISC architecture. http://www.w6rz.net/cl4010.png http://www.w6rz.net/cl4010.png

tails4e 7y ago

This may be true in some cases, but from my experience (nearly 20 years in custom IC design) good design is still more difficult than verification. Verification seems to have an element of over complication. I've seen simple, totally independent sub components take many man weeks of verification, not because it needed to, but because the verification approach, even for simple things can be overly complex. So sometimes the reason verification can take more resources is not because its harder, its because the approach for that particular problem was not efficient. Of course like anything there are extremes on bot sides - but good design must be both correct, efficient, and meet timing/power goals, while verification must just be correct.

jhallenworld 7y ago

I have been there, it's horrible. Often traces are in binary, and I spent time making them more readable, to take off the intellectual load of just decoding what you see from the actual debugging. These days I design Verilog for FPGAs without a verification department. I've learned to code defensively, to use what I know is robust without creating complex corner cases. This often involves clear handshaking and dataflow. Anyway, it's more fun than fixing bugs deep in the weeds of a complex system.