14 ms·

VRoom A high end RISC-V implementation

154 points by cmurf 5y ago

nynx 5y ago
The Architectural presentation linked from the GitHub repository for this project is an incredibly good resource on how these kinds of things are designed.
- avianes 5y ago
  Yes, there is a huge lack of open and approachable information sources in micro-architecture. Be aware though, the micro-architecture used here is very interesting but differs in many ways from state of the art industrial high-end micro-architectures for superscalar out-of-order speculative processor. I am quite curious about how the author came up with these choices
  - Taniwha 5y ago
    Well, everyone was building tiny RISCVs, I kind of thought "can I make a Xeon+ class RISCV if I throw gates at the problem ?" :-) Seriously though I started out with the intent of building a 4/8 instruction/clock decoder, and an O-O execution pipe that could keep up - with the end goal of at least 4+ instruction s/clock average (we peak now at 8) - the renamer, dual register file, and commitQ are the core of what's probably different here
    - avianes 5y ago
      Yes, the "dual register file" is probably the most intriguing to me. This looks like a renaming scheme used in some old micro-architecture (Intel Core 2 maybe) where ROB receives transient results and acts as a physical regfile, at commit reg value are copied to a arch regfile. But in your uarch the physical regfile is decoupled from ROB, which must correspond to your commitQ. I wonder if this solution is viable for a very large uarch (8 way) because read ports to copy reg value from pysical regfile to arch regfile are additional read ports that can be avoided with other (more complex) renaming scheme. These additional read ports can be expensive on a regfile that already has a bunch of ports. Any thoughts about this? But I haven't read much of your code yet, that's just a raw observation
      - Taniwha 5y ago
        the commitQ entries are smart enough to 'see' the commits into the architectural file and request the data from it's current location It does mean lots of register read ports .... but you can duplicate register files at some point (reducing read ports but keeping the write ports) (you want to keep them close to the ALUs/multipliers/etc) - in some ways these are more implementation issues rather than 'architectural'
        avianes 5y ago
        I see, there are indeed solutions like regfile duplication to handle large port number but it's expensive when physical regfile becomes large. I still think that the uarch's job is to ensure minimal implementation cost ;). Thank you for your opinion and thought process, it's very valuable !
        Taniwha 5y ago
        I think that one has to separate out architecture and implementation a bit, they're obviously a deeply intertwingled dance - but you have to start with the architecture and tweak from there to get the best result in the end - I'm probably halfway through that process now, starting to introduce deeper timing constraints to flesh out that stuff BTW once great thing that sort of falls out of this architecture is that the commit register file gets shared between the integer and FP registers (and probably vector registers too), and duping just that may be an interesting architectural way to go
Symmetry 5y ago
The architectural details here were pretty interesting: https://moonbaseotago.github.io/talk/index.html https://moonbaseotago.github.io/talk/index.html It would be nice to get actual performance numbers rather than just frequency scaled Dhrystone but I suppose we have to be patient.
- Taniwha 5y ago
  Dhystone's just a place to start, it helps me make quick tweaks, and I'm at that stage of the process - it's particularly good because it's somewhat at odds with my big wide decoders - VRoom! can decode bundles of up to 8 instructions per clock, while Dhrystone has lots of twisty branches, only decodes ~3.7 instructions per bundle - it's a great test for the architecture by pushing at the things it might not be as good as. Having said that I'm about reaching the end of the point where it's the only thing - being able to run bigger longer benchmarks is one of the reasons for bringing up linux on the big FPGA on AWS
  - Taniwha 5y ago
    I'll add that freq scaled Dhrystone (DMIPS/MHz) is a particularly useful number because it helps you compare architectures rather than just clocks - you can figure out questions like "If I can make this run at 5GHz how will it compare with X?"
    - brucehoult 5y ago
      You could use verilator simulating VRoom! as a benchmark. I haven't actually looked at the generated code, but I imagine it's thousands upon thousands of instructions in a row with no conditional branching at all. CHOMP.
themerone 5y ago
What does GPL mean for a chip design? I understand how it applies to the HDL, but I doubt that it obligates you have to open your code to users of physical chips.
- dmoreno 5y ago
  IANAL, but as far as I know it's very important it's GPLv3 which means the antitivoization clause, which means that hardware that uses this firmware must provide full source code and a way to let you use your own firmware. If somehow this code is not in a firmware... No idea.
  - Taniwha 5y ago
    (this is a light derail but related) some of this design IS firmware - the lowest level bootstrap is encoded into an internal ROM - currently it's a very dumb bootstrap, a real implementation would boot from a number of possible sources. The sources are there on github. All the ARM systems you can buy today have a similar embedded boot loader - almost all of them do not release that source, because it's the root of their secure boot chain. IMHO this code should be public (but not the keys)
- bragr 5y ago
  Also IANAL, but as I understand it, the HDL would compile down to a sequence of gates, and presumably we'd treat that the same way as a binary - a "Non-Source Form" as the GPL calls it. So anyone that receives a copy of those gates (either as a binary blob for a FPGA, or pre-flashed on a FPGA, or made on actual silicon) would be entitled to the source as per GPL3 section 6 "Conveying Non-Source Forms". I don't think the GPL anti-tivoization clause has much bearing there other than presumably you'd have to provide the full tool chain that resulted in the final gates - presumably this would affect companies producing actual chips the most since you couldn't have any propriety optimization or layout steps in producing the actual chip design, though also no DRM for FPGAs (is that even a thing?)
- Someone 5y ago
  I guess you could argue that, if you bought a device with this CPU, you should be able to replace the CPU with one of your own that’s derived from this one. I think that’s the spirit of the GPL in a hardware context, but I don’t think it’s a given (by a long stretch) that courts would accept that argument. A somewhat clearer case would be if you bought a device that implements a GPL licensed design in a FPGA. I think you could argue such devices cannot disable the reprogrammability of the FPGA.
titzer 5y ago
This is a very ambitious project, so respect and good luck. I am wondering if the performance will pan out in practice, as it doesn't seem to have a very deep pipeline, so getting high clockspeeds may be a challenge. In particular the 5 clock branch mispredict penalty suggest the pipeline design is fairly simple. Production CPUs live and die by the gate depth and hit/miss latency of caches and predictors. A longer pipeline is the typical answer to gate delay issues. Cache design (and register file design!) is also super subtle; L1 is extremely important.
- evilos 5y ago
  They mention in their arch slides that they expect to add at least 2 more pipeline stages to hit higher clocks.
- Taniwha 5y ago
  As mentioned here I expect that reality will intrude and the pipe will get bigger - of course good a BTC (and spending lots of gates on it) is important because that's what mitigates that deep pipe. I haven't published my latest work (end of the week) I have a minor bump to ~6.5 DMips/MHz - Dhrystone isn't everything but it's still proving a useful tool to tweak the architecture (which is what's going on now)
minroot 5y ago
Any recommendations for resources on learning to makes things like this in general?
- Symmetry 5y ago
  Computer Architecture: A Quantitative Approach[1] is the textbook that gets recommended the most on the topic, I believe.
  - tsmi 5y ago
    If you're at the point in your career where you're not sure which is the right textbook then "A Quantitative Approach" is likely to be really tough to get through. Computer Organization and Design, by the same authors, is considered a better choice for a first book. I personally loved it and couldn't put it down the first time I read it. https://www.elsevier.com/books/computer-organization-and-design-risc-v-edition/patterson/978-0-12-820331-6 https://www.elsevier.com/books/computer-organization-and-des...
    - minroot 5y ago
      Any recommendation for books on (System)Verilog
      - tsmi 5y ago
        I personally am not into the verilog specific books. For me HDLs are hardware description languages, so first you learn to design digital hardware, then you learn to describe them. For that I highly recommend: https://www.cambridge.org/us/academic/subjects/engineering/circuits-and-systems/digital-design-systems-approach?format=HB&isbn=9780521199506 https://www.cambridge.org/us/academic/subjects/engineering/c... Great first book on the subject.
      - sitkack 5y ago
        You might like "Digital Design and Computer Architecture, RISC-V Edition" by Harris and Harris. https://www.google.com/books/edition/Digital_Design_and_Computer_Architecture/g34yzgEACAAJ?hl=en https://www.google.com/books/edition/Digital_Design_and_Comp... This book definitely skews pragmatic, hands on and doesn't assume much. Covers both VHDL and Verilog. Has sections on branch prediction, register renaming, etc.
tasty_freeze 5y ago
For a few years I worked with the guy behind this project, Paul Campbell. He is a fearless coder, and moves between hardware and software design with equal ease. An example of his crazy coding chops, he was frustrated by the lack of verilog licenses at the place he worked back in the early 90s. His solution was to whip up a compliant verilog simulator, then wrote a screen saver that would pick up verification tasks from a pending queue. They had many macs around the office that were powered 24/7, and they could chew through a lot of work during the 16 hours a day when nobody was sitting in front of them. When someone sat down at their computer in the morning or came back from lunch, the screen saver would just abandon the simulation job it was running and that job would go back to the queue of work waiting to be completed.
- colejohnson66 5y ago
  So, BOINC before BOINC?
  - jasonwatkinspdx 5y ago
    A lot of people have come up with something similar. Someone I know implemented the Condor scheduler to run models on workstations at night at a hedge fund. That Condor scheduler dates to the 80s. Smaller 3d animation studios commonly do this too.
- evilos 5y ago
  That is terrifying.
- thechao 5y ago
  Synthesizable verilog is a very small language compared to system verilog — especially in the 90s. Off the top of my head I know of six "just real quick" verilog simulators that I've worked with (one of which I wrote). I'm not sure how I feel about them. On one hand, I hate dealing with licenses; on the other hand, now you've got to worry that your custom simulation matches behavior with the synthesis tools. A lot of the "nonstandard" interpretation for synthesizable verilog from the bigs comes from practical understanding of the behavior for a given node. Most of that is captured in ABC files ... but not all of it.
Taniwha 5y ago
Author here (Paul Campbell) - AMA
- tsmi 5y ago
  Have you considered making an ASIC of your design? https://efabless.com/open_shuttle_program https://efabless.com/open_shuttle_program
  - Taniwha 5y ago
    It's likely too big for those programs - I am (just now) starting a build with the Open Lane/Sky tools not with the intent of actually taping out but more to squeeze the architectural timing (above the slow FPGA I've been using for bringing up Linux) so I can find the places where I'm being stupidly unreasonable about timing (working on my own I can't afford a Synopsys license)
    - tsmi 5y ago
      Gotcha. Did you run into any issues with yosys given that it has limited system verilog support? Ibex needed to add a pass with sv2v https://github.com/lowRISC/ibex/tree/master/syn https://github.com/lowRISC/ibex/tree/master/syn
      - Taniwha 5y ago
        I'm just starting this week, I've recently switched to some use of SV interfaces and it does not like arrays of them - sv2v seems the way to go - but even without that yosys goes bang! somethings too big Vivado compiles the same stuff - I rearchitected the bit that might obviously be doing this but no luck so far.
        mithro 5y ago
        Checkout Surelog - https://antmicro.com/blog/2020/12/ibex-support-in-verilator-yosys-via-uhdm-surelog/ https://antmicro.com/blog/2020/12/ibex-support-in-verilator-... and the more recent usage in https://github.com/siliconcompiler/siliconcompiler https://github.com/siliconcompiler/siliconcompiler
blacklion 5y ago
> Eventually we'll do some instruction combining using this information (best place may be at entry to I$0 trace cache), or possibly at the rename stage So much for "we will do only simplest of commands and u-op fusing will fix performance". It is why I'm very suspicious about this argument from RISC-V proponents.
- Taniwha 5y ago
  I think that we need lots of trace before we decide which ops make sense to combine
  - blacklion 5y ago
    As far as I understand, RISC-V proponents want to have "recommended" command sequences for compilers, to avoid situation when different RISC-V CPUs will need different compilations. If different RISC-V implementations have different "fuseable" command sequences, we will be in dreadful situation when you will need exact "-mcpu" for decent performance and binary packages will be very unoptimal. And such "conventions" are bad idea, like comments in code, IMHO. It can not be checked by tools, etc.
    - ncmncm 5y ago
      It is always frustrating when you have put in the work to optimize code, and turn out to have pessimized it for the next chip over. The extremum for this is getting a 10x performance boost by using, e.g., POPCNT, and suffering instead a 10-100x pessimization because POPCNT is trapped and emulated.
      - indolering 5y ago
        Isn't the point of the RISC-V extension mechanism is to eliminate instruction emulation?
        ncmncm 5y ago
        I'm not sure "the point" is a well-defined term in this context. Are you guessing that the extension is optional specifically so that nobody will need to emulate things they can't afford to implement in hardware? But trapping and emulating is explicitly allowed. Maybe it should be possible to ask at runtime whether an extension is emulated. Maybe it is? But I have not seen any way to tell. I guess a program could run it a thousand times and see how long it takes... It would be a serious nuisance to need to do that for each optimization, and then provide alternate implementations of algorithms that don't depend on the missing features. This is why leaving popcount out of the core instruction set is such a nuisance. It is cheap in hardware, and very slow to emulate. On organically-evolved ISAs, there are about N variants that correspond to releases. You can decide what is the oldest variant M you want to support, and use anything that is implemented in targets >=M; and the number of <M machines declines more or less exponentially with time. With RISC-V, there are instead N=2^V variants, at all times, increasing exponentially with time. You too frequently don't know if your program might need to run on one that lacks feature X. So you (1) arbitrarily fail on an unknown fraction of targets, (2) fail on some and run badly on some others, with instructions you relied on for optimization instead emulated very slowly, (3) run non-optimally on all targets, or (4) have variant versions of (parts of) your program configured to substitute at runtime, for each of K features that might be missing. None of these choices is tenable. The notion of "profiles" appears meant to reduce the load of this problem, but that makes it even more complicated.
deleted 5y ago
[deleted]
codedokode 5y ago
The presentation was interesting; but I would like to write an idea that is tangentially related to this CPU. I noticed that modern CPUs are optimized for legacy monolith OS kernels like Linux or Windows. But having a large, multimegabyte kernel is a bad idea from a security standpoint. A single mistake or intentional error in some rarely used component (like a temperature sensor driver) can get attacker full access to the system. Again, an error in any part of the monolith kernel can cause system failure. And Linux kernel doesn't even use static analysis to find bugs! It is obvious that using microkernels could solve many of the issues above. But microkernels tend to have poor performance. One of the reasons for this could be high context switch latency. CPUs with high context switch latency are only good for legacy OSes and not ready for better future kernels. Therefore, either we will find a way to make context switches fast or we will have to stay with large, insecure kernels full of vulnerabilities. So I was thinking what could be done here. For example, one thing that could be improved is to get rid of address space switch. It causes flushes of various caches and it hurts performance. Instead, we could always use the single mapping from virtual to physical addresses, but allocate each process different virtual address range. To implement this, we could add two registers, which would hold minumum and maximum accessible virtual addresses. It should be easy to check the address against them to prevent speculative out of bounds memory accesses. By the way, 32-bit x86 architecture had segments, that could be used to divide single address space between processes. Another thing that can take time is saving/restoring registers on context switch. One way to solve the problem could be to use multiple banks (say, 64 banks) of registers that can be quickly switched, another way would be to zero out registers on return from kernel and let processes save them if they need it. Or am I wrong somewhere and fast context switches cannot be implemented this way?
- wrs 5y ago
  Long ago, we in the Newton project at Apple had that idea. We (in conjunction with ARM) were defining the first ARM MMU, so we took the opportunity to implement “domains” of memory protection mappings that could be quickly swapped at a context switch. So you get multiple threads in the same address space, but with independent R/W permission mappings. I think a few other ARM customers were intrigued by the security possibilities, but the vast majority were more like “what is this bizarre thing, I just want to run Unix”, so the feature disappeared eventually. Here’s some ARM documentation if you want to pull this thread: https://developer.arm.com/documentation/dui0056'/latest/'caches-and-tightly-coupled-memories/memory-management-units/memory-access-permissions-and-domains https://developer.arm.com/documentation/dui0056'/latest/'cac...
sylware 5y ago
deleted 5y ago
[deleted]

154 points by cmurf 5y ago

nynx 5y ago

The Architectural presentation linked from the GitHub repository for this project is an incredibly good resource on how these kinds of things are designed.

avianes 5y ago
Yes, there is a huge lack of open and approachable information sources in micro-architecture. Be aware though, the micro-architecture used here is very interesting but differs in many ways from state of the art industrial high-end micro-architectures for superscalar out-of-order speculative processor. I am quite curious about how the author came up with these choices
- Taniwha 5y ago
  Well, everyone was building tiny RISCVs, I kind of thought "can I make a Xeon+ class RISCV if I throw gates at the problem ?" :-) Seriously though I started out with the intent of building a 4/8 instruction/clock decoder, and an O-O execution pipe that could keep up - with the end goal of at least 4+ instruction s/clock average (we peak now at 8) - the renamer, dual register file, and commitQ are the core of what's probably different here
  - avianes 5y ago
    Yes, the "dual register file" is probably the most intriguing to me. This looks like a renaming scheme used in some old micro-architecture (Intel Core 2 maybe) where ROB receives transient results and acts as a physical regfile, at commit reg value are copied to a arch regfile. But in your uarch the physical regfile is decoupled from ROB, which must correspond to your commitQ. I wonder if this solution is viable for a very large uarch (8 way) because read ports to copy reg value from pysical regfile to arch regfile are additional read ports that can be avoided with other (more complex) renaming scheme. These additional read ports can be expensive on a regfile that already has a bunch of ports. Any thoughts about this? But I haven't read much of your code yet, that's just a raw observation
    - Taniwha 5y ago
      the commitQ entries are smart enough to 'see' the commits into the architectural file and request the data from it's current location It does mean lots of register read ports .... but you can duplicate register files at some point (reducing read ports but keeping the write ports) (you want to keep them close to the ALUs/multipliers/etc) - in some ways these are more implementation issues rather than 'architectural'
      - avianes 5y ago
        I see, there are indeed solutions like regfile duplication to handle large port number but it's expensive when physical regfile becomes large. I still think that the uarch's job is to ensure minimal implementation cost ;). Thank you for your opinion and thought process, it's very valuable !
        Taniwha 5y ago
        I think that one has to separate out architecture and implementation a bit, they're obviously a deeply intertwingled dance - but you have to start with the architecture and tweak from there to get the best result in the end - I'm probably halfway through that process now, starting to introduce deeper timing constraints to flesh out that stuff BTW once great thing that sort of falls out of this architecture is that the commit register file gets shared between the integer and FP registers (and probably vector registers too), and duping just that may be an interesting architectural way to go

Symmetry 5y ago

The architectural details here were pretty interesting: https://moonbaseotago.github.io/talk/index.html https://moonbaseotago.github.io/talk/index.html It would be nice to get actual performance numbers rather than just frequency scaled Dhrystone but I suppose we have to be patient.

Taniwha 5y ago
Dhystone's just a place to start, it helps me make quick tweaks, and I'm at that stage of the process - it's particularly good because it's somewhat at odds with my big wide decoders - VRoom! can decode bundles of up to 8 instructions per clock, while Dhrystone has lots of twisty branches, only decodes ~3.7 instructions per bundle - it's a great test for the architecture by pushing at the things it might not be as good as. Having said that I'm about reaching the end of the point where it's the only thing - being able to run bigger longer benchmarks is one of the reasons for bringing up linux on the big FPGA on AWS
- Taniwha 5y ago
  I'll add that freq scaled Dhrystone (DMIPS/MHz) is a particularly useful number because it helps you compare architectures rather than just clocks - you can figure out questions like "If I can make this run at 5GHz how will it compare with X?"
  - brucehoult 5y ago
    You could use verilator simulating VRoom! as a benchmark. I haven't actually looked at the generated code, but I imagine it's thousands upon thousands of instructions in a row with no conditional branching at all. CHOMP.

themerone 5y ago

What does GPL mean for a chip design? I understand how it applies to the HDL, but I doubt that it obligates you have to open your code to users of physical chips.

dmoreno 5y ago
IANAL, but as far as I know it's very important it's GPLv3 which means the antitivoization clause, which means that hardware that uses this firmware must provide full source code and a way to let you use your own firmware. If somehow this code is not in a firmware... No idea.
- Taniwha 5y ago
  (this is a light derail but related) some of this design IS firmware - the lowest level bootstrap is encoded into an internal ROM - currently it's a very dumb bootstrap, a real implementation would boot from a number of possible sources. The sources are there on github. All the ARM systems you can buy today have a similar embedded boot loader - almost all of them do not release that source, because it's the root of their secure boot chain. IMHO this code should be public (but not the keys)
bragr 5y ago
Also IANAL, but as I understand it, the HDL would compile down to a sequence of gates, and presumably we'd treat that the same way as a binary - a "Non-Source Form" as the GPL calls it. So anyone that receives a copy of those gates (either as a binary blob for a FPGA, or pre-flashed on a FPGA, or made on actual silicon) would be entitled to the source as per GPL3 section 6 "Conveying Non-Source Forms". I don't think the GPL anti-tivoization clause has much bearing there other than presumably you'd have to provide the full tool chain that resulted in the final gates - presumably this would affect companies producing actual chips the most since you couldn't have any propriety optimization or layout steps in producing the actual chip design, though also no DRM for FPGAs (is that even a thing?)
Someone 5y ago
I guess you could argue that, if you bought a device with this CPU, you should be able to replace the CPU with one of your own that’s derived from this one. I think that’s the spirit of the GPL in a hardware context, but I don’t think it’s a given (by a long stretch) that courts would accept that argument. A somewhat clearer case would be if you bought a device that implements a GPL licensed design in a FPGA. I think you could argue such devices cannot disable the reprogrammability of the FPGA.

titzer 5y ago

This is a very ambitious project, so respect and good luck. I am wondering if the performance will pan out in practice, as it doesn't seem to have a very deep pipeline, so getting high clockspeeds may be a challenge. In particular the 5 clock branch mispredict penalty suggest the pipeline design is fairly simple. Production CPUs live and die by the gate depth and hit/miss latency of caches and predictors. A longer pipeline is the typical answer to gate delay issues. Cache design (and register file design!) is also super subtle; L1 is extremely important.

evilos 5y ago
They mention in their arch slides that they expect to add at least 2 more pipeline stages to hit higher clocks.
Taniwha 5y ago
As mentioned here I expect that reality will intrude and the pipe will get bigger - of course good a BTC (and spending lots of gates on it) is important because that's what mitigates that deep pipe. I haven't published my latest work (end of the week) I have a minor bump to ~6.5 DMips/MHz - Dhrystone isn't everything but it's still proving a useful tool to tweak the architecture (which is what's going on now)

minroot 5y ago

Any recommendations for resources on learning to makes things like this in general?

Symmetry 5y ago
Computer Architecture: A Quantitative Approach[1] is the textbook that gets recommended the most on the topic, I believe.
- tsmi 5y ago
  If you're at the point in your career where you're not sure which is the right textbook then "A Quantitative Approach" is likely to be really tough to get through. Computer Organization and Design, by the same authors, is considered a better choice for a first book. I personally loved it and couldn't put it down the first time I read it. https://www.elsevier.com/books/computer-organization-and-design-risc-v-edition/patterson/978-0-12-820331-6 https://www.elsevier.com/books/computer-organization-and-des...
  - minroot 5y ago
    Any recommendation for books on (System)Verilog
    - tsmi 5y ago
      I personally am not into the verilog specific books. For me HDLs are hardware description languages, so first you learn to design digital hardware, then you learn to describe them. For that I highly recommend: https://www.cambridge.org/us/academic/subjects/engineering/circuits-and-systems/digital-design-systems-approach?format=HB&isbn=9780521199506 https://www.cambridge.org/us/academic/subjects/engineering/c... Great first book on the subject.
    - sitkack 5y ago
      You might like "Digital Design and Computer Architecture, RISC-V Edition" by Harris and Harris. https://www.google.com/books/edition/Digital_Design_and_Computer_Architecture/g34yzgEACAAJ?hl=en https://www.google.com/books/edition/Digital_Design_and_Comp... This book definitely skews pragmatic, hands on and doesn't assume much. Covers both VHDL and Verilog. Has sections on branch prediction, register renaming, etc.

tasty_freeze 5y ago

For a few years I worked with the guy behind this project, Paul Campbell. He is a fearless coder, and moves between hardware and software design with equal ease. An example of his crazy coding chops, he was frustrated by the lack of verilog licenses at the place he worked back in the early 90s. His solution was to whip up a compliant verilog simulator, then wrote a screen saver that would pick up verification tasks from a pending queue. They had many macs around the office that were powered 24/7, and they could chew through a lot of work during the 16 hours a day when nobody was sitting in front of them. When someone sat down at their computer in the morning or came back from lunch, the screen saver would just abandon the simulation job it was running and that job would go back to the queue of work waiting to be completed.

colejohnson66 5y ago
So, BOINC before BOINC?
- jasonwatkinspdx 5y ago
  A lot of people have come up with something similar. Someone I know implemented the Condor scheduler to run models on workstations at night at a hedge fund. That Condor scheduler dates to the 80s. Smaller 3d animation studios commonly do this too.
evilos 5y ago
That is terrifying.
thechao 5y ago
Synthesizable verilog is a very small language compared to system verilog — especially in the 90s. Off the top of my head I know of six "just real quick" verilog simulators that I've worked with (one of which I wrote). I'm not sure how I feel about them. On one hand, I hate dealing with licenses; on the other hand, now you've got to worry that your custom simulation matches behavior with the synthesis tools. A lot of the "nonstandard" interpretation for synthesizable verilog from the bigs comes from practical understanding of the behavior for a given node. Most of that is captured in ABC files ... but not all of it.

Taniwha 5y ago

Author here (Paul Campbell) - AMA

tsmi 5y ago
Have you considered making an ASIC of your design? https://efabless.com/open_shuttle_program https://efabless.com/open_shuttle_program
- Taniwha 5y ago
  It's likely too big for those programs - I am (just now) starting a build with the Open Lane/Sky tools not with the intent of actually taping out but more to squeeze the architectural timing (above the slow FPGA I've been using for bringing up Linux) so I can find the places where I'm being stupidly unreasonable about timing (working on my own I can't afford a Synopsys license)
  - tsmi 5y ago
    Gotcha. Did you run into any issues with yosys given that it has limited system verilog support? Ibex needed to add a pass with sv2v https://github.com/lowRISC/ibex/tree/master/syn https://github.com/lowRISC/ibex/tree/master/syn
    - Taniwha 5y ago
      I'm just starting this week, I've recently switched to some use of SV interfaces and it does not like arrays of them - sv2v seems the way to go - but even without that yosys goes bang! somethings too big Vivado compiles the same stuff - I rearchitected the bit that might obviously be doing this but no luck so far.
      - mithro 5y ago
        Checkout Surelog - https://antmicro.com/blog/2020/12/ibex-support-in-verilator-yosys-via-uhdm-surelog/ https://antmicro.com/blog/2020/12/ibex-support-in-verilator-... and the more recent usage in https://github.com/siliconcompiler/siliconcompiler https://github.com/siliconcompiler/siliconcompiler

blacklion 5y ago

> Eventually we'll do some instruction combining using this information (best place may be at entry to I$0 trace cache), or possibly at the rename stage So much for "we will do only simplest of commands and u-op fusing will fix performance". It is why I'm very suspicious about this argument from RISC-V proponents.

Taniwha 5y ago
I think that we need lots of trace before we decide which ops make sense to combine
- blacklion 5y ago
  As far as I understand, RISC-V proponents want to have "recommended" command sequences for compilers, to avoid situation when different RISC-V CPUs will need different compilations. If different RISC-V implementations have different "fuseable" command sequences, we will be in dreadful situation when you will need exact "-mcpu" for decent performance and binary packages will be very unoptimal. And such "conventions" are bad idea, like comments in code, IMHO. It can not be checked by tools, etc.
  - ncmncm 5y ago
    It is always frustrating when you have put in the work to optimize code, and turn out to have pessimized it for the next chip over. The extremum for this is getting a 10x performance boost by using, e.g., POPCNT, and suffering instead a 10-100x pessimization because POPCNT is trapped and emulated.
    - indolering 5y ago
      Isn't the point of the RISC-V extension mechanism is to eliminate instruction emulation?
      - ncmncm 5y ago
        I'm not sure "the point" is a well-defined term in this context. Are you guessing that the extension is optional specifically so that nobody will need to emulate things they can't afford to implement in hardware? But trapping and emulating is explicitly allowed. Maybe it should be possible to ask at runtime whether an extension is emulated. Maybe it is? But I have not seen any way to tell. I guess a program could run it a thousand times and see how long it takes... It would be a serious nuisance to need to do that for each optimization, and then provide alternate implementations of algorithms that don't depend on the missing features. This is why leaving popcount out of the core instruction set is such a nuisance. It is cheap in hardware, and very slow to emulate. On organically-evolved ISAs, there are about N variants that correspond to releases. You can decide what is the oldest variant M you want to support, and use anything that is implemented in targets >=M; and the number of <M machines declines more or less exponentially with time. With RISC-V, there are instead N=2^V variants, at all times, increasing exponentially with time. You too frequently don't know if your program might need to run on one that lacks feature X. So you (1) arbitrarily fail on an unknown fraction of targets, (2) fail on some and run badly on some others, with instructions you relied on for optimization instead emulated very slowly, (3) run non-optimally on all targets, or (4) have variant versions of (parts of) your program configured to substitute at runtime, for each of K features that might be missing. None of these choices is tenable. The notion of "profiles" appears meant to reduce the load of this problem, but that makes it even more complicated.

deleted 5y ago

[deleted]

codedokode 5y ago

The presentation was interesting; but I would like to write an idea that is tangentially related to this CPU. I noticed that modern CPUs are optimized for legacy monolith OS kernels like Linux or Windows. But having a large, multimegabyte kernel is a bad idea from a security standpoint. A single mistake or intentional error in some rarely used component (like a temperature sensor driver) can get attacker full access to the system. Again, an error in any part of the monolith kernel can cause system failure. And Linux kernel doesn't even use static analysis to find bugs! It is obvious that using microkernels could solve many of the issues above. But microkernels tend to have poor performance. One of the reasons for this could be high context switch latency. CPUs with high context switch latency are only good for legacy OSes and not ready for better future kernels. Therefore, either we will find a way to make context switches fast or we will have to stay with large, insecure kernels full of vulnerabilities. So I was thinking what could be done here. For example, one thing that could be improved is to get rid of address space switch. It causes flushes of various caches and it hurts performance. Instead, we could always use the single mapping from virtual to physical addresses, but allocate each process different virtual address range. To implement this, we could add two registers, which would hold minumum and maximum accessible virtual addresses. It should be easy to check the address against them to prevent speculative out of bounds memory accesses. By the way, 32-bit x86 architecture had segments, that could be used to divide single address space between processes. Another thing that can take time is saving/restoring registers on context switch. One way to solve the problem could be to use multiple banks (say, 64 banks) of registers that can be quickly switched, another way would be to zero out registers on return from kernel and let processes save them if they need it. Or am I wrong somewhere and fast context switches cannot be implemented this way?

wrs 5y ago
Long ago, we in the Newton project at Apple had that idea. We (in conjunction with ARM) were defining the first ARM MMU, so we took the opportunity to implement “domains” of memory protection mappings that could be quickly swapped at a context switch. So you get multiple threads in the same address space, but with independent R/W permission mappings. I think a few other ARM customers were intrigued by the security possibilities, but the vast majority were more like “what is this bizarre thing, I just want to run Unix”, so the feature disappeared eventually. Here’s some ARM documentation if you want to pull this thread: https://developer.arm.com/documentation/dui0056'/latest/'caches-and-tightly-coupled-memories/memory-management-units/memory-access-permissions-and-domains https://developer.arm.com/documentation/dui0056'/latest/'cac...

sylware 5y ago

deleted 5y ago

[deleted]