6 ms·

Huh, did not expect our press release to end up on HN, but I'm the PI on this project, so AMA, I guess. Will jump into the questions already asked here and prov

Huh, did not expect our press release to end up on HN, but I'm the PI on this project, so AMA, I guess. Will jump into the questions already asked here and provide some context.

Rochus 6y ago
What was the technology replaced by Julia (i.e. the systems which were 100 or 1000 times slower)? Is the whole factor 1000 because of Julia/LLVM optimizations, or is there faster hardware, or was the old system an interpreter?
- Sukera 6y ago
  This is about funding for future work, not payment for achieved bonuses, it seems.
- KenoFischer 6y ago
  The 1000x goal here specially refers to algorithmic accelerations. Depending on your baseline simulator, there may also be improvements because Julia's simulation capabilities are extremely fast, but those gains are multiplicative, since we're using our own simulator as a baseline for measuring the algorithmic improvements. Just to be clear though the 1000x is a goal. We've demonstrated the same techniques at about 100x algorithmic improvement (for close to 1000x overall), but haven't quite gotten to that point yet in this domain. It is still a research project at this point. We are quite actively exploring what to do with just the simulator components though, because we think there's a good opportunity to replace some legacy tooling in this space even before factoring in the algorithmic improvements.
dnautics 6y ago
A long long long time ago, I wrote this (currently very unmaintained, and I have no interest in maintaining) julia project, don't know if this is useful to you, but it's pretty clear that there is a LOT of potential for julia in this domain, though if I'm not mistaken, the HDL-stuff is one level above what you're doing in your project -- it would be nice if these sorts of things could be composable: https://github.com/interplanetary-robot/Verilog.jl https://github.com/interplanetary-robot/Verilog.jl
- KenoFischer 6y ago
  We're not doing design in Julia at the moment, just simulation of existing domains, so we're just reading in netlists from other tools. Eventually I do think Julia would be good at HDL, and I too have an abandoned HDL DSL (that I basically used for an FPGA demo and nothing else), but I don't think HDL is an area where Julia would be able to provide 100x improvements over existing tools (which is about where we want to be to start something new). Of course, if Julia takes over in the space, doing HDL in the same language as your simulations would have advantages :).
  - deleted 6y ago
    [deleted]
  - dnautics 6y ago
    you mean quantifiable 100x improvements ;-). Chisel, which I would argue is the best HDL tool, still produces completely inscrutable verilog, so if you don't like the way it does things you might have... a challenge rejiggering it. I imagine there's like "100x" DX improvement you could generate between say best practices on CI, deployment, portability of design, running your design on accelerated hardware with ease, etc.
javajosh 6y ago
How was the process of applying to DARPA? What kind of effort was involved, and what was the experience like?
- KenoFischer 6y ago
  We have a fairly academic background, so we know how to write these grant proposals and have been doing it for a while. It's definitely a specialized skill, but there's no magic to it. You have to know how to do the paperwork and budgets though. This particular one was an extreme fast track program, so there were 30 days between each of program announcement, proposal due date, performer selection and final contract negotiation date. Nothing in government usually happens that fast - I was quite shocked that it actually happened in the stated time frame.
kayson 6y ago
I've been doing analog integrated circuit design for a decade and I'm somewhat skeptical that this is practical. I hope you can convince me otherwise because this would be great to avoid multi week simulations. From your other comments, it seems that the general principle is to create a simple model that captures most of the behavior then apply corrections on top of that. Can you elaborate on how the model is trained? Is it just from netlist/models? Or are you running a regular transient simulation? Or do you need a special transient setup? What happens when your circuit's operating point varies wildly during operation? Presumably you'd have to train a larger model over the entire range of operating points but it seems to me that it would require extensive simulation to just collect the operating point data, which somewhat defeats the purpose. You've also got simulation corners, where you'd have another massive set of permutations to generate models for... My other big concern is accuracy. Chris commented that you're achieving these speed ups at 99% accuracy. Does that mean your results are within 1% of the "full" simulation? Or that the intrinsic simulation error is 1%? For the former, that result is extremely dependent on the particular simulation and desired output. I'm curious if you have preliminary results on real circuits? If it's the latter... That's not enough. Even for mediocre circuits you're looking at >60dB relative accuracy requirements. Potentially >100dB for some high performance applications. The default relative tolerance is 1e-3 and we often reduce that by an order of magnitude or two... There are certainly mixed signal simulations where we care more about functionality than performance, so it could help there. But as a matter of practice we usually already have verified verilog models for all analog blocks anyways. And lastly, I'm curious how much does this have to do specifically with Julia the programming language? Is it just an "all part of the family" kind of thing? There's no claim that something intrinsic in the language is providing speed up, right? It's just a conveniently easy way to implement?
- KenoFischer 6y ago
  > From your other comments, it seems that the general principle is to create a simple model that captures most of the behavior then apply corrections on top of that. It's not applying corrections as in doing a taylor expansion and then adding terms to it, it's basically a projection of the systems. The basic details are in the CTESN paper, though I think there's improvements that haven't been published yet to make it work in this domain. Another point to note is that sometimes the "simpler" model actually as more equations, but the equations are of a form that is much faster to simulate, because you can relax some stiffness in the original set of equations. > Can you elaborate on how the model is trained? Is it just from netlist/models? Or are you running a regular transient simulation? Or do you need a special transient setup? The details are complicated, but the basic approach is the usual ML thing: You pick some highly parameterizable model and projection, and then take gradients to tweak the parameters until the projection reproduces what you want, except in this case it's obviously a continuous time problem and you don't need to feed in data, because you can just use the original simulation in your loss function. For some more advanced techniques you do benefit from having the whole simulator be differentiable even on the baseline, but conceptually it's not required and you could do it blackbox with a regular transient simulator. > What happens when your circuit's operating point varies wildly during operation? You can choose what parameters to fix and which to keep variable over the training process. It is probably true that in some (e.g. chaotic) models this means that the surrogate generation is too hard and will fail to converge for reasonable sizes of the model. We don't have enough experience yet to give you a clear answer of when this happens or not - figuring that out is part of the research effort here. > My other big concern is accuracy. Chris commented that you're achieving these speed ups at 99% accuracy. Does that mean your results are within 1% of the "full" simulation? I'm not sure I quite understand the distinction that you're drawing and measuring errors in a sensible way here is actually somewhat non-trivial. I believe the error rates Chris quoted are from a smooth distance metric between the signal generated by a full simulation and those generated by the surrogate. These tradeoffs are tunable of course, so for a particular application you can decide how much accuracy loss you can live with. Of course, there will be infeasible points for this technique but hopefully it's useful. I can't speak for DARPA, but if you look at the original call for proposals, you will see that they were asking for accuracy targets in the single-digit percent, so we think at those targets the 1000% speed-up is more than achievable. > There are certainly mixed signal simulations where we care more about functionality than performance, so it could help there. But as a matter of practice we usually already have verified verilog models for all analog blocks anyways. One way to think of this is as automatic generation of those verilog models from the netlist for the mixed signal use case. It is certainly still at the research phase though - where it's applicable will highly depend on what the performance/accuracy trade-offs look like and how expensive the thing is to train. > And lastly, I'm curious how much does this have to do specifically with Julia the programming language? Is it just an "all part of the family" kind of thing? There's no claim that something intrinsic in the language is providing speed up, right? It's just a conveniently easy way to implement? We've spent many millions of dollars building some extremely sophisticated modeling & simulation tooling in Julia, most of it open source, so we get to piggy back on that here to get a very modern simulator with all kinds of bells and whistles "for free", including the surrogatization capabilities. We are expecting speedups on real-world problems just by using this framework for baseline simulation also, but I don't have data on that yet, so I don't have any concrete claims to make. As for the question whether such a speedup is intrinsic to the language, as always the answer is yes and no. Julia's design makes it extremely easy to write very high performance code. Both of those things are important. Being easy to use, allows you to do go further down the algorithmic rabbit hole before you start hitting the complexity limit :).
azalemeth 6y ago
Is there a good way of doing analytic analysis on circuits with a sensible GUI, if restricted to passive components? I have a few interesting problems that I'd like to analytically explore, but the best I've been able to find is OpenModellica piped into something like Maxima or sympy. Mathematica looks like it has a nice tool -- system builder -- but it costs a lot and I don't have a license. Julia would of course be ideal for this, and I very much like where you project is going!
- KenoFischer 6y ago
  The underlying representation of the simulator we have is symbolic, so you can have it spit out a https://github.com/JuliaSymbolics/Symbolics.jl https://github.com/JuliaSymbolics/Symbolics.jl representation of the equations that make up the circuit and feed it into a symbolic integrator from there. Not a focus for us, but I don't see why it wouldn't work.

Huh, did not expect our press release to end up on HN, but I'm the PI on this project, so AMA, I guess. Will jump into the questions already asked here and provide some context.

Rochus 6y ago

What was the technology replaced by Julia (i.e. the systems which were 100 or 1000 times slower)? Is the whole factor 1000 because of Julia/LLVM optimizations, or is there faster hardware, or was the old system an interpreter?

Sukera 6y ago
This is about funding for future work, not payment for achieved bonuses, it seems.
KenoFischer 6y ago
The 1000x goal here specially refers to algorithmic accelerations. Depending on your baseline simulator, there may also be improvements because Julia's simulation capabilities are extremely fast, but those gains are multiplicative, since we're using our own simulator as a baseline for measuring the algorithmic improvements. Just to be clear though the 1000x is a goal. We've demonstrated the same techniques at about 100x algorithmic improvement (for close to 1000x overall), but haven't quite gotten to that point yet in this domain. It is still a research project at this point. We are quite actively exploring what to do with just the simulator components though, because we think there's a good opportunity to replace some legacy tooling in this space even before factoring in the algorithmic improvements.

dnautics 6y ago

A long long long time ago, I wrote this (currently very unmaintained, and I have no interest in maintaining) julia project, don't know if this is useful to you, but it's pretty clear that there is a LOT of potential for julia in this domain, though if I'm not mistaken, the HDL-stuff is one level above what you're doing in your project -- it would be nice if these sorts of things could be composable: https://github.com/interplanetary-robot/Verilog.jl https://github.com/interplanetary-robot/Verilog.jl

KenoFischer 6y ago
We're not doing design in Julia at the moment, just simulation of existing domains, so we're just reading in netlists from other tools. Eventually I do think Julia would be good at HDL, and I too have an abandoned HDL DSL (that I basically used for an FPGA demo and nothing else), but I don't think HDL is an area where Julia would be able to provide 100x improvements over existing tools (which is about where we want to be to start something new). Of course, if Julia takes over in the space, doing HDL in the same language as your simulations would have advantages :).
- deleted 6y ago
  [deleted]
- dnautics 6y ago
  you mean quantifiable 100x improvements ;-). Chisel, which I would argue is the best HDL tool, still produces completely inscrutable verilog, so if you don't like the way it does things you might have... a challenge rejiggering it. I imagine there's like "100x" DX improvement you could generate between say best practices on CI, deployment, portability of design, running your design on accelerated hardware with ease, etc.

javajosh 6y ago

How was the process of applying to DARPA? What kind of effort was involved, and what was the experience like?

KenoFischer 6y ago
We have a fairly academic background, so we know how to write these grant proposals and have been doing it for a while. It's definitely a specialized skill, but there's no magic to it. You have to know how to do the paperwork and budgets though. This particular one was an extreme fast track program, so there were 30 days between each of program announcement, proposal due date, performer selection and final contract negotiation date. Nothing in government usually happens that fast - I was quite shocked that it actually happened in the stated time frame.

kayson 6y ago

I've been doing analog integrated circuit design for a decade and I'm somewhat skeptical that this is practical. I hope you can convince me otherwise because this would be great to avoid multi week simulations. From your other comments, it seems that the general principle is to create a simple model that captures most of the behavior then apply corrections on top of that. Can you elaborate on how the model is trained? Is it just from netlist/models? Or are you running a regular transient simulation? Or do you need a special transient setup? What happens when your circuit's operating point varies wildly during operation? Presumably you'd have to train a larger model over the entire range of operating points but it seems to me that it would require extensive simulation to just collect the operating point data, which somewhat defeats the purpose. You've also got simulation corners, where you'd have another massive set of permutations to generate models for... My other big concern is accuracy. Chris commented that you're achieving these speed ups at 99% accuracy. Does that mean your results are within 1% of the "full" simulation? Or that the intrinsic simulation error is 1%? For the former, that result is extremely dependent on the particular simulation and desired output. I'm curious if you have preliminary results on real circuits? If it's the latter... That's not enough. Even for mediocre circuits you're looking at >60dB relative accuracy requirements. Potentially >100dB for some high performance applications. The default relative tolerance is 1e-3 and we often reduce that by an order of magnitude or two... There are certainly mixed signal simulations where we care more about functionality than performance, so it could help there. But as a matter of practice we usually already have verified verilog models for all analog blocks anyways. And lastly, I'm curious how much does this have to do specifically with Julia the programming language? Is it just an "all part of the family" kind of thing? There's no claim that something intrinsic in the language is providing speed up, right? It's just a conveniently easy way to implement?

KenoFischer 6y ago
> From your other comments, it seems that the general principle is to create a simple model that captures most of the behavior then apply corrections on top of that. It's not applying corrections as in doing a taylor expansion and then adding terms to it, it's basically a projection of the systems. The basic details are in the CTESN paper, though I think there's improvements that haven't been published yet to make it work in this domain. Another point to note is that sometimes the "simpler" model actually as more equations, but the equations are of a form that is much faster to simulate, because you can relax some stiffness in the original set of equations. > Can you elaborate on how the model is trained? Is it just from netlist/models? Or are you running a regular transient simulation? Or do you need a special transient setup? The details are complicated, but the basic approach is the usual ML thing: You pick some highly parameterizable model and projection, and then take gradients to tweak the parameters until the projection reproduces what you want, except in this case it's obviously a continuous time problem and you don't need to feed in data, because you can just use the original simulation in your loss function. For some more advanced techniques you do benefit from having the whole simulator be differentiable even on the baseline, but conceptually it's not required and you could do it blackbox with a regular transient simulator. > What happens when your circuit's operating point varies wildly during operation? You can choose what parameters to fix and which to keep variable over the training process. It is probably true that in some (e.g. chaotic) models this means that the surrogate generation is too hard and will fail to converge for reasonable sizes of the model. We don't have enough experience yet to give you a clear answer of when this happens or not - figuring that out is part of the research effort here. > My other big concern is accuracy. Chris commented that you're achieving these speed ups at 99% accuracy. Does that mean your results are within 1% of the "full" simulation? I'm not sure I quite understand the distinction that you're drawing and measuring errors in a sensible way here is actually somewhat non-trivial. I believe the error rates Chris quoted are from a smooth distance metric between the signal generated by a full simulation and those generated by the surrogate. These tradeoffs are tunable of course, so for a particular application you can decide how much accuracy loss you can live with. Of course, there will be infeasible points for this technique but hopefully it's useful. I can't speak for DARPA, but if you look at the original call for proposals, you will see that they were asking for accuracy targets in the single-digit percent, so we think at those targets the 1000% speed-up is more than achievable. > There are certainly mixed signal simulations where we care more about functionality than performance, so it could help there. But as a matter of practice we usually already have verified verilog models for all analog blocks anyways. One way to think of this is as automatic generation of those verilog models from the netlist for the mixed signal use case. It is certainly still at the research phase though - where it's applicable will highly depend on what the performance/accuracy trade-offs look like and how expensive the thing is to train. > And lastly, I'm curious how much does this have to do specifically with Julia the programming language? Is it just an "all part of the family" kind of thing? There's no claim that something intrinsic in the language is providing speed up, right? It's just a conveniently easy way to implement? We've spent many millions of dollars building some extremely sophisticated modeling & simulation tooling in Julia, most of it open source, so we get to piggy back on that here to get a very modern simulator with all kinds of bells and whistles "for free", including the surrogatization capabilities. We are expecting speedups on real-world problems just by using this framework for baseline simulation also, but I don't have data on that yet, so I don't have any concrete claims to make. As for the question whether such a speedup is intrinsic to the language, as always the answer is yes and no. Julia's design makes it extremely easy to write very high performance code. Both of those things are important. Being easy to use, allows you to do go further down the algorithmic rabbit hole before you start hitting the complexity limit :).

azalemeth 6y ago

Is there a good way of doing analytic analysis on circuits with a sensible GUI, if restricted to passive components? I have a few interesting problems that I'd like to analytically explore, but the best I've been able to find is OpenModellica piped into something like Maxima or sympy. Mathematica looks like it has a nice tool -- system builder -- but it costs a lot and I don't have a license. Julia would of course be ideal for this, and I very much like where you project is going!

KenoFischer 6y ago
The underlying representation of the simulator we have is symbolic, so you can have it spit out a https://github.com/JuliaSymbolics/Symbolics.jl https://github.com/JuliaSymbolics/Symbolics.jl representation of the equations that make up the circuit and feed it into a symbolic integrator from there. Not a focus for us, but I don't see why it wouldn't work.