3 ms·

The work is solid, the approach of domain-specific tokenizer is definitely effective. But the topic seems to be somehow misleading, as I expect the researchers

by enoch2090 3y ago

The work is solid, the approach of domain-specific tokenizer is definitely effective. But the topic seems to be somehow misleading, as I expect the researchers to somehow find a way to connect LLMs for direct design of chips, while this paper focuses on three topics: 1. Engineering assistant chatbot, 2. EDA tool script generation, and 3. Bug summarization and analysis. Where (1) is mostly irrelavent with the core of "design", and only (2) is directly related with a useful workflow. What's more, it seems that they are only generating at most 10 lines of Python/Tcl code (more line "commands" in this case) for the EDA tools.

namibj 3y ago
Their paper from ~6 weeks earlier is on actual LLM-designed chips, though "just" from an LLM benchmark POV: https://research.nvidia.com/index.php/publication/2023-09_verilogeval-evaluating-large-language-models-verilog-code-generation https://research.nvidia.com/index.php/publication/2023-09_ve...
- pbazarnik 3y ago
  It looks like availability of good quality training sets will be a stumbling block for LLM use in Verilog chip design since pretraining with other programming language corpus is not transferable. (see below for quote from Nvidia paper) A lot of high quality Verilog is locked in licensed, close source IP blocks covered by NDAs. HDLBits problems are a toy level complexity circuits suitable for Verilog 101 course material. I would set a benchmark for serious HDL design LLM at reaching ability to implement AXI bus components with specified by user functionality, e.g. AXI4 Slave (address, data widths, burst capability) with memory implemented as banked synchronous SRAM. https://arxiv.org/pdf/2309.07544.pdf https://arxiv.org/pdf/2309.07544.pdf * Despite the fact that multi models undergo pretraining on an extensive corpus of multi-lingual code data, they exhibit only marginal enhancements of approximately 3% when applied to Verilog coding task. This observation potentially suggests that there is limited positive knowledge transfer between software programming languages like C++ and hardware descriptive languages such as Verilog. This highlights the significance of pretraining on substantial Verilog corpora, as it can significantly enhance model performance in Verilog-related tasks *
  - therealcamino 3y ago
    I love the phrase "limited positive knowledge transfer between software programming languages like C++ and hardware descriptive languages such as Verilog." Maybe someday the LLM's will just read the spec and come up with new designs for which they have no examples!
- enoch2090 3y ago
  Ooh, this is what I've been looking for! But I agree with pbazarnik that their benchmark are still tackling with toy-level problems. Bits operations, Karnaugh map and other stuff are just one-level of transformation logic, really want to see more in-depth examples such as high-speed circuit optimization (say a clk over 600MHz), complex filters and PID controls, etc. Wouldn't hurt even if they only achieve 20% correctness on these problems, as long as the approach can work once every few times.

namibj 3y ago

Their paper from ~6 weeks earlier is on actual LLM-designed chips, though "just" from an LLM benchmark POV: https://research.nvidia.com/index.php/publication/2023-09_verilogeval-evaluating-large-language-models-verilog-code-generation https://research.nvidia.com/index.php/publication/2023-09_ve...

pbazarnik 3y ago
It looks like availability of good quality training sets will be a stumbling block for LLM use in Verilog chip design since pretraining with other programming language corpus is not transferable. (see below for quote from Nvidia paper) A lot of high quality Verilog is locked in licensed, close source IP blocks covered by NDAs. HDLBits problems are a toy level complexity circuits suitable for Verilog 101 course material. I would set a benchmark for serious HDL design LLM at reaching ability to implement AXI bus components with specified by user functionality, e.g. AXI4 Slave (address, data widths, burst capability) with memory implemented as banked synchronous SRAM. https://arxiv.org/pdf/2309.07544.pdf https://arxiv.org/pdf/2309.07544.pdf * Despite the fact that multi models undergo pretraining on an extensive corpus of multi-lingual code data, they exhibit only marginal enhancements of approximately 3% when applied to Verilog coding task. This observation potentially suggests that there is limited positive knowledge transfer between software programming languages like C++ and hardware descriptive languages such as Verilog. This highlights the significance of pretraining on substantial Verilog corpora, as it can significantly enhance model performance in Verilog-related tasks *
- therealcamino 3y ago
  I love the phrase "limited positive knowledge transfer between software programming languages like C++ and hardware descriptive languages such as Verilog." Maybe someday the LLM's will just read the spec and come up with new designs for which they have no examples!
enoch2090 3y ago
Ooh, this is what I've been looking for! But I agree with pbazarnik that their benchmark are still tackling with toy-level problems. Bits operations, Karnaugh map and other stuff are just one-level of transformation logic, really want to see more in-depth examples such as high-speed circuit optimization (say a clk over 600MHz), complex filters and PID controls, etc. Wouldn't hurt even if they only achieve 20% correctness on these problems, as long as the approach can work once every few times.