4 ms·

Tools for Learning LLVM TableGen

150 points by matt_d 3y ago

fooker 3y ago
TableGen has been the most annoying part of working with LLVM, it is unfortunate thy doubled down on it for MLIR. If you want to generate code, define a better language and write a good compiler. TableGen tries to be similar to C++, but is different enough that it is annoying. And god help you if something goes wrong, error messages are straight up misleading.
- UncleOxidant 3y ago
  > TableGen tries to be similar to C++, but is different enough that it is annoying. It seemed to me that TableGen was trying to be a pure functional language more like Haskell than C++.
  - fooker 3y ago
    trying is the keyword here!
- j2kun 3y ago
  It's not entirely doubled down on. There is work on IRDL, a domain-specific language for dialect definitions, and PDL, a language for pattern definitions. In theory, those could replace tablegen one day. I haven't looked into either too deeply, though.
  - fooker 3y ago
    These are research projects you don't want to touch for anything other than publishing papers until they are adopted by $BIGCO and has ~5 man years of engineering effort.
    - j2kun 3y ago
      They are upstreamed in LLVM and work today, though they are feature incomplete. From what I understand, the main reason you shouldn't use them is that development is stalled because the people who were working on them went to work at Modular.
      - fooker 3y ago
        They don't work today, give it a try. The core dialects are upstreamed so that people don't reinvent them, but all tools to utilize any of it aren't. Some of it isn't even open source. For example, nothing else in MLIR uses PDL right now.
- jcranmer 3y ago
  TableGen isn't trying to be similar to C++, it's trying to be a language that makes it easy to specify highly redundant data tables. Now, when you're writing an LLVM backend, these tables are immediately consumed by another tool that tries to write a very complex state machine based on these tables, and the formats of the tables that said tool expects is essentially undocumented and confusing, and the error messages tend to range from confusing to, well, straight up misleading. But technically that's not the fault of TableGen, that's the fault of that TableGen-based tool.
  - fooker 3y ago
    I don't doubt the 'what', clearly it's needed. My point was about the 'how'. See the following example code from the manual and tell me it doesn't resemble C++ if you squint. class PersonName<string name> { assert ... string Name = name; } class Person<string name, int age> : PersonName<name> { assert ... int Age = age; } def Rec20 : Person<"Donald Knuth", 60> { ... }
- pjmlp 3y ago
  Much better than other even more arcane tools, like DSLs based on C macros. It has been ages since I delved into GCC internals, think GCC vs egcs days, but if I am not mistaken it was either DSL based on C macros, or some variation of GIMPLE.
  - fooker 3y ago
    GIMPLE is a joy compared to Tablegen in my opinion! It tries to do one thing well.
    - pjmlp 3y ago
      Sure, I only don't remember if the backend part wasn't a pile of C macros instead, for the code generation configuration.
neverrroot 3y ago
Good all-round post, I know TG is not easy, but it’s quite powerful and I’d rather use it instead of something else that lacks.
UncleOxidant 3y ago
Kind of surprised to see an article about TableGen hit #1 on HN. TableGen is pretty arcane. I had some dealings with it in about 2016 while creating a backend for compiler targeting a new processor architecture. It was very poorly documented then - mostly you looked at what other people had done with other architectures and tried to fit that into what was needed for your new architecture. Anyway, I guess I'd be surprised if even 1,000 people in the world needed to actually do much with TableGen.
- matt_d 3y ago
  FWIW, its use has grown over time, with new applications most significantly including MLIR (Multi-Level IR compiler framework, which allows you to define your own intermediate representations as well as compiler passes operating on them), https://mlir.llvm.org/docs/DefiningDialects/Operations/ https://mlir.llvm.org/docs/DefiningDialects/Operations/ The docs have been improving, https://llvm.org/docs/TableGen/ https://llvm.org/docs/TableGen/ I also like these resources: Lessons in TableGen FOSDEM 2019; Nicolai Hähnle https://fosdem.org/2019/schedule/event/llvm_tablegen/ https://fosdem.org/2019/schedule/event/llvm_tablegen/ - What has TableGen ever done for us?, http://nhaehnle.blogspot.com/2018/02/tablegen-1-what-has-tablegen-ever-done.html http://nhaehnle.blogspot.com/2018/02/tablegen-1-what-has-tab... - Functional Programming, http://nhaehnle.blogspot.com/2018/02/tablegen-2-functional-programming.html http://nhaehnle.blogspot.com/2018/02/tablegen-2-functional-p... - Bits, http://nhaehnle.blogspot.com/2018/02/tablegen-3-bits.html http://nhaehnle.blogspot.com/2018/02/tablegen-3-bits.html - Resolving variables, http://nhaehnle.blogspot.com/2018/03/tablegen-4-resolving-variables.html http://nhaehnle.blogspot.com/2018/03/tablegen-4-resolving-va... - DAGs, http://nhaehnle.blogspot.com/2018/03/tablegen-5-dags.html http://nhaehnle.blogspot.com/2018/03/tablegen-5-dags.html
j2kun 3y ago
I found TableGen frustrating when I first started working with MLIR, in part because I expected it to be a layer of abstraction, but it very much is not. It is (in MLIR) mainly convenience tool for autogenerating boilerplate code. Once I realized that and got used to looking at the generated code, it became much easier.
fearthetelomere 3y ago
As I know nothing about TableGen, can someone explain when you would reach for this tool? Perhaps even links to simple examples?
- UncleOxidant 3y ago
  Unless you're making an LLVM/MLIR compiler backend you're not going to be reaching for this tool.
- pjmlp 3y ago
  Basically this is how you configure a compiler backend, in LLVM's case. It specificies how the whole set of internal data structures for instructions actually map into real CPU instructions, and also guides the register allocation algorithms.
CalChris 3y ago
TableGen factors out a lot of detail, out of the code and into a collection of tables of records which are then processed into .inc files by TableGen backends. Emphasis on factors out as in even if the result is eye wateringly complex, it's much better that that complexity is in another declarative file. This factoring also allows reuse by other tools like debuggers and linkers. Emphasis on a lot of detail because 500,000 lines of instruction, register, scheduling, ... information is encoding a lot of detail. Processed, --print-records generates 87M bytes of text just for the X86 backend. Take the simple BPF backend. It has 5 TableGen files: BPF.td BPFInstrFormats.td BPFInstrInfo.td BPFRegisterInfo.td BPFCallingConv.td TableGen streams these files and processes the unloveable TableGen language over them producing a collection of records. The backend's CMakeFile.txt tells TableGen to generate .inc include files. Each of these -gen flags invokes a TableGen backend to generate a specific include. tablegen(LLVM BPFGenAsmMatcher.inc -gen-asm-matcher) tablegen(LLVM BPFGenAsmWriter.inc -gen-asm-writer) tablegen(LLVM BPFGenCallingConv.inc -gen-callingconv) tablegen(LLVM BPFGenDAGISel.inc -gen-dag-isel) tablegen(LLVM BPFGenDisassemblerTables.inc -gen-disassembler) tablegen(LLVM BPFGenInstrInfo.inc -gen-instr-info) tablegen(LLVM BPFGenMCCodeEmitter.inc -gen-emitter) tablegen(LLVM BPFGenRegisterInfo.inc -gen-register-info) tablegen(LLVM BPFGenSubtargetInfo.inc -gen-subtarget) So here's the deal. Yeah, TableGen is ungainly but then the problem that it solves is truly massive. If you were to redo it now with what we know now, the architecture would be very similar and the language would be only slightly better. TableGen solves a really hard problem, maybe not perfectly but well.
- lifthrasiir 3y ago
  I wonder if that could have been achieved without an actual DSL. TableGen ultimately produces definitions for record types and a list of fully expanded records in those types, but those "types" don't have to be actual types during the generation phase---you can instead have a generic associative array that gets validated when every record has been generated. A distinction between `class` and `multiclass` can be removed by using an ordinary function that can generate one or more records. So I feel that everything can be done without the DSL, except for a bit nicer syntax.
- mananaysiempre 3y ago
mingodad 3y ago
Looking at https://llvm.org/docs/TableGen/ProgRef.html https://llvm.org/docs/TableGen/ProgRef.html I've just added a partially working grammar/parser for tablegen in https://mingodad.github.io/parsertl-playground/playground/ https://mingodad.github.io/parsertl-playground/playground/ an Yacc/Lex compatible online editor/tester (select "Tablegen-LLVM parser (partially working)" from "Examples" then click "Parse" to see a parser tree for the content in "Input source"). Any feedback is welcome !

150 points by matt_d 3y ago

TableGen has been the most annoying part of working with LLVM, it is unfortunate thy doubled down on it for MLIR. If you want to generate code, define a better language and write a good compiler. TableGen tries to be similar to C++, but is different enough that it is annoying. And god help you if something goes wrong, error messages are straight up misleading.

UncleOxidant 3y ago
> TableGen tries to be similar to C++, but is different enough that it is annoying. It seemed to me that TableGen was trying to be a pure functional language more like Haskell than C++.
- fooker 3y ago
  trying is the keyword here!
j2kun 3y ago
It's not entirely doubled down on. There is work on IRDL, a domain-specific language for dialect definitions, and PDL, a language for pattern definitions. In theory, those could replace tablegen one day. I haven't looked into either too deeply, though.
- fooker 3y ago
  These are research projects you don't want to touch for anything other than publishing papers until they are adopted by $BIGCO and has ~5 man years of engineering effort.
  - j2kun 3y ago
    They are upstreamed in LLVM and work today, though they are feature incomplete. From what I understand, the main reason you shouldn't use them is that development is stalled because the people who were working on them went to work at Modular.
    - fooker 3y ago
      They don't work today, give it a try. The core dialects are upstreamed so that people don't reinvent them, but all tools to utilize any of it aren't. Some of it isn't even open source. For example, nothing else in MLIR uses PDL right now.
jcranmer 3y ago
TableGen isn't trying to be similar to C++, it's trying to be a language that makes it easy to specify highly redundant data tables. Now, when you're writing an LLVM backend, these tables are immediately consumed by another tool that tries to write a very complex state machine based on these tables, and the formats of the tables that said tool expects is essentially undocumented and confusing, and the error messages tend to range from confusing to, well, straight up misleading. But technically that's not the fault of TableGen, that's the fault of that TableGen-based tool.
- fooker 3y ago
  I don't doubt the 'what', clearly it's needed. My point was about the 'how'. See the following example code from the manual and tell me it doesn't resemble C++ if you squint. class PersonName<string name> { assert ... string Name = name; } class Person<string name, int age> : PersonName<name> { assert ... int Age = age; } def Rec20 : Person<"Donald Knuth", 60> { ... }
pjmlp 3y ago
Much better than other even more arcane tools, like DSLs based on C macros. It has been ages since I delved into GCC internals, think GCC vs egcs days, but if I am not mistaken it was either DSL based on C macros, or some variation of GIMPLE.
- fooker 3y ago
  GIMPLE is a joy compared to Tablegen in my opinion! It tries to do one thing well.
  - pjmlp 3y ago
    Sure, I only don't remember if the backend part wasn't a pile of C macros instead, for the code generation configuration.

neverrroot 3y ago

Good all-round post, I know TG is not easy, but it’s quite powerful and I’d rather use it instead of something else that lacks.

UncleOxidant 3y ago

Kind of surprised to see an article about TableGen hit #1 on HN. TableGen is pretty arcane. I had some dealings with it in about 2016 while creating a backend for compiler targeting a new processor architecture. It was very poorly documented then - mostly you looked at what other people had done with other architectures and tried to fit that into what was needed for your new architecture. Anyway, I guess I'd be surprised if even 1,000 people in the world needed to actually do much with TableGen.

matt_d 3y ago
FWIW, its use has grown over time, with new applications most significantly including MLIR (Multi-Level IR compiler framework, which allows you to define your own intermediate representations as well as compiler passes operating on them), https://mlir.llvm.org/docs/DefiningDialects/Operations/ https://mlir.llvm.org/docs/DefiningDialects/Operations/ The docs have been improving, https://llvm.org/docs/TableGen/ https://llvm.org/docs/TableGen/ I also like these resources: Lessons in TableGen FOSDEM 2019; Nicolai Hähnle https://fosdem.org/2019/schedule/event/llvm_tablegen/ https://fosdem.org/2019/schedule/event/llvm_tablegen/ - What has TableGen ever done for us?, http://nhaehnle.blogspot.com/2018/02/tablegen-1-what-has-tablegen-ever-done.html http://nhaehnle.blogspot.com/2018/02/tablegen-1-what-has-tab... - Functional Programming, http://nhaehnle.blogspot.com/2018/02/tablegen-2-functional-programming.html http://nhaehnle.blogspot.com/2018/02/tablegen-2-functional-p... - Bits, http://nhaehnle.blogspot.com/2018/02/tablegen-3-bits.html http://nhaehnle.blogspot.com/2018/02/tablegen-3-bits.html - Resolving variables, http://nhaehnle.blogspot.com/2018/03/tablegen-4-resolving-variables.html http://nhaehnle.blogspot.com/2018/03/tablegen-4-resolving-va... - DAGs, http://nhaehnle.blogspot.com/2018/03/tablegen-5-dags.html http://nhaehnle.blogspot.com/2018/03/tablegen-5-dags.html

j2kun 3y ago

I found TableGen frustrating when I first started working with MLIR, in part because I expected it to be a layer of abstraction, but it very much is not. It is (in MLIR) mainly convenience tool for autogenerating boilerplate code. Once I realized that and got used to looking at the generated code, it became much easier.

fearthetelomere 3y ago

As I know nothing about TableGen, can someone explain when you would reach for this tool? Perhaps even links to simple examples?

UncleOxidant 3y ago
Unless you're making an LLVM/MLIR compiler backend you're not going to be reaching for this tool.
pjmlp 3y ago
Basically this is how you configure a compiler backend, in LLVM's case. It specificies how the whole set of internal data structures for instructions actually map into real CPU instructions, and also guides the register allocation algorithms.

CalChris 3y ago

TableGen factors out a lot of detail, out of the code and into a collection of tables of records which are then processed into .inc files by TableGen backends. Emphasis on factors out as in even if the result is eye wateringly complex, it's much better that that complexity is in another declarative file. This factoring also allows reuse by other tools like debuggers and linkers. Emphasis on a lot of detail because 500,000 lines of instruction, register, scheduling, ... information is encoding a lot of detail. Processed, --print-records generates 87M bytes of text just for the X86 backend. Take the simple BPF backend. It has 5 TableGen files: BPF.td BPFInstrFormats.td BPFInstrInfo.td BPFRegisterInfo.td BPFCallingConv.td TableGen streams these files and processes the unloveable TableGen language over them producing a collection of records. The backend's CMakeFile.txt tells TableGen to generate .inc include files. Each of these -gen flags invokes a TableGen backend to generate a specific include. tablegen(LLVM BPFGenAsmMatcher.inc -gen-asm-matcher) tablegen(LLVM BPFGenAsmWriter.inc -gen-asm-writer) tablegen(LLVM BPFGenCallingConv.inc -gen-callingconv) tablegen(LLVM BPFGenDAGISel.inc -gen-dag-isel) tablegen(LLVM BPFGenDisassemblerTables.inc -gen-disassembler) tablegen(LLVM BPFGenInstrInfo.inc -gen-instr-info) tablegen(LLVM BPFGenMCCodeEmitter.inc -gen-emitter) tablegen(LLVM BPFGenRegisterInfo.inc -gen-register-info) tablegen(LLVM BPFGenSubtargetInfo.inc -gen-subtarget) So here's the deal. Yeah, TableGen is ungainly but then the problem that it solves is truly massive. If you were to redo it now with what we know now, the architecture would be very similar and the language would be only slightly better. TableGen solves a really hard problem, maybe not perfectly but well.

lifthrasiir 3y ago
I wonder if that could have been achieved without an actual DSL. TableGen ultimately produces definitions for record types and a list of fully expanded records in those types, but those "types" don't have to be actual types during the generation phase---you can instead have a generic associative array that gets validated when every record has been generated. A distinction between `class` and `multiclass` can be removed by using an ordinary function that can generate one or more records. So I feel that everything can be done without the DSL, except for a bit nicer syntax.
mananaysiempre 3y ago

mingodad 3y ago

Looking at https://llvm.org/docs/TableGen/ProgRef.html https://llvm.org/docs/TableGen/ProgRef.html I've just added a partially working grammar/parser for tablegen in https://mingodad.github.io/parsertl-playground/playground/ https://mingodad.github.io/parsertl-playground/playground/ an Yacc/Lex compatible online editor/tester (select "Tablegen-LLVM parser (partially working)" from "Examples" then click "Parse" to see a parser tree for the content in "Input source"). Any feedback is welcome !