4 ms·

TableGen factors out a lot of detail, out of the code and into a collection of tables of records which are then processed into .inc files by TableGen backends.

TableGen factors out a lot of detail, out of the code and into a collection of tables of records which are then processed into .inc files by TableGen backends. Emphasis on factors out as in even if the result is eye wateringly complex, it's much better that that complexity is in another declarative file. This factoring also allows reuse by other tools like debuggers and linkers. Emphasis on a lot of detail because 500,000 lines of instruction, register, scheduling, ... information is encoding a lot of detail. Processed, --print-records generates 87M bytes of text just for the X86 backend. Take the simple BPF backend. It has 5 TableGen files: BPF.td BPFInstrFormats.td BPFInstrInfo.td BPFRegisterInfo.td BPFCallingConv.td TableGen streams these files and processes the unloveable TableGen language over them producing a collection of records. The backend's CMakeFile.txt tells TableGen to generate .inc include files. Each of these -gen flags invokes a TableGen backend to generate a specific include. tablegen(LLVM BPFGenAsmMatcher.inc -gen-asm-matcher) tablegen(LLVM BPFGenAsmWriter.inc -gen-asm-writer) tablegen(LLVM BPFGenCallingConv.inc -gen-callingconv) tablegen(LLVM BPFGenDAGISel.inc -gen-dag-isel) tablegen(LLVM BPFGenDisassemblerTables.inc -gen-disassembler) tablegen(LLVM BPFGenInstrInfo.inc -gen-instr-info) tablegen(LLVM BPFGenMCCodeEmitter.inc -gen-emitter) tablegen(LLVM BPFGenRegisterInfo.inc -gen-register-info) tablegen(LLVM BPFGenSubtargetInfo.inc -gen-subtarget) So here's the deal. Yeah, TableGen is ungainly but then the problem that it solves is truly massive. If you were to redo it now with what we know now, the architecture would be very similar and the language would be only slightly better. TableGen solves a really hard problem, maybe not perfectly but well.

lifthrasiir 3y ago
I wonder if that could have been achieved without an actual DSL. TableGen ultimately produces definitions for record types and a list of fully expanded records in those types, but those "types" don't have to be actual types during the generation phase---you can instead have a generic associative array that gets validated when every record has been generated. A distinction between `class` and `multiclass` can be removed by using an ordinary function that can generate one or more records. So I feel that everything can be done without the DSL, except for a bit nicer syntax.
mananaysiempre 3y ago
> If you were to redo it now with what we know now, the architecture would be very similar and the language would be only slightly better. Some parts are, of course, a matter of taste (compiling in backend code rather than having them be separate programs consuming an easy-to-parse canonical form; admittedly that’s in line with the LLVM ethos). But also, honestly, the language—late binding and all—seems a bit like a more ad hoc version of CUE. If I were to look for a tool for a TableGen-shaped problem, that’d probably be the direction I would be thinking in.

lifthrasiir 3y ago

I wonder if that could have been achieved without an actual DSL. TableGen ultimately produces definitions for record types and a list of fully expanded records in those types, but those "types" don't have to be actual types during the generation phase---you can instead have a generic associative array that gets validated when every record has been generated. A distinction between `class` and `multiclass` can be removed by using an ordinary function that can generate one or more records. So I feel that everything can be done without the DSL, except for a bit nicer syntax.

mananaysiempre 3y ago

> If you were to redo it now with what we know now, the architecture would be very similar and the language would be only slightly better. Some parts are, of course, a matter of taste (compiling in backend code rather than having them be separate programs consuming an easy-to-parse canonical form; admittedly that’s in line with the LLVM ethos). But also, honestly, the language—late binding and all—seems a bit like a more ad hoc version of CUE. If I were to look for a tool for a TableGen-shaped problem, that’d probably be the direction I would be thinking in.