4 ms·

I'm curious how your build process works with the codegen approach. Do you have the codegen run a pass everytime you build the project? Or is it a manual proces

by rwbt 3y ago

I'm curious how your build process works with the codegen approach. Do you have the codegen run a pass everytime you build the project? Or is it a manual process? I've always wanted to use Lua to generate boilerplate C code (like typesafe vectors) instead of relying on macros, so might actually start thinking in this direction going forward.

turtledragonfly 3y ago
Not the person you asked, but this sort of thing is pretty easy with templating languages, such as Perl's Template::Toolkit[1] or Python's Jinja2[2]. The advantage of a system like that over a plain programming language (such as Lua you mentioned) is that there's less escaping and string.format(xxx) sorts of things. The output text is the primary thing and the templating language has its own special syntax to "activate" it when you need it, rather than the other way around. That being said, Lua's square-bracket strings are pretty good for bulk text, too (: [1] http://template-toolkit.org/about.html http://template-toolkit.org/about.html [2] https://jinja.palletsprojects.com/en/3.1.x/templates/#synopsis https://jinja.palletsprojects.com/en/3.1.x/templates/#synops...
- rwbt 3y ago
  Template-Toolkit seems like the right tool for the job. Thanks for sharing.
JonChesterfield 3y ago
Instead of code generation as a special case, let it be the default. That kills most of the complexity. src/ contains lots of code in lots of languages, all describing code generators of varying complexity. src/foo.c uses 'cp' as the code generator. src/bar.c.py uses 'python'. gen/ contains the result of the ad hoc code generators. Same tree layout as src, one to one map from those files. All of that is C. Files that get #included and don't make sense standalone I tend to suffix .data, e.g. src/table.data.lua runs to create gen/table.data. obj/ contains all the C compiled to object code or llvm bitcode, depending on the toolchain. I originally did that with the idea that I could distribute gen/ as the source code for people who wanted to build the project without dealing with the complexity of the build system, strongly inspired by sqlite's amalgamation. Sqlite is built from a chaotic collection of TCL and C code generators but you don't see that as a user of sqlite.c. It turns out that debugging the stuff under gen/ when things go wrong is really easy. Valgrind / gdb show you the boring code that the generators stamped out. So in practice I keep that separation because it makes it easier to trace through the system when it behaves unexpectedly. Lua does that really well. Good multiline string literal support. A template string that you call gsub() on a few times then dump to stdout is very quick to put together. You've got string format for more complicated things, maybe using one of the string interpolation implementations found on their wiki.

by rwbt 3y ago

turtledragonfly 3y ago

Not the person you asked, but this sort of thing is pretty easy with templating languages, such as Perl's Template::Toolkit[1] or Python's Jinja2[2]. The advantage of a system like that over a plain programming language (such as Lua you mentioned) is that there's less escaping and string.format(xxx) sorts of things. The output text is the primary thing and the templating language has its own special syntax to "activate" it when you need it, rather than the other way around. That being said, Lua's square-bracket strings are pretty good for bulk text, too (: [1] http://template-toolkit.org/about.html http://template-toolkit.org/about.html [2] https://jinja.palletsprojects.com/en/3.1.x/templates/#synopsis https://jinja.palletsprojects.com/en/3.1.x/templates/#synops...

rwbt 3y ago
Template-Toolkit seems like the right tool for the job. Thanks for sharing.

JonChesterfield 3y ago

Instead of code generation as a special case, let it be the default. That kills most of the complexity. src/ contains lots of code in lots of languages, all describing code generators of varying complexity. src/foo.c uses 'cp' as the code generator. src/bar.c.py uses 'python'. gen/ contains the result of the ad hoc code generators. Same tree layout as src, one to one map from those files. All of that is C. Files that get #included and don't make sense standalone I tend to suffix .data, e.g. src/table.data.lua runs to create gen/table.data. obj/ contains all the C compiled to object code or llvm bitcode, depending on the toolchain. I originally did that with the idea that I could distribute gen/ as the source code for people who wanted to build the project without dealing with the complexity of the build system, strongly inspired by sqlite's amalgamation. Sqlite is built from a chaotic collection of TCL and C code generators but you don't see that as a user of sqlite.c. It turns out that debugging the stuff under gen/ when things go wrong is really easy. Valgrind / gdb show you the boring code that the generators stamped out. So in practice I keep that separation because it makes it easier to trace through the system when it behaves unexpectedly. Lua does that really well. Good multiline string literal support. A template string that you call gsub() on a few times then dump to stdout is very quick to put together. You've got string format for more complicated things, maybe using one of the string interpolation implementations found on their wiki.