10 ms·

Cog: Use Python in your source files to generate pieces of code (2021)

158 points by goranmoomin 3y ago

binary132 3y ago
I think people really just want a better metaprogramming system
sakesun 3y ago
Wow. Cog is almost 20 years now. Still cool today. Perhaps we should try to put GPT prompt into the Cog comment these days.
url00 3y ago
I never expected to see this here! I used this in my earlier programming days. A great idea that sadly never caught on enough to allow for tooling integration with popular editors/IDEs. I still love the concept and the source code is super readable. Makes me want to revisit it…
adriangrigore 3y ago
I use shell in the same way https://mkws.sh/docs https://mkws.sh/docs for static site generation!
fake-name 3y ago
How is this different then literally any other templating language?
- lifthrasiir 3y ago
  Normally code generation via templates is a part of the build process, but this is not always ideal for all uses. For example some template is simply too slow to run, or needs some specialized dependency which is otherwise not required. Or a particular code can be manually written, but you want to further document how that code could have been automatically generated. Cog templates are intended to persist, so that the output can safely replace the input and be directly checked into the repository without any additional dependency, making them useful for such situations. (This concept is not that unusual, by the way. CPython itself has a preprocessor called the Argument Clinic [1] which was in fact inspired by Cog.) [1] https://devguide.python.org/development-tools/clinic/ https://devguide.python.org/development-tools/clinic/
- simonw 3y ago
  See comment here: https://news.ycombinator.com/item?id=38623182 https://news.ycombinator.com/item?id=38623182 The clever thing about Cog is that it's a templating language which is designed to be hidden in comments inside files (such as Markdown) that don't usually support an embedded templating language.
selcuka 3y ago
I'm not sure if I'm missing anything, but if you ever need something like this, it generally means that you have a serious issue in your design. I can understand templating languages (e.g. jinja2) which add dynamic logic using a high level language (e.g. Python) to a dumb markdown language (e.g. HTML), but I don't see the need to generate code for another high level language.
- nine_k 3y ago
 Before the introduction of generics quite a few projects used source code generation in Go projects, to make them manageable. (Lisps, OTOH, have macro capabilities that do the same thing elegantly behind the scenes.)
- simonw 3y ago
 It's mainly useful for automating chunks of documentation. Markdown files on GitHub for example don't support any form of templating language - but Cog is a template language which can be hidden away in  in your Markdown, and then executed using a separate command: cog -r README.md So it's a templating language for files that don't usually support templating languages, but that do support comments.
- lifthrasiir 3y ago
 It is not for dynamic logic, it is more like a code generation tool. The inlined output means that you don't have to rerun the embedded code every time, which is useful for many cases where templating as a part of the build process would be inadaquate or cumbersome.
- faizshah 3y ago
 There’s definitely use cases for example API client generation.
- svilen_dobrev 3y ago
 then you have not gone high enough in abstraction/interoperability within some code food-chain . i had python code that generated another python code which then generated C (and C++) for wrapping CORBA.. 20 years ago. Like 1000 LOC generated 5k that generated about 100K C.. No way to make that by hand - and keep it consistent.. Unless, one has to employ large number of people for the sake of it..
devsda 3y ago
Interesting. I know IDE snippets are not the same, but I wish that they can do more than token replacement and have features that enable us to do the same. Neovim has few snippet plugins that are powerful enough(after setup) to do similar code-gen but I think LSP servers might be more suitable to do this(based on a dsl may be) in an IDE agnostic way.
simonw 3y ago
I discovered Cog a couple of years ago and I use it on dozens of my own projects now. It's a really neat thing to have around. One trick I use a lot is including the output of the "--help" command in the README files for CLI tools that I release. Here's an example: https://github.com/simonw/ospeak/blob/main/README.md#ospeak---help https://github.com/simonw/ospeak/blob/main/README.md#ospeak-... View source on that file and you'll see this: ## ospeak --help  ``` Usage: ospeak [OPTIONS] [TEXT] CLI tool for running text through OpenAI Text to speech Set the OPENAI_API_KEY environment variable to your OpenAI API key to avoid using the --token option every time. Example usage: ospeak "Everyone deserves a pelican" --voice alloy -x 1.5 Options: --version Show the version and exit. -v, --voice [alloy|echo|fable|onyx|nova|shimmer|all] Voice to use -o, --output FILE Save audio to this file on disk -x, --speed FLOAT RANGE Speed of the voice [0.25<=x<=4.0] -s, --speak Speak the text even when saving to a file --token TEXT OpenAI API key --help Show this message and exit. ```  I can now run "cog -r README.md" any time I want to update that block of text in the README. I also have this line in my GitHub Actions test.yml workflow: https://github.com/simonw/ospeak/blob/73a449d3f006737e72e7a0c5ae47516d7ede766e/.github/workflows/test.yml#L29 https://github.com/simonw/ospeak/blob/73a449d3f006737e72e7a0... cog --check README.md That way my tests will fail if I make a change to the help and forget to update the README. I like reflecting my help output in my documentation mainly because it's a useful way to review if the help makes sense without having to actually run the command. I maintain these much larger pages using Cog in a similar way: - https://docs.datasette.io/en/stable/cli-reference.html https://docs.datasette.io/en/stable/cli-reference.html - https://sqlite-utils.datasette.io/en/stable/cli-reference.html https://sqlite-utils.datasette.io/en/stable/cli-reference.ht... - https://llm.datasette.io/en/stable/help.html https://llm.datasette.io/en/stable/help.html I wrote a bit more about this pattern here: https://til.simonwillison.net/python/cog-to-update-help-in-readme https://til.simonwillison.net/python/cog-to-update-help-in-r...
dang 3y ago
Related: Cog: Use pieces of Python code as generators in your source files - https://news.ycombinator.com/item?id=19566384 https://news.ycombinator.com/item?id=19566384 - April 2019 (72 comments)
deleted 3y ago
[deleted]
throwawaaarrgh 3y ago
I wrote a thing like this once, only it would find any matching multiline regex, send it to any external command you wanted, and substitute the result. That way you could replace any text with any kind of processing program depending on your needs. Too bad I wrote it at work, was probably ditched when I left.
- atoav 3y ago
  Was it only a twmporary fix? Then no, nothing is more permanent than a twmporary fix.
- simonw 3y ago
  I built a feature like that which works using the Python AST recently: https://github.com/simonw/symbex/blob/main/README.md#replacing-a-matched-symbol-by-running-a-command https://github.com/simonw/symbex/blob/main/README.md#replaci...
deleted 3y ago
[deleted]
pseudony 3y ago
Wrote something similar, but while you can mix generated and hand-written code, the actual python code for that is moved out of the file. https://jwdevantier.github.io/gcgen/ https://jwdevantier.github.io/gcgen/
- rjeli 3y ago
  this looks nice, but I feel like mixing the python into the source is actually the reason I like cog so much. It feels weird at first, but having it all inline and checking it in together is simpler to work with IMO
pjungwir 3y ago
I've read technical books that say the code samples are actually executed to avoid typos. I wonder if you could use cog for that. Probably there are LaTeX packages that can do it, but what if I'm writing in markdown?
- lf-non 3y ago
 I don't think that would fall within the scope of a utility like cog. I wrote a minimal tool long back which I used to extract parts of code from my test suite as json which I could then use within my documentation generator to embed these snippets as example. The examples then are guaranteed to be correct because they are quite literally pulled from the project's test suite. The util is written i n ts but should work with any lang. that supports C or Python style comments. https://github.com/lorefnon/snippet-collector https://github.com/lorefnon/snippet-collector
- lvncelot 3y ago
 This is a perfect use case for org[1] and org-babel[2] and actually one of the reasons I've switched from vim to emacs. I wonder whether there's something similar that executes code fences in markdown? After a cursory search I found markdown-exec[3], which seems to be a more editor-agnostic solution, although I don't know whether it can compete with org-babel in terms of language support and being maintained. [1] https://orgmode.org/ https://orgmode.org/ [2] https://orgmode.org/worg/org-contrib/babel/ https://orgmode.org/worg/org-contrib/babel/ [3] https://github.com/pawamoy/markdown-exec https://github.com/pawamoy/markdown-exec
 - chriswarbo 3y ago
 > I wonder whether there's something similar that executes code fences in markdown? I wrote a script to do this called panpipe: https://github.com/warbo/panpipe https://github.com/warbo/panpipe It uses Pandoc, so it supports many formats (I've personally used it with Markdown code fences and with HTML <code> elements). The idea is pretty simple and powerful: if a code block has an attribute/annotation like `pipe="foo"` then that value is executed as a shell command, and the contents of the code block gets piped through it and replaced by the stdout. This supports running basically all programming languages, by giving their interpreter as the value of `pipe`. I use this for my web site ( chriswarbo.net ) to execute code in PHP, Haskell, Racket, Bash, Coq, etc. It can also run things indirectly, e.g. if you want to compile and run C you could do something like this: Here is my C program: ```{pipe="tee myfile.c"} int main() etc etc. ``` Here is its output: ```{pipe="sh"} gcc -o myfile myfile.c 1>&2 || exit 1 ./myfile ``` A more scalable alternative, for documents which will run many such snippets, is to write a helper script to use as our command, e.g. ```{pipe="cat > runC && chmod +x runC"} #!/usr/bin/env bash set -e cat > myfile.c gcc -o myfile myfile.c 1>&2 ./myfile ``` Here is the output of my C program: ```{pipe="./runC"} int main() etc. etc. ``` I've also written a companion script called panhandle: https://github.com/warbo/panhandle https://github.com/warbo/panhandle Panhandle allows the contents of a code block (annotated with class `unwrap`) to be spliced into its containing document. This is useless on its own (since we could just write that part of the document directly, rather than putting it inside a code block); but it allows us to generate markup using panpipe, and insert it into the document. For example, the following is a HTML page containing a element and a <code> element; panpipe can execute the contents of the <code> element, which generates HTML for an <img> element, but that HTML is stuck as the text of a <code> element. Panhandle lets us "unwrap" it, to become a real <img> element in the document: Here is a checkerboard: <code class="unwrap" pipe="sh | pandoc -f html -t json"> printf '<img src="data:image/png;base64,' { echo "P1 101 100" for N in $(seq 10100); do echo $(( N % 2 )); done } > checker.pbm convert checker.pbm checker.png base64 -w0 < checker.png printf '" />' </code> Note that panhandle needs to parse the content of a code block in order to unwrap it; to keep things simple and unambiguous it only supports Pandoc JSON format (a serialisation of Pandoc's internal AST). Since the above shell code generates HTML, we pipe it through `pandoc -f html -t json` to get the required JSON format. This whole approach is described at http://www.chriswarbo.net/projects/activecode http://www.chriswarbo.net/projects/activecode
kasajian 3y ago
I would want this for C, C within C. You can just write C code in a C file that runs at "compile time" to create more C code. And I want it to be built-in to Prettier, so all this happens when I press Ctrl+S to save the file.
- egeozcan 3y ago
  From what I hear, perhaps Zig could be something for you? Of course, that's no use if you just want to keep writing C but perhaps worth a try?
- deleted 3y ago
  [deleted]
  - deleted 3y ago
    [deleted]
lifeisstillgood 3y ago
I get it but I don't get it. I have seen cog before and respect ned so I assume there are sound uses, but the demo versions are trivial and Instruggle to see how I can use it for more complex cases. My thoughts are the python code will be most effective if it can read config or access chunks of data outside the source (ie auto-generating fetter setter style stuff). I am trying to see how much is dangerous and how much is trivial or better handled pre-commit
- jmopp 3y ago
  For my work I often run a lot of queries with a similar structure, but not repetitive enough that I can just have one query. What I normally do is paste the query template in a Python REPL and write some code to generate all my queries, which I then output to a file that I can call before ending work and leave running through the night. This would allow me to do the same thing, but in a more reproducible way that I can, for instance, share with my teammates.
- pseudony 3y ago
  Having written something similar ( https://pypi.org/project/gcgen/ https://pypi.org/project/gcgen/ ), it is sometimes hard to provide examples without being overly specific. However, some "meta" examples: * If you have an authoritative source - a protobuf/openai/some other structured specification, then this is helpful (if there isn't a specific generator already). * Whenever you're knee-deep in meta-classes in Python, generating code from annotations in Java or similar - then this might actually be better - the code will likely be simpler and therefore easier for teammates to maintain as well.
- DriftRegion 3y ago
  > My thoughts are the python code will be most effective if it can read config or access chunks of data outside the source Agreed. I've used jinja2 quite a bit for templating C with structured data. It is very effective. Ned writes that he also considered templating engines but was not satisfied when trying to implement more complex code generation inline. I feel like I understand Ned's thought process here but I strongly disagree with his conclusion unless it's for personal use only. Cog is subject to the lisp curse.
Bimos 3y ago
> If dedent is True, then common initial white space is removed from the lines in sOut before adding them to the output. If trimblanklines is True, then an initial and trailing blank line are removed from sOut before adding them to the output. Together, these option arguments make it easier to use multi-line strings, and they only are useful for multi-line strings I've recently learnt Julia and known how multiline strings in Julia works: normal " allow multiline strings, while """ will automatically remove the first blank line and all indents.
mytailorisrich 3y ago
It's quite common to use scripts to generate code. My experience is that this is best kept as standalone scripts executed only when needed with the resulting generated code committed to source control (scripts are also of course under source control). Code generated on the fly during the build process and never committed to source control is a pain, IMHO.
- planede 3y ago
  Committed generated code can also be a pain. Now you invite manual changes to the generated code, which is probably a pain to maintain if you also want to maintain and reuse the generation script in parallel.
  - mytailorisrich 3y ago
    No, committed generated code does not invite manual changes. This should obviously be prevented and part of your standard software development process and guidelines. That's why autogenerated source files usually have a big warning in their headers, and you would kill any manual changes in code review with a telling off of the dev, anyway. Very simple to enforce and maintain. At least you have the whole compiled code base under source control, you can guarantee that you have access to the exact source files used for previous builds, and debugging is generally easier. You have the generated code subject to your normal code review process and everyone can easily inspect it. You also have a separation of concerns and source files, embedding Python scripts into your, say, C files is conplexifying everything for little, if any, benefits.
  - simonw 3y ago
    You can use the "cog --check" command in CI to prevent this from happening. On every commit, your tests can rerun the code generator and confirm that it makes no changes to the code. If it does, the tests fail. That way if anyone manually edits generated code (which they should not be doing) it will be caught by CI.
- tasuki 3y ago
  I really don't want any generated code committed to my source control! By its nature, this generated code is repetitive. People will go and manually change a thing in the generated code in ten thousand places. Then, when I have to make a change, I have to: 1. Either manually change my thing in ten thousand places (which is a pain) 2. Or use the code generation for my change and also make sure I don't break the previous peoples' manual changes. I don't think this is possible. > Code generated on the fly during the build process and never committed to source control is a pain, IMHO. Maybe a pain to debug, but much less of a pain to maintain/change!
rogual 3y ago
Cog is great. I use it all the time in my personal projects. It really shines in C++. Some uses I've found for it: * Generate long literal lists, with less tedium and fewer bugs: snprintf(buf, sizeof buf, "%f %f %f %f\n" "%f %f %f %f\n" "%f %f %f %f\n" "%f %f %f %f\n" "%f %f %f %f\n", /* [[[cog for i in range(20): cog.outl(f"matrix[{i}]" + (',' if i != 19 else '')) ]]] */ matrix[0], matrix[1], (...) * Automatically instantiate classes from a set of files: /* [[[cog for path in glob('src/editor/Panel*.cc'): name = stem(path) instance = name[0].lower() + name[1:] outl(f'{name} {instance};') ]]] */ PanelInspector inspector; PanelGame game; PanelTextures textures; PanelModules modules; PanelProject project; PanelInput input; PanelPerformance performance; // [[[end]]] * Given a list of Python strings, generate a C++ enum, and functions to convert it to and from a string. * Parse a C++ source file for struct definitions, and generate code to debug-dump them, inspect them in a GUI, save/load them... alright, Clang might be a better tool for this, but Cog will do in a pinch. All this is stuff that a more expressive language could do without external tools, but even in such a language, you won't usually see what a macro is being expanded to without some sort of tooling. Cog puts it right there in the file for you, which I often appreciate.
- pjc50 3y ago
  Great examples. Is source generation becoming "trendy"? Microsoft have recently added more of it to C#, in order to support their AOT efforts. It's a bit infrastructure-heavy as it seems to be intended for point support for specific things, rather than a general purpose "write code to generate code" tool. But I wonder if there would be a market for such a thing. The architecture currently does not allow you to edit user source code, just add more code. I.e. it's not a preprocessor, it just has readonly access to the AST. e.g. the regex generator https://learn.microsoft.com/en-us/dotnet/standard/base-types/regular-expression-source-generators https://learn.microsoft.com/en-us/dotnet/standard/base-types... (Reference blog post https://andrewlock.net/creating-a-source-generator-part-1-creating-an-incremental-source-generator/ https://andrewlock.net/creating-a-source-generator-part-1-cr... )
  - humanrebar 3y ago
globular-toast 3y ago
So it's a pre-processor? Any reason they don't call it that?
tarruda 3y ago
In contrast to this, "comptime" is one of Zig's innovations which completely removes the need for preprocessors and code generators without leaving language you're programming in.
- battesonb 3y ago
  I look over at Zig with envy every time I have to write and, inevitably, look up how to write a Rust macro.
  - winter_blue 3y ago
    Will Rust ever get Zig-style comptime support? Is that something that’s in the pipeline?
gchamonlive 3y ago
I used to do something similar with Jinja2 templates and ansible. This combo was really good for augmenting bash scripts. Also, the fact you can create ansible modules in python is perfect for when you need more flexibility for data transformation without resourcing to obscure j2 incantations.
baq 3y ago
so basically php, but preserves the code used to generate the template. which is completely fine, good idea, well executed!
speps 3y ago
Surprised it's not more well-known but Visual Studio (MSBuild really) has had support for this kind of code generation for a while, it's called T4 (probably in reference to the GNU M4 utility). It can be used to generate files on save as well, so as the template is edited, it updates the output files automatically. [0] https://learn.microsoft.com/en-us/visualstudio/modeling/design-time-code-generation-by-using-t4-text-templates https://learn.microsoft.com/en-us/visualstudio/modeling/desi...
- mjd 3y ago
  Just FYI, m4 was in the original Unix in 1977, years before the GNU project began. (It was invented by Kernighan and Ritchie.) Thanks for the VS tip.
- _a_a_a_ 3y ago
  I needed code generation for C# and looked at T4, decided it was less capable than M4 (and obviously less portable, a big plus for Microsoft in tying you into their environment) and went with python instead. That was a while back, maybe T4 has improved but I still wouldn't use it.
mjd 3y ago
Something in the article I don't understand: > Cog lets you use the full power of Python for text generation, without a templating system dumbing down your tools for you. I would have said that Cog is a templating engine. What distinction is the author drawing here?
- codetrotter 3y ago
  I think the difference is that some templating engines have a DSL that lets you do things like loop over arrays and such, and that compared to those Cog is different.
- lifthrasiir 3y ago
  Many if not most templating engines do not allow an arbitrary code because of security implications.
- PurpleRamen 3y ago
  Cog seem to be the type that just evals normal python.code. While many other template-engines replicate code-structures. For example, in Jinja2 you would write something like "{% for i in data %}{{i}}{% endfor %}", which looks kinda like python, and utilizes python-object, but is not real python-code.
  - HelloNurse 3y ago
    Good code at the expense of "syntactic noise" in templates: a suitable tradeoff for the typical use case of short and precise templates with advanced logic backing them (e.g. the DCT coefficient tables linked in another comment) rather than large outputs with simple logic to keep the templates readable (e.g. a dynamic web page, the typical use of Jinja).
dmcq2 3y ago
I used Spyce for a while years ago https://spyce.sourceforge.net/ https://spyce.sourceforge.net/ it stopped being maintained and I abandoned it but it had a number of ideas which I think work well. It changed the Python so semicolons and square brackets were used instead of indentation, one could also use straight Python. It generated HTML but I'm sure could be adapted easily.
eequah9L 3y ago
I used m4 to that effect to generate some set-like and map-like strongly-typed classes for a j2me project. Must have been 15 years ago. For C and C++ in particular, X macros are another technique that can be used to generate boilerplate nicely. I use that quite a bit to this day, even in C++, where it's not needed as much due to templates.
Arelius 3y ago
I built/use something similar using S7 scheme inspired by NaughtyDog's use of PLT Scheme/Rackey. I really like that it's easy to embed S7 into a single binary that's easy to build and distribute. Sadly, I mostly built it for my own usage and havent documented it well. https://github.com/ScatteredRay/s7pp https://github.com/ScatteredRay/s7pp
abcde777666 3y ago
Feels like one of those 'neat on paper, bad in practice' kinds of ideas. I can imagine it being pretty bad for code readability if not used judiciously, and it adds an extra layer of complexity and troubleshooting when things go wrong. Maybe I'm just a purist.
- nedbat 3y ago
  Have you read the other comments that show examples like inserting help output into your docs?

158 points by goranmoomin 3y ago

I think people really just want a better metaprogramming system

Wow. Cog is almost 20 years now. Still cool today. Perhaps we should try to put GPT prompt into the Cog comment these days.

url00 3y ago

I never expected to see this here! I used this in my earlier programming days. A great idea that sadly never caught on enough to allow for tooling integration with popular editors/IDEs. I still love the concept and the source code is super readable. Makes me want to revisit it…

adriangrigore 3y ago

I use shell in the same way https://mkws.sh/docs https://mkws.sh/docs for static site generation!

fake-name 3y ago

How is this different then literally any other templating language?

lifthrasiir 3y ago
Normally code generation via templates is a part of the build process, but this is not always ideal for all uses. For example some template is simply too slow to run, or needs some specialized dependency which is otherwise not required. Or a particular code can be manually written, but you want to further document how that code could have been automatically generated. Cog templates are intended to persist, so that the output can safely replace the input and be directly checked into the repository without any additional dependency, making them useful for such situations. (This concept is not that unusual, by the way. CPython itself has a preprocessor called the Argument Clinic [1] which was in fact inspired by Cog.) [1] https://devguide.python.org/development-tools/clinic/ https://devguide.python.org/development-tools/clinic/
simonw 3y ago
See comment here: https://news.ycombinator.com/item?id=38623182 https://news.ycombinator.com/item?id=38623182 The clever thing about Cog is that it's a templating language which is designed to be hidden in comments inside files (such as Markdown) that don't usually support an embedded templating language.

selcuka 3y ago

I'm not sure if I'm missing anything, but if you ever need something like this, it generally means that you have a serious issue in your design. I can understand templating languages (e.g. jinja2) which add dynamic logic using a high level language (e.g. Python) to a dumb markdown language (e.g. HTML), but I don't see the need to generate code for another high level language.

nine_k 3y ago
Before the introduction of generics quite a few projects used source code generation in Go projects, to make them manageable. (Lisps, OTOH, have macro capabilities that do the same thing elegantly behind the scenes.)
simonw 3y ago
It's mainly useful for automating chunks of documentation. Markdown files on GitHub for example don't support any form of templating language - but Cog is a template language which can be hidden away in  in your Markdown, and then executed using a separate command: cog -r README.md So it's a templating language for files that don't usually support templating languages, but that do support comments.
lifthrasiir 3y ago
It is not for dynamic logic, it is more like a code generation tool. The inlined output means that you don't have to rerun the embedded code every time, which is useful for many cases where templating as a part of the build process would be inadaquate or cumbersome.
faizshah 3y ago
There’s definitely use cases for example API client generation.
svilen_dobrev 3y ago
then you have not gone high enough in abstraction/interoperability within some code food-chain . i had python code that generated another python code which then generated C (and C++) for wrapping CORBA.. 20 years ago. Like 1000 LOC generated 5k that generated about 100K C.. No way to make that by hand - and keep it consistent.. Unless, one has to employ large number of people for the sake of it..

devsda 3y ago

Interesting. I know IDE snippets are not the same, but I wish that they can do more than token replacement and have features that enable us to do the same. Neovim has few snippet plugins that are powerful enough(after setup) to do similar code-gen but I think LSP servers might be more suitable to do this(based on a dsl may be) in an IDE agnostic way.

simonw 3y ago

I discovered Cog a couple of years ago and I use it on dozens of my own projects now. It's a really neat thing to have around. One trick I use a lot is including the output of the "--help" command in the README files for CLI tools that I release. Here's an example: https://github.com/simonw/ospeak/blob/main/README.md#ospeak---help https://github.com/simonw/ospeak/blob/main/README.md#ospeak-... View source on that file and you'll see this: ## ospeak --help  ``` Usage: ospeak [OPTIONS] [TEXT] CLI tool for running text through OpenAI Text to speech Set the OPENAI_API_KEY environment variable to your OpenAI API key to avoid using the --token option every time. Example usage: ospeak "Everyone deserves a pelican" --voice alloy -x 1.5 Options: --version Show the version and exit. -v, --voice [alloy|echo|fable|onyx|nova|shimmer|all] Voice to use -o, --output FILE Save audio to this file on disk -x, --speed FLOAT RANGE Speed of the voice [0.25<=x<=4.0] -s, --speak Speak the text even when saving to a file --token TEXT OpenAI API key --help Show this message and exit. ```  I can now run "cog -r README.md" any time I want to update that block of text in the README. I also have this line in my GitHub Actions test.yml workflow: https://github.com/simonw/ospeak/blob/73a449d3f006737e72e7a0c5ae47516d7ede766e/.github/workflows/test.yml#L29 https://github.com/simonw/ospeak/blob/73a449d3f006737e72e7a0... cog --check README.md That way my tests will fail if I make a change to the help and forget to update the README. I like reflecting my help output in my documentation mainly because it's a useful way to review if the help makes sense without having to actually run the command. I maintain these much larger pages using Cog in a similar way: - https://docs.datasette.io/en/stable/cli-reference.html https://docs.datasette.io/en/stable/cli-reference.html - https://sqlite-utils.datasette.io/en/stable/cli-reference.html https://sqlite-utils.datasette.io/en/stable/cli-reference.ht... - https://llm.datasette.io/en/stable/help.html https://llm.datasette.io/en/stable/help.html I wrote a bit more about this pattern here: https://til.simonwillison.net/python/cog-to-update-help-in-readme https://til.simonwillison.net/python/cog-to-update-help-in-r...

dang 3y ago

Related: Cog: Use pieces of Python code as generators in your source files - https://news.ycombinator.com/item?id=19566384 https://news.ycombinator.com/item?id=19566384 - April 2019 (72 comments)

deleted 3y ago

[deleted]

throwawaaarrgh 3y ago

I wrote a thing like this once, only it would find any matching multiline regex, send it to any external command you wanted, and substitute the result. That way you could replace any text with any kind of processing program depending on your needs. Too bad I wrote it at work, was probably ditched when I left.

atoav 3y ago
Was it only a twmporary fix? Then no, nothing is more permanent than a twmporary fix.
simonw 3y ago
I built a feature like that which works using the Python AST recently: https://github.com/simonw/symbex/blob/main/README.md#replacing-a-matched-symbol-by-running-a-command https://github.com/simonw/symbex/blob/main/README.md#replaci...

deleted 3y ago

[deleted]

pseudony 3y ago

Wrote something similar, but while you can mix generated and hand-written code, the actual python code for that is moved out of the file. https://jwdevantier.github.io/gcgen/ https://jwdevantier.github.io/gcgen/

rjeli 3y ago
this looks nice, but I feel like mixing the python into the source is actually the reason I like cog so much. It feels weird at first, but having it all inline and checking it in together is simpler to work with IMO

pjungwir 3y ago

I've read technical books that say the code samples are actually executed to avoid typos. I wonder if you could use cog for that. Probably there are LaTeX packages that can do it, but what if I'm writing in markdown?

lf-non 3y ago
I don't think that would fall within the scope of a utility like cog. I wrote a minimal tool long back which I used to extract parts of code from my test suite as json which I could then use within my documentation generator to embed these snippets as example. The examples then are guaranteed to be correct because they are quite literally pulled from the project's test suite. The util is written i n ts but should work with any lang. that supports C or Python style comments. https://github.com/lorefnon/snippet-collector https://github.com/lorefnon/snippet-collector
lvncelot 3y ago
This is a perfect use case for org[1] and org-babel[2] and actually one of the reasons I've switched from vim to emacs. I wonder whether there's something similar that executes code fences in markdown? After a cursory search I found markdown-exec[3], which seems to be a more editor-agnostic solution, although I don't know whether it can compete with org-babel in terms of language support and being maintained. [1] https://orgmode.org/ https://orgmode.org/ [2] https://orgmode.org/worg/org-contrib/babel/ https://orgmode.org/worg/org-contrib/babel/ [3] https://github.com/pawamoy/markdown-exec https://github.com/pawamoy/markdown-exec
- chriswarbo 3y ago
 > I wonder whether there's something similar that executes code fences in markdown? I wrote a script to do this called panpipe: https://github.com/warbo/panpipe https://github.com/warbo/panpipe It uses Pandoc, so it supports many formats (I've personally used it with Markdown code fences and with HTML <code> elements). The idea is pretty simple and powerful: if a code block has an attribute/annotation like `pipe="foo"` then that value is executed as a shell command, and the contents of the code block gets piped through it and replaced by the stdout. This supports running basically all programming languages, by giving their interpreter as the value of `pipe`. I use this for my web site ( chriswarbo.net ) to execute code in PHP, Haskell, Racket, Bash, Coq, etc. It can also run things indirectly, e.g. if you want to compile and run C you could do something like this: Here is my C program: ```{pipe="tee myfile.c"} int main() etc etc. ``` Here is its output: ```{pipe="sh"} gcc -o myfile myfile.c 1>&2 || exit 1 ./myfile ``` A more scalable alternative, for documents which will run many such snippets, is to write a helper script to use as our command, e.g. ```{pipe="cat > runC && chmod +x runC"} #!/usr/bin/env bash set -e cat > myfile.c gcc -o myfile myfile.c 1>&2 ./myfile ``` Here is the output of my C program: ```{pipe="./runC"} int main() etc. etc. ``` I've also written a companion script called panhandle: https://github.com/warbo/panhandle https://github.com/warbo/panhandle Panhandle allows the contents of a code block (annotated with class `unwrap`) to be spliced into its containing document. This is useless on its own (since we could just write that part of the document directly, rather than putting it inside a code block); but it allows us to generate markup using panpipe, and insert it into the document. For example, the following is a HTML page containing a element and a <code> element; panpipe can execute the contents of the <code> element, which generates HTML for an <img> element, but that HTML is stuck as the text of a <code> element. Panhandle lets us "unwrap" it, to become a real <img> element in the document: Here is a checkerboard: <code class="unwrap" pipe="sh | pandoc -f html -t json"> printf '<img src="data:image/png;base64,' { echo "P1 101 100" for N in $(seq 10100); do echo $(( N % 2 )); done } > checker.pbm convert checker.pbm checker.png base64 -w0 < checker.png printf '" />' </code> Note that panhandle needs to parse the content of a code block in order to unwrap it; to keep things simple and unambiguous it only supports Pandoc JSON format (a serialisation of Pandoc's internal AST). Since the above shell code generates HTML, we pipe it through `pandoc -f html -t json` to get the required JSON format. This whole approach is described at http://www.chriswarbo.net/projects/activecode http://www.chriswarbo.net/projects/activecode

kasajian 3y ago

I would want this for C, C within C. You can just write C code in a C file that runs at "compile time" to create more C code. And I want it to be built-in to Prettier, so all this happens when I press Ctrl+S to save the file.

egeozcan 3y ago
From what I hear, perhaps Zig could be something for you? Of course, that's no use if you just want to keep writing C but perhaps worth a try?
deleted 3y ago
[deleted]
- deleted 3y ago
  [deleted]

lifeisstillgood 3y ago

I get it but I don't get it. I have seen cog before and respect ned so I assume there are sound uses, but the demo versions are trivial and Instruggle to see how I can use it for more complex cases. My thoughts are the python code will be most effective if it can read config or access chunks of data outside the source (ie auto-generating fetter setter style stuff). I am trying to see how much is dangerous and how much is trivial or better handled pre-commit

jmopp 3y ago
For my work I often run a lot of queries with a similar structure, but not repetitive enough that I can just have one query. What I normally do is paste the query template in a Python REPL and write some code to generate all my queries, which I then output to a file that I can call before ending work and leave running through the night. This would allow me to do the same thing, but in a more reproducible way that I can, for instance, share with my teammates.
pseudony 3y ago
Having written something similar ( https://pypi.org/project/gcgen/ https://pypi.org/project/gcgen/ ), it is sometimes hard to provide examples without being overly specific. However, some "meta" examples: * If you have an authoritative source - a protobuf/openai/some other structured specification, then this is helpful (if there isn't a specific generator already). * Whenever you're knee-deep in meta-classes in Python, generating code from annotations in Java or similar - then this might actually be better - the code will likely be simpler and therefore easier for teammates to maintain as well.
DriftRegion 3y ago
> My thoughts are the python code will be most effective if it can read config or access chunks of data outside the source Agreed. I've used jinja2 quite a bit for templating C with structured data. It is very effective. Ned writes that he also considered templating engines but was not satisfied when trying to implement more complex code generation inline. I feel like I understand Ned's thought process here but I strongly disagree with his conclusion unless it's for personal use only. Cog is subject to the lisp curse.

Bimos 3y ago

> If dedent is True, then common initial white space is removed from the lines in sOut before adding them to the output. If trimblanklines is True, then an initial and trailing blank line are removed from sOut before adding them to the output. Together, these option arguments make it easier to use multi-line strings, and they only are useful for multi-line strings I've recently learnt Julia and known how multiline strings in Julia works: normal " allow multiline strings, while """ will automatically remove the first blank line and all indents.

mytailorisrich 3y ago

It's quite common to use scripts to generate code. My experience is that this is best kept as standalone scripts executed only when needed with the resulting generated code committed to source control (scripts are also of course under source control). Code generated on the fly during the build process and never committed to source control is a pain, IMHO.

planede 3y ago
Committed generated code can also be a pain. Now you invite manual changes to the generated code, which is probably a pain to maintain if you also want to maintain and reuse the generation script in parallel.
- mytailorisrich 3y ago
  No, committed generated code does not invite manual changes. This should obviously be prevented and part of your standard software development process and guidelines. That's why autogenerated source files usually have a big warning in their headers, and you would kill any manual changes in code review with a telling off of the dev, anyway. Very simple to enforce and maintain. At least you have the whole compiled code base under source control, you can guarantee that you have access to the exact source files used for previous builds, and debugging is generally easier. You have the generated code subject to your normal code review process and everyone can easily inspect it. You also have a separation of concerns and source files, embedding Python scripts into your, say, C files is conplexifying everything for little, if any, benefits.
- simonw 3y ago
  You can use the "cog --check" command in CI to prevent this from happening. On every commit, your tests can rerun the code generator and confirm that it makes no changes to the code. If it does, the tests fail. That way if anyone manually edits generated code (which they should not be doing) it will be caught by CI.
tasuki 3y ago
I really don't want any generated code committed to my source control! By its nature, this generated code is repetitive. People will go and manually change a thing in the generated code in ten thousand places. Then, when I have to make a change, I have to: 1. Either manually change my thing in ten thousand places (which is a pain) 2. Or use the code generation for my change and also make sure I don't break the previous peoples' manual changes. I don't think this is possible. > Code generated on the fly during the build process and never committed to source control is a pain, IMHO. Maybe a pain to debug, but much less of a pain to maintain/change!

rogual 3y ago

Cog is great. I use it all the time in my personal projects. It really shines in C++. Some uses I've found for it: * Generate long literal lists, with less tedium and fewer bugs: snprintf(buf, sizeof buf, "%f %f %f %f\n" "%f %f %f %f\n" "%f %f %f %f\n" "%f %f %f %f\n" "%f %f %f %f\n", /* [[[cog for i in range(20): cog.outl(f"matrix[{i}]" + (',' if i != 19 else '')) ]]] */ matrix[0], matrix[1], (...) * Automatically instantiate classes from a set of files: /* [[[cog for path in glob('src/editor/Panel*.cc'): name = stem(path) instance = name[0].lower() + name[1:] outl(f'{name} {instance};') ]]] */ PanelInspector inspector; PanelGame game; PanelTextures textures; PanelModules modules; PanelProject project; PanelInput input; PanelPerformance performance; // [[[end]]] * Given a list of Python strings, generate a C++ enum, and functions to convert it to and from a string. * Parse a C++ source file for struct definitions, and generate code to debug-dump them, inspect them in a GUI, save/load them... alright, Clang might be a better tool for this, but Cog will do in a pinch. All this is stuff that a more expressive language could do without external tools, but even in such a language, you won't usually see what a macro is being expanded to without some sort of tooling. Cog puts it right there in the file for you, which I often appreciate.

pjc50 3y ago
Great examples. Is source generation becoming "trendy"? Microsoft have recently added more of it to C#, in order to support their AOT efforts. It's a bit infrastructure-heavy as it seems to be intended for point support for specific things, rather than a general purpose "write code to generate code" tool. But I wonder if there would be a market for such a thing. The architecture currently does not allow you to edit user source code, just add more code. I.e. it's not a preprocessor, it just has readonly access to the AST. e.g. the regex generator https://learn.microsoft.com/en-us/dotnet/standard/base-types/regular-expression-source-generators https://learn.microsoft.com/en-us/dotnet/standard/base-types... (Reference blog post https://andrewlock.net/creating-a-source-generator-part-1-creating-an-incremental-source-generator/ https://andrewlock.net/creating-a-source-generator-part-1-cr... )
- humanrebar 3y ago

globular-toast 3y ago

So it's a pre-processor? Any reason they don't call it that?

tarruda 3y ago

In contrast to this, "comptime" is one of Zig's innovations which completely removes the need for preprocessors and code generators without leaving language you're programming in.

battesonb 3y ago
I look over at Zig with envy every time I have to write and, inevitably, look up how to write a Rust macro.
- winter_blue 3y ago
  Will Rust ever get Zig-style comptime support? Is that something that’s in the pipeline?

gchamonlive 3y ago

I used to do something similar with Jinja2 templates and ansible. This combo was really good for augmenting bash scripts. Also, the fact you can create ansible modules in python is perfect for when you need more flexibility for data transformation without resourcing to obscure j2 incantations.

baq 3y ago

so basically php, but preserves the code used to generate the template. which is completely fine, good idea, well executed!

speps 3y ago

Surprised it's not more well-known but Visual Studio (MSBuild really) has had support for this kind of code generation for a while, it's called T4 (probably in reference to the GNU M4 utility). It can be used to generate files on save as well, so as the template is edited, it updates the output files automatically. [0] https://learn.microsoft.com/en-us/visualstudio/modeling/design-time-code-generation-by-using-t4-text-templates https://learn.microsoft.com/en-us/visualstudio/modeling/desi...

mjd 3y ago
Just FYI, m4 was in the original Unix in 1977, years before the GNU project began. (It was invented by Kernighan and Ritchie.) Thanks for the VS tip.
_a_a_a_ 3y ago
I needed code generation for C# and looked at T4, decided it was less capable than M4 (and obviously less portable, a big plus for Microsoft in tying you into their environment) and went with python instead. That was a while back, maybe T4 has improved but I still wouldn't use it.

mjd 3y ago

Something in the article I don't understand: > Cog lets you use the full power of Python for text generation, without a templating system dumbing down your tools for you. I would have said that Cog is a templating engine. What distinction is the author drawing here?

codetrotter 3y ago
I think the difference is that some templating engines have a DSL that lets you do things like loop over arrays and such, and that compared to those Cog is different.
lifthrasiir 3y ago
Many if not most templating engines do not allow an arbitrary code because of security implications.
PurpleRamen 3y ago
Cog seem to be the type that just evals normal python.code. While many other template-engines replicate code-structures. For example, in Jinja2 you would write something like "{% for i in data %}{{i}}{% endfor %}", which looks kinda like python, and utilizes python-object, but is not real python-code.
- HelloNurse 3y ago
  Good code at the expense of "syntactic noise" in templates: a suitable tradeoff for the typical use case of short and precise templates with advanced logic backing them (e.g. the DCT coefficient tables linked in another comment) rather than large outputs with simple logic to keep the templates readable (e.g. a dynamic web page, the typical use of Jinja).

dmcq2 3y ago

I used Spyce for a while years ago https://spyce.sourceforge.net/ https://spyce.sourceforge.net/ it stopped being maintained and I abandoned it but it had a number of ideas which I think work well. It changed the Python so semicolons and square brackets were used instead of indentation, one could also use straight Python. It generated HTML but I'm sure could be adapted easily.

eequah9L 3y ago

I used m4 to that effect to generate some set-like and map-like strongly-typed classes for a j2me project. Must have been 15 years ago. For C and C++ in particular, X macros are another technique that can be used to generate boilerplate nicely. I use that quite a bit to this day, even in C++, where it's not needed as much due to templates.

Arelius 3y ago

I built/use something similar using S7 scheme inspired by NaughtyDog's use of PLT Scheme/Rackey. I really like that it's easy to embed S7 into a single binary that's easy to build and distribute. Sadly, I mostly built it for my own usage and havent documented it well. https://github.com/ScatteredRay/s7pp https://github.com/ScatteredRay/s7pp

abcde777666 3y ago

Feels like one of those 'neat on paper, bad in practice' kinds of ideas. I can imagine it being pretty bad for code readability if not used judiciously, and it adds an extra layer of complexity and troubleshooting when things go wrong. Maybe I'm just a purist.

nedbat 3y ago
Have you read the other comments that show examples like inserting help output into your docs?