3 ms·

IMX if you're going to all the trouble of generating linear bytecode, might as well leverage ctypes and go all the way to linear machine code.

eesmith 2y ago
The linked-to example uses a query-dependent expression which resists easy pre-compilation. The author says the overhead of building a Python expression as a string then byte compiling it is too high, which would also hold for a ctypes/cffi implementation. In my case I had a lot of config dictionaries like: default_output_args = { "tsv": {"sep": "tab", "quoting": "always"}, "csv": {"sep": "comma", "quoting": "always"}, "json": {"indent": None, "allow_nan": True}, ... } and a generic API like: def open_writer(destination, format="csv", output_args=None): ... which would let 'format' specify the output format, with the corresponding default arguments, and with the output_args able to override the defaults. I found the generic API too annoying when I knew I wanted a given format, so I added format-specific APIs with kwargs set to the default values, like: def open_csv_writer(destination, sep="tab", quoting="always"): ... def open_json_writer(destination, indent=None, allow_nan=True): ... And I wanted to keep the generic API and format-specific APIs synchronized, because the generic API is useful too. My original version processed default_output_args to generate the byte code for the different writer functions, setting up the function definitions correctly. This is not something which can be done with ctypes.
- 082349872349872 2y ago
  > resists easy pre-compilation Right, I was (in pre W^X days; compare Windows' BitBlt) compiling machine code onto the stack and executing it.
  - eesmith 2y ago
    I still know people doing on-the-fly query compilation. They control the hardware it runs on, so they don't have to worry about that.

eesmith 2y ago

The linked-to example uses a query-dependent expression which resists easy pre-compilation. The author says the overhead of building a Python expression as a string then byte compiling it is too high, which would also hold for a ctypes/cffi implementation. In my case I had a lot of config dictionaries like: default_output_args = { "tsv": {"sep": "tab", "quoting": "always"}, "csv": {"sep": "comma", "quoting": "always"}, "json": {"indent": None, "allow_nan": True}, ... } and a generic API like: def open_writer(destination, format="csv", output_args=None): ... which would let 'format' specify the output format, with the corresponding default arguments, and with the output_args able to override the defaults. I found the generic API too annoying when I knew I wanted a given format, so I added format-specific APIs with kwargs set to the default values, like: def open_csv_writer(destination, sep="tab", quoting="always"): ... def open_json_writer(destination, indent=None, allow_nan=True): ... And I wanted to keep the generic API and format-specific APIs synchronized, because the generic API is useful too. My original version processed default_output_args to generate the byte code for the different writer functions, setting up the function definitions correctly. This is not something which can be done with ctypes.

082349872349872 2y ago
> resists easy pre-compilation Right, I was (in pre W^X days; compare Windows' BitBlt) compiling machine code onto the stack and executing it.
- eesmith 2y ago
  I still know people doing on-the-fly query compilation. They control the hardware it runs on, so they don't have to worry about that.