4 ms·
Cool concept. Right off the bat I see some big issues with the generated CPython documentation: > This provides a register-based virtual machine that executes
by teraflop 3y ago
Cool concept. Right off the bat I see some big issues with the generated CPython documentation:
> This provides a register-based virtual machine that executes the bytecode through simple opcodes.
Python's VM is stack-based, not register-based.
> The tiered interpreter in …/ceval.c can compile bytecode sequences into "traces" of optimized microoperations.
No such functionality exists in CPython, as far as I know.
> The dispatch loop switches on opcodes, calling functions to manipulate the operand stack. It implements stack manipulation with macros.
No it doesn't. If you look at the bytecode interpreter, it's full of plain old statements like `stack_pointer += 1;`.
> The tiered interpreter is entered from a label. It compiles the bytecode sequence into a trace of "micro-operations" stored in the code object. These micro-ops are then executed in a tight loop in the trace for faster interpretation.
As mentioned above, this seems to be a complete hallucination.
> During initialization, …/pylifecycle.c performs several important steps: [...] It creates the main interpreter object and thread
No, the code in this file creates an internal thread state object, corresponding to the already-running thread that calls it.
> References: Python/clinic/import.c.h The module implements finding and loading modules from the file system and cached bytecode.
This is kinda sorta technically correct, but the description never mentions the crucial fact that most of this C code only exists to bootstrap and support the real import machinery, which is written in Python, not C. (Also, the listed source file is the wrong one: it just contains auto-generated function wrappers, not the actual implementations.)
> Core data structure modules like …/arraymodule.c provide efficient implementations of homogeneous multidimensional arrays
Python's built-in array module provides only one-dimensional arrays.
And so on.
- oshams 3y agoThank you for this feedback. We actually have an Auto Wiki v2 in the works which is even higher quality, would be interesting to see how it changes when that comes out.
- faizshah 3y agoCan you talk a little bit about the crawler or what information are you feeding the agent about the repostory? My main concern is that this is just hallucinating the documentation and that with the more well known repos like React it can pull the data from training data like blogs etc. I think the concept is really great would just like to understand especially for enterprise use cases.
- oshams 3y agoIt's purely based on the code and we force the LLM to cite the code it describes to cut back on hallucination. We are adding a self verification steps (like chain-of-verification / https://arxiv.org/abs/2309.11495 https://arxiv.org/abs/2309.11495) in Wiki V2.
- GrinningFool 3y agoI would expect something like this to output only factually correct documentation since it would be used as reference; but it sounds like even under the upcoming v2 that's not the case?
- HL33tibCe7 3y agoAutogenerating documentation that could then be corrected by a human could still have value, in fairness
- krupan 3y agoReally? How much value would you put on that actually? How would you price that if you were selling it? Would I get a little refund for each mistake I find and have to correct?
- GrinningFool 3y agoFor one-and-done, I agree. But if you're looking at dynamic code base and you want to keep your documentation in sync, having to make 'manually correct the generated output' part of your build process can get expensive. (And tedious for the poor grunt who's got to do the work.)
- oshams 3y agoWe think chain-of-verification and fine tuning will help a lot see the other response. We are really excited to launch v2 ASAP so we can address these concerns.
- nerdponx 3y agoGreat example of plausible but completely incorrect outputs from an AI model that would go largely undetected by a non-expert human.