4 ms·
I didn't actually even test tool calling. I have two test cases that I use for all models: one is a floating-point equality function, which is quite difficult t
by jwr 1y ago
I didn't actually even test tool calling. I have two test cases that I use for all models: one is a floating-point equality function, which is quite difficult to get right, and another is a core.async pack-into-batches! function which has the following docstring:
"Take items from `input-ch` and group them into `batch-size` vectors. Put these onto `output-ch`. Once items
start arriving, if `batch-size` items do not arrive within `inactivity-timeout`, put the current incomplete
batch onto `output-ch`. If an anomaly is received, passes it on to `output-ch` and closes all channels. If
`input-ch` is closed, closes `output-ch`.
If `flush-predicate-fn` is provided, it will get called with two parameters: the currently accumulated
batch (guaranteed to have at least one item) and the next item. If the function returns a truthy value, the
batch will get flushed immediately.
If `convert-batch-fn` is provided, it will get called with the currently accumulated batch (guaranteed to
have at least one item) and its return value will be put onto `output-ch`. Anomalies bypass
`convert-batch-fn` and get put directly onto `output-ch` (which gets closed immediately afterwards)."
In other words, not obvious.
I ask the model to review the code and tell me if there are improvements that can be made. Big (online) models can do a pretty good job with the floating point equality function, and suggest something at least in the ballpark for the async code. Small models rarely get everything right, but some of their observations are good.