5 ms·
I am subscribed to o1 Pro and am working on a little Rust crate. I asked both o1 Pro and Deepseek R1 to write e2e tests given all of the code in the repo (usin
by mohsen1 2y ago
I am subscribed to o1 Pro and am working on a little Rust crate.
I asked both o1 Pro and Deepseek R1 to write e2e tests given all of the code in the repo (using yek[1]).
o1 Pro code: https://github.com/bodo-run/clap-config-file/pull/3 https://github.com/bodo-run/clap-config-file/pull/3
Deepseek R1: https://github.com/bodo-run/clap-config-file/pull/4 https://github.com/bodo-run/clap-config-file/pull/4
My judgement is that Deepseek wrote better tests. This repo is small enough for making a judgement by reviewing the code.
Neither pass tests.
[1] https://github.com/bodo-run/yek https://github.com/bodo-run/yek
- terhechte 2y agoI have a set of tests that I can run against different models implemented in different languages (e.g. the same tests in Rust, Ts, Python, Swift), and out of these languages, all models have by far the most difficulty with Rust. The scores are notably higher for the same tests in other languages. I'm currently preparing the whole thing for release to share, but its not ready yet because some urgent work-work came up.
- colonial 2y agoCan confirm anecdotally. Even R1 (the full, official version with web search enabled) crashes out hard on my personal Rust benchmark - it refers to multiple items (methods, constants) that don't exist and fails to import basic necessary traits like io::Read. Embarrassing, and does little to challenge my belief that these models will never reliably advance beyond boilerplate. (My particular test is to ask for an ICMP BPF that does some simple constant comparisons. Correctly implemented, this only takes 6 sock_filters.)