3 ms·

>Testing is the first layer of defense. My system now includes 1,300+ tests — from unit tests to minimal integration tests (e.g., proposer + acceptor only), all

>Testing is the first layer of defense. My system now includes 1,300+ tests — from unit tests to minimal integration tests (e.g., proposer + acceptor only), all the way to multi-replica full integration tests with injected failures. See the project status. I know LOC is a silly metric, but ~1300 tests for 130k lines averages out to a test per 100 lines - isn't this awfully low for a highly complex piece of code, even discounting the fact that it's vibecoded? 100 LOC can carry a lot of logic for a single test, even for just happy paths.

risyachka 5mo ago
I may have missed it but are those tests written by person or generated? Otherwise how do you know they even test anything (like actually test, not appear to test)
- sashank_1509 5mo ago
  It’s all written by AI and you can’t tell for sure if the tests are good. You can eyeball some but eyeballing 50k lines of code takes a lot of time. You just trust AI and YOLO, find errors later
- flossly 5mo ago
  Ask AI to find the weakest tests. :) No joke: it works for me. I have a 45kLOC prod code (just code, no comments, no blanks), tested by a 30kLOC test code containing 1600 tests (that run in 30secs). I helped with the test infrastructure/architecture. Sometimes I had to write the first few tests of a particular kind, but now Claude TDDs for me. A fair share of my CLAUDE.md instructs in how I like my tests, when to write them (first), different types of tests (unit, faked-services, db, e2e, etc.) Asking Claude to find weak tests has helped a lot in getting here. I also do review AI-gen'd code, pretty much line-by-line, before accepting it.
kawogi 5mo ago
IIUC only 50k LoC are non-test code, which improves the metric. Whether that's enough tests still depends on the code. If most are getters and setters, the coverage might be ok.
embedding-shape 5mo ago
Considering the domain being distributed systems, and aiming to implement "a Rust-based multi-Paxos consensus engine that not only implements all the features of Azure’s Replicated State Library (RSL)", I don't think we even have to look so deep into it, it's severely lacking tests. If you're building a distributed system and you don't have more tests and testing code than actual code, by an order of magnitude most likely, then you're missing test coverage.

risyachka 5mo ago

I may have missed it but are those tests written by person or generated? Otherwise how do you know they even test anything (like actually test, not appear to test)

sashank_1509 5mo ago
It’s all written by AI and you can’t tell for sure if the tests are good. You can eyeball some but eyeballing 50k lines of code takes a lot of time. You just trust AI and YOLO, find errors later
flossly 5mo ago
Ask AI to find the weakest tests. :) No joke: it works for me. I have a 45kLOC prod code (just code, no comments, no blanks), tested by a 30kLOC test code containing 1600 tests (that run in 30secs). I helped with the test infrastructure/architecture. Sometimes I had to write the first few tests of a particular kind, but now Claude TDDs for me. A fair share of my CLAUDE.md instructs in how I like my tests, when to write them (first), different types of tests (unit, faked-services, db, e2e, etc.) Asking Claude to find weak tests has helped a lot in getting here. I also do review AI-gen'd code, pretty much line-by-line, before accepting it.

kawogi 5mo ago

IIUC only 50k LoC are non-test code, which improves the metric. Whether that's enough tests still depends on the code. If most are getters and setters, the coverage might be ok.

embedding-shape 5mo ago

Considering the domain being distributed systems, and aiming to implement "a Rust-based multi-Paxos consensus engine that not only implements all the features of Azure’s Replicated State Library (RSL)", I don't think we even have to look so deep into it, it's severely lacking tests. If you're building a distributed system and you don't have more tests and testing code than actual code, by an order of magnitude most likely, then you're missing test coverage.