4 ms·

This is a great pitch, and I don't want to come across as negative, but I feel like a statement like "we found all bugs" can only be true with a very narrow def

This is a great pitch, and I don't want to come across as negative, but I feel like a statement like "we found all bugs" can only be true with a very narrow definition of bug. The most pernicious, hard-to-find bugs that I've come across have all been around the business logic of an application, rather than it hitting into an error state. I'm thinking of the category where you have something like "a database is currently reporting a completed transaction against a customer, but no completed purchase item, how should it be displayed on the customer recent transactions page?". Implementing something where "a thing will appear and not crash" in those cases is one thing, but making sure that it actually makes sense as a choice given all the context of everyone elses choices everywhere else in the stack is a lot harder. Or to take a database, something along the lines of "our query planner produces a really suboptimal plan in this edge-case". Neither of those types of problems could ever be automatically detected, because they aren't issues of the programming reaching an error state- the issue is figuring out in the first place what "correct" actually is for you application. Maybe I'm setting the bar too high for what a "bug" is, but I guess my point is, its one thing to fantasize about having zero bugs, its another to build software in the real world. I probably still settle for 0 run time errors though to be fair. . .

moritonal 3y ago
Good summary of the hard part of being a software developer that deals with clients.
- Aachen 3y ago
  What software developer does not deal with clients (and makes a living)?
  - ejb999 3y ago
    lots of software developers never deal with clients (clients as in the people who will actually use the software) - most of them in fact, in any of the big companies I have worked for anyway...and that is probably not a good thing. I myself, prefer to work with the people who will actually use what I build - get a better product hat way.
adamauckland 3y ago
I consider a "bug" to be "it was supposed to do something and failed". Issues around business logic are not failures of the system, the system worked to spec, the spec was not comprehensive enough and now we iterate.
- repelsteeltje 3y ago
  ...And now we could probably start debating your narrow definition of "system". ;-)
  - pipo234 3y ago
    Most of the software I've built doesn't have "a spec.", but let me zoom in on specs. around streaming media. MPEG DASH, CMAF or even the base media file format (ISO/IEC 14496-12) at times can be pretty vague. In practice, this frequently turns up in actual interoperability issues where it's pretty difficult to point out which of two products is according to spec and which one has a bug. So yes, I totally agree with GP and would actually go further: a phrase like "we found all the bugs in the database" is nonsense and makes the article less credible.
- Aachen 3y ago
  What do you call it when the spec is wrong? Like clearly actually wrong, such as when someone copied a paragraph from one CRUD-describing page to the next and forgot to change the word "thing1" to "thing2" in the delete description. Because I'd call that a bug. A spec bug, but a bug. It's no feature request to make the code based on the newer page delete thing2 rather than thing1, it's fixing a defect
  - deleted 3y ago
    [deleted]
  - pinkmuffinere 3y ago
    Ya, I would like a word for this as well. I naturally refer to this category of error as bug, but this occasionally leads to significant conflict with others at work. I now default to calling _almost everything_ a feature request, which is obviously dumb but less likely to get me into trouble. If there is a better word for "it does exactly what we planned, but what we planned was wrong" I would love to adopt it.
amw-zero 3y ago
I do think that it was a mistake to use the word "all" and imply that there are absolutely no bugs in FoundationDB. However, FoundationDB is truly known as having advanced the state of the art for testing practices: https://apple.github.io/foundationdb/testing.html https://apple.github.io/foundationdb/testing.html. So in normal cases this would reek of someone being arrogant / overconfident, but here they really have gotten very close to zero bugs.
- spinningD20 3y ago
  The other issue I would point out is that building a database, while impressive with their quality, is still fundamentally different than an application or set of applications like a larger SaaS offering would involve (api, web, mobile, etc). Like the difference between API and UI test strategies, where API has much more clearly defined and standardized inputs and outputs. To be clear, I am not saying that you can't define all inputs and outputs of a "complete SaaS product offering stack", because you likely could, though if it's already been built by someone that doesn't have these things in mind, then it's a different problem space to find bugs. As someone who has spent the last 15 years championing quality strategy for companies and training folks of varying roles on how to properly assess risk, it does indeed feel like this has a more narrow scope of "bug" as a definition, in the sort of way that a developer could try to claim that robust unit tests would catch "any" bugs, or even most of them. The types of risk to a software's quality have larger surface areas than at that level.
  - amw-zero 3y ago
    There's a lot of assertions that I throw into business applications that would be very useful to test in this way. So I don't think this only applies to testing databases. Also, when properties are difficult to think of, that often means that a model of the behavior might be more appropriate to test against, e.g. https://concerningquality.com/model-based-testing/ https://concerningquality.com/model-based-testing/. It would take a bit of design work to get this to play nicely with the Antithesis approach, but it's definitely doable.
    - 3y ago
nlavezzo 3y ago
I think the reference to "all the bugs" here is basically that our insanely brutal deterministic testing system was not finding any more bugs after 100's of thousands of runs. Can't prove a negative obviously, but the fact that we'd gotten to that "all green" status gave us a ton of confidence to push forward in feature development, believing we were building on something solid - which, time has shown we were.
- dap 3y ago
  Thanks -- that's very clarifying! But isn't this circular? The lack of bugs is used as evidence of the effectiveness of the testing approach, but the testing approach is validated by...not finding any more bugs in the software?
  - FridgeSeal 3y ago
    Yeah but if your software is running in an environment that controls for a lot of non-determinism and can simulate various kinds of failures and degradations at varying rates, and do it all in accelerated time and your software is still working correctly; I think it’d be somewhat reasonable to assert that maybe the testing setup has done a pretty good job.
    - dap 3y ago
      Agreed, the approach sounds very interesting and I can see how it could be very effective! I'd love to try it on my own stuff. That's why it's so surprising (to me) to claim that the approach found nearly every bug in something as complicated as a production distributed database. My career experience tells me (quite strongly) that can't possibly be true.
dap 3y ago
The best definition I've heard for "bug" is "software not working as documented". Of course, a lot of software is lacking documentation -- and those are doc bugs. But I like this definition because even when the docs are incomplete, the definition guides you to ask: would I really document that the software behaves like this or would I change the behavior [and document that]? It's harder (at least for me) to sweep goofy behavior under the rug.
oconnor663 3y ago
To be fair, the line right after that is "I know, I know, that's an insane thing to say."
pshc 3y ago
I feel like business logic bugs live on a separate layer, the application layer, and it's not fair to count those against the database itself. I agree that suboptimal query planning would be a database-layer bug, a defect which could easily be missed by the bug-testing framework.

moritonal 3y ago

Good summary of the hard part of being a software developer that deals with clients.

Aachen 3y ago
What software developer does not deal with clients (and makes a living)?
- ejb999 3y ago
  lots of software developers never deal with clients (clients as in the people who will actually use the software) - most of them in fact, in any of the big companies I have worked for anyway...and that is probably not a good thing. I myself, prefer to work with the people who will actually use what I build - get a better product hat way.

adamauckland 3y ago

I consider a "bug" to be "it was supposed to do something and failed". Issues around business logic are not failures of the system, the system worked to spec, the spec was not comprehensive enough and now we iterate.

repelsteeltje 3y ago
...And now we could probably start debating your narrow definition of "system". ;-)
- pipo234 3y ago
  Most of the software I've built doesn't have "a spec.", but let me zoom in on specs. around streaming media. MPEG DASH, CMAF or even the base media file format (ISO/IEC 14496-12) at times can be pretty vague. In practice, this frequently turns up in actual interoperability issues where it's pretty difficult to point out which of two products is according to spec and which one has a bug. So yes, I totally agree with GP and would actually go further: a phrase like "we found all the bugs in the database" is nonsense and makes the article less credible.
Aachen 3y ago
What do you call it when the spec is wrong? Like clearly actually wrong, such as when someone copied a paragraph from one CRUD-describing page to the next and forgot to change the word "thing1" to "thing2" in the delete description. Because I'd call that a bug. A spec bug, but a bug. It's no feature request to make the code based on the newer page delete thing2 rather than thing1, it's fixing a defect
- deleted 3y ago
  [deleted]
- pinkmuffinere 3y ago
  Ya, I would like a word for this as well. I naturally refer to this category of error as bug, but this occasionally leads to significant conflict with others at work. I now default to calling _almost everything_ a feature request, which is obviously dumb but less likely to get me into trouble. If there is a better word for "it does exactly what we planned, but what we planned was wrong" I would love to adopt it.

amw-zero 3y ago

I do think that it was a mistake to use the word "all" and imply that there are absolutely no bugs in FoundationDB. However, FoundationDB is truly known as having advanced the state of the art for testing practices: https://apple.github.io/foundationdb/testing.html https://apple.github.io/foundationdb/testing.html. So in normal cases this would reek of someone being arrogant / overconfident, but here they really have gotten very close to zero bugs.

spinningD20 3y ago
The other issue I would point out is that building a database, while impressive with their quality, is still fundamentally different than an application or set of applications like a larger SaaS offering would involve (api, web, mobile, etc). Like the difference between API and UI test strategies, where API has much more clearly defined and standardized inputs and outputs. To be clear, I am not saying that you can't define all inputs and outputs of a "complete SaaS product offering stack", because you likely could, though if it's already been built by someone that doesn't have these things in mind, then it's a different problem space to find bugs. As someone who has spent the last 15 years championing quality strategy for companies and training folks of varying roles on how to properly assess risk, it does indeed feel like this has a more narrow scope of "bug" as a definition, in the sort of way that a developer could try to claim that robust unit tests would catch "any" bugs, or even most of them. The types of risk to a software's quality have larger surface areas than at that level.
- amw-zero 3y ago
  There's a lot of assertions that I throw into business applications that would be very useful to test in this way. So I don't think this only applies to testing databases. Also, when properties are difficult to think of, that often means that a model of the behavior might be more appropriate to test against, e.g. https://concerningquality.com/model-based-testing/ https://concerningquality.com/model-based-testing/. It would take a bit of design work to get this to play nicely with the Antithesis approach, but it's definitely doable.
  - 3y ago

nlavezzo 3y ago

I think the reference to "all the bugs" here is basically that our insanely brutal deterministic testing system was not finding any more bugs after 100's of thousands of runs. Can't prove a negative obviously, but the fact that we'd gotten to that "all green" status gave us a ton of confidence to push forward in feature development, believing we were building on something solid - which, time has shown we were.

dap 3y ago
Thanks -- that's very clarifying! But isn't this circular? The lack of bugs is used as evidence of the effectiveness of the testing approach, but the testing approach is validated by...not finding any more bugs in the software?
- FridgeSeal 3y ago
  Yeah but if your software is running in an environment that controls for a lot of non-determinism and can simulate various kinds of failures and degradations at varying rates, and do it all in accelerated time and your software is still working correctly; I think it’d be somewhat reasonable to assert that maybe the testing setup has done a pretty good job.
  - dap 3y ago
    Agreed, the approach sounds very interesting and I can see how it could be very effective! I'd love to try it on my own stuff. That's why it's so surprising (to me) to claim that the approach found nearly every bug in something as complicated as a production distributed database. My career experience tells me (quite strongly) that can't possibly be true.

dap 3y ago

The best definition I've heard for "bug" is "software not working as documented". Of course, a lot of software is lacking documentation -- and those are doc bugs. But I like this definition because even when the docs are incomplete, the definition guides you to ask: would I really document that the software behaves like this or would I change the behavior [and document that]? It's harder (at least for me) to sweep goofy behavior under the rug.

oconnor663 3y ago

To be fair, the line right after that is "I know, I know, that's an insane thing to say."

pshc 3y ago

I feel like business logic bugs live on a separate layer, the application layer, and it's not fair to count those against the database itself. I agree that suboptimal query planning would be a database-layer bug, a defect which could easily be missed by the bug-testing framework.