3 ms·

> a platform that takes your software and hunts for bugs in it Ok but, what actually IS it? It seems like it is a cloud service that will run integration test

> a platform that takes your software and hunts for bugs in it Ok but, what actually IS it? It seems like it is a cloud service that will run integration tests. I have to figure out how to deploy to this special environment and I still have to write those integration tests using special libraries. But even after all that integration refactoring, how is this supposed to help me find actual bugs that I wouldn't already have found in my own environment with my own integration tests?

mrinterweb 3y ago
I came away with the same questions.
spartanatreyu 3y ago
The article says they created a deterministic hypervisor that runs all pseudorandom behavior from a starting seed to enable perfect re-playability. But that's all we know so far. I'm assuming there'll be some sort of fuzz testing, and static analysis or some defining actions that your software can perform. Honestly it sounds a lot like it has a lot of crossover with what the Vale language is trying to solve: https://vale.dev/ https://vale.dev/, but focused on trying to get existing software to that state instead of creating a new language to make new software already be at that state by default.
nlavezzo 3y ago
I'd suggest taking a dive into the docs - there is quite a lot there that should address some of these questions. That said, Antithesis doesn't require you to write manual tests, integration or otherwise. It requires your software system to be packaged in containers, which is fairly straightforward, and then requires a workload to be written which will emulate the normal functioning of the software system. So for example an e-commerce store would have product views, cart adds, checkouts, etc. With this, Antithesis can start testing (running your workload, varying inputs, injecting faults, etc) the software and looking for violations of test properties. There are many (60+) test properties that come "out of the box" such as crashes, out of memory, etc. You can (and should) also define custom properties that are unique to your system, as this will surface more problems. As your tests run, violations of test properties are reported, with lots of useful debug information included. Test runs that are particularly interesting can have a lot of extra analysis done, due to our ability to "rewind" and change inputs, get artifacts, add logging, etc.
- shuntress 3y ago
  "Workloads" seem to be effectively equivalent to integration tests. I don't mean to poke holes but I'm having trouble seeing the value add here. If I have to deploy to some new environment anyways and I have to tailor the "Workloads" anyways why would I pay extra for vendor lock-in? The type of devious bug this is promising to find would be something like: "The DB silently drops timezone from Dates because of the column type. This results in unexpected data being returned for users in different timezones from the server" I just don't see how repeatably calling the API with an expanding set of random inputs helps find something like that.
- catskul2 3y ago
  IMO "Read the docs" is not a reasonable response to "what is it?" Typically someone wants to know the most basic question before devoting time to diving into the docs.
maronato 3y ago
Reading their docs it seems you got it pretty much correctly. You write integration tests (aka “workloads”) and then they run it under different scenarios. This means they use their hypervisor to change random seeds, make http requests fail or take too long, break connections between servers, change the order of server responses, and all sorts of wild things you don’t usually control, but that happen in the real world. Then they compare the expected workload responses and figure out which conditions break your systems. That’s why they sell yearly contracts - you’re supposed to pay them to keep your workloads running continuously all year to try all sorts of different combinations of failures.

mrinterweb 3y ago

I came away with the same questions.

spartanatreyu 3y ago

The article says they created a deterministic hypervisor that runs all pseudorandom behavior from a starting seed to enable perfect re-playability. But that's all we know so far. I'm assuming there'll be some sort of fuzz testing, and static analysis or some defining actions that your software can perform. Honestly it sounds a lot like it has a lot of crossover with what the Vale language is trying to solve: https://vale.dev/ https://vale.dev/, but focused on trying to get existing software to that state instead of creating a new language to make new software already be at that state by default.

nlavezzo 3y ago

I'd suggest taking a dive into the docs - there is quite a lot there that should address some of these questions. That said, Antithesis doesn't require you to write manual tests, integration or otherwise. It requires your software system to be packaged in containers, which is fairly straightforward, and then requires a workload to be written which will emulate the normal functioning of the software system. So for example an e-commerce store would have product views, cart adds, checkouts, etc. With this, Antithesis can start testing (running your workload, varying inputs, injecting faults, etc) the software and looking for violations of test properties. There are many (60+) test properties that come "out of the box" such as crashes, out of memory, etc. You can (and should) also define custom properties that are unique to your system, as this will surface more problems. As your tests run, violations of test properties are reported, with lots of useful debug information included. Test runs that are particularly interesting can have a lot of extra analysis done, due to our ability to "rewind" and change inputs, get artifacts, add logging, etc.

shuntress 3y ago
"Workloads" seem to be effectively equivalent to integration tests. I don't mean to poke holes but I'm having trouble seeing the value add here. If I have to deploy to some new environment anyways and I have to tailor the "Workloads" anyways why would I pay extra for vendor lock-in? The type of devious bug this is promising to find would be something like: "The DB silently drops timezone from Dates because of the column type. This results in unexpected data being returned for users in different timezones from the server" I just don't see how repeatably calling the API with an expanding set of random inputs helps find something like that.
catskul2 3y ago
IMO "Read the docs" is not a reasonable response to "what is it?" Typically someone wants to know the most basic question before devoting time to diving into the docs.

maronato 3y ago

Reading their docs it seems you got it pretty much correctly. You write integration tests (aka “workloads”) and then they run it under different scenarios. This means they use their hypervisor to change random seeds, make http requests fail or take too long, break connections between servers, change the order of server responses, and all sorts of wild things you don’t usually control, but that happen in the real world. Then they compare the expected workload responses and figure out which conditions break your systems. That’s why they sell yearly contracts - you’re supposed to pay them to keep your workloads running continuously all year to try all sorts of different combinations of failures.