3 ms·

For operations that don't have any side effects, I can definitely see how you could use the Science library. I'm curious though if there are any strategies fol

For operations that don't have any side effects, I can definitely see how you could use the Science library. I'm curious though if there are any strategies folks use for experiments that do have side effects like updating a database or modifying files on disk.

Nacraile 11y ago
The first thing to do is to try to minimize the scope of mutating operations: e.g, decompose a monolithic read-write operation into independent load, transform, and store. You can then easily test as much as possible (the load and the store) side-by-side. The parts that must have side effects are still hard, but at least there are fewer of them. (One variant of this would be to build your code such that you can always intercept side-effects, and then block the new code's side effects and compare with the old code) The next thing is to try to take advantage of idempotence (and, by extension, try to make as many of your mutating operations idempotent as possible): there's nothing completely risk free, but you can at least verify idempotence for either ordering of new and old code, and if you're factored right, you can run both paths on the same input and verify they have the same side-effects and output. Finally, making the observation that the new code must in general be backwards-compatible with the old code, and both versions need to be able to run concurrently (because this situation will always exist during deployment): in the worst case, you can always start with a limited deployment of the new path, which limits the amount of damage done if the new code is bad.
bcbrown 11y ago
Point the control to the real database, and the experiment to an unused database. After each request through Scientist, the two databases should be identical. Or, make the output be the SQL command used to mutate the database. If the two outputs are different, then you've found something that needs investigation.

Nacraile 11y ago

The first thing to do is to try to minimize the scope of mutating operations: e.g, decompose a monolithic read-write operation into independent load, transform, and store. You can then easily test as much as possible (the load and the store) side-by-side. The parts that must have side effects are still hard, but at least there are fewer of them. (One variant of this would be to build your code such that you can always intercept side-effects, and then block the new code's side effects and compare with the old code) The next thing is to try to take advantage of idempotence (and, by extension, try to make as many of your mutating operations idempotent as possible): there's nothing completely risk free, but you can at least verify idempotence for either ordering of new and old code, and if you're factored right, you can run both paths on the same input and verify they have the same side-effects and output. Finally, making the observation that the new code must in general be backwards-compatible with the old code, and both versions need to be able to run concurrently (because this situation will always exist during deployment): in the worst case, you can always start with a limited deployment of the new path, which limits the amount of damage done if the new code is bad.

bcbrown 11y ago

Point the control to the real database, and the experiment to an unused database. After each request through Scientist, the two databases should be identical. Or, make the output be the SQL command used to mutate the database. If the two outputs are different, then you've found something that needs investigation.