6 ms·
Scientist: A Ruby library for carefully refactoring critical paths
- Manfred 7y agoFeeding production data into two different paths to compare them is obviously valuable. Does anyone have any good reasons why this has to happen in online code and can't happen against database replicas in offline mode? I always feel that it would be great to be able to do this with code that has side-effects (eg. anything that changes the database) but I've never seen a general purpose solution for this. The README mentions using a write replica, but how do you deal with data drifting in case of bad writes?
- FrancoisBosun 7y agoMaybe because of the large increase in complexity?
- dyeje 7y agoThey wrote a good blog post about using this gem back in 2016: https://github.blog/2016-02-03-scientist/ https://github.blog/2016-02-03-scientist/
- jauco 7y agoThere’s goreplay[0]. But once you start using that you quickly find that many parts of your app use random data (such as uuids) or clock data (timestamps etc) getting these synced between prod and replay or ignored on replay is quite a hassle. [0]: https://goreplay.org/ https://goreplay.org/
- hamandcheese 7y agoAnother possibility might be to do the side effects in a transaction that gets rolled back?
- ydnaclementine 7y agoUsing this at work last week to refactor a database call to a new service call
- tobyhinloopen 7y agoAlternative title: “Scientist: A Ruby library for changing production code while still not having to write tests”
- hinkley 7y agoIn TDD they might call these pinning tests. This kind of testing isn’t just good for refactors. You can also use it for rewrites or substantial architectural changes. I think github did something like this while trying to make the storage format more efficient.
- hamandcheese 7y agoWhich is really valuable if you don’t have tests, or your test suite doesn’t give you much confidence, is too brittle to make the changes without breaking the tests, or maybe the code is structured so that it’s really hard to test all the paths that occur in production.
- Xylakant 7y agoEven in a well-tested codebase some integration points are extremely hard to test, notably external services that you connect to. Rewriting these to use a different service or just a different way of connecting to a service would be an example where this ends up being useful.
- jbarnette 7y agoBy the time we extracted Scientist, the code we were refactoring had pretty good test coverage. But even the best test suite is an imperfect model of production. The first section of the README briefly mentions this: "Let's pretend you're changing the way you handle permissions in a large web app. Tests can help guide your refactoring, but you really want to compare the current and refactored behaviors under load."
- jamesfzhang 7y agoI've use Scientist for every major refactor I've done--IMO it's one of the best examples of a library that does one thing extremely well and has a clean interface.
- spraak 7y agoUnfortunately the Node version leaves a bad taste in my mouth.
- dang 7y ago"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- nchase 7y agohttps://www.npmjs.com/package/scientist https://www.npmjs.com/package/scientist Is it this one, or another?
- aflag 7y agoHow to handle the case when you want to replace a function that makes changes to a database? Is this library aimed only towards functions which don't make changes to the system's state?
- blairanderson 7y agoHow would you handle this case WITHOUT this library?
- aflag 7y agoI'm not experienced in running live experiments. So, I don't know.
- sentrysapper 7y agois there a library like this for PHP?
- e12e 7y agoSee https://github.com/github/scientist/blob/master/README.md#alternatives https://github.com/github/scientist/blob/master/README.md#al...