3 ms·
As a counter, you have 10 scrapers with the same login code. You find one site has issues when they deploy occasionally and go and add retry logic to it, then n
by IanCal 7y ago
As a counter, you have 10 scrapers with the same login code. You find one site has issues when they deploy occasionally and go and add retry logic to it, then next week find the same on another site. You go and copy paste it over all 10 or leave it. You find later the retry causes rate limiting problems so you again have to fix it in 10 places, or maybe someone fixes it in one and not the others because the implementations have drifted despite solving the same problem.
Or, you use the same code for all and if the sites change the login flow then you add a custom one for the site which is different.
Fundamentally, I don't agree with the argument that I should copy and paste code now to avoid the possibility of copying and pasting the same code later.
- Huggernaut 7y agoIt would seem like retry is working at a different level of abstraction than logging in and that code should not live together. As such, there should be the opportunity to inject a role player that understands what it means to retry. The base case of retrying is `zero retries` (or `one try`).
- IanCal 7y ago> As such, there should be the opportunity to inject a role player that understands what it means to retry. In each of the 10 scrapers? If it's common code for the retry, now all my login code needs to raise the same type of exception or otherwise signal in the same way that the login has failed. Except instead of the abstracted retry code needing to work with one login function it needs to work with ten independently maintained functions that happen to be the same currently.
- hderms 7y agoRetrying and stuff like that could either be handled with inversion-of-control which requires building a set of interfaces that provide stuff like 'login()' which is made available to some kind of orchestrator that handles retrying and other metalogic. This requires understanding some new domain specific abstraction really well or providing ample "escape hatches" in the even the abstraction doesn't work for all cases Or, in my opinion, handling it in the typical way functional programming does, you'd have your stateful computations like login represented as functions returning IO, which you can easily use off the shelf functionality to rate limit and retry (like cats effect, fs2, etc... In scala). This kind of programming isn't as mainstream as it could be, but if you can build retry once and use it for pretty much any side effecting computation, you wouldn't feel a need to DRY up things that should be separate in an attempt to share code.