4 ms·

I struggle to find the correct descriptor for a counter-example, wherein You Really Want Success for the process as a whole, but it is acceptable for a sliver o

I struggle to find the correct descriptor for a counter-example, wherein You Really Want Success for the process as a whole, but it is acceptable for a sliver of it to fail, in the the context of ETL. I have an ETL I am told (I switched jobs) that is still working, from 2008. It was built to be a tank, and I also did another forbidden thing: Pokemon Exception Handling. It's a guideline, not a law of physics, and it is fine to resort to a general error catch when you really don't know every possible error (and let's be honest, if you have enough libraries in the mix, some surprises will happen) and you want the other 99.999% of the data to go through. Yes, this one little thing didn't load, and let's log that, let's examine that and figure out how to prevent that going forward, but overall, the rest of the program must continue. How did it get so tanklike? Every time a little bit failed and it got logged, I figured out what went wrong, fixed it, and then tried to generalize a class of similar errors. After a while, I got into Things I Was Told Would Never Happen in the data we ingested and programmed for when never happened. Reader, never came a little sooner than expected. Anyway, I largely agree with the idea but there are places where you want the exact opposite, and I think it is important to look for those places lest this heuristic become so stiff it can lose utility.

e9 4y ago
Funny thing, I had same experience. I also built robust ETL. It was for ingesting and manipulating financial data from 30+ different banks and I also did Pokemon Exception Handling to make it robust. In general my philosophy is: I don't want to wake up at 4am unless it's urgent. What can I do to gracefully handle failures to achieve that goal?
- at_a_remove 4y ago
  Exactly, thank you. At the time I was transitioning from a jack of all trades person to strictly a programmer. I despise getting called in the middle of the night. My feeling is that, if you have a foundational business process like this, it should be designed to be maintained if there is a serious problem, and it ought to keep working. I know, haha, "the Internet is a series of tubes" but I really wanted this thing to be like a chunk of very uninteresting ductwork that just moves air from one place to another: it should just do its job with as much fanfare.

e9 4y ago

Funny thing, I had same experience. I also built robust ETL. It was for ingesting and manipulating financial data from 30+ different banks and I also did Pokemon Exception Handling to make it robust. In general my philosophy is: I don't want to wake up at 4am unless it's urgent. What can I do to gracefully handle failures to achieve that goal?

at_a_remove 4y ago
Exactly, thank you. At the time I was transitioning from a jack of all trades person to strictly a programmer. I despise getting called in the middle of the night. My feeling is that, if you have a foundational business process like this, it should be designed to be maintained if there is a serious problem, and it ought to keep working. I know, haha, "the Internet is a series of tubes" but I really wanted this thing to be like a chunk of very uninteresting ductwork that just moves air from one place to another: it should just do its job with as much fanfare.