3 ms·
In one of the better examples, someone at a very large tech company told me that one of their major data pipelines had effectively no monitoring in it while it
by Diederich 7y ago
In one of the better examples, someone at a very large tech company told me that one of their major data pipelines had effectively no monitoring in it while it ran, and it took many hours to complete. The monitoring before and after the pipeline was excellent, but they had no visibility while it was running.
He said there were cases where the pipeline had effectively failed early, but continued to run throughout most of the normal run period. If they had some in-pipeline monitoring, they could have intervened earlier and helped it succeed.