3 ms·
FWIW, our Artifacts tool can track references to data that exists in other systems, or store the data directly which allows us to ensure there are no "dark matt
by slewis 6y ago
FWIW, our Artifacts tool can track references to data that exists in other systems, or store the data directly which allows us to ensure there are no "dark matter" users like you mentioned. It's a flexible system.
- jdoliner 6y agoThe key difference is that it's something you _can_ do, rather than something that just happens automatically. It's not about what you're able to do, it's about what you can forget to do. Storing references to external systems is something that people do in Pachyderm as well, but we discourage it because those external systems generally aren't immutable. So they can be deleted, or even worse change, at which point your lineage tracking is actually misleading you rather than informing you.