3 ms·
I agree that fixing the pipeline is indeed the correct decision, but I've created this tool to provide the detection. In a complex environment, you often don't
by arseniibr 9mo ago
I agree that fixing the pipeline is indeed the correct decision, but I've created this tool to provide the detection.
In a complex environment, you often don't control the upstream ingestion methods used by every team. They might use git lfs, wget, huggingface-cli, or custom caching layers.
Relying solely on the hope that every downstream consumer correctly handles Git LFS is dangerous. This tool acts as a detector to catch those inevitable human or tooling errors before they crash the production.
- embedding-shape 9mo ago> This tool acts as a detector to catch those inevitable human or tooling errors before they crash the production. Again, that sounds like a bigger issue, that a repository using Git LFS can somehow "crash the production", that's where I'd add resilience first. But as mentioned in another comment, I don't have the full view of your infrastructure, maybe it has to work like that for whatever reason, so YMMV.