3 ms·

Petabytes of training data is only one application of PyTorch, which is going to use tens of thousands of containers, but... Inference, development cycles, any

Petabytes of training data is only one application of PyTorch, which is going to use tens of thousands of containers, but... Inference, development cycles, any of the application domains of PyTorch that don't involve training frontier models... all of those are complicated by excessive container layers. But mostly dev really sucks with writing out an extra 10GB for a small code change.

a_t48 6mo ago
Going to self promote one last time here - I've built a fix for this, at least for the registry/image export side, at https://clipper.dev https://clipper.dev. Docker(Hub) can't share large files between layers, but I can.