3 ms·

This looks kind of interesting: https://github.com/huggingface/pytorch-pretrained-BigGAN/blob/6ae20a35a051816d66811d85597033623a8ac888/pytorch_pretrained_bigga

This looks kind of interesting: https://github.com/huggingface/pytorch-pretrained-BigGAN/blob/6ae20a35a051816d66811d85597033623a8ac888/pytorch_pretrained_biggan/model.py#L29 https://github.com/huggingface/pytorch-pretrained-BigGAN/blo... When you look further down you find: https://github.com/huggingface/pytorch-pretrained-BigGAN/blob/6ae20a35a051816d66811d85597033623a8ac888/pytorch_pretrained_biggan/model.py#L253-L281 https://github.com/huggingface/pytorch-pretrained-BigGAN/blo... And that's just a quick search for s3 in the repo. It would not surprise me in the least to discover a `from_pretrained` that points at one of the s3 resources being pulled. There's probably other stuff like that as well in the code that could be causing equally nasty heartache .. especially if non-persistent containers are involved.... (This is a WAG aka Wild A Guess) EDIT: Dug a little more and found: https://github.com/search?q=org%3Ahuggingface+s3&type=Code https://github.com/search?q=org%3Ahuggingface+s3&type=Code Unless I'm mistaken here, there's a crap ton of code that could be downloading models at runtime ... Which seems significantly less than ideal ...

emeraldd 7y ago
Based on this, they might not even realize they are downloading this stuff ...
madisonmay 7y ago
It's also possible it's part of a docker build step or similar. Even if they're aren't downloading models at run time they may be loading s3 if their pytorch-transformers lib docker cache gets invalidated frequently.
- kortex 7y ago
  Docker builds are crazy wasteful in terms of bandwidth and compute. Right now I'm struggling with a project that builds ITK on demand every heckin time. I'm working through how to best integrate apt-cacher-ng, sccache, and a pip cacher. It costs me nothing to hit apt or pypi, but like, somebody is paying that bill. A little perspective goes a long way. I wonder if I could do something to just proxy all requests and cache those on a whitelist and stick it on my CI network.

emeraldd 7y ago

Based on this, they might not even realize they are downloading this stuff ...

madisonmay 7y ago

It's also possible it's part of a docker build step or similar. Even if they're aren't downloading models at run time they may be loading s3 if their pytorch-transformers lib docker cache gets invalidated frequently.

kortex 7y ago
Docker builds are crazy wasteful in terms of bandwidth and compute. Right now I'm struggling with a project that builds ITK on demand every heckin time. I'm working through how to best integrate apt-cacher-ng, sccache, and a pip cacher. It costs me nothing to hit apt or pypi, but like, somebody is paying that bill. A little perspective goes a long way. I wonder if I could do something to just proxy all requests and cache those on a whitelist and stick it on my CI network.