4 ms·

We all agree about how much important debian is for all tech community. Why is it so difficult to scale their snapshot service? Serving static files at scale is

We all agree about how much important debian is for all tech community. Why is it so difficult to scale their snapshot service? Serving static files at scale is a solved problem. Am I missing something? Can't a cloud provider help them out?

lrem 5y ago
Isn't Debian opinionated about the freedom of the stack it stands on? Would the community be happy to build a dependency on a vendor?
- goodpoint 5y ago
  Debian is opinionated about software freedom - thankfully. But it's OK to accept donations in hardware or money as long as there are no strings attached.
BiteCode_dev 5y ago
It's very, very, expensive. I don't know the details for debian, but pypi.org, hosting python packages, costs $800k/month: https://twitter.com/dstufft/status/1236331765846990848 https://twitter.com/dstufft/status/1236331765846990848 I imaging debian is also super expensive, and so scaling that must not be easy. Every decision could be thousands of dollars.
- est31 5y ago
  Is that with at least some attempt at building a CDN? Generally cloud providers don't charge for traffic between hosts in the same availability zone. One could think about putting a slave into each major availability zone of various cloud providers. The main service would then only be used to create HTTP redirects to the specific slave, or if none exists, or the package isn't replicated yet at the slave, just answer directly. Even if such a system isn't built, with that kind of money on the table you could get a team of FANG scale developers to build it for you.
- rightbyte 5y ago
  This lazy constant pulling of dependencies by CI systems and containers is not very substainable. pypi should set up limits and make people use some cache proxy.
  - ericbarrett 5y ago
    I worked at a place (briefly) where the CI process was pulling down 3-4GiB of container images for every run. Same repos, same sites, every check-in on every branch pushed to GitHub set up a blank environment and pulled everything down afresh. Then the build process yoinked dozens of packages, again with no cache. It must have consumed terabytes a day. Madness!
    - choeger 5y ago
      I am curious about how such a cache could be setup reliably. 1. Proxy inside the company network that intercepts every request to pypi? Doesn't work well due to https, I guess. 2. Replacing pypi in your project description with your own mirror? Might work, but at least slows down every build outside of your network. Also needs to be replicated for npm, cargo, maven, docker, ... 3. Start a userspace wrapper before starting the build that transparently redirects every request. That would be the best solution, IMO. But how do I intercept https requests of child processes? Technically, it must be possible, but is there such a tool?
      - BiteCode_dev 5y ago
        For 2.: pip can be configured to use a local warehouse, and there are warehouses that transparently proxy to pypi, but caches any previously fetched result. E.G: https://pypi.org/project/proxypypi/ https://pypi.org/project/proxypypi/ Since you control it, and it's read only from the outside, you can actually expose it even outside of your network. But indeed, it must be replicated for npm, cargo, maven, docker... There is a startup idea here :)
      - user5994461 5y ago
        > Also needs to be replicated for npm, cargo, maven, docker, ... All these frameworks come with a tool to run an internal repo/mirror. There are also commercial offerings (like artifactory) that cover all these languages in a single tool. For python, just set PIP_INDEX in the CI environment to point to the internal mirror and it will be used automatically. It's very easy. By default the downloaded wheels are cached in ~/cache/somewhere. Isolated builds don't benefit from it if they start with a fresh filesystem on every run, consider mounting a writable directory to use as cache, the speedup is more than worth it.
AtNightWeCode 5y ago
All the CI/CD build agents with no cache and so on. This is a general problem for all tech. For the web, cache is cheap but as far as I know there is no equal way to cache builds as cheap. I think there needs to be a redesign in how dependencies work in most programming languages. Deterministic builds have been such a game changer and I think that CPU vs bandwidth may be the next big area to explore when it comes to compiling code.

lrem 5y ago

Isn't Debian opinionated about the freedom of the stack it stands on? Would the community be happy to build a dependency on a vendor?

goodpoint 5y ago
Debian is opinionated about software freedom - thankfully. But it's OK to accept donations in hardware or money as long as there are no strings attached.

BiteCode_dev 5y ago

It's very, very, expensive. I don't know the details for debian, but pypi.org, hosting python packages, costs $800k/month: https://twitter.com/dstufft/status/1236331765846990848 https://twitter.com/dstufft/status/1236331765846990848 I imaging debian is also super expensive, and so scaling that must not be easy. Every decision could be thousands of dollars.

est31 5y ago
Is that with at least some attempt at building a CDN? Generally cloud providers don't charge for traffic between hosts in the same availability zone. One could think about putting a slave into each major availability zone of various cloud providers. The main service would then only be used to create HTTP redirects to the specific slave, or if none exists, or the package isn't replicated yet at the slave, just answer directly. Even if such a system isn't built, with that kind of money on the table you could get a team of FANG scale developers to build it for you.
rightbyte 5y ago
This lazy constant pulling of dependencies by CI systems and containers is not very substainable. pypi should set up limits and make people use some cache proxy.
- ericbarrett 5y ago
  I worked at a place (briefly) where the CI process was pulling down 3-4GiB of container images for every run. Same repos, same sites, every check-in on every branch pushed to GitHub set up a blank environment and pulled everything down afresh. Then the build process yoinked dozens of packages, again with no cache. It must have consumed terabytes a day. Madness!
  - choeger 5y ago
    I am curious about how such a cache could be setup reliably. 1. Proxy inside the company network that intercepts every request to pypi? Doesn't work well due to https, I guess. 2. Replacing pypi in your project description with your own mirror? Might work, but at least slows down every build outside of your network. Also needs to be replicated for npm, cargo, maven, docker, ... 3. Start a userspace wrapper before starting the build that transparently redirects every request. That would be the best solution, IMO. But how do I intercept https requests of child processes? Technically, it must be possible, but is there such a tool?
    - BiteCode_dev 5y ago
      For 2.: pip can be configured to use a local warehouse, and there are warehouses that transparently proxy to pypi, but caches any previously fetched result. E.G: https://pypi.org/project/proxypypi/ https://pypi.org/project/proxypypi/ Since you control it, and it's read only from the outside, you can actually expose it even outside of your network. But indeed, it must be replicated for npm, cargo, maven, docker... There is a startup idea here :)
    - user5994461 5y ago
      > Also needs to be replicated for npm, cargo, maven, docker, ... All these frameworks come with a tool to run an internal repo/mirror. There are also commercial offerings (like artifactory) that cover all these languages in a single tool. For python, just set PIP_INDEX in the CI environment to point to the internal mirror and it will be used automatically. It's very easy. By default the downloaded wheels are cached in ~/cache/somewhere. Isolated builds don't benefit from it if they start with a fresh filesystem on every run, consider mounting a writable directory to use as cache, the speedup is more than worth it.

AtNightWeCode 5y ago

All the CI/CD build agents with no cache and so on. This is a general problem for all tech. For the web, cache is cheap but as far as I know there is no equal way to cache builds as cheap. I think there needs to be a redesign in how dependencies work in most programming languages. Deterministic builds have been such a game changer and I think that CPU vs bandwidth may be the next big area to explore when it comes to compiling code.