5 ms·

>Running our test suite 2-3 times over the span of a couple hours would trigger the rate limit… Not to single you out, but not too hard to imagine why companie

>Running our test suite 2-3 times over the span of a couple hours would trigger the rate limit… Not to single you out, but not too hard to imagine why companies start enabling rate limiting. I oftentimes wonder the obscene amount of bandwidth being consumed by this developer automation. This page[0] indicates that PyPi serves 900 terabytes daily. Seems obvious we need more turn-key solutions that enable caching for all of the popular package distributions (PyPi, NPM, Docker, etc). Too hard to configure and everyone will default to the path of least resistance (ie hammering the origin servers). [0] https://dustingram.com/articles/2021/04/14/powering-the-python-package-index-in-2021/ https://dustingram.com/articles/2021/04/14/powering-the-pyth...

glenjamin 5y ago
The fun thing about the docker rate limit is that the layer tarballs are stored on S3 in us-east-1. If you do a docker pull inside us-east-1, you’ll get a direct S3 link instead of a proxied download, and thus neither you nor docker need to pay AWS anything for bandwidth!
- yjftsjthsd-h 5y ago
  > If you do a docker pull inside us-east-1, you’ll get a direct S3 link instead of a proxied download, and thus neither you nor docker need to pay AWS anything for bandwidth! I thought cross-account data still cost something; is that not the case?
  - dantiberian 5y ago
    > You pay for all bandwidth into and out of Amazon S3, except for the following: > Data transferred from an Amazon S3 bucket to any AWS service(s) within the same AWS Region as the S3 bucket (including to a different account in the same AWS Region). https://aws.amazon.com/s3/pricing/ https://aws.amazon.com/s3/pricing/
    - yjftsjthsd-h 5y ago
      Ooooh very neat thanks for sharing:) That has... interesting potential...
      - acdha 5y ago
        Also consider enabling S3 gateway endpoints: https://docs.aws.amazon.com/vpc/latest/privatelink/vpce-gateway.html https://docs.aws.amazon.com/vpc/latest/privatelink/vpce-gate... That avoids paying NAT Gateway charges for traffic to S3 and in my testing also made a bit of a latency reduction, which came in handy once when I had a one-off data small file migration.
        yjftsjthsd-h 5y ago
        Thank you:)
shoo 5y ago
If you work in a large company, for security reasons you may already be required to use in-house mirrors of libraries or container images. There are commercial products for doing this, e.g. https://jfrog.com/artifactory/ https://jfrog.com/artifactory/ All the command line package management tools then need to be configured to point at the custom in-house package mirror, rather than connecting to the default public package server over the internet.
adamgordonbell 5y ago
Agreed! I don't mean to complain about docker, they build great products.

glenjamin 5y ago

The fun thing about the docker rate limit is that the layer tarballs are stored on S3 in us-east-1. If you do a docker pull inside us-east-1, you’ll get a direct S3 link instead of a proxied download, and thus neither you nor docker need to pay AWS anything for bandwidth!

yjftsjthsd-h 5y ago
> If you do a docker pull inside us-east-1, you’ll get a direct S3 link instead of a proxied download, and thus neither you nor docker need to pay AWS anything for bandwidth! I thought cross-account data still cost something; is that not the case?
- dantiberian 5y ago
  > You pay for all bandwidth into and out of Amazon S3, except for the following: > Data transferred from an Amazon S3 bucket to any AWS service(s) within the same AWS Region as the S3 bucket (including to a different account in the same AWS Region). https://aws.amazon.com/s3/pricing/ https://aws.amazon.com/s3/pricing/
  - yjftsjthsd-h 5y ago
    Ooooh very neat thanks for sharing:) That has... interesting potential...
    - acdha 5y ago
      Also consider enabling S3 gateway endpoints: https://docs.aws.amazon.com/vpc/latest/privatelink/vpce-gateway.html https://docs.aws.amazon.com/vpc/latest/privatelink/vpce-gate... That avoids paying NAT Gateway charges for traffic to S3 and in my testing also made a bit of a latency reduction, which came in handy once when I had a one-off data small file migration.
      - yjftsjthsd-h 5y ago
        Thank you:)

shoo 5y ago

If you work in a large company, for security reasons you may already be required to use in-house mirrors of libraries or container images. There are commercial products for doing this, e.g. https://jfrog.com/artifactory/ https://jfrog.com/artifactory/ All the command line package management tools then need to be configured to point at the custom in-house package mirror, rather than connecting to the default public package server over the internet.

adamgordonbell 5y ago

Agreed! I don't mean to complain about docker, they build great products.