10 ms·
Amazon Cloud Traffic Is Suffocating Fedora's Mirrors
- INTPenis 2y agoI think Digitalocean has their own package mirror for their image. AWS is just being ignorant. If I were in charge of Fedora infrastructure I'd block them and send them instructions on how to setup a mirror.
- kbolino 2y agoThough there are probably things AWS could do anyway, this could well be caused by a large customer using a custom AMI, and not because of anything Amazon did or didn't do.
- hinkley 2y agoEscalating delays can help with this. Get it to be slow enough that people notice. XML schemas have had a similar history, tanking w3c.org servers.
- recursive 2y agoThey totally deserved it for making namespaces that "just happen to be" URLs. XML is insane.
- kevindamm 2y agowell, but they're URIs. See, the difference is right there. An identifier not a location. Nobody should ever confuse the two! /s of course
- IshKebab 2y agoWhy even are they URLs? The only reasonable suggestion I could find is that it was part of an abandoned or poorly adopted idea to also host the schema at that URL.
- TillE 2y agoProbably the same sort of thought process that led to the convention of Java packages being named com.example.whatever. It identifies the creator and gives you some structure to create a unique identifier. Lot of half-baked ideas floating around in the early years of the commercial internet, but the Java thing held up better.
- agwa 2y agoIt's a way to ensure global uniqueness. In the end they're just compared byte-for-byte as strings.
- IshKebab 2y agoPrecisely, so why are they URLs and not something like Java's packages? Or just a URL without the `http:// http://`?
- agwa 2y agoSnarky answer: because they're the W3C and are high on their own supply ;-) But it does allow more flexibility. If you don't want to be tied to a domain name, you can use a URN with a UUID like urn:uuid:4603d9d3-e895-4000-9077-0ab0f2776e1e
- IshKebab 2y agoYou could also just do `com.mydomain.uid.4603d....` if you wanted. But yeah I think your snarky answer is probably more than a little true!
- hinkley 2y agoIn addition to the uniqueness others have mentioned, where do you find the canonical definition of the schema if it doesn’t have a url? So they just cut out the mapping and made them one.
- recursive 2y ago
- rodgerd 2y agoAh I remember the good old days of strict XML parsers that would fail if they didn't have Internet access to pull the schema in.
- hinkley 2y agoWhich you didn’t figure out for a while and so your CI/CD pipeline and dev code/build/test cycles hammer their servers for months. Then you prefetch to fix that problem, and now some slow calls you hadn’t gotten to the bottom of suddenly aren’t.
- agwa 2y agoSchemas != namespaces. I'm sure some braindead software out there attempts to retrieve namespace URIs, but it would surely be a drop in the bucket compared to traffic for schemas/DTDs (which are intended to be retrieved).
- jcrawfordor 2y agoThe fact that this is EPEL strongly suggests that it was set up by an AWS user, not by Amazon themselves. EPEL is not used by default in any common AWS AMIs. Perhaps it is an Amazon Linux user who enabled EPEL via Amazon's package, but it's not supported in the most recent version of AL so Amazon seems to have addresses that issue anyway.
- Havoc 2y ago>that it was set up by an AWS user, A user with "five million additional systems" on AWS?
- vesrah 2y agoA long time ago I knew a guy that uploaded a Counter-Strike patch to his ISP personal hosting and ended up on the official mirror list. Ended up taking down the ISP iirc.
- Reason077 2y ago> “A user with "five million additional systems" on AWS?” Someone is going to be in for a big surprise when they get their AWS bill this month and realise there’s an infinite-loop bug in their instance spawning script.
- thedudeabides97 2y ago[dead]
- facialwipe 2y agoIt’s clearly either a large contract that would have been negotiated before any instances were spun up, or Amazon themselves.
- stonogo 2y agoI don't think it's one user; I think it's a ton of them. Want to use Let's Encrypt in your Openshift-on-AWS deployment? certbot's in EPEL, along with a lot of other quality-of-life stuff for log-shipping, monitoring, etc.
- blitzar 2y agoNahh that seems needlessly cruel they should continue to serve them at 100k speed.
- ajross 2y agoSurely AWS knows how to set up a mirror. It's just a mistake, they'll surely correct it. Also simply blogging about it (which gets amplified by Phoronix, then by HN) is a better strategy for getting their attention than blocking.
- INTPenis 2y agoI worked in ops for 20+ years. If someone blocks you it becomes an incident, a post mortem and you learn your lesson. If someone blogs about it, or e-mails you, it gets added to a todo list and might get fixed in a few weeks by a disinterested intern.
- papichulo2023 2y agoMaybe they have a commercial relationship and dont want to harm it because a bug?
- ajross 2y agoPer the original blog: > ADDENDUM (2024-05-30T01:08+00:00): Multiple Amazon engineers reached out after I posted this and there is work on identifying what is causing this issue. Thank you to all the people who are burning the midnight oil on this. You worked in ops, but not in a context where your employer could get shamed by IBM in public on the pages of Phoronix and HN. Call it "cloud scale" ops, I guess.
- pquki4 2y agoHow do you know "it's just a mistake"?
- skywhopper 2y agoI doubt Amazon builds the Fedora images. So if they’re pointed to the wrong place, that’s not AWS’s fault.
- phirephly 2y agoI've made a point of calling out Digital Ocean in Linux mirroring talks as the gold standard for being a good citizen; run their own internal mirrors, which are FAST, making it a value add feature for them as well.
- xd1936 2y agoI wish apt, dnf/rpm, flatpak, etc utilized a decentralized distribution option, like IPFS or BEP46 Mutable Torrents. It would be neat if the project leads seeded new package update hashes, volunteers ran seedboxes instead of http mirrors, and clients had (default-on?) seeding of package binaries in addition to only downloading. It would be neat to see the open source community contributing to support each other's experience.
- spookie 2y agoAt the end of the day, there's a trust issue. The way distribution works nowadays, mitigates a lot of those. Making it decentralised would be a step backwards.
- ciupicri 2y agoAren't the RPMs signed?
- BSDobelix 2y agoYou know that packages are signed right? That's why everyone can be a libreoffice or ArchLinux Mirror...or Fedora?
- deleted 2y ago[deleted]
- aaomidi 2y agoAnd because of this, these mirrors are often non https so your ISP can actually intercept and provide the data to you directly. This is why torrents would actually work fine in this model.
- Cyphase 2y agoWhere intercept usually means point DNS for the mirror domains to the ISP's local mirror, and ISP could mean your cloud provider.
- 2y ago
- renewiltord 2y agoChecks out. The normal stuff is mirrored but not EPEL https://repost.aws/knowledge-center/ec2-enable-epel https://repost.aws/knowledge-center/ec2-enable-epel
- steelframe 2y agoSomething irks me about volunteers spending real money to support all the OSS freeloading businesses. I'm talking about companies with a market cap in the $Billions. Almost none of them can be bothered to kick back even a modicum of financial support to the authors of the software that runs their business, and to add insult to injury, they in fact soak the members of community who distribute the binaries for their bandwidth.
- deleted 2y ago[deleted]
- BSDobelix 2y agoHow about just support community distributions? ...and no, Fedora is non of them. Just stop support brand's and corporation's if you don't get paid?
- mardifoufs 2y agoDoes Redhat pay for fedora's infra? Genuine question and I'm not saying it's better either. It's just that it would be weird for fedora to operate on volunteer/donated infra when it's quite important to red hat, considering it's the "upstream" (not sure if that's the correct term here) of RHEL
- steelframe 2y agoFrom TFA: > The massive uptick in Fedora/EPEL activity puts additional pressure on Fedora web proxies for mirror data and then the mirrors themselves that tend to be volunteer run. Much of this new traffic is coming from the Amazon/AWS cloud.
- mardifoufs 2y agoAh I read the article but I wasn't sure what they meant by volunteer (for example individuals only or corporate too) run in this context. Your interpretation makes sense though!
- quaintdev 2y ago
- skissane 2y agoSuppose you have an Artifactory server that mirrors/caches a lot of public stuff so one is (hopefully) a good citizen and don’t spam public mirrors with constant requests for the same thing. But every tool has its own config to set to use the Artifactory. One setting for the OS package manager (which is different for different Linux distributions), another for PyPI, another for NPM (or Yarn or whatever), another for Maven/Gradle, something else for Go, then I need to download this Postgres extension and build it from source - the list goes on. So almost inevitably something gets missed and one ends up not being as good a citizen as one ought, and then one day some random Jenkins job is failing because some external dependency could not be downloaded. I wish there was an easier way. Like some standard mechanism for saying “for this URL use this proxy”. I guess one could just use a proxy server (http_proxy environment variable) but with most things on HTTPS it needs to MITM the TLS which then means you need that certificate installed in the build process - which is another one of those “everything can do it but everything does it differently” problems. And in any event, MITM is a bad smell.
- azornathogron 2y agoProxy auto config (PAC) supports specifying different proxies for different URLs. Unfortunately, a PAC file is just a file that contains a JavaScript function to pick the proxy, so they're crazily over-powered for the task, and support for them isn't very broad. Browsers support them, but I guess most command line tools wouldn't. https://en.m.wikipedia.org/wiki/Proxy_auto-config https://en.m.wikipedia.org/wiki/Proxy_auto-config
- skissane 2y agoAnother solution: a HTTP proxy server listening on localhost to which you sent HTTPS requests using GET https:// https:// instead of CONNECT. Then the proxy server could have all the logic about which requests to handle via the cache versus which to fetch directly. It could also handle authentication to a cache server if that is required. The problem is most clients don’t do GET https:// https://, because in your old-school corporate web proxy use case, the proxy server is remote, and sending HTTPS requests to it over HTTP eliminates the security of HTTPS. If only there was some standard environment variable like artifact_proxy, which had to be a localhost http URI, and which tools would understand as meaning “send HTTP GET to this proxy, even for https:// https://, delegating all the TLS stuff to it, but only if you are trying to download a build artifact, not for any runtime use” The hard part wouldn’t be implementing this idea (the local proxy server and the environment variable), the hard part would be getting all the different tool developers to agree to support it
- lameidoit 2y agoJust upload distributions in the blockchain
- crest 2y agoStart putting ever harsher rate limits on their IP ranges in place until AWS has an actual human reaches out bypassing their so-called "support" channels by making their problem?
- deleted 2y ago[deleted]
- tragiclos 2y agoIt might be more appropriate to link to the original blog post: http://smoogespace.blogspot.com/2024/05/where-did-5-million-epel-7-systems-come.html http://smoogespace.blogspot.com/2024/05/where-did-5-million-...
- kemitchell 2y ago> Amazon Cloud Traffic Is Suffocating Fedora's Mirrors An astounding milestone for the English language. Imagine what this sentence possibly could have meant in 1990.
- MongoTheMad 2y agoNative English speaker here that has a Fedora box as a daily driver. I thought this title referenced cars driving in a south american rain forest.
- yjftsjthsd-h 2y agoSo I know this is only kind of relevant, but... why is EPEL on Fedora mirrors at all? AFAIK EPEL is specifically for RHEL et al. and its packages don't even target Fedora.
- klooney 2y agoIt's built and maintained by the Fedora community
- phirephly 2y agoEPEL is a separate module from fedora-enchilada, but uses the same backend CDN infrastructure and most mirrors tend to carry both /fedora/ and /epel/, so they're not technically the same mirrors, but most mirrors tend to carry both.
- greatgib 2y agoIt always surprised me that no one complained about the current trend of having automated process or build quasi systematically retrieving packages from public repositories like pypi, debian, GitHub, ... Each time a debian image or something is build, or an automated test or GitHub action is run. Without personal cache. A decade ago, each company used to have its own cache of all public packages for CI/CD, but it looks like that bo one cares anymore.
- chriswarbo 2y ago> A decade ago, each company used to have its own cache of all public packages for CI/CD, but it looks like that no one cares anymore. Even worse, many of those processes are running an 'apt-get update' (or equivalent), so there's no way to know what packages they'll get each time!