10 ms·
GitHub is aggressively caching raw.github, breaking many use cases
- amerine 4y agoNot asking this to distract from the problem but can you link to the raw version of a specific commit/ref?
- quickthrower2 4y agoYes. E.g.: https://raw.githubusercontent.com/community/community/a86a03aebe28e15c2671a7f3c71e5a3a1c5eaf90/README.md https://raw.githubusercontent.com/community/community/a86a03...
- TobTobXX 4y agoThere are two hard problems in IT: cache invalidation, naming things and off-by-one errors.
- deleted 4y ago[deleted]
- ruuda 4y agoYou can eliminate the cache invalidation problem by not reusing old names for new things.
- onion2k 4y agoThat doesn't solve cache invalidation; that just means you're always invalidating the cache even in cases where you don't actually want to.
- rightbyte 4y agoYou just solve that by using another level of indirection. Duh.
- ithkuil 4y agoContent addressing causes the name to change only when the content changes, which also means the name doesn't change if the content doesn't change, this by definition you don't have spurious cache invalidations
- jakear 4y agoAnd how do you check what the new name of the content you haven't seen is?
- taeric 4y agoSerious question? There will be a root object that has a stable name through updates. When you write that, you embed current content hashes in it. Someone could get the cached of it, of course. But since it content addresses what it links to, it should avoid incoherent groups of cached data.
- jakear 4y agoI agree this would provide for increased coherency, but the additional indirection is hardly ideal. And really it's just kicking the caching problem up to a super-object that will probably be thrashed much harder than any individual file, and may be larger than each individual file. All in all it's a "certified hard problem", there's a bunch of domain specific nuance an HN thread couldn't hope to capture.
- taeric 4y agoThe extra indirection may or may not be a problem, of course. For a lot of places this is used, it is "index.html," after all. That said, I do find folks bend way too many things into fitting this pattern; such that I do not mean to be dismissive of the criticism.
- ilyt 4y agoBut where client gets the new name it should download for? As long as name client asks for (even if it is reference) is constant, it can have cache problems. Of course it makes it simpler as youc an opt to cache it so short that invalidation is less of an issue, but that's again working around cache invalidation
- jonatron 4y agonewester_new_final_v4_new.txt
- deleted 4y ago[deleted]
- rightbyte 4y agoDoesn't that just move the problems to more "naming things" though?
- ithkuil 4y agoYes, naming things is intimately tied to cache invalidation. That's why the two things are together in that maxim. Not sure why people think it has to so with naming variables or functions...
- codetrotter 4y ago> Not sure why people think it has to so with naming variables or functions... I think it’s open to interpretation. And I think it is valid to apply the saying to naming of variables, functions, and many other things. See https://skeptics.stackexchange.com/questions/19836/has-phil-karlton-ever-said-there-are-only-two-hard-things-in-computer-science https://skeptics.stackexchange.com/questions/19836/has-phil-... Links to https://www.karlton.org/2017/12/naming-things-hard/ https://www.karlton.org/2017/12/naming-things-hard/ Links to a bunch of places. None of the ones I looked at seem to confidently say what was originally meant by the saying. Either way, regardless of what the guy meant when he originally used to say it, it’s allowed to apply a saying to new situations.
- CommitSyn 4y ago> That's why the two things are together in that maxim. I think you missed the joke.
- onlypositive 4y agoDid they?
- throwanem 4y agoNot using old names for new things in this case means abandoning the concept of Git branch tags.
- endymi0n 4y agoSounds a little bit like the standard recipe of any functional programming enthusiast: Just use pure functions! Except that they‘re colliding all the time with that tiny problem that the world isn‘t pure... People expect the same page at the same URL, with different content, tomorrow, so good luck choosing a different one than the one they have bookmarked and that ranks on Google. Caching doesn‘t get any less hard by trying to define the problem away.
- deleted 4y ago[deleted]
- Faaak 4y agoThe Oxford comma would be useful here
- YourDadVPN 4y agoYou pretty much never have a list where "things" is somewhere other than the end.
- tmtvl 4y agoThe edit adding ", concurrency," and updating "two" to "three" arrived before the message, so it wasn't applied.
- joshspankit 4y agoGithub raw seems like the simplest system for solving cache invalidation: invalidate the cache of a changed file when it’s pushed. They have access to both GitHub and the raw service. I know there are usually all sorts of layers between that make interconnectivity logistically complicated, but am I wrong that at the top-level it’s that simple?
- roundandround 4y agoThat is too simple for the feature they are using. The client itself has its own cache and the only way to fully prevent traffic from a client is to tell it content it caches will remain valid for some amount of time into the future. For URLs that return the latest entry there is no valid amount of time known in advance by GitHub unless they want to introduce mandatory publication delays. For URLs of specific change sets, they should never be corrected again and an infinite cache is pretty much valid unless a user overrides good git practices. I think GitHub frequently misidentifies which scenario they are in and when they return 1 day for a current state URL users notice, while when they return 5 minutes for a permanent change set that gets a lot if traffic they lost network capacity.
- stingraycharles 4y agoIsn’t this where stuff like ETAGs are supposed to help? Not completely solve, but at least help reduce the problem a bit more?
- ignoramous 4y agoYes, but cache invalidation is a hard problem, so most services side-step etags (+ if-none-match), only to hit caches elsewhere.
- flangola7 4y ago[flagged]
- joshspankit 4y ago
- airstrike 4y agoThere are actually only two hard problems in computer science: 0) Cache invalidation 1) Naming things 5) Asynchronous callbacks 2) Off-by-one errors 3) Scope creep 6) Bounds checking
- raverbashing 4y agoYou forgot the 'Segmentation Fault' at the last line
- wildcow 4y agoIt is on the list, but was never written to stdout because of a NULL pointer exception.
- marssaxman 4y agoHaven't seen that variant before! Thanks.
- ilyt 4y agoasynchronous callbacks are more of "wrong solution" than hard problem...
- Dylan16807 4y agoBounds checking as separate from off-by-one just means you stop using C arrays. That's not hard. And why point at callbacks specifically? And scope creep is not a computer science problem; it's easy to avoid if people decide to avoid it. So this list is too bloated for the joke to work well, I think. Even before we talk about how off-by-one gets ruined this way.
- latchkey 4y ago7) Yoda logic
- thih9 4y agoYou multithreading have forgotten.
- Retric 4y agoCache invalidation is a large class of multi threading problems, that’s just as much an issue between threads as between servers.
- nhumrich 4y agoAnd the oxford comma as well, apparently
- ravi-delia 4y agoThis is not actually a situation where the Oxford comma disambiguates, though I also had the gut feeling one should be there
- thfuran 4y agoThat's clearly a type of off by one error.
- drstewart 4y ago[flagged]
- bratbag 4y agoHas github fired most people capable of fixing problems?
- drstewart 4y agoGithub did layoffs, yes. We'll see how much longer the site can stay up. This is a death rattle.
- soulofmischief 4y agoGitHub is among many tech companies currently shedding fat from hiring during the pandemic boom. The morality of that is another discussion, but I'd hardly call this a death rattle.
- drstewart 4y agoLet's see you apply that same logic in this thread: https://news.ycombinator.com/item?id=34715890 https://news.ycombinator.com/item?id=34715890
- Kwpolska 4y agoTwitter fired many more engineers, and the reason for those firings was Elon's takeover and temper tantrums. Many more companies had firings this winter and things aren't on fire anywhere but Elon's Twitter.
- bdhcuidbebe 4y agoLol, what a hilariously unfounded coclusion.
- 4y ago
- InvaderFizz 4y agoReading the comment threads on GitHub, some files get a TTL of 300, some get a TTL of 86400. The "why" is certainly an interesting question.
- rgbrenner 4y agoIf the TTL starts at 86400 and then declines to 0 before resetting.. this is a fairly common caching strategy... it ensures the cache will expire for all clients at around the same time. For example, if you want the client's cache to expire at midnight everyday.
- adastra22 4y agoIs that really common? That sounds like a recipe for disaster.
- Kinrany 4y agoYou could use the same mechanism to spread out the load
- doctor_eval 4y agoYeah, sounds like a self-inflicted variant of the Thundering Herd Problem https://en.wikipedia.org/wiki/Thundering_herd_problem https://en.wikipedia.org/wiki/Thundering_herd_problem
- _cenw 4y agoThat's a terrible idea. Source: Implemented global TTL in our own caching DNS in front of kube-dns (which is horrible if you, among other things, have node containers with no DNS caching; i still have a pcap with 20000+ queries for A in s3.amazonaws.com in a 0.2s span) before coredns was a thing. The CPU spikes were huge, but remained hidden for a long time due to metrics resolution. But eventually it got bad enough that clients ended up not getting responses.
- rgbrenner 4y ago
- kzrdude 4y agoWhat level of service should we expect from github? Many of its services are free, can we ask for more when not paying?
- _cenw 4y agoraw.github has seen some extensive (ab)use for bandwidth heavy applications and/or pointing a few thousand clients at it concurrently (including entire IoT fleets), without ever being communicated as your globally consistent free CDN. I'm actually more amazed GitHub will give you a response in the 2xx range at all for these use-cases.
- sourcecodeplz 4y agoThis. I know I've used it as a static file hosting service before. The cache problem was real then too. Can't complain though.
- nubinetwork 4y agoWouldn't an aggressive cache mean it has the most recent content, not the other way around?
- low_tech_punk 4y agoYou mean aggressive cache eviction/invalidation? I think they might be taking about aggressive caching. Subtle difference in words.
- madewulf 4y agoGitHub caching can be a pain. It's been for example impossible to make files completely disappear from an open source repo without deleting the repo or contacting their support.
- Symbiote 4y agoThat's Git garbage collection, not the caching discussed in the post. https://github.blog/2022-09-13-scaling-gits-garbage-collection/ https://github.blog/2022-09-13-scaling-gits-garbage-collecti...
- Laaas 4y agoThis seems reasonable to me. Caches should last longer; if you want to be sure of you get the latest version, rename the file. a good trick is to include the hash of the file in the name to get content-addressing à la IPFS.
- mihaaly 4y agoI thought the content is more important than getting whatever quickly... Rename the file to get what you want, am I the only one finding that a very strange approach?
- croes 4y agoSo we are back to file_v1, file_v1.1, file_v1.2 etc?
- nix23 4y agoLike OpenVMS makes it on Filesystem level where the highest version is the "real" one: Blupblub;1 Blupblub;2 --> Blupblub;2 = Blupblub Actually a good idea ;)
- jack_pp 4y agoNah, file_v1_final_final_REALLY_final
- 9dev 4y agoNo, we are where we should have always been: > GET /file/latest HTTP/1.1 < HTTP/1.1 302 Found < Location /path/to/real/file/hash.tar.gz
- heavyset_go 4y agoHere I thought I was experiencing this because I was doing something wrong.
- mihaaly 4y agoThe old Intel vs. Motorola joke comes to mind from the era of floating point bug in Pentium. Intel and Motorola processors compete: How much is 2 + 2? - asks the Motorola 5! Wrong answer. But I was quick wasn't I?!
- amadeuspagel 4y agoI'm guessing that a lot of people embed these files in their website in one way or another, making aggressive caching necessary.
- kristofferR 4y agoI discovered the issue when I reported an adblock filter list that broke a site. The issue was fixed within minutes, but the broken filter list is still being served. Thankfully, it turned out to be a bug.
- nikeee 4y agoGitHub recommends using GH Pages for that. Raw links cause a higher server load and aren't even being served with the correct mime type.
- Symbiote 4y agoGithub say the problem will soon be fixed. https://github.com/orgs/community/discussions/46691#discussioncomment-4948308 https://github.com/orgs/community/discussions/46691#discussi...
- usr1106 4y agoA bug: See https://github.com/orgs/community/discussions/46691#discussioncomment-4948308 https://github.com/orgs/community/discussions/46691#discussi... (Well, if we believe the statement "github engineer here". Of course every clown could write that, too)
- Kwpolska 4y agoIf you look at their profile[0], you can see they are a member of the GitHub organization and that they are marked as GitHub staff. [0]: https://github.com/antn https://github.com/antn
- usr1106 4y agoHow do you see that? At leat in my mobile browser nothing jumps into my eye. This user has contributions to @github, but very few. Edit: Found it. Need to click on one of the achievements. Then the layout changes and it appears in the lower left corner under Organizations.
- TJSomething 4y agoIt doesn't appear to be visible on mobile, only desktop.
- pjmlp 4y agoWhat about having their own servers instead of misusing Github?
- iLoveOncall 4y agoIf only GitHub's underlying technology Git had a way to trigger actions when a file is updated and they could use that to invalidate cache! One day maybe?
- remram 4y agoInvalidate browser's cache?
- iLoveOncall 4y agoThis is not the problem as adding query parameters to bypass that doesn't work.
- Aeolun 4y agoThis sounds like a bug, not something they deliberately do.
- zelphirkalt 4y agoUsually I am using permalink or a version containing link for my use cases, so they shouldn't change anyway and I am checking with a checksum. I think my use cases are not affected then.
- smashah 4y agoI hit this problem for my project. On every launch of my lib, it would grab a patches file from the main github repo. I was seeing people having to wait 5 minutes+ and multiple restarts to get the latest pushed file. The solution was to run a custom BunnyCDN instance where I can easily invalidate the caches via a basic API request which happens on a github push of that file. I was surprised to see that sometimes that file was being requested millions of times a day due to attempts to loading it too early. With some optimizations I was able to greatly reduce that by 99%. Having seen all those requests, I understand why github aggressively caches these files. I agree that repo owners should be allowed to invalidate those caches upon an API request - or it should happen automatically on every change.
- homero 4y agoI don't think raw was supposed to be used as your CDN
- Brian_K_White 4y agoIt doesn't matter what it was supposed to be used for. It's bad data in any context, even ones where you can't think of a way to blame the victim for holding it wrong.
- OJFord 4y agoIt's not bad data though, it's just old and cached, which seems fine IMO even if the stale period is longer than you might expect. It's not like it's serving the wrong content for a specific commit ref, it's when you request the file as on a branch, and that branch has recently changed.
- joshxyz 4y agoIsnt the Cache-Control header the solution here? 304 if etag is cached otherwise return the new with 200
- wildcow 4y agoPeople should stop using GitHub as a CDN and just clone their ad whitelist.
- smrtinsert 4y agoWhy wouldn't it be cached? After a push/merge shouldn't it be static until the next one?
- remram 4y agoThere's is no way to forecast when the "next one" is going to be and tell the client to cache for that long.
- atdixon 4y agoCan you just add a random query param value to bust the cache?
- brundolf 4y agoFixed: https://github.com/orgs/community/discussions/46691#discussioncomment-4948987 https://github.com/orgs/community/discussions/46691#discussi...
- dcchambers 4y agoCould this not be resolved by using the file version from a specific git tag instead of using the main branch?