3 ms·

> But just based on what was explained in this blog post, it sounds like they're replicating database updates asynchronously, without waiting for the updates to

by nstart 8y ago

> But just based on what was explained in this blog post, it sounds like they're replicating database updates asynchronously, without waiting for the updates to be acknowledged by slaves before the master allows them to commit. Which means the data on slaves is always slightly out-of-date... Anecdotally I think I see this regularly. Part of the build pipeline I work on requires recursively downloading a particular folder for a given repo. From time to time, I'll have the latest git ref give me data from the previous git ref. Everywhere else on GitHub the information is up-to-date. But not the API endpoint to download files. When this happens it takes roughly 5-7 minutes for the correct information to show up. After reading this and all the other resources people have posted I'm definitely going to see if I can reliably reproduce the error.

richardwhiuk 8y ago
Why would you use an API endpoint to download files in a build pipeline? Surely you should just use a clone of the repository? I could believe that an endpoint to download files looks at a cache, and if you are just using 'master' or 'latest' as opposed to a commit, I could believe that would be out of date.
- nstart 8y ago
  Bit late, but replying to this because my comment does have missing context. Most of the repos we have are large enough that downloading three individual files is faster than cloning the whole repo. Even with a shallow clone with a depth of 1. I've tried the combinations and even tried downloading a zip file of the repo. Still faster to download the three files. And I haven't even added parallel downloads because it's fine as is. But given the issues, why continue to use the API? Because those files are related to the build pipeline logic and they change maybe once in 3-4 months. I can live with an error once in 3-4 months. Recently I was working on changing some of the logic of the pipeline and that's when I ran into it enough to say that there was something odd from Github's end.

by nstart 8y ago

richardwhiuk 8y ago

Why would you use an API endpoint to download files in a build pipeline? Surely you should just use a clone of the repository? I could believe that an endpoint to download files looks at a cache, and if you are just using 'master' or 'latest' as opposed to a commit, I could believe that would be out of date.

nstart 8y ago
Bit late, but replying to this because my comment does have missing context. Most of the repos we have are large enough that downloading three individual files is faster than cloning the whole repo. Even with a shallow clone with a depth of 1. I've tried the combinations and even tried downloading a zip file of the repo. Still faster to download the three files. And I haven't even added parallel downloads because it's fine as is. But given the issues, why continue to use the API? Because those files are related to the build pipeline logic and they change maybe once in 3-4 months. I can live with an error once in 3-4 months. Recently I was working on changing some of the logic of the pipeline and that's when I ran into it enough to say that there was something odd from Github's end.