5 ms·

Why not just use Partial or Shallow Clones? [0] [0] https://github.blog/2020-12-21-get-up-to-speed-with-partial-clone-and-shallow-clone/ https://github.blog/20

Why not just use Partial or Shallow Clones? [0] [0] https://github.blog/2020-12-21-get-up-to-speed-with-partial-clone-and-shallow-clone/ https://github.blog/2020-12-21-get-up-to-speed-with-partial-...

verticalscaler 3y ago
git clone --depth=1 is fast because it doesn't grab the full commits history but OP is getting at a specific singular file directly. It is neat, not aware of a git incantation so fine grained. Hope it gets built into Git directly.
- merlincorey 3y ago
  Ah, thank you for the clarification -- as far as I know that is indeed unique. It's a thimbleful sized shallow clone, one could say.
- paulddraper 3y ago
  You can do a shallow clone and partial checkout.
  - verticalscaler 3y ago
    Cool! Why you gotta tease me like this without busting out a one liner? :) Gemini disagrees with you btw: Unfortunately, directly combining git shallow clone and git partial checkout to grab just one specific file isn't straightforward. Here's why: Shallow clone: This limits the downloaded commit history to a specific depth, but it still fetches all files involved in those commits. While reducing data, it wouldn't restrict files solely based on your needs. Partial checkout: This lets you specify which files to include in your working directory, but it requires a full clone initially. However, you have a few alternative approaches to achieve your goal: 1. Shallow clone + Sparse checkout: Use git clone --depth=<commit_depth> --single-branch=<branch> to shallow clone the specific branch and limit history. Create a .git/info/sparse-checkout file containing only the path to the desired file. Run git read-tree -u to update the index based on the sparse-checkout file. This method downloads a limited history and only keeps the specified file in your working directory. 2. Partial clone with server support (limited availability): Check if the server supports partial clones (currently implemented on Github and some Gitlab self-hosted instances). Use git clone --filter=blob:none <url> for a "blobless" clone that only contains file content, no history or directory structure. Add the desired file path to the .git/info/sparse-checkout file and run git read-tree -u as before. This approach minimizes downloaded data but requires server support and won't work everywhere.
Karellen 3y ago
Because then some process on the server has to fetch/checkout the clone each time there's a push? Whereas reading directly from the repo gets around that.
- p4bl0 3y ago
  But then some process on the server has to fetch multiple files from the repository each time there's a request. It seems this solution consists in getting around doing a bit of work when it is actually necessary (on website update) by doing a lot of work over and over.
  - Karellen 3y ago
    FTA: > This sounds like a lot of steps to serve a single file, but there's two key optimizations which can be made. The first is to cache the root tree's hash in memory, which skips two lookups right at the beginning. The root tree's hash will only change when the latest commit of the branch changes, so it's enough to cache it in memory and have a separate background process periodically re-check the latest commit. > The second optimization is to cache tree objects in-memory using their hash as a key. The object identified by a hash never changes, so this cache is easy to manage, and by caching the tree objects in memory (perhaps with an LRU cache if memory usage is a concern) all round-trips to the remote server can be eliminated, save for the final round-trip for the file itself. Also, the "background process periodically re-check the latest commit" seems like a bit of overkill if the repo is local; just caching and checking the mtime of `refs/heads/main` should be enough to decide whether the root tree needs re-reading.

verticalscaler 3y ago

git clone --depth=1 is fast because it doesn't grab the full commits history but OP is getting at a specific singular file directly. It is neat, not aware of a git incantation so fine grained. Hope it gets built into Git directly.

merlincorey 3y ago
Ah, thank you for the clarification -- as far as I know that is indeed unique. It's a thimbleful sized shallow clone, one could say.
paulddraper 3y ago
You can do a shallow clone and partial checkout.
- verticalscaler 3y ago
  Cool! Why you gotta tease me like this without busting out a one liner? :) Gemini disagrees with you btw: Unfortunately, directly combining git shallow clone and git partial checkout to grab just one specific file isn't straightforward. Here's why: Shallow clone: This limits the downloaded commit history to a specific depth, but it still fetches all files involved in those commits. While reducing data, it wouldn't restrict files solely based on your needs. Partial checkout: This lets you specify which files to include in your working directory, but it requires a full clone initially. However, you have a few alternative approaches to achieve your goal: 1. Shallow clone + Sparse checkout: Use git clone --depth=<commit_depth> --single-branch=<branch> to shallow clone the specific branch and limit history. Create a .git/info/sparse-checkout file containing only the path to the desired file. Run git read-tree -u to update the index based on the sparse-checkout file. This method downloads a limited history and only keeps the specified file in your working directory. 2. Partial clone with server support (limited availability): Check if the server supports partial clones (currently implemented on Github and some Gitlab self-hosted instances). Use git clone --filter=blob:none <url> for a "blobless" clone that only contains file content, no history or directory structure. Add the desired file path to the .git/info/sparse-checkout file and run git read-tree -u as before. This approach minimizes downloaded data but requires server support and won't work everywhere.

Karellen 3y ago

Because then some process on the server has to fetch/checkout the clone each time there's a push? Whereas reading directly from the repo gets around that.

p4bl0 3y ago
But then some process on the server has to fetch multiple files from the repository each time there's a request. It seems this solution consists in getting around doing a bit of work when it is actually necessary (on website update) by doing a lot of work over and over.
- Karellen 3y ago
  FTA: > This sounds like a lot of steps to serve a single file, but there's two key optimizations which can be made. The first is to cache the root tree's hash in memory, which skips two lookups right at the beginning. The root tree's hash will only change when the latest commit of the branch changes, so it's enough to cache it in memory and have a separate background process periodically re-check the latest commit. > The second optimization is to cache tree objects in-memory using their hash as a key. The object identified by a hash never changes, so this cache is easy to manage, and by caching the tree objects in memory (perhaps with an LRU cache if memory usage is a concern) all round-trips to the remote server can be eliminated, save for the final round-trip for the file itself. Also, the "background process periodically re-check the latest commit" seems like a bit of overkill if the repo is local; just caching and checking the mtime of `refs/heads/main` should be enough to decide whether the root tree needs re-reading.