4 ms·

> Large object promisors are special Git remotes that only house large files. I like this approach. If I could configure my repos to use something like S3, I w

> Large object promisors are special Git remotes that only house large files. I like this approach. If I could configure my repos to use something like S3, I would switch away from using LFS. S3 seems like a really good synergy for large blobs in a VCS. The intelligent tiering feature can move data into colder tiers of storage as history naturally accumulates and old things are forgotten. I wouldn't mind a historical checkout taking half a day (i.e., restored from a robotic tape library) if I am pulling in stuff from a decade ago.

account42 1y ago
Why does the git client need specific support for this though? What's stopping a git host from redirecting requests certain objects to a different host and refusing to pack them into bundles today?
deleted 1y ago
[deleted]
a_t48 1y ago
At my current job I've started caching all of our LFS objects in a bucket, for cost reasons. Every time a PR is run, I get the list of objects via `git lfs ls-files`, sync them from gcp, run `git lfs checkout` to actually populate the repo from the object store, and then `git lfs pull` to pick up anything not cached. If there were uncached objects, I push them back up via `gcloud storage rsync`. Simple, doesn't require any configuration for developers (who only ever have to pull new objects), keeps the Github UI unconfused about the state of the repo. I'd initially at spinning up an LFS backends, but this solves the main pain point, for now. Github was charging us an arm and a leg for pulling LFS files for CI, because each checkout is fresh, the caching model is non-ideal (max 10GB cache, impossible to share between branches), so we end up pulling a bunch of data that is unfortunately in LFS, every commit, possibly multiple times. Because of this they happily charge us for all that bandwidth, because they don't provide tools to make it easy to reduce bandwidth (let me pay for more cache size, or warm workers with an entire cache disc, or better cache control, or...). ...and if I want to enable this for developers it's relatively easy, just add a new git hook to do the same set of operations locally.
- gmm1990 1y ago
  Why not run some open source ci locally or the google equivalent ec2, if you’re already going to the trouble of this much customization with running GitHub ci?
  - a_t48 1y ago
    It was half a day of work to make a drop in action.yml that does this. Saved a bunch of money (both in bandwidth and builder minutes), well worth the investment. It really wasn’t a lot of customization. All our builds are on GHA definitions, there’s no way it’s worth it to swap us over to another build system, administer it, etc. Our team is small (two at the time, but hopefully doubling soon!), and there’s barely a dozen people in the whole engineering org. The next hit list item is to move from GH hosted builders to GCE workers to get a warmer docker cache (a bunch of our build time is spent pulling images that haven’t changed) - it will also save a chunk of change (GCE workers are 4x cheaper per minute and the caching will make for faster builds), but the opportunity cost for me tackling that is quite high.
    - fmbb 1y ago
      Doesn’t the official docker build push action support caching with the GitHub Actions cache?
      - a_t48 1y ago
        Yes but one image push for us is >10GB, due to ML dependencies. And even if it is intelligent and supports per layer caching, you can’t share between release branches - https://github.com/docker/build-push-action/issues/862 https://github.com/docker/build-push-action/issues/862. And even if that did work, I’ve found it much more reliable to use the actual docker BuildX disk state than to try and get caching for complex multi stage builds working reliably. I have a case right now where there’s no combination of —cache-to/from flags that will give me a 100% cached rebuild starting from a fresh builder, using only remote cache. I should probably report it to the Docker team, but I don’t have a minimal repro right now and there’s a 10% chance it’s actually my fault.
nullwarp 1y ago
Same and never understood why it wasn't the default from the get go but maybe it wasn't so synonymous when it first came out. I run a small git LFS server because of this and will be happy to switch away the second I can get git to natively support S3.
johnisgood 1y ago
Is S3 related to Amazon?
- flohofwoe 1y ago
  Yeah it's AWS's 'cloud storage service'.
  - dotancohen 1y ago
    It's actually 'Simple Storage Service', hence the acronym S3.
- bayindirh 1y ago
  You can install your own S3 compatible storage system on premises. It can be anything from a simple daemon (Scality, JuiceFS) to a small appliance (TrueNAS) to a full-blown storage cluster (Ceph). OpenStack has it own object storage service (Swift). If you fancy it for your datacenter, big players (Fujitsu, Lenovo, Huawei, HPE) will happily sell you "object storage" systems which also support S3 at very high speeds.
  - yugoslavia4ever 1y ago
    And for CI and local development testing you can use localstack which runs in a docker container and has implementations of most AWS services
    - bayindirh 1y ago
      Oh, that sounds interesting. We don't use AWS, but it's a nice alternative for people using AWS for their underpinnings. Scality's open source S3 Server also can run in a container.
- StopDisinfo910 1y ago
  Yes, S3 is the name of Amazon Object Storage Service. Various players in the industry have started offering solutions with a compatible API which some people abusively call S3 too.
riedel 1y ago
The article mentions alternatives to git lfs like git annex that support S3 already (which IMHO is however still a bit of a pain in the ass on windows due to the symlink workflow). Also dvc plays nicely with git and S3. Gitlab btw also simply offloads git lfs to S3. All have their quirks. I typically opt for LFS as a no-brainer but use the others when it fits the workflow and the infrastructure requirements. Edit: Particularly the hash algorithm and the change detection (also when this happens) makes a difference if you have 2 GB files and not only the 25MB file from the OP
_bent 1y ago
I'm currently running https://github.com/datopian/giftless https://github.com/datopian/giftless to store the LFS files belonging to repos I have on GitHub on my homelab miniio instance. There are a couple other projects that bridge S3 and LFS, though I had the most success with this setup.

account42 1y ago

Why does the git client need specific support for this though? What's stopping a git host from redirecting requests certain objects to a different host and refusing to pack them into bundles today?

deleted 1y ago

[deleted]

a_t48 1y ago

At my current job I've started caching all of our LFS objects in a bucket, for cost reasons. Every time a PR is run, I get the list of objects via `git lfs ls-files`, sync them from gcp, run `git lfs checkout` to actually populate the repo from the object store, and then `git lfs pull` to pick up anything not cached. If there were uncached objects, I push them back up via `gcloud storage rsync`. Simple, doesn't require any configuration for developers (who only ever have to pull new objects), keeps the Github UI unconfused about the state of the repo. I'd initially at spinning up an LFS backends, but this solves the main pain point, for now. Github was charging us an arm and a leg for pulling LFS files for CI, because each checkout is fresh, the caching model is non-ideal (max 10GB cache, impossible to share between branches), so we end up pulling a bunch of data that is unfortunately in LFS, every commit, possibly multiple times. Because of this they happily charge us for all that bandwidth, because they don't provide tools to make it easy to reduce bandwidth (let me pay for more cache size, or warm workers with an entire cache disc, or better cache control, or...). ...and if I want to enable this for developers it's relatively easy, just add a new git hook to do the same set of operations locally.

gmm1990 1y ago
Why not run some open source ci locally or the google equivalent ec2, if you’re already going to the trouble of this much customization with running GitHub ci?
- a_t48 1y ago
  It was half a day of work to make a drop in action.yml that does this. Saved a bunch of money (both in bandwidth and builder minutes), well worth the investment. It really wasn’t a lot of customization. All our builds are on GHA definitions, there’s no way it’s worth it to swap us over to another build system, administer it, etc. Our team is small (two at the time, but hopefully doubling soon!), and there’s barely a dozen people in the whole engineering org. The next hit list item is to move from GH hosted builders to GCE workers to get a warmer docker cache (a bunch of our build time is spent pulling images that haven’t changed) - it will also save a chunk of change (GCE workers are 4x cheaper per minute and the caching will make for faster builds), but the opportunity cost for me tackling that is quite high.
  - fmbb 1y ago
    Doesn’t the official docker build push action support caching with the GitHub Actions cache?
    - a_t48 1y ago
      Yes but one image push for us is >10GB, due to ML dependencies. And even if it is intelligent and supports per layer caching, you can’t share between release branches - https://github.com/docker/build-push-action/issues/862 https://github.com/docker/build-push-action/issues/862. And even if that did work, I’ve found it much more reliable to use the actual docker BuildX disk state than to try and get caching for complex multi stage builds working reliably. I have a case right now where there’s no combination of —cache-to/from flags that will give me a 100% cached rebuild starting from a fresh builder, using only remote cache. I should probably report it to the Docker team, but I don’t have a minimal repro right now and there’s a 10% chance it’s actually my fault.

nullwarp 1y ago

Same and never understood why it wasn't the default from the get go but maybe it wasn't so synonymous when it first came out. I run a small git LFS server because of this and will be happy to switch away the second I can get git to natively support S3.

johnisgood 1y ago

Is S3 related to Amazon?

flohofwoe 1y ago
Yeah it's AWS's 'cloud storage service'.
- dotancohen 1y ago
  It's actually 'Simple Storage Service', hence the acronym S3.
bayindirh 1y ago
You can install your own S3 compatible storage system on premises. It can be anything from a simple daemon (Scality, JuiceFS) to a small appliance (TrueNAS) to a full-blown storage cluster (Ceph). OpenStack has it own object storage service (Swift). If you fancy it for your datacenter, big players (Fujitsu, Lenovo, Huawei, HPE) will happily sell you "object storage" systems which also support S3 at very high speeds.
- yugoslavia4ever 1y ago
  And for CI and local development testing you can use localstack which runs in a docker container and has implementations of most AWS services
  - bayindirh 1y ago
    Oh, that sounds interesting. We don't use AWS, but it's a nice alternative for people using AWS for their underpinnings. Scality's open source S3 Server also can run in a container.
StopDisinfo910 1y ago
Yes, S3 is the name of Amazon Object Storage Service. Various players in the industry have started offering solutions with a compatible API which some people abusively call S3 too.

riedel 1y ago

The article mentions alternatives to git lfs like git annex that support S3 already (which IMHO is however still a bit of a pain in the ass on windows due to the symlink workflow). Also dvc plays nicely with git and S3. Gitlab btw also simply offloads git lfs to S3. All have their quirks. I typically opt for LFS as a no-brainer but use the others when it fits the workflow and the infrastructure requirements. Edit: Particularly the hash algorithm and the change detection (also when this happens) makes a difference if you have 2 GB files and not only the 25MB file from the OP

_bent 1y ago

I'm currently running https://github.com/datopian/giftless https://github.com/datopian/giftless to store the LFS files belonging to repos I have on GitHub on my homelab miniio instance. There are a couple other projects that bridge S3 and LFS, though I had the most success with this setup.