16 ms·

Starting October 19, storage limit will be enforced on all Gitlab Free accounts

167 points by reimertz 4y ago

bm5k 4y ago
Looks like everyone is rushing to check their storage. > something went wrong while loading usage details On my free group’s storage page.
- john_cogs 4y ago
  Yes, the container registry and build artifacts are now correctly displayed in the free plans.
giancarlostoro 4y ago
I'm assuming this is only for the ones they host and not the self-hosted solution. That is insane that anyone uploads terrabytes of data into gitlab, is there an actual valid non-illegal / weird backup choice use case? Is there some big ass GIS open source project out there that could use the attention of GitLab before they nuke some vital data somehow?
- ev1 4y ago
  I'm actually somewhat curious how much of an impact this will actually have - I think I've only seen a 5 GB+ repo once or twice and they were not really source code, but mildly "abusing" github CDN/releases for downloads. At least gitlab is not deleting any data, just rejecting pushes if you're over the limit.
- bedosh 4y ago
  The quota includes not just the git repo, but also other services that gitlab offers. In particular the container and package registries are easy to setup in a way that accumulates a lot of data (e.g. by building and tagging a docker image in every commit and not cleaning up old images). In most cases this will be a misconfiguration and all the quota enforcement is doing is to force people to configure their projects a bit more considerate. As far as I know, the quota itself isn't new, it just wasn't enforced in the past.
  - Beltalowda 4y ago
    GitHub limits the package registry to 500M for free accounts, so 5G doesn't seem so bad. Git repos themselves usually don't take up that much space; all my 135 projects on GitHub are ~250M combined (quick count, could be off a bit, but roughly on that order). Even larger repos at $dayjob with daily commits typically take up a few hundred M at the most. Basically you need to either be in the top 0.1% of highly active accounts or do something specific that requires a lot of disk space, but for most people 5G doesn't seem so unreasonable.
- john_cogs 4y ago
  You are correct in your assumption - the storage limit impacts SaaS only. I added some context on the rollout in a previous comment. https://news.ycombinator.com/item?id=32387597 https://news.ycombinator.com/item?id=32387597
noasaservice 4y ago
So, we either go to Github, where our licenses are abused for their shitty ML. Or we pay $20/month to Gitlab. And I can't figure out how the quotas will intersect with "professional", if at all. For us Open Source devs, neither is a good option. Although I have heard good things about sr.ht / sourcehut. And for the service, it appears to be fair https://sourcehut.org/pricing/ https://sourcehut.org/pricing/
- uo21tp5hoyg 4y ago
  Personally I use codeberg[1] which doesn't have all the fancy bells and whistles but for personal projects it runs like butter. [1] https://codeberg.org/ https://codeberg.org/
- pxeger1 4y ago
  Git works perfectly fine without a web interface like that. It's not popular nowadays, but you can just keep the primary repository on your own server and merge patches by email.
- marcinzm 4y ago
  >So, we either go to Github, where our licenses are abused for their shitty M As long as it's on the internet MS can scrape the data so not being on Github is only a temporary defense.
  - jacooper 4y ago
    But GitHub's TOS can only protect them from GitHub repos. If it has impact on CoPilot or not, is not clear.
    - marcinzm 4y ago
      Except not all contributors to projects signed the TOS since you can upload an existing project with existing non-Github contributors. One person uploading a project does not override or change the license of that project. So MS is already not relying on the TOS for legal protection which means they feel legally they can train on any OSS projects.
T3RMINATED 4y ago
lol768 4y ago
I guess this is the next step to reduce costs after the brakes were put on the "let's delete old OSS repositories" leaked plan. For comparison, I think GitHub just have a cap of 100MB on any single individual file, plus: > We recommend repositories remain small, ideally less than 1 GB, and less than 5 GB is strongly recommended. Smaller repositories are faster to clone and easier to work with and maintain. If your repository excessively impacts our infrastructure, you might receive an email from GitHub Support asking you to take corrective action. We try to be flexible, especially with large projects that have many collaborators, and will work with you to find a resolution whenever possible. Which is a bit wishy-washy, but sounds like there's room for discretion / exceptions to be made there rather than a hard cap at 5GB.
- Bayart 4y ago
  The difference being that the value Microsoft gets from GitHub isn't its revenue, it's its influence. Whereas Gitlab is just another corporate software suite.
  - ollien 4y ago
    I don't have the diffs in front of me, but I'm fairly sure this predates the MS acquisition by quite a while.
    - xtracto 4y ago
      Prior to MS acquisition, Github free offering was really bad: No private repos, no CI pipeline and I think there was not real "team".
      - tonmoy 4y ago
        Do you have a source for this? I vaguely remember GitHub starting to offer private repos when GitLab started getting popular.
        ollien 4y ago
imalerba 4y ago
Reading between lines it also says it going to enforce a 10GB limit on Paid tiers. > Namespaces on a GitLab SaaS paid tier (Premium and Ultimate) have a storage limit on their project repositories. A project’s repository has a storage quota of 10 GB. Even it's not mentioned as a change nor in the timeline, but that limit does not exist currently.
- imalerba 4y ago
  This change is one of the only MRs about these new limits discussed in the open. https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91418 https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91418
- JZerf 4y ago
  Just wanted to point out that it looks like people will be able to go over that limit but will have to pay extra for it according to https://docs.gitlab.com/ee/subscriptions/gitlab_com/index.html#purchase-more-storage-and-transfer https://docs.gitlab.com/ee/subscriptions/gitlab_com/index.ht... . Seems somewhat reasonable.
uo21tp5hoyg 4y ago
Am I reading it right that the original Free Tier had a quota of 45,000GB? That seems absurdly high and not very sustainable (hence the change I assume).
- bombcar 4y ago
  That number sounds like "unlimited, but the system requires a number" kind of thing.
- YorickPeterse 4y ago
  No, originally there was no quota enforced to begin with if my memory serves me right. The limits discussed here are likely meant to gradually tighten up the limits, rather than immediately locking out projects that exceed these limits.
  - uo21tp5hoyg 4y ago
    I guess an infinitely large storage quota is only slightly less sustainable than a 45,000GB one.
    - szundi 4y ago
      I won’t publish my gitlabfs plugin for 1.7k different programming languages then.
- NovemberWhiskey 4y ago
  I'm reading that as a "stop-the-bleeding" number; presumably there is someone out there with a 44TB repository and they want to impose initial quotas that don't actually impact anyone immediately. I guess someone has been backing up their movie collections to Gitlab or something.
  - bluedino 4y ago
    One case I've seen on our local Gitlab server is someone in data science/HPC (accidentally?) adding the output of a solver to their repo. Easily hundreds or thousands of gigabytes of data.
Macha 4y ago
Does the storage measurement work now? The container registry and build artifacts used to not be measured
- john_cogs 4y ago
  Yes, the container registry and build artifacts are now measured for Free users.
  - james-skemp 4y ago
    Mine is still showing negative usage for some repos. The most negative is showing -1.08 GB, both from my profile and from the project page. Repo 130.82 MB Artifacts 12.74 MB Wiki 51.20 KiB Everything else 0 bytes. Been like this for at least a couple weeks. This one is public, but I have others that are private with negative values.
    - james-skemp 4y ago
      Looks like the issue is tracked in https://gitlab.com/gitlab-org/gitlab/-/issues/368326 https://gitlab.com/gitlab-org/gitlab/-/issues/368326 Example of a public project with negative project storage: https://gitlab.com/strivinglife/book-raspberry-pi https://gitlab.com/strivinglife/book-raspberry-pi
  - rwmj 4y ago
    https://gitlab.com/nbdkit/nbdkit/-/usage_quotas https://gitlab.com/nbdkit/nbdkit/-/usage_quotas The headline number is 1.1G but the container registry is 16G. Edit: You should definitely add a simpler way to delete old pipelines. Having to mnnage it yourself through the API is a pain. (https://stackoverflow.com/questions/71513286/clean-up-history-in-gitlab-pipeline https://stackoverflow.com/questions/71513286/clean-up-histor...)
raverbashing 4y ago
How much did data hoarders use Gitlab for their backups?
- T3RMINATED 4y ago
whoomp12342 4y ago
5 gigs? what is this, repo hosting for ants?
- bluedino 4y ago
  That does seem ridiculously small. It's probably fine for 95% of projects though.
  - hinkley 4y ago
    That will depend on how well they handle forks. I could easily see myself exceeding such a limit just issuing bug fixes. Especially since one of the bugs I'm likely to fix is, "why is this library so damned big?"
    - tomjakubowski 4y ago
      Shrinking the library won't make the repo any smaller. You have to erase history to do that.
      - hinkley 4y ago
        Yeah I thought that might be confusing. Artifact size depends on what garbage from the project infrastructure gets included into the artifact. So simple things like not having a fully populated ignore file will cause things to get included into the project or the artifact. People who don't care about file size don't care about file size. If the artifact is ridiculous, sometimes the repo is ridiculous too. Therefore if you take a bunch of projects with outsized artifacts, you are going to have above average repository sizes as well.
    - john_cogs 4y ago
      GitLab team member here. Forks of projects get deduplicated, so only the changes you make will contribute to your storage consumption as long as the fork relationship is maintained.
      - 4y ago
bluedino 4y ago
What kind of projects are using 500GB much less TB's of data?
- kodah 4y ago
  Desktop Environment components, I suspect.
- ryanlitalien 4y ago
  In gaming, you can have a single branch that is 8TB with 1+ million files. But those projects aren't using git.
- Gigachad 4y ago
  Gitlabs own repo is at 33TB (mostly docker images). https://gitlab.com/gitlab-org/gitlab https://gitlab.com/gitlab-org/gitlab
alexb_ 4y ago
I didn't even know there was no storage limit - that seems like an immediate way to get your platform used to store non-code data in very large quantities.
- pantulis 4y ago
  I would guess controls to avoid people storing pirated movies on Gitlab repos and similar stuff have been in place for a long time.
game-of-throws 4y ago
If you build an image for testing on every commit and don't have a retention policy set up, you could be using a massive amount of space without realizing it. I can see why they did this.
- snoopy_telex 4y ago
  Are retention policies easy? I recall fighting on GitHub to setup a sane retention policy awhile back, but haven't revisited in years.
  - sofixa 4y ago
    I wouldn't say easy because regexes are needed for complex ones (e.g. retain -prod images for 6 months, -staging for 24 days) but they are powerful and not that complicated.
  - dnsmichi 4y ago
    GitLab offers ways to manage retention policies for your container registry. Please see the docs for guidance: https://docs.gitlab.com/ee/user/packages/container_registry/reduce_container_registry_storage.html#create-a-cleanup-policy https://docs.gitlab.com/ee/user/packages/container_registry/...
- khuey 4y ago
  Gitlab didn't even have retention policies until sometime in 2020. I have a six year old project that's consuming something like 4TB of space in their container registry.
  - akudha 4y ago
    Woah. 4 TB? What kind of project is it?
    - georgyo 4y ago
      Not op, but a simple node application can very easily make container images that are well over a gigabyte. Build and store that container images on every push and boom, you can explode in usage.
john_cogs 4y ago
GitLab team member here. The impacted users are notified via email and in-app notifications will begin 2022-08-22, so far we've contacted 30,000 users. Only GitLab SaaS users are impacted - the limits are not applicable to self-managed users.
- martypitt 4y ago
  We're affected by this change, and have been trying to get support in forums, but can't get hold of anyone. Build Artifacts are listed as part of what contributes to the quota (which is fair enough), but there's no way (that I could find in the docs) to manage build artifacts that are stored. I suspect we have a large historic storage, which we don't use / need, but there's no way to browse this, no way to verify it, and no way to delete what we don't need. I'm dreading getting a big ole bill in a few months for storage we had no way to opt out of.
  - xiaq 4y ago
    I have a repo that was almost exactly 5GB and it turned out that the storage was also used exclusively by pipeline logs and artifacts. Found this recipe on StackOverflow to just nuke everything and worked for me: https://stackoverflow.com/questions/71513286/clean-up-history-in-gitlab-pipeline https://stackoverflow.com/questions/71513286/clean-up-histor.... You can tweak it a bit if you want to keep the more recent pipeline runs. For some reason I had to run this multiple times to completely remove everything; each run removes about half of all the pipelines and only when there were about 20 remaining did the script remove everything.
  - lbotos 4y ago
    GitLab Support Engineering Manager here. These limits will first be soft limits as they are now. Impacted users will be notified and there will be about 2 months to take action before enforcement starts. After that point, limits will be enforced. If you’d like to investigate now you can use the GitLab API to fetch job artifacts and their storage size: https://docs.gitlab.com/ee/api/jobs.html#list-project-jobs https://docs.gitlab.com/ee/api/jobs.html#list-project-jobs and calculate the summary. I’d suggest collecting all pipeline jobs to be deleted using this API endpoint https://docs.gitlab.com/ee/api/job_artifacts.html#delete-job-artifacts https://docs.gitlab.com/ee/api/job_artifacts.html#delete-job... We are looking to improve the visibility for job artifacts in the UI as we get nearer to the storage limit being enforced.
Kelteseth 4y ago
My Qt/C++ cross-platform FOSS Wallpaper Engine project[1] currently uses 47gb of storage. This is because I compile for every platform and store the artifacts for 4 weeks. Not sure what I will do in the future, because having older builds around to try out without recompiling is always nice. [1] https://gitlab.com/kelteseth/ScreenPlay https://gitlab.com/kelteseth/ScreenPlay
- TheRealPomax 4y ago
  That's... an incredible footprint. Surely even if there was no size limit, deduplicating the build output would have made a lot of sense?
  - TheRealPomax 4y ago
    Having said that: if you run a public open source project that has requirements beyond what a regular free user gets, then why not apply to the open source program[1] (which you almost certainly already qualify for) so you're not space constrained? Quoting the requirements: --- Who qualifies for the GitLab for Open Source Program? In order to be accepted into the GitLab for Open Source Program, applicants must: - Use OSI-approved licenses for their projects: Every project in the applying namespace must be published under an OSI-approved open source license. [2] - Not seek profit: An organization can accept donations to sustain its work, but it can’t seek to make a profit by selling services, by charging for enhancements or add-ons, or by other means. - Be publicly visible: Both the applicant's GitLab.com group or self-managed instance and source code must be publicly visible and publicly available. --- [1] https://about.gitlab.com/handbook/marketing/community-relations/opensource-program/#-gitlab-for-open-source-program https://about.gitlab.com/handbook/marketing/community-relati... [2] https://opensource.org/licenses/alphabetical https://opensource.org/licenses/alphabetical
- shakow 4y ago
  I guess that having old artefacts is nice, but expecting 47GB of storage for free per repo seems ludicrous to me. Maybe it's a generational question, but I would be pretty happy with 1-2GB (i.e. git history, and one artefact per platform).
bachmeier 4y ago
This doesn't affect me, but a better way to handle this would be to sell extra storage at, say, double GitLab's cost. Digital Ocean sells 250 GB object storage at $5/month and $0.02/GB beyond that.
- john_cogs 4y ago
  GitLab team member here. GitLab offers additional storage for $60/year for an additional 10GB of storage and 20GB of transfer/month.
  - oy6amn8RDPw8w9n 4y ago
    Yes, $0.50/GB/month seems like very reasonable pricing /s That's over twice as much as the disks to store that much data cost. Every month.
    - glenneroo 4y ago
      Yeah that really hurts. A single Unity project can easily reach that in a short amount of time. When I get that warning email in a few weeks, I'll be shopping around.
  - bachmeier 4y ago
    I see now that it was mentioned at the end of the post. (Though I did specify "double GitLab's cost".)
unintendedcons 4y ago
Gitlab just can't shooting themselves in the foot. Driving away individuals is apparently their strategy now. Sad. I used to teach new developers starting with Gitlab pages.
eliseumds 4y ago
Fair enough.
deleted 4y ago
[deleted]
encryptluks2 4y ago
5GB isn't much different than the storage limits of other services, but their storage pricing is atrocious. I've seen the writing on the wall for a while and watched as GitLab went from being the cool open source alternative to GitHub to becoming a bloated oversized mess. I know several popular open source projects were offered premium tier upgrades for free. I am curious to see if these changes, especially transfer limits, will impact them enough to move away.
nabakin 4y ago
With this change, the 5 user limit[1], and original intent to delete dormant repositories[2][3], it seems as though GitLab is no longer able to support the free side of its business. GitLab has been touted as more OSS-friendly than GitHub, but a large part of the OSS ecosystem depends on free repositories. With these changes and this trajectory, I can't see myself putting another OSS project on GitLab. It's a shame it's come to this, but I'm confident GitLab didn't make this choice lightly. It must be done in order for them to stay afloat. Thank you GitLab team for your efforts. I hope you guys are successful in your future endeavors. [1] https://about.gitlab.com/blog/2022/03/24/efficient-free-tier/ https://about.gitlab.com/blog/2022/03/24/efficient-free-tier... [2] https://www.theregister.com/2022/08/04/gitlab_data_retention_policy/ https://www.theregister.com/2022/08/04/gitlab_data_retention... [3] https://www.theregister.com/2022/08/05/gitlab_reverses_deletion_policy/ https://www.theregister.com/2022/08/05/gitlab_reverses_delet...
- DoesntMatter22 4y ago
  I think a lot of the problem is how they communicated it. It isn't like "Well hey we have a problem here", it's just "Hey here's this super complicated to this problem you didn't know existed and you can take it or leave it"
jacooper 4y ago
Honestly for me the bigger issue is the upcoming bandwidth limit. There is no clear date to when its going to be enforced. But its definition is so wide its crazy, its basically any egress data except the web interface and shared runners. AFAIK it will also include git clones!, so if your project suddenly gets popular the users clones will cost you many too. Also if you use your own runner, cloning the repo to the runner will also be included in your bandwidth limit. And since it should apply to GitLab pages, it becomes useless for anything the you want to get few visits. Since GitLab is behind cloudflare, you might as well just use Cloudflare Pages at this point.
- karamanolev 4y ago
  > Also if you use your own runner, cloning the repo to the runner will also be included in your bandwidth limit. You make it sounds like it's horrible. I'd offer a different take in that sure, it can be a huge negative if done recklessly on GitLab's part. But if done correctly, it can actually be a good thing. I think CI pipelines doing a full clone from scratch on every build (or npm installs or other bootstraps) is extremely wasteful in terms of resources, so I'd be glad if that reduces it significantly.
  - jacooper 4y ago
    GitLab CI default clone configuration already AFAIK only includes the last 20 commits, so its already slimmed down.
    - karamanolev 4y ago
      If you start from scratch, regardless of history, you're fetching at least the size of the repo. In many cases, that's hundreds of MBs for every build and I'm a huge fan of building for every commit. I find that a colossal waste. I have a very old fashioned and "not recommended" setup with Jenkins that only pulls new commits, because it works out of a persistent working directory. Works wonders for a Django/ES6 project and sips bandwidth. I wish more of the modern containerized, start-from-scratch and so on setups would work in a similar way wrt the bandwidth they use.
elevation 4y ago
Git excels at tracking human keyboard output. A productive developer might write 100KB of code annually so a git repo can represent many developer years of collaborative effort in just a few MB. That is, unless you require git to track large media files, third party BLOBs, or build output. However, sometimes tracking these things are necessary, and since there isn't an obvious companion technology to git for caching large media assets ("blob hub?") or tracking historical build output ("release hub?"), devs abuse git itself. I wish there were a widely accepted stack that would make it easy to keep the source in the source repo, and track the multi-gb blobs by reference.
- foepys 4y ago
  You mean something like git Large File Storage? It comes with git for Windows by default and every Linux distro I know has it in its repos. MacOS also has it in Homebrew. https://git-lfs.github.com/ https://git-lfs.github.com/ It's basically one click/command away.
- WorldMaker 4y ago
  Git LFS is a pretty widely accepted stack for managing binary blobs by reference in git: https://git-lfs.github.com/ https://git-lfs.github.com/ The plugin is installed out of the box in many git distributions now. Many hosts support it today, including Gitlab, which is relevant to this article's discussion: https://docs.gitlab.com/ee/topics/git/lfs/ https://docs.gitlab.com/ee/topics/git/lfs/
  - Dayshine 4y ago
    I have yet to find a host that has at all usable Git LFS pricing though. GitLab is $60/m (edit, per year? unclear in pricing page) for 10GB storage and 20GB/m transfer. So, I can checkout my repo twice a month!? GitHub is similar, with a 1:1 ratio for storage and transfer. So you can checkout your repo once a month.
- adobrawy 4y ago
  > companion technology to git for caching large media assets ("blob hub?") or There is Amazon S3 or any other cloud storage. > tracking historical build output ("release hub?") There are OCI artifact registries. For me, there is rather issue that when all you have is a hammer, everything looks like a nail.
oefrha 4y ago
A lot of discussions about how git repos are supposed to be small are totally missing the point. This storage quota applies to everything, including release artifacts, containers, etc. Forget containers or CI artifacts on every commit, let's look at a very common scenario: using goreleaser to build binaries and deb/rpm/etc. packages for multiple architectures every release. This way a moderately sized Go project can easily consume 50-100MB or more per release. That gives you at most 50-100 releases across all your projects. Using hosted GitLab for open source projects is looking less and less appealing. I also posted about issue trackers on gitlab.com not allowing search without signing in a while back: https://news.ycombinator.com/item?id=32252501 https://news.ycombinator.com/item?id=32252501 Edit: An open source program that upgrades the quota is mentioned elsewhere in the thread: https://about.gitlab.com/solutions/open-source/ https://about.gitlab.com/solutions/open-source/ I don’t use hosted GitLab for my open source work, so no idea how many people get approved.
- john_cogs 4y ago
  Hi, GitLab team member here. The GitLab for Open Source program provides open source projects with Ultimate benefits and higher limits. More info in previous comment: https://news.ycombinator.com/item?id=32387621 https://news.ycombinator.com/item?id=32387621
  - KMnO4 4y ago
    Why do you have to “apply” for these benefits? Shouldn’t providing the source publicly in the open (ie not a private repository) by definition be enough for “open source”?
    - saganus 4y ago
      What happens if I publish the code under a restrictive (non-open source) license? Even though the code is available, that does not mean you can use it however you want, no?
      - SCLeo 4y ago
xaxaxb 4y ago
Haha, great.
jordemort 4y ago
What's the point of tapering it down in stages like this? Between the October 19th quotas and the October 20th quotas, if you wait until the last minute, you have 24 hours to move 37.5TB of data. Then 4 more days to move another 7TB; does that actually help anyone? The proposition of getting that much data out of it at that speed seems a bit unrealistic. Why not just say "the quota will be 5GB on November 9th" and be done with it?
- M3L0NM4N 4y ago
  Definitely sounds like there was some bureaucracy in there.
- john_cogs 4y ago
  The phased enforcement of the limits are a part of the technical rollout plan for this change that was added to our docs. Related comment: https://news.ycombinator.com/item?id=32387597 https://news.ycombinator.com/item?id=32387597. The communication sent to impacted users via email and future in-app notifications includes only the applicable enforcement dates and limits. Issue to follow: https://gitlab.com/gitlab-org/gitlab/-/issues/368150 https://gitlab.com/gitlab-org/gitlab/-/issues/368150
cercatrova 4y ago
I'm just going to move to GitHub at this point.
- jacooper 4y ago
  Yup its looking to be the only platform where projects can realisticly grow. But after CoPilot I still dont want to use it, on the contrary, I'm switching to GitLab because all the free stuff on GitHub are just unrealistic, and are just to play the long game. Also I dont want to invest my time and effort into a proprietary platform, like GitHub Actions, and Gitlab CI and most other core features of GitLab are Open Source. https://sfconservancy.org/blog/2022/jun/30/give-up-github-launch/ https://sfconservancy.org/blog/2022/jun/30/give-up-github-la... If GitLab didn't add the 5 person limit per group most people will be fine, but now with the upcoming bandwidth limit, the 5 person limit, and many others. Its almost impossible to grow on the platform Excexpt with the OSS program, which requires Lawful agreement with each org member.
  - cercatrova 4y ago
    I love GitHub Copilot. In fact, I am more inclined to use GitHub simply to help improve their tool.
djohnston 4y ago
Why are people checking build artefacts into the repo?
- cesarb 4y ago
  Playing devil's advocate: sometimes your build is non-reproducible, annoying to build, or both (for instance, needing a proprietary tool which can only run on a particular developer's laptop because the license is tied to that particular hardware, and which crashes half of the time for no particular reason). Keeping the build artifacts in the repository means you can reproducibly obtain that exact artifact, even years into the future.
- ranguna 4y ago
  This limitation is not just about code it will also affect all artifacts like container and package registry, pipeline artifacts, lfs, the lot.
cookiengineer 4y ago
Anybody found out which projects on gitlab exceed the 45 TB limit? I'm curious what kind of project would even need such a repository size. From a distant view this sounds like heavily mismanaged build artifacts in the project's git history; or abused storage for free CDN of video data or similar.
- SCLeo 4y ago
  Maybe a super active repository with a large build matrix? For instance, a repository with 10,000 commits, 25 artifacts per build, and 200 MB/artifact will take 50 TB. It is still ridiculous though.
  - cookiengineer 4y ago
    Are releases/tags also counting in on that? I was re-reading the announcement but as far as I can tell it only states repository size everywhere. The linked cleanup instruction articles also mostly show repo size (git gc et al) and nothing related to the releases section of a repository. [1] I'm also still unclear about what exactly is part of the package registry storage and whether or not releases in the repository are part of that. [1] https://docs.gitlab.com/ee/user/usage_quotas.html#manage-your-storage-usage https://docs.gitlab.com/ee/user/usage_quotas.html#manage-you...
lakomen 4y ago
- bdcravens 4y ago
  I'm confused at the connection between Microsoft/Github and the decision that Gitlab made here.
  - likortera 4y ago
    I think GP just rushed the free hate
- likortera 4y ago
  You are confusing Gitlab with GitHub?
- MaxDPS 4y ago
  But this is related to GitLab…
kerblang 4y ago
Perhaps now is a good time to recommend the ever-popular BFG to anyone unaware: https://rtyley.github.io/bfg-repo-cleaner/ https://rtyley.github.io/bfg-repo-cleaner/ Also my team's biggest repo is a 2.5 GB checkout but gitlab (self-managed) reports it as 185MB "files" and 353 MB "storage" (no CI/CD artifacts).
- sofixa 4y ago
  Similarly there's also git-filter-repo: https://github.com/newren/git-filter-repo https://github.com/newren/git-filter-repo It's in Python so runs pretty much everywhere *nix out of the box.
- dnsmichi 4y ago
  > erhaps now is a good time to recommend the ever-popular BFG to anyone unaware: https://rtyley.github.io/bfg-repo-cleaner/ https://rtyley.github.io/bfg-repo-cleaner/ Thanks for sharing this tool to help cleanup the Git history. Please be aware that it will rewrite the history, which could be very impactful to existing branches, merge requests and local clones. A similar approach is described in the documentation: https://docs.gitlab.com/ee/user/project/repository/reducing_the_repo_size_using_git.html https://docs.gitlab.com/ee/user/project/repository/reducing_...
rohfle 4y ago
A couple of notes - If I tag a docker image with multiple tags, and then push it to Gitlab, each tag counts towards the storage limits even though SHAs are identical. eg 100MB container tagged with "latest" and "v0.5" uses 200MB of storage. - The storage limit is not per repository, but per namespace. So 5GB free combined for all repositories under your user. If you create a group, then you get 5GB free combined for that group. Does this include forks? Does this include compression server side? - The 10GB egress limit per month includes egress to self-hosted Gitlab Runners in free tier. Consider this with the 400 minutes per month limit on shared runners. These limits feel less like curbing abuse and more like squeezing to see who will jump to premium while reducing operating costs. Is this a consequence to Gitlab hosting on GCP with associated egress and storage costs? Is this a move to improve financials / justify a market cap with fiscal storm clouds on the horizon? Is this being incentivized by $67m in awarded stock between the CFO and 2 directors? Stock history over last year for GTLB (since IPO in 2021?): https://yhoo.it/3QaExCs https://yhoo.it/3QaExCs From the golden era of 2015: https://about.gitlab.com/blog/2015/04/08/gitlab-dot-com-storage-limit-raised-to-10gb-per-repo/ https://about.gitlab.com/blog/2015/04/08/gitlab-dot-com-stor... > To celebrate today's good news we've permanently raised our storage limit per repository on GitLab.com from 5GB to 10GB. As before, public and private repositories on GitLab.com are unlimited, don't have a transfer limit and they include unlimited collaborators.
- jreporter 4y ago
  Hi @rohfle! A couple of clarifications: > If I tag a docker image with multiple tags, and then push it to Gitlab, each tag counts towards the storage limits even though SHAs are identical. eg 100MB container tagged with "latest" and "v0.5" uses 200MB of storage. Any "duplicated" data under a given "node" (be that the root namespace, a group or a project) counts towards the storage usage only once. So images latest and v0.5 would only represent 100MB in their namespace registry usage, not 200MB. > Does this include forks? The registry data is not copied/duplicated when one forks a project. So this is not applicable. But even if it was, as long as the fork and the source are under the same root namespace, any "duplicated" registry data across the two would only count towards the storage usage once. > Does this include compression server side? Yes, the measured size is the size of the compressed artifacts on the storage backend. We should be updating the docs shortly to make these answers more transparent!
balentio 4y ago
Gitlab keeps giving me more reason to go to Gittea.
- nix23 4y ago
  I stay on: https://sourcehut.org/ https://sourcehut.org/ No blingbling no social-media drama, just clean straightforward code hosting.
eddof13 4y ago
Bitbucket anyone? Or Atlassian is no longer an option?
crgwbr 4y ago
Seems like a buried lede here is that limits also now apply to paid accounts. Just checked my team’s name space: we have 700GB of storage used, and gitlab is going to start charging us $0.50/month/GB for everything in excess of 50 GB. On top of the hundreds of $/month we’re already paying in per-seat pricing. That seems absurdly expensive.
- dnsmichi 4y ago
  Thanks for your feedback. I’d suggest starting the analysis to identify the type of most storage being consumed. Maybe CI/CD job artifacts are kept forever and need an expiration configuration [0], or container images in the registry would need cleanup policies [1]. The documentation linked in the FAQ provides more analysis and guides to help. [2] If you need additional help with analyzing the storage usage, please contact the GitLab support team. You can also post cleanup questions on our community forum. [3] [0] https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html#when-job-artifacts-are-deleted https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html#w... [1] https://docs.gitlab.com/ee/user/packages/container_registry/reduce_container_registry_storage.html https://docs.gitlab.com/ee/user/packages/container_registry/... [2] https://about.gitlab.com/pricing/faq-paid-storage-transfer/#managing-your-storage-usage https://about.gitlab.com/pricing/faq-paid-storage-transfer/#... [3] https://forum.gitlab.com/c/gitlab-ci-cd/23 https://forum.gitlab.com/c/gitlab-ci-cd/23

167 points by reimertz 4y ago

bm5k 4y ago

Looks like everyone is rushing to check their storage. > something went wrong while loading usage details On my free group’s storage page.

john_cogs 4y ago
Yes, the container registry and build artifacts are now correctly displayed in the free plans.

giancarlostoro 4y ago

I'm assuming this is only for the ones they host and not the self-hosted solution. That is insane that anyone uploads terrabytes of data into gitlab, is there an actual valid non-illegal / weird backup choice use case? Is there some big ass GIS open source project out there that could use the attention of GitLab before they nuke some vital data somehow?

ev1 4y ago
I'm actually somewhat curious how much of an impact this will actually have - I think I've only seen a 5 GB+ repo once or twice and they were not really source code, but mildly "abusing" github CDN/releases for downloads. At least gitlab is not deleting any data, just rejecting pushes if you're over the limit.
bedosh 4y ago
The quota includes not just the git repo, but also other services that gitlab offers. In particular the container and package registries are easy to setup in a way that accumulates a lot of data (e.g. by building and tagging a docker image in every commit and not cleaning up old images). In most cases this will be a misconfiguration and all the quota enforcement is doing is to force people to configure their projects a bit more considerate. As far as I know, the quota itself isn't new, it just wasn't enforced in the past.
- Beltalowda 4y ago
  GitHub limits the package registry to 500M for free accounts, so 5G doesn't seem so bad. Git repos themselves usually don't take up that much space; all my 135 projects on GitHub are ~250M combined (quick count, could be off a bit, but roughly on that order). Even larger repos at $dayjob with daily commits typically take up a few hundred M at the most. Basically you need to either be in the top 0.1% of highly active accounts or do something specific that requires a lot of disk space, but for most people 5G doesn't seem so unreasonable.
john_cogs 4y ago
You are correct in your assumption - the storage limit impacts SaaS only. I added some context on the rollout in a previous comment. https://news.ycombinator.com/item?id=32387597 https://news.ycombinator.com/item?id=32387597

noasaservice 4y ago

So, we either go to Github, where our licenses are abused for their shitty ML. Or we pay $20/month to Gitlab. And I can't figure out how the quotas will intersect with "professional", if at all. For us Open Source devs, neither is a good option. Although I have heard good things about sr.ht / sourcehut. And for the service, it appears to be fair https://sourcehut.org/pricing/ https://sourcehut.org/pricing/

uo21tp5hoyg 4y ago
Personally I use codeberg[1] which doesn't have all the fancy bells and whistles but for personal projects it runs like butter. [1] https://codeberg.org/ https://codeberg.org/
pxeger1 4y ago
Git works perfectly fine without a web interface like that. It's not popular nowadays, but you can just keep the primary repository on your own server and merge patches by email.
marcinzm 4y ago
>So, we either go to Github, where our licenses are abused for their shitty M As long as it's on the internet MS can scrape the data so not being on Github is only a temporary defense.
- jacooper 4y ago
  But GitHub's TOS can only protect them from GitHub repos. If it has impact on CoPilot or not, is not clear.
  - marcinzm 4y ago
    Except not all contributors to projects signed the TOS since you can upload an existing project with existing non-Github contributors. One person uploading a project does not override or change the license of that project. So MS is already not relying on the TOS for legal protection which means they feel legally they can train on any OSS projects.

T3RMINATED 4y ago

lol768 4y ago

I guess this is the next step to reduce costs after the brakes were put on the "let's delete old OSS repositories" leaked plan. For comparison, I think GitHub just have a cap of 100MB on any single individual file, plus: > We recommend repositories remain small, ideally less than 1 GB, and less than 5 GB is strongly recommended. Smaller repositories are faster to clone and easier to work with and maintain. If your repository excessively impacts our infrastructure, you might receive an email from GitHub Support asking you to take corrective action. We try to be flexible, especially with large projects that have many collaborators, and will work with you to find a resolution whenever possible. Which is a bit wishy-washy, but sounds like there's room for discretion / exceptions to be made there rather than a hard cap at 5GB.

Bayart 4y ago
The difference being that the value Microsoft gets from GitHub isn't its revenue, it's its influence. Whereas Gitlab is just another corporate software suite.
- ollien 4y ago
  I don't have the diffs in front of me, but I'm fairly sure this predates the MS acquisition by quite a while.
  - xtracto 4y ago
    Prior to MS acquisition, Github free offering was really bad: No private repos, no CI pipeline and I think there was not real "team".
    - tonmoy 4y ago
      Do you have a source for this? I vaguely remember GitHub starting to offer private repos when GitLab started getting popular.
      - ollien 4y ago

imalerba 4y ago

Reading between lines it also says it going to enforce a 10GB limit on Paid tiers. > Namespaces on a GitLab SaaS paid tier (Premium and Ultimate) have a storage limit on their project repositories. A project’s repository has a storage quota of 10 GB. Even it's not mentioned as a change nor in the timeline, but that limit does not exist currently.

imalerba 4y ago
This change is one of the only MRs about these new limits discussed in the open. https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91418 https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91418
JZerf 4y ago
Just wanted to point out that it looks like people will be able to go over that limit but will have to pay extra for it according to https://docs.gitlab.com/ee/subscriptions/gitlab_com/index.html#purchase-more-storage-and-transfer https://docs.gitlab.com/ee/subscriptions/gitlab_com/index.ht... . Seems somewhat reasonable.

uo21tp5hoyg 4y ago

Am I reading it right that the original Free Tier had a quota of 45,000GB? That seems absurdly high and not very sustainable (hence the change I assume).

bombcar 4y ago
That number sounds like "unlimited, but the system requires a number" kind of thing.
YorickPeterse 4y ago
No, originally there was no quota enforced to begin with if my memory serves me right. The limits discussed here are likely meant to gradually tighten up the limits, rather than immediately locking out projects that exceed these limits.
- uo21tp5hoyg 4y ago
  I guess an infinitely large storage quota is only slightly less sustainable than a 45,000GB one.
  - szundi 4y ago
    I won’t publish my gitlabfs plugin for 1.7k different programming languages then.
NovemberWhiskey 4y ago
I'm reading that as a "stop-the-bleeding" number; presumably there is someone out there with a 44TB repository and they want to impose initial quotas that don't actually impact anyone immediately. I guess someone has been backing up their movie collections to Gitlab or something.
- bluedino 4y ago
  One case I've seen on our local Gitlab server is someone in data science/HPC (accidentally?) adding the output of a solver to their repo. Easily hundreds or thousands of gigabytes of data.

Macha 4y ago

Does the storage measurement work now? The container registry and build artifacts used to not be measured

john_cogs 4y ago
Yes, the container registry and build artifacts are now measured for Free users.
- james-skemp 4y ago
  Mine is still showing negative usage for some repos. The most negative is showing -1.08 GB, both from my profile and from the project page. Repo 130.82 MB Artifacts 12.74 MB Wiki 51.20 KiB Everything else 0 bytes. Been like this for at least a couple weeks. This one is public, but I have others that are private with negative values.
  - james-skemp 4y ago
    Looks like the issue is tracked in https://gitlab.com/gitlab-org/gitlab/-/issues/368326 https://gitlab.com/gitlab-org/gitlab/-/issues/368326 Example of a public project with negative project storage: https://gitlab.com/strivinglife/book-raspberry-pi https://gitlab.com/strivinglife/book-raspberry-pi
- rwmj 4y ago
  https://gitlab.com/nbdkit/nbdkit/-/usage_quotas https://gitlab.com/nbdkit/nbdkit/-/usage_quotas The headline number is 1.1G but the container registry is 16G. Edit: You should definitely add a simpler way to delete old pipelines. Having to mnnage it yourself through the API is a pain. (https://stackoverflow.com/questions/71513286/clean-up-history-in-gitlab-pipeline https://stackoverflow.com/questions/71513286/clean-up-histor...)

raverbashing 4y ago

How much did data hoarders use Gitlab for their backups?

T3RMINATED 4y ago

whoomp12342 4y ago

5 gigs? what is this, repo hosting for ants?

bluedino 4y ago
That does seem ridiculously small. It's probably fine for 95% of projects though.
- hinkley 4y ago
  That will depend on how well they handle forks. I could easily see myself exceeding such a limit just issuing bug fixes. Especially since one of the bugs I'm likely to fix is, "why is this library so damned big?"
  - tomjakubowski 4y ago
    Shrinking the library won't make the repo any smaller. You have to erase history to do that.
    - hinkley 4y ago
      Yeah I thought that might be confusing. Artifact size depends on what garbage from the project infrastructure gets included into the artifact. So simple things like not having a fully populated ignore file will cause things to get included into the project or the artifact. People who don't care about file size don't care about file size. If the artifact is ridiculous, sometimes the repo is ridiculous too. Therefore if you take a bunch of projects with outsized artifacts, you are going to have above average repository sizes as well.
  - john_cogs 4y ago
    GitLab team member here. Forks of projects get deduplicated, so only the changes you make will contribute to your storage consumption as long as the fork relationship is maintained.
    - 4y ago

bluedino 4y ago

What kind of projects are using 500GB much less TB's of data?

kodah 4y ago
Desktop Environment components, I suspect.
ryanlitalien 4y ago
In gaming, you can have a single branch that is 8TB with 1+ million files. But those projects aren't using git.
Gigachad 4y ago
Gitlabs own repo is at 33TB (mostly docker images). https://gitlab.com/gitlab-org/gitlab https://gitlab.com/gitlab-org/gitlab

alexb_ 4y ago

I didn't even know there was no storage limit - that seems like an immediate way to get your platform used to store non-code data in very large quantities.

pantulis 4y ago
I would guess controls to avoid people storing pirated movies on Gitlab repos and similar stuff have been in place for a long time.

game-of-throws 4y ago

If you build an image for testing on every commit and don't have a retention policy set up, you could be using a massive amount of space without realizing it. I can see why they did this.

snoopy_telex 4y ago
Are retention policies easy? I recall fighting on GitHub to setup a sane retention policy awhile back, but haven't revisited in years.
- sofixa 4y ago
  I wouldn't say easy because regexes are needed for complex ones (e.g. retain -prod images for 6 months, -staging for 24 days) but they are powerful and not that complicated.
- dnsmichi 4y ago
  GitLab offers ways to manage retention policies for your container registry. Please see the docs for guidance: https://docs.gitlab.com/ee/user/packages/container_registry/reduce_container_registry_storage.html#create-a-cleanup-policy https://docs.gitlab.com/ee/user/packages/container_registry/...
khuey 4y ago
Gitlab didn't even have retention policies until sometime in 2020. I have a six year old project that's consuming something like 4TB of space in their container registry.
- akudha 4y ago
  Woah. 4 TB? What kind of project is it?
  - georgyo 4y ago
    Not op, but a simple node application can very easily make container images that are well over a gigabyte. Build and store that container images on every push and boom, you can explode in usage.

john_cogs 4y ago

GitLab team member here. The impacted users are notified via email and in-app notifications will begin 2022-08-22, so far we've contacted 30,000 users. Only GitLab SaaS users are impacted - the limits are not applicable to self-managed users.

martypitt 4y ago
We're affected by this change, and have been trying to get support in forums, but can't get hold of anyone. Build Artifacts are listed as part of what contributes to the quota (which is fair enough), but there's no way (that I could find in the docs) to manage build artifacts that are stored. I suspect we have a large historic storage, which we don't use / need, but there's no way to browse this, no way to verify it, and no way to delete what we don't need. I'm dreading getting a big ole bill in a few months for storage we had no way to opt out of.
- xiaq 4y ago
  I have a repo that was almost exactly 5GB and it turned out that the storage was also used exclusively by pipeline logs and artifacts. Found this recipe on StackOverflow to just nuke everything and worked for me: https://stackoverflow.com/questions/71513286/clean-up-history-in-gitlab-pipeline https://stackoverflow.com/questions/71513286/clean-up-histor.... You can tweak it a bit if you want to keep the more recent pipeline runs. For some reason I had to run this multiple times to completely remove everything; each run removes about half of all the pipelines and only when there were about 20 remaining did the script remove everything.
- lbotos 4y ago
  GitLab Support Engineering Manager here. These limits will first be soft limits as they are now. Impacted users will be notified and there will be about 2 months to take action before enforcement starts. After that point, limits will be enforced. If you’d like to investigate now you can use the GitLab API to fetch job artifacts and their storage size: https://docs.gitlab.com/ee/api/jobs.html#list-project-jobs https://docs.gitlab.com/ee/api/jobs.html#list-project-jobs and calculate the summary. I’d suggest collecting all pipeline jobs to be deleted using this API endpoint https://docs.gitlab.com/ee/api/job_artifacts.html#delete-job-artifacts https://docs.gitlab.com/ee/api/job_artifacts.html#delete-job... We are looking to improve the visibility for job artifacts in the UI as we get nearer to the storage limit being enforced.

Kelteseth 4y ago

My Qt/C++ cross-platform FOSS Wallpaper Engine project[1] currently uses 47gb of storage. This is because I compile for every platform and store the artifacts for 4 weeks. Not sure what I will do in the future, because having older builds around to try out without recompiling is always nice. [1] https://gitlab.com/kelteseth/ScreenPlay https://gitlab.com/kelteseth/ScreenPlay

TheRealPomax 4y ago
That's... an incredible footprint. Surely even if there was no size limit, deduplicating the build output would have made a lot of sense?
- TheRealPomax 4y ago
  Having said that: if you run a public open source project that has requirements beyond what a regular free user gets, then why not apply to the open source program[1] (which you almost certainly already qualify for) so you're not space constrained? Quoting the requirements: --- Who qualifies for the GitLab for Open Source Program? In order to be accepted into the GitLab for Open Source Program, applicants must: - Use OSI-approved licenses for their projects: Every project in the applying namespace must be published under an OSI-approved open source license. [2] - Not seek profit: An organization can accept donations to sustain its work, but it can’t seek to make a profit by selling services, by charging for enhancements or add-ons, or by other means. - Be publicly visible: Both the applicant's GitLab.com group or self-managed instance and source code must be publicly visible and publicly available. --- [1] https://about.gitlab.com/handbook/marketing/community-relations/opensource-program/#-gitlab-for-open-source-program https://about.gitlab.com/handbook/marketing/community-relati... [2] https://opensource.org/licenses/alphabetical https://opensource.org/licenses/alphabetical
shakow 4y ago
I guess that having old artefacts is nice, but expecting 47GB of storage for free per repo seems ludicrous to me. Maybe it's a generational question, but I would be pretty happy with 1-2GB (i.e. git history, and one artefact per platform).

bachmeier 4y ago

This doesn't affect me, but a better way to handle this would be to sell extra storage at, say, double GitLab's cost. Digital Ocean sells 250 GB object storage at $5/month and $0.02/GB beyond that.

john_cogs 4y ago
GitLab team member here. GitLab offers additional storage for $60/year for an additional 10GB of storage and 20GB of transfer/month.
- oy6amn8RDPw8w9n 4y ago
  Yes, $0.50/GB/month seems like very reasonable pricing /s That's over twice as much as the disks to store that much data cost. Every month.
  - glenneroo 4y ago
    Yeah that really hurts. A single Unity project can easily reach that in a short amount of time. When I get that warning email in a few weeks, I'll be shopping around.
- bachmeier 4y ago
  I see now that it was mentioned at the end of the post. (Though I did specify "double GitLab's cost".)

unintendedcons 4y ago

Gitlab just can't shooting themselves in the foot. Driving away individuals is apparently their strategy now. Sad. I used to teach new developers starting with Gitlab pages.

eliseumds 4y ago

Fair enough.

deleted 4y ago

[deleted]

encryptluks2 4y ago

5GB isn't much different than the storage limits of other services, but their storage pricing is atrocious. I've seen the writing on the wall for a while and watched as GitLab went from being the cool open source alternative to GitHub to becoming a bloated oversized mess. I know several popular open source projects were offered premium tier upgrades for free. I am curious to see if these changes, especially transfer limits, will impact them enough to move away.

nabakin 4y ago

With this change, the 5 user limit[1], and original intent to delete dormant repositories[2][3], it seems as though GitLab is no longer able to support the free side of its business. GitLab has been touted as more OSS-friendly than GitHub, but a large part of the OSS ecosystem depends on free repositories. With these changes and this trajectory, I can't see myself putting another OSS project on GitLab. It's a shame it's come to this, but I'm confident GitLab didn't make this choice lightly. It must be done in order for them to stay afloat. Thank you GitLab team for your efforts. I hope you guys are successful in your future endeavors. [1] https://about.gitlab.com/blog/2022/03/24/efficient-free-tier/ https://about.gitlab.com/blog/2022/03/24/efficient-free-tier... [2] https://www.theregister.com/2022/08/04/gitlab_data_retention_policy/ https://www.theregister.com/2022/08/04/gitlab_data_retention... [3] https://www.theregister.com/2022/08/05/gitlab_reverses_deletion_policy/ https://www.theregister.com/2022/08/05/gitlab_reverses_delet...

DoesntMatter22 4y ago
I think a lot of the problem is how they communicated it. It isn't like "Well hey we have a problem here", it's just "Hey here's this super complicated to this problem you didn't know existed and you can take it or leave it"

jacooper 4y ago

Honestly for me the bigger issue is the upcoming bandwidth limit. There is no clear date to when its going to be enforced. But its definition is so wide its crazy, its basically any egress data except the web interface and shared runners. AFAIK it will also include git clones!, so if your project suddenly gets popular the users clones will cost you many too. Also if you use your own runner, cloning the repo to the runner will also be included in your bandwidth limit. And since it should apply to GitLab pages, it becomes useless for anything the you want to get few visits. Since GitLab is behind cloudflare, you might as well just use Cloudflare Pages at this point.

karamanolev 4y ago
> Also if you use your own runner, cloning the repo to the runner will also be included in your bandwidth limit. You make it sounds like it's horrible. I'd offer a different take in that sure, it can be a huge negative if done recklessly on GitLab's part. But if done correctly, it can actually be a good thing. I think CI pipelines doing a full clone from scratch on every build (or npm installs or other bootstraps) is extremely wasteful in terms of resources, so I'd be glad if that reduces it significantly.
- jacooper 4y ago
  GitLab CI default clone configuration already AFAIK only includes the last 20 commits, so its already slimmed down.
  - karamanolev 4y ago
    If you start from scratch, regardless of history, you're fetching at least the size of the repo. In many cases, that's hundreds of MBs for every build and I'm a huge fan of building for every commit. I find that a colossal waste. I have a very old fashioned and "not recommended" setup with Jenkins that only pulls new commits, because it works out of a persistent working directory. Works wonders for a Django/ES6 project and sips bandwidth. I wish more of the modern containerized, start-from-scratch and so on setups would work in a similar way wrt the bandwidth they use.

elevation 4y ago

Git excels at tracking human keyboard output. A productive developer might write 100KB of code annually so a git repo can represent many developer years of collaborative effort in just a few MB. That is, unless you require git to track large media files, third party BLOBs, or build output. However, sometimes tracking these things are necessary, and since there isn't an obvious companion technology to git for caching large media assets ("blob hub?") or tracking historical build output ("release hub?"), devs abuse git itself. I wish there were a widely accepted stack that would make it easy to keep the source in the source repo, and track the multi-gb blobs by reference.

foepys 4y ago
You mean something like git Large File Storage? It comes with git for Windows by default and every Linux distro I know has it in its repos. MacOS also has it in Homebrew. https://git-lfs.github.com/ https://git-lfs.github.com/ It's basically one click/command away.
WorldMaker 4y ago
Git LFS is a pretty widely accepted stack for managing binary blobs by reference in git: https://git-lfs.github.com/ https://git-lfs.github.com/ The plugin is installed out of the box in many git distributions now. Many hosts support it today, including Gitlab, which is relevant to this article's discussion: https://docs.gitlab.com/ee/topics/git/lfs/ https://docs.gitlab.com/ee/topics/git/lfs/
- Dayshine 4y ago
  I have yet to find a host that has at all usable Git LFS pricing though. GitLab is $60/m (edit, per year? unclear in pricing page) for 10GB storage and 20GB/m transfer. So, I can checkout my repo twice a month!? GitHub is similar, with a 1:1 ratio for storage and transfer. So you can checkout your repo once a month.
adobrawy 4y ago
> companion technology to git for caching large media assets ("blob hub?") or There is Amazon S3 or any other cloud storage. > tracking historical build output ("release hub?") There are OCI artifact registries. For me, there is rather issue that when all you have is a hammer, everything looks like a nail.

oefrha 4y ago

A lot of discussions about how git repos are supposed to be small are totally missing the point. This storage quota applies to everything, including release artifacts, containers, etc. Forget containers or CI artifacts on every commit, let's look at a very common scenario: using goreleaser to build binaries and deb/rpm/etc. packages for multiple architectures every release. This way a moderately sized Go project can easily consume 50-100MB or more per release. That gives you at most 50-100 releases across all your projects. Using hosted GitLab for open source projects is looking less and less appealing. I also posted about issue trackers on gitlab.com not allowing search without signing in a while back: https://news.ycombinator.com/item?id=32252501 https://news.ycombinator.com/item?id=32252501 Edit: An open source program that upgrades the quota is mentioned elsewhere in the thread: https://about.gitlab.com/solutions/open-source/ https://about.gitlab.com/solutions/open-source/ I don’t use hosted GitLab for my open source work, so no idea how many people get approved.

john_cogs 4y ago
Hi, GitLab team member here. The GitLab for Open Source program provides open source projects with Ultimate benefits and higher limits. More info in previous comment: https://news.ycombinator.com/item?id=32387621 https://news.ycombinator.com/item?id=32387621
- KMnO4 4y ago
  Why do you have to “apply” for these benefits? Shouldn’t providing the source publicly in the open (ie not a private repository) by definition be enough for “open source”?
  - saganus 4y ago
    What happens if I publish the code under a restrictive (non-open source) license? Even though the code is available, that does not mean you can use it however you want, no?
    - SCLeo 4y ago

xaxaxb 4y ago

Haha, great.

jordemort 4y ago

What's the point of tapering it down in stages like this? Between the October 19th quotas and the October 20th quotas, if you wait until the last minute, you have 24 hours to move 37.5TB of data. Then 4 more days to move another 7TB; does that actually help anyone? The proposition of getting that much data out of it at that speed seems a bit unrealistic. Why not just say "the quota will be 5GB on November 9th" and be done with it?

M3L0NM4N 4y ago
Definitely sounds like there was some bureaucracy in there.
john_cogs 4y ago
The phased enforcement of the limits are a part of the technical rollout plan for this change that was added to our docs. Related comment: https://news.ycombinator.com/item?id=32387597 https://news.ycombinator.com/item?id=32387597. The communication sent to impacted users via email and future in-app notifications includes only the applicable enforcement dates and limits. Issue to follow: https://gitlab.com/gitlab-org/gitlab/-/issues/368150 https://gitlab.com/gitlab-org/gitlab/-/issues/368150

cercatrova 4y ago

I'm just going to move to GitHub at this point.

jacooper 4y ago
Yup its looking to be the only platform where projects can realisticly grow. But after CoPilot I still dont want to use it, on the contrary, I'm switching to GitLab because all the free stuff on GitHub are just unrealistic, and are just to play the long game. Also I dont want to invest my time and effort into a proprietary platform, like GitHub Actions, and Gitlab CI and most other core features of GitLab are Open Source. https://sfconservancy.org/blog/2022/jun/30/give-up-github-launch/ https://sfconservancy.org/blog/2022/jun/30/give-up-github-la... If GitLab didn't add the 5 person limit per group most people will be fine, but now with the upcoming bandwidth limit, the 5 person limit, and many others. Its almost impossible to grow on the platform Excexpt with the OSS program, which requires Lawful agreement with each org member.
- cercatrova 4y ago
  I love GitHub Copilot. In fact, I am more inclined to use GitHub simply to help improve their tool.

djohnston 4y ago

Why are people checking build artefacts into the repo?

cesarb 4y ago
Playing devil's advocate: sometimes your build is non-reproducible, annoying to build, or both (for instance, needing a proprietary tool which can only run on a particular developer's laptop because the license is tied to that particular hardware, and which crashes half of the time for no particular reason). Keeping the build artifacts in the repository means you can reproducibly obtain that exact artifact, even years into the future.
ranguna 4y ago
This limitation is not just about code it will also affect all artifacts like container and package registry, pipeline artifacts, lfs, the lot.

cookiengineer 4y ago

Anybody found out which projects on gitlab exceed the 45 TB limit? I'm curious what kind of project would even need such a repository size. From a distant view this sounds like heavily mismanaged build artifacts in the project's git history; or abused storage for free CDN of video data or similar.

SCLeo 4y ago
Maybe a super active repository with a large build matrix? For instance, a repository with 10,000 commits, 25 artifacts per build, and 200 MB/artifact will take 50 TB. It is still ridiculous though.
- cookiengineer 4y ago
  Are releases/tags also counting in on that? I was re-reading the announcement but as far as I can tell it only states repository size everywhere. The linked cleanup instruction articles also mostly show repo size (git gc et al) and nothing related to the releases section of a repository. [1] I'm also still unclear about what exactly is part of the package registry storage and whether or not releases in the repository are part of that. [1] https://docs.gitlab.com/ee/user/usage_quotas.html#manage-your-storage-usage https://docs.gitlab.com/ee/user/usage_quotas.html#manage-you...

lakomen 4y ago

bdcravens 4y ago
I'm confused at the connection between Microsoft/Github and the decision that Gitlab made here.
- likortera 4y ago
  I think GP just rushed the free hate
likortera 4y ago
You are confusing Gitlab with GitHub?
MaxDPS 4y ago
But this is related to GitLab…

kerblang 4y ago

Perhaps now is a good time to recommend the ever-popular BFG to anyone unaware: https://rtyley.github.io/bfg-repo-cleaner/ https://rtyley.github.io/bfg-repo-cleaner/ Also my team's biggest repo is a 2.5 GB checkout but gitlab (self-managed) reports it as 185MB "files" and 353 MB "storage" (no CI/CD artifacts).

sofixa 4y ago
Similarly there's also git-filter-repo: https://github.com/newren/git-filter-repo https://github.com/newren/git-filter-repo It's in Python so runs pretty much everywhere *nix out of the box.
dnsmichi 4y ago
> erhaps now is a good time to recommend the ever-popular BFG to anyone unaware: https://rtyley.github.io/bfg-repo-cleaner/ https://rtyley.github.io/bfg-repo-cleaner/ Thanks for sharing this tool to help cleanup the Git history. Please be aware that it will rewrite the history, which could be very impactful to existing branches, merge requests and local clones. A similar approach is described in the documentation: https://docs.gitlab.com/ee/user/project/repository/reducing_the_repo_size_using_git.html https://docs.gitlab.com/ee/user/project/repository/reducing_...

rohfle 4y ago

A couple of notes - If I tag a docker image with multiple tags, and then push it to Gitlab, each tag counts towards the storage limits even though SHAs are identical. eg 100MB container tagged with "latest" and "v0.5" uses 200MB of storage. - The storage limit is not per repository, but per namespace. So 5GB free combined for all repositories under your user. If you create a group, then you get 5GB free combined for that group. Does this include forks? Does this include compression server side? - The 10GB egress limit per month includes egress to self-hosted Gitlab Runners in free tier. Consider this with the 400 minutes per month limit on shared runners. These limits feel less like curbing abuse and more like squeezing to see who will jump to premium while reducing operating costs. Is this a consequence to Gitlab hosting on GCP with associated egress and storage costs? Is this a move to improve financials / justify a market cap with fiscal storm clouds on the horizon? Is this being incentivized by $67m in awarded stock between the CFO and 2 directors? Stock history over last year for GTLB (since IPO in 2021?): https://yhoo.it/3QaExCs https://yhoo.it/3QaExCs From the golden era of 2015: https://about.gitlab.com/blog/2015/04/08/gitlab-dot-com-storage-limit-raised-to-10gb-per-repo/ https://about.gitlab.com/blog/2015/04/08/gitlab-dot-com-stor... > To celebrate today's good news we've permanently raised our storage limit per repository on GitLab.com from 5GB to 10GB. As before, public and private repositories on GitLab.com are unlimited, don't have a transfer limit and they include unlimited collaborators.

jreporter 4y ago
Hi @rohfle! A couple of clarifications: > If I tag a docker image with multiple tags, and then push it to Gitlab, each tag counts towards the storage limits even though SHAs are identical. eg 100MB container tagged with "latest" and "v0.5" uses 200MB of storage. Any "duplicated" data under a given "node" (be that the root namespace, a group or a project) counts towards the storage usage only once. So images latest and v0.5 would only represent 100MB in their namespace registry usage, not 200MB. > Does this include forks? The registry data is not copied/duplicated when one forks a project. So this is not applicable. But even if it was, as long as the fork and the source are under the same root namespace, any "duplicated" registry data across the two would only count towards the storage usage once. > Does this include compression server side? Yes, the measured size is the size of the compressed artifacts on the storage backend. We should be updating the docs shortly to make these answers more transparent!

balentio 4y ago

Gitlab keeps giving me more reason to go to Gittea.

nix23 4y ago
I stay on: https://sourcehut.org/ https://sourcehut.org/ No blingbling no social-media drama, just clean straightforward code hosting.

eddof13 4y ago

Bitbucket anyone? Or Atlassian is no longer an option?

crgwbr 4y ago

Seems like a buried lede here is that limits also now apply to paid accounts. Just checked my team’s name space: we have 700GB of storage used, and gitlab is going to start charging us $0.50/month/GB for everything in excess of 50 GB. On top of the hundreds of $/month we’re already paying in per-seat pricing. That seems absurdly expensive.

dnsmichi 4y ago
Thanks for your feedback. I’d suggest starting the analysis to identify the type of most storage being consumed. Maybe CI/CD job artifacts are kept forever and need an expiration configuration [0], or container images in the registry would need cleanup policies [1]. The documentation linked in the FAQ provides more analysis and guides to help. [2] If you need additional help with analyzing the storage usage, please contact the GitLab support team. You can also post cleanup questions on our community forum. [3] [0] https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html#when-job-artifacts-are-deleted https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html#w... [1] https://docs.gitlab.com/ee/user/packages/container_registry/reduce_container_registry_storage.html https://docs.gitlab.com/ee/user/packages/container_registry/... [2] https://about.gitlab.com/pricing/faq-paid-storage-transfer/#managing-your-storage-usage https://about.gitlab.com/pricing/faq-paid-storage-transfer/#... [3] https://forum.gitlab.com/c/gitlab-ci-cd/23 https://forum.gitlab.com/c/gitlab-ci-cd/23