7 ms·
Someone asked "Would this have made sense for a company like GitLab if they didn't have the corporate backing of something like MS?" and deleted their comment w
by sytse 7y ago
Someone asked "Would this have made sense for a company like GitLab if they didn't have the corporate backing of something like MS?" and deleted their comment while I was writing the answer below:
Being the canonical registry for a language (Rubygems) or technology (DockerHub) tends to be a huge expense.
The main expenses are cloud costs (bandwidth and storage) and security (defense and curation).
I've not seen examples of organizations turning this into a great business by itself. For example Rubygems is sponsored by RubyCentral http://rubycentral.org/ http://rubycentral.org/ who organize the annual RubyConf and RailsConf software conferences.
Please note that running a non-canonical registry is a good business. JFrog does well with Artifactory https://jfrog.com/artifactory/ https://jfrog.com/artifactory/ and we have the GitLab Package Registry https://docs.gitlab.com/ee/user/packages/ https://docs.gitlab.com/ee/user/packages/ that includes a dependency proxy and we're working on a dependency firewall.
- gowld 7y agoWhy is non canonical better? Because they get to charge a fee?
- kortilla 7y agoYep, in particular the use case is private packages.
- sytse 7y agoYep, non-canonical registries are used by business and they are more likely to pay. For example our dependency firewall will be a paid feature that delays updates from packages that are recently updated under suspicious circumstances like: author changed, author information changed, different programming language, activity after a long period of inactivity, large change to the code, etc.
- JMTQp8lwXL 7y agoI am curious what proportion of compromised packages fall under the umbrella of scenarios you have outlined.
- sytse 7y agoMe too. If someone has data on predictors of compromised packages I’m all ears.
- deleted 7y ago[deleted]
- smcnally 7y ago> delays updates from packages that are recently updated under suspicious circumstances This is an example of the security curation you note earlier. People reviewing flags alongside the code, remediation during the delay, highlighting package authors you like are some of why “Being the canonical registry … tends to be a huge expense.” How many people build and run GitLab Package Registry?
- sytse 7y agoWe don’t know exactly how many people use it since it is open source but I think between 100,000 and 1 million people. The docker container registry is the most popular. The security curation examples I mentioned are intended to use automated signals so there wouldn’t be an expense for labor. In fact we intend to run them separately on every self managed GitLab installation.
- deleted 7y ago[deleted]
- wp381640 7y agoCan anybody comment on how hosting a package repo for someone large may help peering? I'd imagine they would see significant upstream traffic and globally but not certain if that makes an impact
- sytse 7y agoI think hosting a package repo will generally create more downloads (costs money) than uploads (helps with peering but costs storage) because the average number of downloads for a package is more than 1.
- oaiey 7y agoSpeaking of... The NuGet registry in GitLab deserves an option to run without authentication from other builds and/or the "public" accessing it. Currently, it does not even handle the build job token and I have to cross connect builds using my personal access token (which then again is accessible to other maintainers of a repo).
- blyry 7y agoWe stopped using gitlabs artifact repositories for this exact reason!
- tr-gitlab 7y agoWe'd love for you to try again. We do support using the job token with our NPM, Maven and Conan repositories. What do you currently use?
- tr-gitlab 7y agoGitLab PM here - Agreed that is an important feature and one we have scheduled for milestone 13.1 which will be released in June 2020. As a workaround, you can add your access token as an environment variable and use that to publish/install packages via CI/CD. And if you are interested in contributing to GitLab that issue is a great way get started.
- gramakri 7y agoIt's a mystery how DockerHub remains free. The network and storage costs must be massive. Docker also sold it's enterprise business, so I am not sure who is paying for all this.
- diggan 7y agoIf you're running a package registry, you'd be painfully unaware of your own requirements if you're using a host/cloud that you need to pay for the amount of bandwidth. Hosting a package registry on AWS for example, unless you figure out a way of seriously reducing the amount of traffic (which seems to be against working towards making a popular registry), is suicidal because of the bandwidth costs.
- deleted 7y ago[deleted]
- yhoiseth 7y agoThere's also https://packagist.com/ https://packagist.com/. I don't know how well the business is doing.
- beberlei 7y agoThis massively depends on the architecture of the package manager though. Composer (PHPs primary package manager) only serves the metadata and directs the installer to the already packaged ZIPs hosted on Github/Bitbucket/Gitlab. This allowed them to serve the whole community as a small team of 2-4 for almost a decade now without any outside funding.
- drewbug01 7y ago> This massively depends on the architecture of the package manager though. Composer (PHPs primary package manager) only serves the metadata See also, CocoaPods. But we're really talking about two things here as if they're the same, and I think it's because the language we use for them isn't clear: package managers (like CocoaPods and Composer) and full-blown package registries (like RubyGems and NPM). I mean, the difference between them is slight; and there's overlap. They both resolve package dependencies and sort out problems, for example. But they're such very, very different things! It's not really fair to compare these two approaches and say things about Composer like "this allows them to serve the whole community with a team of 2-4 people": they're just not directly comparable. Those 2-4 people + the entire GitHub staff is what supports the whole community. Now - don't get me wrong: I think it's fine that GitHub does this! I'm not saying that CocoaPods and Composer are making the wrong choice here at all. In fact, I'm proud to work for a company that provides this service to everyone. But it hand-waves away a lot of stuff as just an "architecture" decision, when in reality it's a decision not to do the hardest part - which is hosting, because it requires gobs of money and dedicated staff and people on-call 24/7. Framing it as just a different "architecture" implies that registries like NPM are making the wrong choices by hosting their own downloads, which I think is not nearly that simple. Because what happens if GitHub or Bitbucket decided to not allow this type of usage? It would be devastating for the community - so I think it's very appropriate for package managers to make a decision about whether or not they also want to be a registry. They're considering what could happen down the road and what that would mean. CJ's comments on all of this, re: entropic, are really good: https://www.youtube.com/watch?v=MO8hZlgK5zc https://www.youtube.com/watch?v=MO8hZlgK5zc
- g8oz 7y agoComposer is amazing and I'm going to say far better than what I've seen in JavaScript and python world.