10 ms·

We repurposed NPM to publish and distribute Go binaries for our internal CLI

62 points by boxbag 6y ago

cube00 6y ago
Given NPM registries are presumably optimized to store text, I wonder how this scales and if other repositories that do expect binaries (eg. Maven, NuGet) compare.
- jacobwg 6y ago
  NPM packages are uploaded / stored as `.tar.gz` files, so they can contain anything (alongside a package.json metadata file) - some public NPM packages contain binary files like images or compiled assets too. The registry doesn't really care about much beyond the package.json. I've done something similar to the author at a previous company, distributing an internal development CLI written in Go, though in our case we published to the GitHub Packages NPM registry rather than private npmjs.com, so we could authenticate our developers via their GitHub accounts. Including multiple binaries inside the published package, one for macOS and one for Linux (we had no Windows devs), did increase the package size, but the whole gzipped .tar.gz was around 20-30mb, so not terrible from a registry storage standpoint. Publishing to an NPM registry, vs just hosting a binary for our devs to download, was convenient since we could use the registry API to check for updates and notify the user that an update was available. Users could then update via a subcommand, which just pulled the new package from the registry.
monadic3 6y ago
What, did curl not work?
- bmurphy1976 6y ago
  I haven't read the article yet, but I would assume they want versioning, history, checksumming and verification. All that stuff is worthwhile, it's just unfortunate npm is such a heavy dependency to achieve it.
  - derision 6y ago
    Not that heavy if they're already using it though
    - tstrimple 6y ago
      It also only gets really bad with deep dependency trees. If it's just your stuff, you probably avoid most of the issues.
  - throwawaygulf 6y ago
    If only there was a system software package management tool that could do all of that... YUM is almost old enough to drink in Europe.
  - kevin_thibedeau 6y ago
    Debian apt gives all these things and would be less of a heavy dependency.
    - zdragnar 6y ago
      If they only needed linux support, that would have worked too
    - aprdm 6y ago
      The problem with .deb and .rpm is that their spec file format is horrendous. I've packaged multiple RPMs and every time I have to do it is a complete PITA, lots of quirks, macros, flags, etc. The official docs are like a deep man page and do not have simple use cases, like, create a pkg for a binary or create a pkg for a binary with systemd or how to distribute a pkg of .so IMO if the .deb and .rpm maintainers invested some time in modernizing their official pages with lots of quick starts and guides it would bring a lot of developers to their side. (And in this specific case they needed more than Linux support)
    - deleted 6y ago
      [deleted]
    - 6y ago
- bigiain 6y ago
  "we ran into the issue of distributing the CLI tool to our team of engineers without the need of distributing a new set of tokens or making the binary public" Seems a neat enough hack to make their newer shit available internally using existing tools/security.
lxe 6y ago
I need a medium account to read this for some reason.
- tstrimple 6y ago
  > "This is your last free member-only story this month. Sign up for Medium and get an extra one" Well, that's about the end of medium I think.
  - kevin_thibedeau 6y ago
    Turn off Javascript. They don't (yet) have an SPA and it degrades appropriately for disabled users.
    - tstrimple 6y ago
      I'm not going to expend any effort to circumvent medium's nonsense. The content there isn't of consistently high enough quality to justify jumping through any hoops. Any barriers they put up will just result in me not consuming content hosted there.
      - freedomben 6y ago
        Open a private tab (or in Chrome "incognito") and it will work fine.
  - simongr3dal 6y ago
    I wonder how “publishers” feel about that. I would feel rather angry if Medium dictated that people couldn’t access my blog posts.
    - tempodox 6y ago
      But that's exactly what to expect when people donate their writings to a platform they have no control over. There are always those who need to learn the hard way.
anonacct37 6y ago
I wish commentors would read the article. Why didn't they use the os package manager? Which OS? Their build clearly builds for osx, windows, and Linux.
- bfung 6y ago
  Or since it is an engineering CLI tool, why not have engineers: 1. clone cli repo 2. run goreleaser 3. copy binary to desired path sure, it's not 1 step (can be condensed with a build script), but it does teach the other engineers how to roll their own cli tool and contribute back if they fix a bug in it. Or if the blog indicated that not all engineers understand golang and build tools, and there's too many teams - that'd be good for blog context. Hopefully the author didn't spend much time in figuring out or being too clever in making this "hack".
  - emeraldd 6y ago
    As the number of tools grow and the number of teams using/developing them grow this becomes more and more untenable. You not only have to maintain the tool itself, but also the build stacks associated with them and it becomes the task of the user to make sure they are running the most recent version. This leads to more complex support situations. Basically, you need a deployment methodology that allows for: * Ease of use by "novice" developers * Only introduces minimal overhead for those developers * Makes it easy to upgrade or remove a tool you installed 6 months ago * Doesn't require a tool development team to come up with their own solution This is a surprisingly tricky to do on a diverse environment with 1000+ internal users.
- sagonar 6y ago
  NPM could handle different OS, I would guess they run more than 1 os. (It is also a tool which the team is used to handle, as they have been using node + npm)
turminal 6y ago
This just saddens me deeply. Of all the ways this could have been done and all the package managers they could have used they picked npm. I bet some day we'll have a Linux distro with npm as the system package manager here on HN.
- xena 6y ago
  https://node-os.com/ https://node-os.com/
  - turminal 6y ago
    Dang, I wish I could say I'm surprised, but I guess it was to be expected.
- detaro 6y ago
  It saddens you that someone used a tool they already use instead of adding another thing to distribute a small binary?
  - turminal 6y ago
    I guess my comment appears rude, which wasn't intended, I just find npm a troublesome piece of software that is passable as a package manager for the javascript ecosystem, but not a tool one solves problems with if there are viable alternatives.
    - murukesh_s 6y ago
      why is it trouble? why can't they improve it even if its trouble?
    - deleted 6y ago
      [deleted]
    - folkrav 6y ago
      When you have one thing to nail and already own a hammer, why go and buy another one? Even if it's not a good one, it's gonna do the job, and be the least trouble.
dboreham 6y ago
Next: Docker images!
lmm 6y ago
I was very tempted to reuse a Maven repository for publishing the handful of small Python utilities we had. It's a standardised format, we already had the server set up, we already had different kinds of artifacts in there and multiple languages, and frankly Python-on-Maven dependency management can't be worse than what standard Python dependency management looks like.
- coredog64 6y ago
  I kinda do that today. We’re primarily a Java shop, but we’re using Python for FaaS. So we package up our code and dependencies into a zip and use Maven to push the resulting archive into our common repo.
pjmlp 6y ago
Go supports native packages natively via go get, why the hell bring npm into the picture?!?
- woutr_be 6y ago
  > As the project completed we ran into the issue of distributing the CLI tool to our team of engineers without the need of distributing a new set of tokens or making the binary public. From the article itself, it seems like they wanted to re-use their existing process without disruptions.
- noselasd 6y ago
  You'd use go get as a developer, not to update workstations and production machines.
  - pjmlp 6y ago
    Same applies to using npm on workstations and production machines.
lrossi 6y ago
If all you have is a hammer, everything looks like a nail. [0] [0] Abraham H. Maslow (1966). The Psychology of Science.
- deleted 6y ago
  [deleted]
Fresh20210107 6y ago
Why use a language, created to keep interns safe and technical fellows from leaving, to begin with?
siscia 6y ago
At CERN we have an internal system to distribute software. It is mostly used to distribute software to production systems in the datacenters or to distribute nightlies and toolchains to developers. It replace standard package managers on most production systems. It is a FUSE filesystem, where we can install software once and have it distributed to a practically unlimited number of clients thanks to a simple and performant cache system. The filesystem is called CernVM-FileSystem, CVMFS for short.[1] I maintain a public instance of it under https://packages.redbeardlab.com https://packages.redbeardlab.com It contains "common" software, like standard unix tools, compilers, webservers, etc... I wrote a bit about it: https://redbeardlab.com/2020/12/20/announcing-packages-redbeardlab-com-a-public-cvmfs-repository-for-common-software/ https://redbeardlab.com/2020/12/20/announcing-packages-redbe... https://redbeardlab.com/2020/12/21/bootstrapping-packages-redbeardlab-com/ https://redbeardlab.com/2020/12/21/bootstrapping-packages-re... https://redbeardlab.com/2020/12/21/packages-redbeardlab-com-for-golang/ https://redbeardlab.com/2020/12/21/packages-redbeardlab-com-... https://redbeardlab.com/2020/12/23/packages-redbeardlab-com-for-node-js-community/ https://redbeardlab.com/2020/12/23/packages-redbeardlab-com-... [1]: https://cernvm.cern.ch/fs/ https://cernvm.cern.ch/fs/
- beojan 6y ago
  One of the best things about CVMFS is the idea of LCG views which are coherent installations of a large number of packages. This more or less eliminates the need for package management and guarantees that someone else can reproduce your environment. It's like Stackage LTS releases, but everything's installed already.
  - siscia 6y ago
    Unfortunately I am not familiar enough with LCG and LCGmake to actually build an outside-CERN release. Fortunately Nix helped in creating the same coherent installation of packages. Then `source /cvmfs/packages.redbeardbal.com/setup.sh` give you the same feel of an LCG view, but with more widespread packages.
nerdponx 6y ago
There is obviously a need here that is not met by current tools. I believe that Conda is the closest thing to an answer to this need. It works on Windows, Mac and Linux. It supports packaging arbitrary binaries. It lets you specify dependencies that can be arbitrary libraries for arbitrary languages. And it has a relatively simple package format, build process, and repository format. It's not the best piece of software when it comes to user experience, and its niche is still very much data science. But it is a glimpse of what should be possible, but is not currently available.
notretarded 6y ago
>We recently migrated our internal CLI tool from Node JS to Golang Why? Out of things to do? Wanted to look productive?

62 points by boxbag 6y ago

Given NPM registries are presumably optimized to store text, I wonder how this scales and if other repositories that do expect binaries (eg. Maven, NuGet) compare.

jacobwg 6y ago
NPM packages are uploaded / stored as `.tar.gz` files, so they can contain anything (alongside a package.json metadata file) - some public NPM packages contain binary files like images or compiled assets too. The registry doesn't really care about much beyond the package.json. I've done something similar to the author at a previous company, distributing an internal development CLI written in Go, though in our case we published to the GitHub Packages NPM registry rather than private npmjs.com, so we could authenticate our developers via their GitHub accounts. Including multiple binaries inside the published package, one for macOS and one for Linux (we had no Windows devs), did increase the package size, but the whole gzipped .tar.gz was around 20-30mb, so not terrible from a registry storage standpoint. Publishing to an NPM registry, vs just hosting a binary for our devs to download, was convenient since we could use the registry API to check for updates and notify the user that an update was available. Users could then update via a subcommand, which just pulled the new package from the registry.

monadic3 6y ago

What, did curl not work?

bmurphy1976 6y ago
I haven't read the article yet, but I would assume they want versioning, history, checksumming and verification. All that stuff is worthwhile, it's just unfortunate npm is such a heavy dependency to achieve it.
- derision 6y ago
  Not that heavy if they're already using it though
  - tstrimple 6y ago
    It also only gets really bad with deep dependency trees. If it's just your stuff, you probably avoid most of the issues.
- throwawaygulf 6y ago
  If only there was a system software package management tool that could do all of that... YUM is almost old enough to drink in Europe.
- kevin_thibedeau 6y ago
  Debian apt gives all these things and would be less of a heavy dependency.
  - zdragnar 6y ago
    If they only needed linux support, that would have worked too
  - aprdm 6y ago
    The problem with .deb and .rpm is that their spec file format is horrendous. I've packaged multiple RPMs and every time I have to do it is a complete PITA, lots of quirks, macros, flags, etc. The official docs are like a deep man page and do not have simple use cases, like, create a pkg for a binary or create a pkg for a binary with systemd or how to distribute a pkg of .so IMO if the .deb and .rpm maintainers invested some time in modernizing their official pages with lots of quick starts and guides it would bring a lot of developers to their side. (And in this specific case they needed more than Linux support)
  - deleted 6y ago
    [deleted]
  - 6y ago
bigiain 6y ago
"we ran into the issue of distributing the CLI tool to our team of engineers without the need of distributing a new set of tokens or making the binary public" Seems a neat enough hack to make their newer shit available internally using existing tools/security.

lxe 6y ago

I need a medium account to read this for some reason.

tstrimple 6y ago
> "This is your last free member-only story this month. Sign up for Medium and get an extra one" Well, that's about the end of medium I think.
- kevin_thibedeau 6y ago
  Turn off Javascript. They don't (yet) have an SPA and it degrades appropriately for disabled users.
  - tstrimple 6y ago
    I'm not going to expend any effort to circumvent medium's nonsense. The content there isn't of consistently high enough quality to justify jumping through any hoops. Any barriers they put up will just result in me not consuming content hosted there.
    - freedomben 6y ago
      Open a private tab (or in Chrome "incognito") and it will work fine.
- simongr3dal 6y ago
  I wonder how “publishers” feel about that. I would feel rather angry if Medium dictated that people couldn’t access my blog posts.
  - tempodox 6y ago
    But that's exactly what to expect when people donate their writings to a platform they have no control over. There are always those who need to learn the hard way.

anonacct37 6y ago

I wish commentors would read the article. Why didn't they use the os package manager? Which OS? Their build clearly builds for osx, windows, and Linux.

bfung 6y ago
Or since it is an engineering CLI tool, why not have engineers: 1. clone cli repo 2. run goreleaser 3. copy binary to desired path sure, it's not 1 step (can be condensed with a build script), but it does teach the other engineers how to roll their own cli tool and contribute back if they fix a bug in it. Or if the blog indicated that not all engineers understand golang and build tools, and there's too many teams - that'd be good for blog context. Hopefully the author didn't spend much time in figuring out or being too clever in making this "hack".
- emeraldd 6y ago
  As the number of tools grow and the number of teams using/developing them grow this becomes more and more untenable. You not only have to maintain the tool itself, but also the build stacks associated with them and it becomes the task of the user to make sure they are running the most recent version. This leads to more complex support situations. Basically, you need a deployment methodology that allows for: * Ease of use by "novice" developers * Only introduces minimal overhead for those developers * Makes it easy to upgrade or remove a tool you installed 6 months ago * Doesn't require a tool development team to come up with their own solution This is a surprisingly tricky to do on a diverse environment with 1000+ internal users.
sagonar 6y ago
NPM could handle different OS, I would guess they run more than 1 os. (It is also a tool which the team is used to handle, as they have been using node + npm)

turminal 6y ago

This just saddens me deeply. Of all the ways this could have been done and all the package managers they could have used they picked npm. I bet some day we'll have a Linux distro with npm as the system package manager here on HN.

xena 6y ago
https://node-os.com/ https://node-os.com/
- turminal 6y ago
  Dang, I wish I could say I'm surprised, but I guess it was to be expected.
detaro 6y ago
It saddens you that someone used a tool they already use instead of adding another thing to distribute a small binary?
- turminal 6y ago
  I guess my comment appears rude, which wasn't intended, I just find npm a troublesome piece of software that is passable as a package manager for the javascript ecosystem, but not a tool one solves problems with if there are viable alternatives.
  - murukesh_s 6y ago
    why is it trouble? why can't they improve it even if its trouble?
  - deleted 6y ago
    [deleted]
  - folkrav 6y ago
    When you have one thing to nail and already own a hammer, why go and buy another one? Even if it's not a good one, it's gonna do the job, and be the least trouble.

dboreham 6y ago

Next: Docker images!

lmm 6y ago

I was very tempted to reuse a Maven repository for publishing the handful of small Python utilities we had. It's a standardised format, we already had the server set up, we already had different kinds of artifacts in there and multiple languages, and frankly Python-on-Maven dependency management can't be worse than what standard Python dependency management looks like.

coredog64 6y ago
I kinda do that today. We’re primarily a Java shop, but we’re using Python for FaaS. So we package up our code and dependencies into a zip and use Maven to push the resulting archive into our common repo.

pjmlp 6y ago

Go supports native packages natively via go get, why the hell bring npm into the picture?!?

woutr_be 6y ago
> As the project completed we ran into the issue of distributing the CLI tool to our team of engineers without the need of distributing a new set of tokens or making the binary public. From the article itself, it seems like they wanted to re-use their existing process without disruptions.
noselasd 6y ago
You'd use go get as a developer, not to update workstations and production machines.
- pjmlp 6y ago
  Same applies to using npm on workstations and production machines.

lrossi 6y ago

If all you have is a hammer, everything looks like a nail. [0] [0] Abraham H. Maslow (1966). The Psychology of Science.

deleted 6y ago
[deleted]

Fresh20210107 6y ago

Why use a language, created to keep interns safe and technical fellows from leaving, to begin with?

siscia 6y ago

At CERN we have an internal system to distribute software. It is mostly used to distribute software to production systems in the datacenters or to distribute nightlies and toolchains to developers. It replace standard package managers on most production systems. It is a FUSE filesystem, where we can install software once and have it distributed to a practically unlimited number of clients thanks to a simple and performant cache system. The filesystem is called CernVM-FileSystem, CVMFS for short.[1] I maintain a public instance of it under https://packages.redbeardlab.com https://packages.redbeardlab.com It contains "common" software, like standard unix tools, compilers, webservers, etc... I wrote a bit about it: https://redbeardlab.com/2020/12/20/announcing-packages-redbeardlab-com-a-public-cvmfs-repository-for-common-software/ https://redbeardlab.com/2020/12/20/announcing-packages-redbe... https://redbeardlab.com/2020/12/21/bootstrapping-packages-redbeardlab-com/ https://redbeardlab.com/2020/12/21/bootstrapping-packages-re... https://redbeardlab.com/2020/12/21/packages-redbeardlab-com-for-golang/ https://redbeardlab.com/2020/12/21/packages-redbeardlab-com-... https://redbeardlab.com/2020/12/23/packages-redbeardlab-com-for-node-js-community/ https://redbeardlab.com/2020/12/23/packages-redbeardlab-com-... [1]: https://cernvm.cern.ch/fs/ https://cernvm.cern.ch/fs/

beojan 6y ago
One of the best things about CVMFS is the idea of LCG views which are coherent installations of a large number of packages. This more or less eliminates the need for package management and guarantees that someone else can reproduce your environment. It's like Stackage LTS releases, but everything's installed already.
- siscia 6y ago
  Unfortunately I am not familiar enough with LCG and LCGmake to actually build an outside-CERN release. Fortunately Nix helped in creating the same coherent installation of packages. Then `source /cvmfs/packages.redbeardbal.com/setup.sh` give you the same feel of an LCG view, but with more widespread packages.

nerdponx 6y ago

There is obviously a need here that is not met by current tools. I believe that Conda is the closest thing to an answer to this need. It works on Windows, Mac and Linux. It supports packaging arbitrary binaries. It lets you specify dependencies that can be arbitrary libraries for arbitrary languages. And it has a relatively simple package format, build process, and repository format. It's not the best piece of software when it comes to user experience, and its niche is still very much data science. But it is a glimpse of what should be possible, but is not currently available.

notretarded 6y ago

>We recently migrated our internal CLI tool from Node JS to Golang Why? Out of things to do? Wanted to look productive?