9 ms·

Backyourstack: discover and sponsor your open-source dependencies

277 points by vvoyer 8y ago

adrianN 8y ago
A similar tool with a focus on license compliance is fossology: https://www.fossology.org/ https://www.fossology.org/
JanisL 8y ago
Interesting project, would be great if this supported more than just JS projects, I'd be very interested to see some of the dependencies that my current business relies upon.
- Vinnl 8y ago
  They're working on that, contributions for C++ and Ruby have apparently already been received: https://medium.com/open-collective/open-collective-august-2018-newsletter-438db9761159#3c02 https://medium.com/open-collective/open-collective-august-20...
- michaelmior 8y ago
  Based on the fact that you can also upload composer.json files, I assume it also supports PHP.
- pferde 8y ago
  You mean you do not already know your software dependencies? A bit careless for running a business, if you ask me.
  - Spacemolte 8y ago
    Consider how many layers of dependencies are in use today, and you have no idea who the commenter is, what position he has in the business you assume he is 'running'.
    - pferde 8y ago
      If they're running that business, they are either paying people who can tell them the dependencies (if not, those people are being paid too much), or they are responsible for the code themselves (small co. or early startup), and should already know, or with small effort be able to identify all the dependencies whenever necessary. Anything else would be irresponsible. If they're not running that business and have enough access to the code to be able to use this service, they're probably a developer who again should be able to know about the dependencies. Or perhaps they're in a different position, and only want to know about this for curiosity's sake - in which case, I guess this could be useful. But still, you should be able to ask about this in-house. And as for the "layers" argument - if there are too many layers of dependencies to keep track of, something is very, very wrong with the technology you are using. (And yes, I do consider modern web tech completely insane.)
      - Vinnl 8y ago
        Which technology are you using that does not have this problem?
        ajdlinux 8y ago
        I work on the opposite end from the web stack - firmware development - and even we have layers upon layers of dependencies that are problematic to track.
        pferde 8y ago
        The point is not that the deps are problematic to track - that is to be expected with any larger project. The point is actually doing the tracking, and being aware when the deps change, new one gets added, etc., and being able to determine what it means for the business.
        Vinnl 8y ago
        So what technology are you using that makes that viable?
  - JanisL 8y ago
    My initial impression of the product was that it was a way to see which packages you depend on are looking for support in terms of funding/support. On a non-technical level this would be really good information for us to know because it would really help us make better open source investments. On a technical level we do the best we can to get to know all the package dependencies and the various other things that we have to be aware of such as security and license issues (we are aware that not everyone does this). But honestly even when you decide to take this seriously it is still hard. Because of transitive dependencies and long chains of dependencies in the libraries you use, it is hard to be able to have a high confidence that you know everything you depend on. Essentially this requires tooling of some form or another to have any chance. With our in-house code we know what our direct dependencies are and we usually can track down most of what we need fairly quickly because we control our build environments. However when we go consult for a clients who have large codebases that we have never seen before it takes a while to track everything down. Sometimes if people have customized build systems this can be really hard to do. We have our own static analysis tools to help with this too. Even without all the online package management services it's still a hard problem, consider the case of the humble makefile (http://lcgapp.cern.ch/project/architecture/recursive_make.pdf http://lcgapp.cern.ch/project/architecture/recursive_make.pd...) Add in dependencies on remote computers and this gets harder. Take for example Python with it's huge number of different ways of installing a package, we have some tooling to check things but it's probably not 100% accurate because of the various ways in which Python packaging is broken exacerbated by the various ways in which people have worked around these shortcomings in the past. Pipenv has helped with the lock files but not everything is using those. The power of good tools for package analysis is clear and we use whatever we can. I hope you will see that this is actually a hard problem in a business sense as it costs substantial amounts of time for a business to create tooling for these things and the customer is likely unable to assess the benefits of this directly. We have an obligation and a desire to bring value to our customers and this means we will sometimes have to prioritize a 100% coverage of package information less than other objectives if the client demands it (for example fixing mission critical bugs may be a higher priority). In an ideal world we would love to know every aspect of the stack that we run on but as time goes on the increasing complexity of the systems we use makes this harder.
- ariehkovler 8y ago
  I'm working with a pre-launch company that's building a tool to map companies' stacks (among other things). They've shared some of their research with me from and it's insane. Large software companies have no idea what dependencies, libraries, or even languages are in their stacks. Companies are using multiple versions of the same library, replicated in a bunch of different repos. Different teams in he same company end up re-implementing solutions over and over, sometimes in a whole other framework than the team next to them because nobody knows what anyone else is using. Makes compliance a nightmare. It's wild out there.
  - justicz 8y ago
    It’s even more wild when you consider that these companies are relying on all of these dependencies to be secure, and to lack intentional backdoors.
- XiZhao 8y ago
  Kevin from https://fossa.io/ https://fossa.io/ here. We have an open source project just for this: https://github.com/fossas/fossa-cli https://github.com/fossas/fossa-cli. It currently supports roughly 20+ build systems and languages, and pairs with our web service for license and vulnerability discovery. Would love your feedback.
  - mbesto 8y ago
    Wait, can you do this as a one-time event without a paid subscription to your service?
  - remram 8y ago
    You claim this is open-source, however I don't see an easy way to run this myself without relying on your infrastructure and signup up for an account there. Your "analyse locally" option requires an API key and is therefore very much misnamed.
- tajen 8y ago
  There’s a bit of irony in doing a crawler to give a few dollars to a JS library, when a lot of us run Linux (either on desktop or server) of abolutely immense value. What would one do without SSH? without Apache? without V8? without the Linux core? without Git? It’s great to fund the libraries, but don’t forget your OS, IDE, compiler, runtime etc.
mirekrusin 8y ago
It would be nice if you could paste package-lock file for private repos.
- kubami 8y ago
  You can do that! Check out the section below the button.
fs111 8y ago
Can we add javascript to the title? It is a bit misleading without it.
- onion2k 8y ago
  composer.json files are PHP.
- deadbunny 8y ago
  Yup, tested it out on a few python projects I have/use and it has 0 dependencies listed which is very, very wrong.
  - znarfor 8y ago
    We effectively don't support Python yet but it should not be far away. It's 100% Open Source and we're looking for contributions. https://github.com/opencollective/backyourstack/issues/34 https://github.com/opencollective/backyourstack/issues/34
    - ajdlinux 8y ago
      Understandable that you're adding support for languages and packaging ecosystems as you go, and JS + PHP is as good a place as any to start, but it would be helpful if this were explicitly highlighted. For those who are unfamiliar with the ecosystems in question, package.json and composer.json are just filenames that don't actually tell you anything.
      - anoncoward111 8y ago
        It would be helpful, and the site creator should list it. However, let's not treat this as a fault. They're offering you a free service and can't launch with every issue already resolved before being identified.
        deleted 8y ago
        [deleted]
hucker 8y ago
This is a really good idea! I assume support for more languages / package managers is coming?
kanzure 8y ago
I've been wondering whether a for-pay alternative to the open-source ecosystem could be developed. The problem seems to be that open-source gratis software contributes nearly zero friction to a company building out its tech, so any alternative would have to compete against that near-zero friction. I just don't see each company negotiating separate prices with 100,000 package maintainers to use all of their software on a custom linux distro just for one of their internal servers or whatever. It's a tremendous amount of friction for each company to bear. If that friction could be eliminated, while keeping a requirement to pay for use of the software, then I think a non-gratis ecosystem could dwarf the gratis software world within two or three years from its launch.
- wongarsu 8y ago
  How would that be different from the regular Microsoft Ecosystem? I can spin up a quick testserver using Windows Server, running an IIS webserver and Microsoft SQL Server, with my software stack written in C#, programmed in Visual Studio. It covers pretty much everything I could need, and involves ordering à la carte instead of negotiations. Obviously it costs more and unless I spin up a cloud server I can't "just spin up a server" without making sure I have enough licences etc. But AWS/GCP/Azure solve that mostly.
  - kanzure 8y ago
    My understanding is that Microsoft does not auto-publish+sell third-party libraries in a package repository. So the difference is basically the distinction made in the old cathedral/bazaar analogies.
    - wongarsu 8y ago
      They do have nuget for packages, but that doesn't have an integrated payment system. Despite this, there are plenty of paid C# packages you can buy around the web.
  - rpeden 8y ago
    You can do pretty much the same with ASP.NET core + Kestrel, too. Substitute Postgres for SQL Server if you want to stay all open source. But even with SQL Server, you can deploy the whole stack on Linux.
oliwarner 8y ago
> If you want to analyze non-public repositories, sign in with your GitHub account Do people really expose their or their employer's source code to random third party convenience services? I do understand the convenience factor here, I just think it's dodgy to encourage developers to be so flippant with privileged access.
- dschep 8y ago
  It also has the option to upload a package.json which is far less exposure, and can easily be tweaked to omit anything sensitive.
  - oliwarner 8y ago
    It's still not no-exposure, and it's still getting a developer in the mindset that just sending out files on a whim is okay. I'm not wholly against this sort of stuff, I'm sure we've used similar links in the past for CI and coverage, but it seems to be the end of the slope, where we're handing out access to our stuff for something so frivolous. This is the same sort of mechanism that got everybody and their dog's copies of Windows XP infected with trojans in the early 2000s. "Sure, I'll install that toolbar, just let me see Britney naked". This could be a local, auditable script that fetched a static list of projects seeking funding.
    - thatjsguy 8y ago
      To be fair, I don’t think this is intentionally bad privacy-wise. I think that, uh, a lot of the JavaScript set doesn’t actually know how to write code that doesn’t involve uploading shit to websites. When all you have is a hammer...
- is_true 8y ago
  That's a bit too much. Shouldn't even be an option.
nickjj 8y ago
I tried putting in my github account on the home page just to see what would happen. I didn't create an account or sign in, but it created a public profile on your domain using my name without my consent. Is there any way to remove that?
- znarfor 8y ago
  In that case, it's just reading the public data from GitHub on demand, there is no account, nothing stored.
  - nickjj 8y ago
    Ah ok, it looked like maybe it was created and cached and then publicly accessible at backyourstack.com/foo since that's the URL it forwards you to. You might want to anonymize that URL for accounts who don't sign up just for clarity.
deleted 8y ago
[deleted]
jhare 8y ago
I think we need less "awareness" and promotion, just more work on peoples' parts. Seems these projects are asking for a magic bullet to improve their stacks but find it's missing in their own time and efforts
cavneb 8y ago
This is absolutely amazing!!
rubbingalcohol 8y ago
This is a great tool, and going to keep tabs on it for future use. I appreciated the package.json upload for my private repos. Kudos to whoever built it!
themtutty 8y ago
The site offers to analyze your composer.json file, but doesn't seem to identify even popular libs like Monolog and Doctrine
- znarfor 8y ago
  Monolog and Doctrine should be properly detected. After that, we need to match detected dependencies with fundraising strategies (today Open Collective, in the future Patreon and others). Maybe this is why they're not appearing in the "Projects Requiring Funding" section. Feel free to submit an issue on GitHub and we can look at that in details.
phyzome 8y ago
« 56 repositories depending on 0 Open Source projects. » Well, that's not true. Maybe you could indicate which package systems you actually are able to analyze?
- znarfor 8y ago
  You're right, it's a common feedback we have, we even have an issue opened for that https://github.com/opencollective/backyourstack/issues/57 https://github.com/opencollective/backyourstack/issues/57 We explain what we support in the FAQ but that's not enough. What languages / packages managers are you expecting?
  - phyzome 8y ago
    Common ones, I suppose! I have projects involving Java, Python, and Clojure. Of those, I would expect the first two to be covered by a generic dependency analyzer. (It's fine if it doesn't, but it would be nice to know up front.)
hnruss 8y ago
Great idea! It's nice to see an easy way to support open-source software. Regarding funding open-source software: Companies I've worked for have all been OK with purchasing licenses for software that saves development time. They've also been careful to abide by software license terms. I'm surprised that more open-source libraries/frameworks don't require the purchase of a commercial license in order to use them commercially.

277 points by vvoyer 8y ago

A similar tool with a focus on license compliance is fossology: https://www.fossology.org/ https://www.fossology.org/

Interesting project, would be great if this supported more than just JS projects, I'd be very interested to see some of the dependencies that my current business relies upon.

Vinnl 8y ago
They're working on that, contributions for C++ and Ruby have apparently already been received: https://medium.com/open-collective/open-collective-august-2018-newsletter-438db9761159#3c02 https://medium.com/open-collective/open-collective-august-20...
michaelmior 8y ago
Based on the fact that you can also upload composer.json files, I assume it also supports PHP.
pferde 8y ago
You mean you do not already know your software dependencies? A bit careless for running a business, if you ask me.
- Spacemolte 8y ago
  Consider how many layers of dependencies are in use today, and you have no idea who the commenter is, what position he has in the business you assume he is 'running'.
  - pferde 8y ago
    If they're running that business, they are either paying people who can tell them the dependencies (if not, those people are being paid too much), or they are responsible for the code themselves (small co. or early startup), and should already know, or with small effort be able to identify all the dependencies whenever necessary. Anything else would be irresponsible. If they're not running that business and have enough access to the code to be able to use this service, they're probably a developer who again should be able to know about the dependencies. Or perhaps they're in a different position, and only want to know about this for curiosity's sake - in which case, I guess this could be useful. But still, you should be able to ask about this in-house. And as for the "layers" argument - if there are too many layers of dependencies to keep track of, something is very, very wrong with the technology you are using. (And yes, I do consider modern web tech completely insane.)
    - Vinnl 8y ago
      Which technology are you using that does not have this problem?
      - ajdlinux 8y ago
        I work on the opposite end from the web stack - firmware development - and even we have layers upon layers of dependencies that are problematic to track.
        pferde 8y ago
        The point is not that the deps are problematic to track - that is to be expected with any larger project. The point is actually doing the tracking, and being aware when the deps change, new one gets added, etc., and being able to determine what it means for the business.
        Vinnl 8y ago
        So what technology are you using that makes that viable?
- JanisL 8y ago
  My initial impression of the product was that it was a way to see which packages you depend on are looking for support in terms of funding/support. On a non-technical level this would be really good information for us to know because it would really help us make better open source investments. On a technical level we do the best we can to get to know all the package dependencies and the various other things that we have to be aware of such as security and license issues (we are aware that not everyone does this). But honestly even when you decide to take this seriously it is still hard. Because of transitive dependencies and long chains of dependencies in the libraries you use, it is hard to be able to have a high confidence that you know everything you depend on. Essentially this requires tooling of some form or another to have any chance. With our in-house code we know what our direct dependencies are and we usually can track down most of what we need fairly quickly because we control our build environments. However when we go consult for a clients who have large codebases that we have never seen before it takes a while to track everything down. Sometimes if people have customized build systems this can be really hard to do. We have our own static analysis tools to help with this too. Even without all the online package management services it's still a hard problem, consider the case of the humble makefile (http://lcgapp.cern.ch/project/architecture/recursive_make.pdf http://lcgapp.cern.ch/project/architecture/recursive_make.pd...) Add in dependencies on remote computers and this gets harder. Take for example Python with it's huge number of different ways of installing a package, we have some tooling to check things but it's probably not 100% accurate because of the various ways in which Python packaging is broken exacerbated by the various ways in which people have worked around these shortcomings in the past. Pipenv has helped with the lock files but not everything is using those. The power of good tools for package analysis is clear and we use whatever we can. I hope you will see that this is actually a hard problem in a business sense as it costs substantial amounts of time for a business to create tooling for these things and the customer is likely unable to assess the benefits of this directly. We have an obligation and a desire to bring value to our customers and this means we will sometimes have to prioritize a 100% coverage of package information less than other objectives if the client demands it (for example fixing mission critical bugs may be a higher priority). In an ideal world we would love to know every aspect of the stack that we run on but as time goes on the increasing complexity of the systems we use makes this harder.
ariehkovler 8y ago
I'm working with a pre-launch company that's building a tool to map companies' stacks (among other things). They've shared some of their research with me from and it's insane. Large software companies have no idea what dependencies, libraries, or even languages are in their stacks. Companies are using multiple versions of the same library, replicated in a bunch of different repos. Different teams in he same company end up re-implementing solutions over and over, sometimes in a whole other framework than the team next to them because nobody knows what anyone else is using. Makes compliance a nightmare. It's wild out there.
- justicz 8y ago
  It’s even more wild when you consider that these companies are relying on all of these dependencies to be secure, and to lack intentional backdoors.
XiZhao 8y ago
Kevin from https://fossa.io/ https://fossa.io/ here. We have an open source project just for this: https://github.com/fossas/fossa-cli https://github.com/fossas/fossa-cli. It currently supports roughly 20+ build systems and languages, and pairs with our web service for license and vulnerability discovery. Would love your feedback.
- mbesto 8y ago
  Wait, can you do this as a one-time event without a paid subscription to your service?
- remram 8y ago
  You claim this is open-source, however I don't see an easy way to run this myself without relying on your infrastructure and signup up for an account there. Your "analyse locally" option requires an API key and is therefore very much misnamed.
tajen 8y ago
There’s a bit of irony in doing a crawler to give a few dollars to a JS library, when a lot of us run Linux (either on desktop or server) of abolutely immense value. What would one do without SSH? without Apache? without V8? without the Linux core? without Git? It’s great to fund the libraries, but don’t forget your OS, IDE, compiler, runtime etc.

mirekrusin 8y ago

It would be nice if you could paste package-lock file for private repos.

kubami 8y ago
You can do that! Check out the section below the button.

fs111 8y ago

Can we add javascript to the title? It is a bit misleading without it.

onion2k 8y ago
composer.json files are PHP.
deadbunny 8y ago
Yup, tested it out on a few python projects I have/use and it has 0 dependencies listed which is very, very wrong.
- znarfor 8y ago
  We effectively don't support Python yet but it should not be far away. It's 100% Open Source and we're looking for contributions. https://github.com/opencollective/backyourstack/issues/34 https://github.com/opencollective/backyourstack/issues/34
  - ajdlinux 8y ago
    Understandable that you're adding support for languages and packaging ecosystems as you go, and JS + PHP is as good a place as any to start, but it would be helpful if this were explicitly highlighted. For those who are unfamiliar with the ecosystems in question, package.json and composer.json are just filenames that don't actually tell you anything.
    - anoncoward111 8y ago
      It would be helpful, and the site creator should list it. However, let's not treat this as a fault. They're offering you a free service and can't launch with every issue already resolved before being identified.
      - deleted 8y ago
        [deleted]

hucker 8y ago

This is a really good idea! I assume support for more languages / package managers is coming?

kanzure 8y ago

I've been wondering whether a for-pay alternative to the open-source ecosystem could be developed. The problem seems to be that open-source gratis software contributes nearly zero friction to a company building out its tech, so any alternative would have to compete against that near-zero friction. I just don't see each company negotiating separate prices with 100,000 package maintainers to use all of their software on a custom linux distro just for one of their internal servers or whatever. It's a tremendous amount of friction for each company to bear. If that friction could be eliminated, while keeping a requirement to pay for use of the software, then I think a non-gratis ecosystem could dwarf the gratis software world within two or three years from its launch.

wongarsu 8y ago
How would that be different from the regular Microsoft Ecosystem? I can spin up a quick testserver using Windows Server, running an IIS webserver and Microsoft SQL Server, with my software stack written in C#, programmed in Visual Studio. It covers pretty much everything I could need, and involves ordering à la carte instead of negotiations. Obviously it costs more and unless I spin up a cloud server I can't "just spin up a server" without making sure I have enough licences etc. But AWS/GCP/Azure solve that mostly.
- kanzure 8y ago
  My understanding is that Microsoft does not auto-publish+sell third-party libraries in a package repository. So the difference is basically the distinction made in the old cathedral/bazaar analogies.
  - wongarsu 8y ago
    They do have nuget for packages, but that doesn't have an integrated payment system. Despite this, there are plenty of paid C# packages you can buy around the web.
- rpeden 8y ago
  You can do pretty much the same with ASP.NET core + Kestrel, too. Substitute Postgres for SQL Server if you want to stay all open source. But even with SQL Server, you can deploy the whole stack on Linux.

oliwarner 8y ago

> If you want to analyze non-public repositories, sign in with your GitHub account Do people really expose their or their employer's source code to random third party convenience services? I do understand the convenience factor here, I just think it's dodgy to encourage developers to be so flippant with privileged access.

dschep 8y ago
It also has the option to upload a package.json which is far less exposure, and can easily be tweaked to omit anything sensitive.
- oliwarner 8y ago
  It's still not no-exposure, and it's still getting a developer in the mindset that just sending out files on a whim is okay. I'm not wholly against this sort of stuff, I'm sure we've used similar links in the past for CI and coverage, but it seems to be the end of the slope, where we're handing out access to our stuff for something so frivolous. This is the same sort of mechanism that got everybody and their dog's copies of Windows XP infected with trojans in the early 2000s. "Sure, I'll install that toolbar, just let me see Britney naked". This could be a local, auditable script that fetched a static list of projects seeking funding.
  - thatjsguy 8y ago
    To be fair, I don’t think this is intentionally bad privacy-wise. I think that, uh, a lot of the JavaScript set doesn’t actually know how to write code that doesn’t involve uploading shit to websites. When all you have is a hammer...
is_true 8y ago
That's a bit too much. Shouldn't even be an option.

nickjj 8y ago

I tried putting in my github account on the home page just to see what would happen. I didn't create an account or sign in, but it created a public profile on your domain using my name without my consent. Is there any way to remove that?

znarfor 8y ago
In that case, it's just reading the public data from GitHub on demand, there is no account, nothing stored.
- nickjj 8y ago
  Ah ok, it looked like maybe it was created and cached and then publicly accessible at backyourstack.com/foo since that's the URL it forwards you to. You might want to anonymize that URL for accounts who don't sign up just for clarity.

deleted 8y ago

[deleted]

jhare 8y ago

I think we need less "awareness" and promotion, just more work on peoples' parts. Seems these projects are asking for a magic bullet to improve their stacks but find it's missing in their own time and efforts

cavneb 8y ago

This is absolutely amazing!!

rubbingalcohol 8y ago

This is a great tool, and going to keep tabs on it for future use. I appreciated the package.json upload for my private repos. Kudos to whoever built it!

themtutty 8y ago

The site offers to analyze your composer.json file, but doesn't seem to identify even popular libs like Monolog and Doctrine

znarfor 8y ago
Monolog and Doctrine should be properly detected. After that, we need to match detected dependencies with fundraising strategies (today Open Collective, in the future Patreon and others). Maybe this is why they're not appearing in the "Projects Requiring Funding" section. Feel free to submit an issue on GitHub and we can look at that in details.

phyzome 8y ago

« 56 repositories depending on 0 Open Source projects. » Well, that's not true. Maybe you could indicate which package systems you actually are able to analyze?

znarfor 8y ago
You're right, it's a common feedback we have, we even have an issue opened for that https://github.com/opencollective/backyourstack/issues/57 https://github.com/opencollective/backyourstack/issues/57 We explain what we support in the FAQ but that's not enough. What languages / packages managers are you expecting?
- phyzome 8y ago
  Common ones, I suppose! I have projects involving Java, Python, and Clojure. Of those, I would expect the first two to be covered by a generic dependency analyzer. (It's fine if it doesn't, but it would be nice to know up front.)

hnruss 8y ago

Great idea! It's nice to see an easy way to support open-source software. Regarding funding open-source software: Companies I've worked for have all been OK with purchasing licenses for software that saves development time. They've also been careful to abide by software license terms. I'm surprised that more open-source libraries/frameworks don't require the purchase of a commercial license in order to use them commercially.