6 ms·

One issue I see with Copilot is that they get free access to all open-source data on GitHub, but using GitHub APIs to download the data yourself isn't possible

One issue I see with Copilot is that they get free access to all open-source data on GitHub, but using GitHub APIs to download the data yourself isn't possible (rate limiting). This is an unfair advantage. Copilot is not only making money off of open-source, they are making money off of open-source in a way others can't. I would love to see a lawsuit which requires GitHub to provide their full Copilot dataset.

keithnz 4y ago
why use the API? why not just use git to get the code? All you need the API for is repository discovery
- rcoder 4y ago
  I'm gonna guess that Microsoft GitHub (tm) would shut you down pretty quickly if you tried to clone tens or hundreds of thousands of repos in a short window of time, b/c of course that's sketchy/abusive use of their infrastructure, right? But of course if the data is already sitting in object storage inside your cloud environment and all you have to do is run some MapReduce jobs to get at it... Hence: unfair, anticompetitive, intellectual-property-right-abusing behavior. Microsoft GitHub (tm) can prevent anyone else from running the kinds of analysis they do by simple "operational security", while running literally any kind of analysis, model training, etc. they want. Don't like it? But their commercial services and products so you can run Microsoft GitHub (tm) on your very own Microsoft Azure (tm) infrastructure, using Microsoft Visual Studio Code (tm) and Microsoft GitHub Codespaces (tm) so work on _your_ code privately. Best of all, you can still still take advantage of the huge library of "free" code offered by Microsoft GitHub Copilot (tm) to ensure your private, proprietary codebase still has all of the advantages of Open Source Software, brought to you exclusively by the Microsoft GitHub Platform (tm).
  - ketralnis 4y ago
    I don't understand. Your favourite boba joint can email every one of their customers a coupon. That's "unfair" to the other boba joints without access to their mailing list too, right? You're just describing a regular old competitive advantage
  - visarga 4y ago
    There are accessible code datasets that contain massive scrapes of Github.
  - EvanAnderson 4y ago
    > I'm gonna guess that Microsoft GitHub (tm) would shut you down pretty quickly if you tried to clone tens or hundreds of thousands of repos in a short window of time, b/c of course that's sketchy/abusive use of their infrastructure, right? ArchiveTeam has a distributed Github archive project[0]. It's unclear what the status is right now. It seems like a worthwhile idea. [0] https://wiki.archiveteam.org/index.php/GitHub#Archive_Team_project https://wiki.archiveteam.org/index.php/GitHub#Archive_Team_p...
  - prpl 4y ago
    Actually they don’t. I’ve cloned thousands of repos before (tried to archive conda-forge org for a project). I’ve also built many parallel repo downloaders for CI reasons. You can clone repos all day pretty much with little rate limiting. I haven’t pushed parallelism past 64 per host though
beachy 4y ago
Github does plenty of stuff that you can't do. e.g. the contributions graph as just one example that comes to mind. That's not unfair, and not the basis for a lawsuit, it's just business. > Copilot is not only making money off of open-source, they are making money off of open-source in a way others can't. Of course! That's why MS paid squillions to buy Github.
- treis 4y ago
  You're not supposed to be able to use dominance in one market (git hosting) to gain dominance in another (AI powered code suggestions).
  - wfme 4y ago
    What do you mean "You're not supposed to"? Is there some law that forbids this? From my (potentially naive) POV this seems to be roughly equivalent to asking physics professors to stay away from mathematics since they are likely to have some relevant cross-domain expertise.
    - mkr-hn 4y ago
      Microsoft has been nailed for this exact kind of thing in the past. This isn't hypothetical.
      - naikrovek 4y ago
        it only runs afowl of antitrust laws if it solidifies an unfair monopoly and/or uses market position as a monopoly holder to prevent fair competition from emerging. most monopolies are completely legal. small town with only one gas station or one grocery store? 100% monopoly, 100% legal. there are plenty of alternatives to github, paid, free, as a service, and self-hosted. GitHub has a large market share, but no monopoly.
        mkr-hn 4y ago
        This would, of course, need to be determined in court. Microsoft argued it had plenty of competition when the DOJ came for it over Windows and bundling. Like I said, this isn't unexplored territory for Microsoft. This is the company of Embrace, Extend, Extinguish and the Halloween documents. This isn't a company with plenty of goodwill in its sails launching a hip new product. They're not Do No Evil era Google or Apple riding high on the iPod. We can't just pretend there isn't a history.
citizenkeen 4y ago
It’s not making money off of open source, it’s making money off of hosting open source.
- armchairhacker 4y ago
  Yes. But the people who agree to let GitHub host their code are doing so with the expectation that it's "open source", i.e. freely accessible. If we define "open source" as "you can't necessarily use this to train an AI", then Copilot itself is illegal because it's using code without permission. If we define "open source" as "you can use this to train an AI", then Copilot it legal, but GitHub may be illegally misrepresenting itself as a host for "open source", as the policy it hosts code under isn't truly open-source. If we define "open source" as "you can't necessarily use this to train an AI" but then GitHub's policy explicitly states "by using us as a hosting provider, you give us permission to use your code to train AI" then they are in the clear. But I doubt they have that clause or at least had it when Copilot was first revealed.
- LtWorf 4y ago
  No? How much money would it make without using open source?
mtlynch 4y ago
Not a supporter of Copilot, but I think it's pretty easy to access the same data through BigQuery: >The Google BigQuery Public Datasets program now offers a full snapshot of the content of more than 2.8 million open source GitHub repositories in BigQuery. Thanks to our new collaboration with GitHub, you'll have access to analyze the source code of almost 2 billion files with a simple (or complex) SQL query. https://cloud.google.com/blog/topics/public-datasets/github-on-bigquery-analyze-all-the-open-source-code https://cloud.google.com/blog/topics/public-datasets/github-...
- wyaeld 4y ago
  There is a distinction between being able to access the source code, and a tool giving it to you without any context of the underlying license it is governed by.
  - mtlynch 4y ago
    GP was saying that GitHub has an unfair advantage in that they have instant access to all GitHub code, whereas everyone else is rate limited. I'm pointing out that this limitation is not meaningful because everyone can access all GitHub hosted source code through BigQuery, where they won't be rate limited. I'm not comparing BigQuery to Copilot.
Arbortheus 4y ago
That’s the problem with big tech. When they own all the roads, it makes organic competition almost impossible, as they will always be 2 steps ahead.
- nicksiscoe 4y ago
  Not to be anti-capitalist, but you’re describing capitalism lol
pornel 4y ago
Don't forget that GitHub is a closed-source proprietary commercial service. Their freemium product is useful to many open-source projects and communities, but you do not have any more rights to use Microsoft's GitHub than you have to Microsoft's Windows.
zo1 4y ago
I think the very least they could do is dump the list of URLs from which they took the source files. The community will figure out the rest.
Aeolun 4y ago
> using GitHub APIs to download the data yourself isn't possible Is it? Data storage would be prohibitive, but I can see ways to download the entirety of Github in a few weeks/months (assuming my size estimate is accurate).
- Aperocky 4y ago
  It's probably the reverse - the data can probably fit in a few commercially available hard drives, the API calls to discover and download all repository without running afoul of the rate limiters and whatever other anti-crawling strategies can take years if you're running it single threaded (a single IP can effectively be considered single threaded).
  - XorNot 4y ago
    There are publicly published datasets, updated regularly, and you can stream all the events happening on Github (which is how secret leaks happen) so you can stay abreast of new repositories. https://www.gharchive.org/ https://www.gharchive.org/ http://ghtorrent.org/ http://ghtorrent.org/ also available on GCP as a dataset, provided by Github itself: https://console.cloud.google.com/marketplace/details/github/github-repos?filter=solution-type:dataset&id=46ee22ab-2ca4-4750-81a7-3ee0f0150dcb https://console.cloud.google.com/marketplace/details/github/...
naikrovek 4y ago
competitive advantage is not a monopoly, nor is it unfair.
resonious 4y ago
Anyone can start their own GitHub competitor and do whatever they want with the source code that ends up on it. GitHub pays the bills and lets us freely upload whatever we want to their service, so it seems a bit entitled to complain about what features or data they provide or don't provide.
- bamboozled 4y ago
  I pay for Github because I thought it was a nice, reputable service who wouldn't go through my stuff without asking me first.
  - toastal 4y ago
    Whew... that username checks out.
    - bamboozled 4y ago
      You've done nothing to refute my point and so it still stands. I pay GitHub / Microsoft to host my code, and that's all I expect them to do with it, host it, as securely as possible. It sounds like Microsoft are doing more than this so what's your actual deal...if you have one?
      - ShamelessC 4y ago
        Read the EULA, not the marketing copy, next time.
        bamboozled 4y ago
        I did, what did I miss? I just re-read it and it's full of statements which are totally inline with what I expected: GitHub considers the contents of private repositories to be confidential to you. GitHub will protect the contents of private repositories from unauthorized use, access, or disclosure in the same manner that we would use to protect our own confidential information of a similar nature and in no event with less than a reasonable degree of care.

keithnz 4y ago

why use the API? why not just use git to get the code? All you need the API for is repository discovery

rcoder 4y ago
I'm gonna guess that Microsoft GitHub (tm) would shut you down pretty quickly if you tried to clone tens or hundreds of thousands of repos in a short window of time, b/c of course that's sketchy/abusive use of their infrastructure, right? But of course if the data is already sitting in object storage inside your cloud environment and all you have to do is run some MapReduce jobs to get at it... Hence: unfair, anticompetitive, intellectual-property-right-abusing behavior. Microsoft GitHub (tm) can prevent anyone else from running the kinds of analysis they do by simple "operational security", while running literally any kind of analysis, model training, etc. they want. Don't like it? But their commercial services and products so you can run Microsoft GitHub (tm) on your very own Microsoft Azure (tm) infrastructure, using Microsoft Visual Studio Code (tm) and Microsoft GitHub Codespaces (tm) so work on _your_ code privately. Best of all, you can still still take advantage of the huge library of "free" code offered by Microsoft GitHub Copilot (tm) to ensure your private, proprietary codebase still has all of the advantages of Open Source Software, brought to you exclusively by the Microsoft GitHub Platform (tm).
- ketralnis 4y ago
  I don't understand. Your favourite boba joint can email every one of their customers a coupon. That's "unfair" to the other boba joints without access to their mailing list too, right? You're just describing a regular old competitive advantage
- visarga 4y ago
  There are accessible code datasets that contain massive scrapes of Github.
- EvanAnderson 4y ago
  > I'm gonna guess that Microsoft GitHub (tm) would shut you down pretty quickly if you tried to clone tens or hundreds of thousands of repos in a short window of time, b/c of course that's sketchy/abusive use of their infrastructure, right? ArchiveTeam has a distributed Github archive project[0]. It's unclear what the status is right now. It seems like a worthwhile idea. [0] https://wiki.archiveteam.org/index.php/GitHub#Archive_Team_project https://wiki.archiveteam.org/index.php/GitHub#Archive_Team_p...
- prpl 4y ago
  Actually they don’t. I’ve cloned thousands of repos before (tried to archive conda-forge org for a project). I’ve also built many parallel repo downloaders for CI reasons. You can clone repos all day pretty much with little rate limiting. I haven’t pushed parallelism past 64 per host though

beachy 4y ago

Github does plenty of stuff that you can't do. e.g. the contributions graph as just one example that comes to mind. That's not unfair, and not the basis for a lawsuit, it's just business. > Copilot is not only making money off of open-source, they are making money off of open-source in a way others can't. Of course! That's why MS paid squillions to buy Github.

treis 4y ago
You're not supposed to be able to use dominance in one market (git hosting) to gain dominance in another (AI powered code suggestions).
- wfme 4y ago
  What do you mean "You're not supposed to"? Is there some law that forbids this? From my (potentially naive) POV this seems to be roughly equivalent to asking physics professors to stay away from mathematics since they are likely to have some relevant cross-domain expertise.
  - mkr-hn 4y ago
    Microsoft has been nailed for this exact kind of thing in the past. This isn't hypothetical.
    - naikrovek 4y ago
      it only runs afowl of antitrust laws if it solidifies an unfair monopoly and/or uses market position as a monopoly holder to prevent fair competition from emerging. most monopolies are completely legal. small town with only one gas station or one grocery store? 100% monopoly, 100% legal. there are plenty of alternatives to github, paid, free, as a service, and self-hosted. GitHub has a large market share, but no monopoly.
      - mkr-hn 4y ago
        This would, of course, need to be determined in court. Microsoft argued it had plenty of competition when the DOJ came for it over Windows and bundling. Like I said, this isn't unexplored territory for Microsoft. This is the company of Embrace, Extend, Extinguish and the Halloween documents. This isn't a company with plenty of goodwill in its sails launching a hip new product. They're not Do No Evil era Google or Apple riding high on the iPod. We can't just pretend there isn't a history.

citizenkeen 4y ago

It’s not making money off of open source, it’s making money off of hosting open source.

armchairhacker 4y ago
Yes. But the people who agree to let GitHub host their code are doing so with the expectation that it's "open source", i.e. freely accessible. If we define "open source" as "you can't necessarily use this to train an AI", then Copilot itself is illegal because it's using code without permission. If we define "open source" as "you can use this to train an AI", then Copilot it legal, but GitHub may be illegally misrepresenting itself as a host for "open source", as the policy it hosts code under isn't truly open-source. If we define "open source" as "you can't necessarily use this to train an AI" but then GitHub's policy explicitly states "by using us as a hosting provider, you give us permission to use your code to train AI" then they are in the clear. But I doubt they have that clause or at least had it when Copilot was first revealed.
LtWorf 4y ago
No? How much money would it make without using open source?

mtlynch 4y ago

Not a supporter of Copilot, but I think it's pretty easy to access the same data through BigQuery: >The Google BigQuery Public Datasets program now offers a full snapshot of the content of more than 2.8 million open source GitHub repositories in BigQuery. Thanks to our new collaboration with GitHub, you'll have access to analyze the source code of almost 2 billion files with a simple (or complex) SQL query. https://cloud.google.com/blog/topics/public-datasets/github-on-bigquery-analyze-all-the-open-source-code https://cloud.google.com/blog/topics/public-datasets/github-...

wyaeld 4y ago
There is a distinction between being able to access the source code, and a tool giving it to you without any context of the underlying license it is governed by.
- mtlynch 4y ago
  GP was saying that GitHub has an unfair advantage in that they have instant access to all GitHub code, whereas everyone else is rate limited. I'm pointing out that this limitation is not meaningful because everyone can access all GitHub hosted source code through BigQuery, where they won't be rate limited. I'm not comparing BigQuery to Copilot.

Arbortheus 4y ago

That’s the problem with big tech. When they own all the roads, it makes organic competition almost impossible, as they will always be 2 steps ahead.

nicksiscoe 4y ago
Not to be anti-capitalist, but you’re describing capitalism lol

pornel 4y ago

Don't forget that GitHub is a closed-source proprietary commercial service. Their freemium product is useful to many open-source projects and communities, but you do not have any more rights to use Microsoft's GitHub than you have to Microsoft's Windows.

zo1 4y ago

I think the very least they could do is dump the list of URLs from which they took the source files. The community will figure out the rest.

Aeolun 4y ago

> using GitHub APIs to download the data yourself isn't possible Is it? Data storage would be prohibitive, but I can see ways to download the entirety of Github in a few weeks/months (assuming my size estimate is accurate).

Aperocky 4y ago
It's probably the reverse - the data can probably fit in a few commercially available hard drives, the API calls to discover and download all repository without running afoul of the rate limiters and whatever other anti-crawling strategies can take years if you're running it single threaded (a single IP can effectively be considered single threaded).
- XorNot 4y ago
  There are publicly published datasets, updated regularly, and you can stream all the events happening on Github (which is how secret leaks happen) so you can stay abreast of new repositories. https://www.gharchive.org/ https://www.gharchive.org/ http://ghtorrent.org/ http://ghtorrent.org/ also available on GCP as a dataset, provided by Github itself: https://console.cloud.google.com/marketplace/details/github/github-repos?filter=solution-type:dataset&id=46ee22ab-2ca4-4750-81a7-3ee0f0150dcb https://console.cloud.google.com/marketplace/details/github/...

naikrovek 4y ago

competitive advantage is not a monopoly, nor is it unfair.

resonious 4y ago

Anyone can start their own GitHub competitor and do whatever they want with the source code that ends up on it. GitHub pays the bills and lets us freely upload whatever we want to their service, so it seems a bit entitled to complain about what features or data they provide or don't provide.

bamboozled 4y ago
I pay for Github because I thought it was a nice, reputable service who wouldn't go through my stuff without asking me first.
- toastal 4y ago
  Whew... that username checks out.
  - bamboozled 4y ago
    You've done nothing to refute my point and so it still stands. I pay GitHub / Microsoft to host my code, and that's all I expect them to do with it, host it, as securely as possible. It sounds like Microsoft are doing more than this so what's your actual deal...if you have one?
    - ShamelessC 4y ago
      Read the EULA, not the marketing copy, next time.
      - bamboozled 4y ago
        I did, what did I miss? I just re-read it and it's full of statements which are totally inline with what I expected: GitHub considers the contents of private repositories to be confidential to you. GitHub will protect the contents of private repositories from unauthorized use, access, or disclosure in the same manner that we would use to protect our own confidential information of a similar nature and in no event with less than a reasonable degree of care.