7 ms·

> Microsoft GitHub is the largest collection of open source code in the world. Microsoft GitHub is in a unique and dominant positions to host and access and dis

> Microsoft GitHub is the largest collection of open source code in the world. Microsoft GitHub is in a unique and dominant positions to host and access and distribute most of the open-source code in the world No, it's not in a "unique and dominant position". Open source code is freely available online, it's almost trivial to build a bot to scrape OS code from anywhere on the web (GitHub included). The comparison to the Google Books antitrust falls down completely, Google had a dominant position because it had the resources to scan all books. Anyone can build a collection of almost all open source code. Further to that, all these models (GPT and Image generation) are trained on scraped data, trying to suggest that only GitHub/Microsoft could do it defeats the purpose of trying to establish what the legal rights are over training models with scraped data. We need test cases and precedent, but trying to use this as one is not going to work. Edit: It took me 15 seconds to find that there is a Google Big Query dataset of open source code for GitHub: https://cloud.google.com/blog/topics/public-datasets/github-on-bigquery-analyze-all-the-open-source-code https://cloud.google.com/blog/topics/public-datasets/github-... and thats been further curated on Hugging Face: https://huggingface.co/datasets/codeparrot/github-code https://huggingface.co/datasets/codeparrot/github-code GitHub / Microsoft do not have a monopoly on this data.

marginalia_nu 3y ago
> It's almost trivial to built a bot to scrape OS code from anywhere on the web. Seems like a logistical nightmare to me. Git repos interact spectacularly poorly with web scraping in general.
- mewpmewp2 3y ago
  I would've said you should download only archives, but really I think commits are also very important data since that shows the actual changes in the code which would be very useful to train AI to suggest changes to the code.
  - marginalia_nu 3y ago
    There are valid non-evil reasons for git hosts to want to throttle and put up obstacles toward scraping as well, both via crawlers or 'git clone' or whatever. These are very expensive operations.
- flockonus 3y ago
  It appears to be the exact opposite to me, `git clone --depth 1 ...` will give you a code that you can know exactly how to parse, vs. webpages that have all sorts of semantical issues.
  - marginalia_nu 3y ago
    Git clone is a very expensive operation. Git hosts generally will try to prohibit mass git clone:ing for this reason.
    - blowski 3y ago
      What makes it so expensive? I’d always assumed it downloaded the .git directory statically, and the computational bits were down by the local client.
      - ablob 3y ago
        I'd assume this is in relation to how much other operations cost. With 'git clone' you at least download the whole repository. Compare that to 'git fetch', which is essentially a lookup at the last-modified timestamp.
        marginalia_nu 3y ago
        Yeah. Git repositories can grow very large very quickly. A single clone here and there isn't too bad, but if you're scraping tens of thousands of projects, you can easily rack up terabytes in disk and network access.
        deleted 3y ago
        [deleted]
- moneywoes 3y ago
  How so? Can’t someone just download the zip file and make a queue of downloads or does GitHub rate limit?
cassianoleal 3y ago
> We need test cases and president I imagine you meant "precedent".
jackdaniel 3y ago
This is addressed in the same paragraph - you can't scan/download "whole" github because you'll be throttled.
- williamcotton 3y ago
  There’s no need to crawl for your own dataset: https://pile.eleuther.ai/ https://pile.eleuther.ai/
  - hanselot 3y ago
    @article{pile, title={The {P}ile: An 800GB Dataset of Diverse Text for Language Modeling}, author={Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and Presser, Shawn and Leahy, Connor}, journal={arXiv preprint arXiv:2101.00027}, year={2020} } So if I understand this correctly, the Pile is for code from 2020 backwards? If I wanted anything released in the past 3 years, say something in the SOTA AI space (where a month is a lifetime), I would need the scraper again? I don't follow how this can compare to direct, live, unrestricted access. I suppose this is just my own hatred of Microsoft shining through. Of course we should accept the status quo, because how dare we suggest Microsoft could operate in a manner that is anti-competitive. For anyone else trying to catch up, just rent a datacenter, write a crawler, deal with all the intricacies of keeping it in sync in real-time. This sounds trivial, simple even. I wonder why nobody is doing it? Perhaps everyone doesn't have access to petabytes of storage space, unlimited bandwidth, unlimited proxy-jumps etc. So the alternative is to buy github?
    - williamcotton 3y ago
      I wonder why nobody is doing it? Perhaps everyone doesn't have access to petabytes of storage space, unlimited bandwidth, unlimited proxy-jumps etc. There are multiple private companies and public institutions that are currently training LLMs. The work that it required to train an LLM is actually in support of fair use, just as it was with regards to Google scanning books.
- 3y ago
mschild 3y ago
There certainly are other spaces where open source code is hosted and available, but the default for most is GitHub. I think it's in a similar position to Google 10 years ago. Sure there are other search engines, but Google is by and large the standard one. That does put Microsoft in the unique position to have direct unfettered access to any and all open source code on GitHub without restrictions. Unless you or I get the same kind of direct access without rate limiting and antibot protection, then they do dominate and have an advantage over everyone else.
- reissbaker 3y ago
  Not sure if you posted before the edits, but I'm pretty convinced by them, seeing as how there are multiple alternatives with the same data.
- scarface74 3y ago
  it’s really not that hard to git clone git set origin… It’s much harder to copy Google’s index.
  - ChatGTP 3y ago
    You think it's practical to do this with almost all the public repos on Github?
    - scott_w 3y ago
      That's not Github's fault or Github's problem, from an antitrust perspective. If they went out of their way to make it difficult, you might have an argument but, as far as I know, they aren't. It's just practically difficult by the nature of the problem.
      - zanellato19 3y ago
        They rate limit, so they do make it difficult though
eternalban 3y ago
> The comparison to the Google Books antitrust falls down completely, Google had a dominant position because it had the resources to scan all books. Anyone can build a collection of almost all open source code. Copying a file is not the same thing as "scanning" a book. To scan you first need to get your hands on the book (the download part) and then use industrial scanners to scan them. So apple-apple comparison here is scanning <-> training & scanned collection of books <-> trained model, and finally the portals to the loot: Google Books <~> Github+VSC. Not everyone has the resources to actually process -- that is train the 'model' -- using the publicly available 'data'. Most also don't also own Github and VSC platforms to field their model. In fact, is anyone other than microsoft in a position to both scrape OSS, train a coding AI, and then include that tool in dominant software development platforms?
bilqis 3y ago
Github also has access to private repositories.
- samwillis 3y ago
  They don't use privet repositories to train Copilot.
  - bilqis 3y ago
    They say they don’t
    - esrauch 3y ago
      Why would they possibly lie about that?
      - ChatGTP 3y ago
        Because they do shady shit, like, by default Copilot would "sample" code for training while using it. Maybe this is no longer the default, maybe it still is, but it was the default. This type of thing erodes trust? Why should my proprietary code be used for training by default? I was really annoyed by this.
        scrollaway 3y ago
        OpenAI is not the same company as GitHub, and it has always been pretty clear that chats on ChatGPT are recorded and used for training (unless you now opt out).
        ChatGTP 3y ago
        Not sure why you're bringing OpenAI into it. My comment and the article is about "Copilot" I'm talking about when using "Github Copilot" and you ask for a code suggestion, it would send the "context" back to GitHub / Microsoft and use that code as training. Your comment is interesting to me though because there does seem to be a surprisingly large amount of defending OpenAI going on. Almost seems automatic now.
rjmunro 3y ago
> Google had a dominant position because it had the resources to scan all books. I thought Google had a dominant position because they signed an exclusive deal with the authors guild that explicitly gave them a dominant position. Anyone else could set up a project to go round libraries and scan books. Google has put more money into it than other organisations, but The Internet Archive has about 20 million scans (https://archive.org/details/texts https://archive.org/details/texts).
goodpoint 3y ago
> No, it's not in a "unique and dominant position". Open source code is freely available online, it's almost trivial to build a bot to scrape OS code from anywhere on the web (GitHub included). Absolutely wrong. GitHub is doing way more than just hosting code. It hosts bugtrackers, CI and much more. For most FOSS project it's the ONLY place where you can go and submit a bug report. It's not just a repository, it's a communication tool and refuses to interoperate with other platform. This is monopoly, just like NPM and Linkedin. Microsoft never changes.
toastal 3y ago
Microsoft GitHub has access to all the commits you force pushed away or branch you deleted. We have no reason to believe that it’s actually gone with no transparency and the source code being closed.

marginalia_nu 3y ago

> It's almost trivial to built a bot to scrape OS code from anywhere on the web. Seems like a logistical nightmare to me. Git repos interact spectacularly poorly with web scraping in general.

mewpmewp2 3y ago
I would've said you should download only archives, but really I think commits are also very important data since that shows the actual changes in the code which would be very useful to train AI to suggest changes to the code.
- marginalia_nu 3y ago
  There are valid non-evil reasons for git hosts to want to throttle and put up obstacles toward scraping as well, both via crawlers or 'git clone' or whatever. These are very expensive operations.
flockonus 3y ago
It appears to be the exact opposite to me, `git clone --depth 1 ...` will give you a code that you can know exactly how to parse, vs. webpages that have all sorts of semantical issues.
- marginalia_nu 3y ago
  Git clone is a very expensive operation. Git hosts generally will try to prohibit mass git clone:ing for this reason.
  - blowski 3y ago
    What makes it so expensive? I’d always assumed it downloaded the .git directory statically, and the computational bits were down by the local client.
    - ablob 3y ago
      I'd assume this is in relation to how much other operations cost. With 'git clone' you at least download the whole repository. Compare that to 'git fetch', which is essentially a lookup at the last-modified timestamp.
      - marginalia_nu 3y ago
        Yeah. Git repositories can grow very large very quickly. A single clone here and there isn't too bad, but if you're scraping tens of thousands of projects, you can easily rack up terabytes in disk and network access.
      - deleted 3y ago
        [deleted]
moneywoes 3y ago
How so? Can’t someone just download the zip file and make a queue of downloads or does GitHub rate limit?

cassianoleal 3y ago

> We need test cases and president I imagine you meant "precedent".

jackdaniel 3y ago

This is addressed in the same paragraph - you can't scan/download "whole" github because you'll be throttled.

williamcotton 3y ago
There’s no need to crawl for your own dataset: https://pile.eleuther.ai/ https://pile.eleuther.ai/
- hanselot 3y ago
  @article{pile, title={The {P}ile: An 800GB Dataset of Diverse Text for Language Modeling}, author={Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and Presser, Shawn and Leahy, Connor}, journal={arXiv preprint arXiv:2101.00027}, year={2020} } So if I understand this correctly, the Pile is for code from 2020 backwards? If I wanted anything released in the past 3 years, say something in the SOTA AI space (where a month is a lifetime), I would need the scraper again? I don't follow how this can compare to direct, live, unrestricted access. I suppose this is just my own hatred of Microsoft shining through. Of course we should accept the status quo, because how dare we suggest Microsoft could operate in a manner that is anti-competitive. For anyone else trying to catch up, just rent a datacenter, write a crawler, deal with all the intricacies of keeping it in sync in real-time. This sounds trivial, simple even. I wonder why nobody is doing it? Perhaps everyone doesn't have access to petabytes of storage space, unlimited bandwidth, unlimited proxy-jumps etc. So the alternative is to buy github?
  - williamcotton 3y ago
    I wonder why nobody is doing it? Perhaps everyone doesn't have access to petabytes of storage space, unlimited bandwidth, unlimited proxy-jumps etc. There are multiple private companies and public institutions that are currently training LLMs. The work that it required to train an LLM is actually in support of fair use, just as it was with regards to Google scanning books.
3y ago

mschild 3y ago

There certainly are other spaces where open source code is hosted and available, but the default for most is GitHub. I think it's in a similar position to Google 10 years ago. Sure there are other search engines, but Google is by and large the standard one. That does put Microsoft in the unique position to have direct unfettered access to any and all open source code on GitHub without restrictions. Unless you or I get the same kind of direct access without rate limiting and antibot protection, then they do dominate and have an advantage over everyone else.

reissbaker 3y ago
Not sure if you posted before the edits, but I'm pretty convinced by them, seeing as how there are multiple alternatives with the same data.
scarface74 3y ago
it’s really not that hard to git clone git set origin… It’s much harder to copy Google’s index.
- ChatGTP 3y ago
  You think it's practical to do this with almost all the public repos on Github?
  - scott_w 3y ago
    That's not Github's fault or Github's problem, from an antitrust perspective. If they went out of their way to make it difficult, you might have an argument but, as far as I know, they aren't. It's just practically difficult by the nature of the problem.
    - zanellato19 3y ago
      They rate limit, so they do make it difficult though

eternalban 3y ago

> The comparison to the Google Books antitrust falls down completely, Google had a dominant position because it had the resources to scan all books. Anyone can build a collection of almost all open source code. Copying a file is not the same thing as "scanning" a book. To scan you first need to get your hands on the book (the download part) and then use industrial scanners to scan them. So apple-apple comparison here is scanning <-> training & scanned collection of books <-> trained model, and finally the portals to the loot: Google Books <~> Github+VSC. Not everyone has the resources to actually process -- that is train the 'model' -- using the publicly available 'data'. Most also don't also own Github and VSC platforms to field their model. In fact, is anyone other than microsoft in a position to both scrape OSS, train a coding AI, and then include that tool in dominant software development platforms?

bilqis 3y ago

Github also has access to private repositories.

samwillis 3y ago
They don't use privet repositories to train Copilot.
- bilqis 3y ago
  They say they don’t
  - esrauch 3y ago
    Why would they possibly lie about that?
    - ChatGTP 3y ago
      Because they do shady shit, like, by default Copilot would "sample" code for training while using it. Maybe this is no longer the default, maybe it still is, but it was the default. This type of thing erodes trust? Why should my proprietary code be used for training by default? I was really annoyed by this.
      - scrollaway 3y ago
        OpenAI is not the same company as GitHub, and it has always been pretty clear that chats on ChatGPT are recorded and used for training (unless you now opt out).
        ChatGTP 3y ago
        Not sure why you're bringing OpenAI into it. My comment and the article is about "Copilot" I'm talking about when using "Github Copilot" and you ask for a code suggestion, it would send the "context" back to GitHub / Microsoft and use that code as training. Your comment is interesting to me though because there does seem to be a surprisingly large amount of defending OpenAI going on. Almost seems automatic now.

rjmunro 3y ago

> Google had a dominant position because it had the resources to scan all books. I thought Google had a dominant position because they signed an exclusive deal with the authors guild that explicitly gave them a dominant position. Anyone else could set up a project to go round libraries and scan books. Google has put more money into it than other organisations, but The Internet Archive has about 20 million scans (https://archive.org/details/texts https://archive.org/details/texts).

goodpoint 3y ago

> No, it's not in a "unique and dominant position". Open source code is freely available online, it's almost trivial to build a bot to scrape OS code from anywhere on the web (GitHub included). Absolutely wrong. GitHub is doing way more than just hosting code. It hosts bugtrackers, CI and much more. For most FOSS project it's the ONLY place where you can go and submit a bug report. It's not just a repository, it's a communication tool and refuses to interoperate with other platform. This is monopoly, just like NPM and Linkedin. Microsoft never changes.

toastal 3y ago

Microsoft GitHub has access to all the commits you force pushed away or branch you deleted. We have no reason to believe that it’s actually gone with no transparency and the source code being closed.