4 ms·

Processing 40 TB of code from ~10M projects with a server and Go for $100 (2019)

KolmogorovComp 4y ago
Oops linked to an anchor in the body and cannot edit. Correct link: https://boyter.org/posts/an-informal-survey-of-10-million-github-bitbucket-gitlab-projects/ https://boyter.org/posts/an-informal-survey-of-10-million-gi...
deleted 4y ago
[deleted]
jwilk 4y ago
Previously: https://news.ycombinator.com/item?id=21121735 https://news.ycombinator.com/item?id=21121735 (80 comments)
deleted 4y ago
[deleted]
thethirdone 4y ago
Does anyone know why "pawn" is considered an offensive word? It seems likely it ranks highly on the offensive word list because of chess engines.
- yaddaor 4y ago
  Phonetically it sounds like "porn".
- ok_dad 4y ago
  I also noticed "knob" and "xxx" near the top of that list (and others) for which I understand why they are considered "offensive" in some contexts, but in code those are quite non-offensive (unless, of course, the context is offensive, haha).
throwawaygog6 4y ago
>>> If someone wants to host the raw files to allow others to download it let me know. It is a 83 GB tar.gz file which uncompressed is just over 1 TB in size. Anyone knows if it is possible to download similar data set for youtube and reddit? I have ideas for search engine based on it, but I don't want to write/maintain scraper scripts.
- testesttest4 4y ago
  I have most of the historical reddit data except ~year I use to train ML models. Let me see if I can find a public link for you...
- teraflop 4y ago
  There's a very large dataset of Reddit posts and comments at https://files.pushshift.io/reddit/ https://files.pushshift.io/reddit/
layer8 4y ago
Average number of Makefiles is 1518.1769808494607? That can’t be right?
- CamJN 4y ago
  The ”average person eats 3 spiders a year” factoid is actually just statistical error. The average person eats 0 spiders per year. Spiders Georg, who lives in a cave & eats over 10,000 each day, is an outlier and should not have been counted”
  - layer8 4y ago
    Of course, however given that there are 1.2 million repositories with makefiles, the listed average would mean that they contain 1.8 billion makefiles in total. I doubt that that’s the case. Also, according to the page, only 59 million of all files are named “Makefile”. I suspect that the file language recognition has major false positives for the Makefile language.
- trevor-e 4y ago
  Yea this clearly seems like a bug or some repo skewing the results. Also a good reminder why the median should often be shown alongside average values.
layer8 4y ago
It would be interesting to plot some of the statistics by age of repository.

Oops linked to an anchor in the body and cannot edit. Correct link: https://boyter.org/posts/an-informal-survey-of-10-million-github-bitbucket-gitlab-projects/ https://boyter.org/posts/an-informal-survey-of-10-million-gi...

deleted 4y ago

[deleted]

jwilk 4y ago

Previously: https://news.ycombinator.com/item?id=21121735 https://news.ycombinator.com/item?id=21121735 (80 comments)

deleted 4y ago

[deleted]

thethirdone 4y ago

Does anyone know why "pawn" is considered an offensive word? It seems likely it ranks highly on the offensive word list because of chess engines.

yaddaor 4y ago
Phonetically it sounds like "porn".
ok_dad 4y ago
I also noticed "knob" and "xxx" near the top of that list (and others) for which I understand why they are considered "offensive" in some contexts, but in code those are quite non-offensive (unless, of course, the context is offensive, haha).

throwawaygog6 4y ago

>>> If someone wants to host the raw files to allow others to download it let me know. It is a 83 GB tar.gz file which uncompressed is just over 1 TB in size. Anyone knows if it is possible to download similar data set for youtube and reddit? I have ideas for search engine based on it, but I don't want to write/maintain scraper scripts.

testesttest4 4y ago
I have most of the historical reddit data except ~year I use to train ML models. Let me see if I can find a public link for you...
teraflop 4y ago
There's a very large dataset of Reddit posts and comments at https://files.pushshift.io/reddit/ https://files.pushshift.io/reddit/

layer8 4y ago

Average number of Makefiles is 1518.1769808494607? That can’t be right?

CamJN 4y ago
The ”average person eats 3 spiders a year” factoid is actually just statistical error. The average person eats 0 spiders per year. Spiders Georg, who lives in a cave & eats over 10,000 each day, is an outlier and should not have been counted”
- layer8 4y ago
  Of course, however given that there are 1.2 million repositories with makefiles, the listed average would mean that they contain 1.8 billion makefiles in total. I doubt that that’s the case. Also, according to the page, only 59 million of all files are named “Makefile”. I suspect that the file language recognition has major false positives for the Makefile language.
trevor-e 4y ago
Yea this clearly seems like a bug or some repo skewing the results. Also a good reminder why the median should often be shown alongside average values.

layer8 4y ago

It would be interesting to plot some of the statistics by age of repository.