10 ms·

Processing 40TB of code from 10M projects with a dedicated server and Go

238 points by boyter 7y ago

robbya 7y ago
> If someone wants to host the raw files to allow others to download it let me know. It is a 83 GB tar.gz file which uncompressed is just over 1 TB in size. Some of the cloud providers have free hosting for public data sets (people who use the data incur cost to download/process the data). I'm not sure if this would qualify. * https://aws.amazon.com/opendata/public-datasets/ https://aws.amazon.com/opendata/public-datasets/ * https://azure.microsoft.com/en-us/services/open-datasets/ https://azure.microsoft.com/en-us/services/open-datasets/
- lunixbochs 7y ago
  I similarly have a 127GB tar.xz of about 2TB of code/files from the top GitHub repositories (filtered by highest starred repositories per primary language) as of around August 2017 if anyone wants it or wants to host it.
- z3t4 7y ago
  Make a bit torrent
- trevyn 7y ago
  You could also use an S3 or GCS “requester pays” bucket, which means you never get charged for data transfer. Would cost about $2/month for the 83GB storage, but that’s it. https://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html https://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPay... https://cloud.google.com/storage/docs/requester-pays https://cloud.google.com/storage/docs/requester-pays
- bauruine 7y ago
  Or you could just put it on some Hetzner box. I would do that but I'm not sure how to contact him.
- polygot 7y ago
  I'm very interested in this data; if there is a torrent, I would be able to seed it if needed.
Scarbutt 7y ago
Missing content for: https://boyter.org/posts/an-informal-survey-of-10-million-github-bitbucket-gitlab-projects/#the-most-complex-code-is-written-in-what-language https://boyter.org/posts/an-informal-survey-of-10-million-gi...
ojkelly 7y ago
As much as I love the confirmation that .yml is the correct extension, I’m most amazed by the fact that there’s over a TRILLION lines of code public on GitHub. That’s an astronomical number. And that’s only what we can all see. Can’t imagine how much more is private. Also, it look’s like 20% of code is comments. Which feels just about right.
- taf2 7y ago
  Surely copy & paste played a major role in this.
  - ojkelly 7y ago
    Yep that and forking. But if you fork a repo and change a small thing, I’d it still a duplicate of the other repo? I don’t think the raw number is what’s so impressive, as much as the fact that there’s more code public on GitHub than I could comprehend in a lifetime.
  - foobarqwertz 7y ago
    I am waiting for the results showing how much copy pasta is on the plate.
- brazzledazzle 7y ago
  People actually bike shed over .yaml vs .yml? If you thought tabs vs spaces was a useless debate...
  - pixelbash 7y ago
    Never been comfortable with extension names longer than 3 letters long for some reason..
    - wazoox 7y ago
      MS-DOS. The reason is MS-DOS.
mirekrusin 7y ago
This is great. Btw. seeing "jquery" most-common filename 20,015,171 times - it probably skews js LoC (artificial LoC of dependencies).
jeremyjh 7y ago
This is a cool project and a great write-up. Some of the complexity numbers struck me as pretty off though. I know he caveats you can only compare files of the same language, but I took a quick look at the code[1] and at least as of now it looks like the complexity statements are just kind of copy/pasted, so keywords not seen in Java/C such as "match" are missing from Rust, OCaml, and Scala, "case" is missing from Haskell and Elixir etc, causing these languages to be much lower in complexity than they should be just based on standard control flow statements. [1] https://github.com/boyter/scc/blob/master/languages.json https://github.com/boyter/scc/blob/master/languages.json
- boyter 7y ago
  In all seriousness if you are a language expert for those languages please submit a PR or at least raise an issue saying what keywords should be in there. I am happy to include it into the next release should it produce more accurate numbers.
Groxx 7y ago
First I've heard of "Taco Bell Programming"[1]. I love it, this needs to become a normal part of my lexicon. Also heck yes, it completely mirrors my experience - a few dozen tools combine in fairly simple ways to make absurdly useful results. The problem is finding the useful N-dozen elemental tools you'll use. [1]: http://widgetsandshit.com/teddziuba/2010/10/taco-bell-programming.html http://widgetsandshit.com/teddziuba/2010/10/taco-bell-progra...
- john-radio 7y ago
  > I could have done the whole thing Taco Bell style if I had only manned up and broken out sed, but I pussied out and wrote some Python. And with one dumb word choice, I suddenly can't share this otherwise good piece with most audiences.
  - homonculus1 7y ago
    Seriously? That's no harsher language than calling someone a dick. Do you have to keep an ultra-clinical PR image for some reason or are we really that puritanical?
    - john-radio 7y ago
      The networks that I belong to and benefit from are that "puritanical," if you want to choose a really stupid word for avoiding insulting half the population for no reason, yeah.
  - fivefive55 7y ago
    The website it's linked on is called widgetsandshit.com. I think that's a bigger problem if you're that worried.
    - john-radio 7y ago
      Nothing offensive about shit.
  - mhuffman 7y ago
tylerl 7y ago
Can't you do all this same research in just a few seconds for like pennies, just by sending a few sql queries to Google's BigQuery? I'm pretty sure we've had stories about that here. https://medium.com/google-cloud/github-on-bigquery-analyze-all-the-code-b3576fd2b150 https://medium.com/google-cloud/github-on-bigquery-analyze-a...
- jmngomes 7y ago
  I partially agree with you; unless the goal was to "do this in Go", the choice of tools seems odd/inneficient. Spark would have a been a simple option to do this kind of processing, with less lines of code, and could also run on "spare compute". Same goes for the "How does one process 10 million JSON files taking up just over 1 TB of disk space in an S3 bucket?": there are appropriate file formats for storing and querying big datasets, text/json is simply the least efficient option and likely the cause of the "$2.50 USD per query" number...
jtwaleson 7y ago
Haha, is this a coincidence or did you take inspiration from my 2016 blog post? "Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS" http://blog.waleson.com/2016/01/parsing-10tb-of-metadata-26m-domains.html http://blog.waleson.com/2016/01/parsing-10tb-of-metadata-26m... We used a lot of the same tech: python, golang, S3, 32 core machines. Anyway, nice read and good use of Hetzner.
chewxy 7y ago
A little bit of promo here: Ben will be giving a talk at GopherconAU later this year. - tickets are on sale now at https://gophercon.com.au https://gophercon.com.au
ran3824692 7y ago
plug https://www.softwareheritage.org/ https://www.softwareheritage.org/
- trevyn 7y ago
  Is this available to download in bulk?
lowsenberg 7y ago
> The front-end varnish box for instance is doing the square root of zero most of the time. Nicely written. :)
breakerbox 7y ago
I thought the most common file names had some peculiar results. > the filename “15” & “s15” made the top 50 most popular filenames Anyone know why? https://boyter.org/posts/an-informal-survey-of-10-million-github-bitbucket-gitlab-projects/#what-are-the-most-common-filenames https://boyter.org/posts/an-informal-survey-of-10-million-gi...
- GrumpyNl 7y ago
  I would like to know to, any insights?
  - breakerbox 7y ago
    The only thing I’ve put together from light searching on GitHub might be to do with the fact that 15 and s15 can be used to describe school years (s for spring 15 semester) and a lot of people post assignments to GitHub. 15 is also a common number in coding problem sets that people post to GitHub. It’s a stretch but it might be something to do with 2015 being a year that has a lot of coursework commits, and 15 is a pretty low number (i.e. higher probability a student posts solutions to #1-15 than #1-19). The same goes for schoolwork. Many people do online MIT and Stanford courses that are from previous years ~2012-2016, but there has been less time elapsed for students to post answers from 2016,17,18,19) This is mostly conjecture, so I hope someone has a better answer!
    - WilliamEdward 7y ago
      advent of code also came out in 2015... so, maybe?
- dmit 7y ago
  s15 seems to be skewed by a bunch of JavaScript projects including Font Awesome, which stores the bathtub icon as s15.js for some reason (note that s1 through s14 are unused, so it's not just consecutive numeric ids). https://fontawesome.com/v4.7.0/icon/bath https://fontawesome.com/v4.7.0/icon/bath Plain 15 is trickier. GitHub's search is too fuzzy to see a pattern at a quick glance, but again it seems that JavaScript is the culprit with a disproportionately high number of filename:15 results compared to filename:14 or filename:16.
z3t4 7y ago
Nice to see that Hypecript is far from engulfing JavaScript. Hopefully the marketing team will not start to delete old JavaScript repos because of this. :P
deleted 7y ago
[deleted]
reacharavindh 7y ago
> I actually wrote a Python solution at first, but having to install the pip dependencies on my clean varnish box seemed like a bad idea and it keep breaking in odd ways which I didn’t feel like debugging. Amen! This is why I am learning Go at the moment and considering using it instead of Python for admin and data processing tasks on a fleet of servers. The single binary deployment makes it a lot easier for users to adopt. Python misses out on a lot of use because of the inability to do this. And No! I do not want to pip install a lot of stuff on the servers just to be able to run this script once. Heck, some of these servers don't even have access to public internet to be able to pip install whatever. Yes, I have looked into pyinstaller and Nuitka. They threw up some errors that were indicative of deeper issues that I didnt feel like a good use of my time to debug. I'd rather choose a language that has this as a priority/design goal instead.
- deleted 7y ago
  [deleted]
- EdwardDiego 7y ago
  Yeah, I was writing a Docker image built around Prometheus' jmx_exporter intended to be used as a mixin for our Java apps, and part of what I needed to do was provide a simple script to preprocess separate config files to produce the config used by jmx_exporter. My initial thought was Python, but it needed a couple of 3rd party dependencies, and there wasn't an overly clean _and_ simple way to copy the script in from the mixin and run locally. So I shrugged, rewrote my script in Go, and then used a multi-stage build to copy in the binary and nothing else. Ending up with a single statically linked binary was cool, even if Go does some stuff that made my eyebrows quirk a tad (I still can't believe that an idiomatic set in Go is map[T]struct{}...)
  - twic 7y ago
    > I was writing a Docker image built around Prometheus' jmx_exporter intended to be used as a mixin for our Java apps, and part of what I needed to do was provide a simple script to preprocess separate config files to produce the config used by jmx_exporter Did you consider, and stop me if this suggestion is completely wild, Java?
gitgud 7y ago
One query that always interested me is how much duplication is in the Open-Source community. Like how many boiler-plate files are copied into projects. It seems generating a `react/angular` project produces many files, that almost never get changed, so it would be interesting to know the most duplicated files... I think this could also give valuable insight into how to make the language/frameworks better or simpler...
throwaway77384 7y ago
Glad to see Go Template make the list of languages with the most curse words in them. NOT Go, but Go Template!! A subset of Go is causing more anguish than all of Python and only a sliver less than all of Rust. Says it all really ;)
08-15 7y ago
Why would anyone store 1TB of data in highly redundant JSON? Stored as protobuf, I estimate it would be 8x smaller. A custom binary format would be smaller again. Not only is that smaller, which saves storage and transfer cost, it's also proportionally faster to process.
gilleain 7y ago
A couple of the large files are explainable: - The largest .c file is actually the CATH database, not sure why it has that extension - The large .cpp file is actually C++ but has a kind of 'data as code' approach defining bonds between atoms
tuananh 7y ago
when cpu utilization is high for a long period of time like this, dedicated servers always make more sense than cloud.

238 points by boyter 7y ago

> If someone wants to host the raw files to allow others to download it let me know. It is a 83 GB tar.gz file which uncompressed is just over 1 TB in size. Some of the cloud providers have free hosting for public data sets (people who use the data incur cost to download/process the data). I'm not sure if this would qualify. * https://aws.amazon.com/opendata/public-datasets/ https://aws.amazon.com/opendata/public-datasets/ * https://azure.microsoft.com/en-us/services/open-datasets/ https://azure.microsoft.com/en-us/services/open-datasets/

lunixbochs 7y ago
I similarly have a 127GB tar.xz of about 2TB of code/files from the top GitHub repositories (filtered by highest starred repositories per primary language) as of around August 2017 if anyone wants it or wants to host it.
z3t4 7y ago
Make a bit torrent
trevyn 7y ago
You could also use an S3 or GCS “requester pays” bucket, which means you never get charged for data transfer. Would cost about $2/month for the 83GB storage, but that’s it. https://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html https://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPay... https://cloud.google.com/storage/docs/requester-pays https://cloud.google.com/storage/docs/requester-pays
bauruine 7y ago
Or you could just put it on some Hetzner box. I would do that but I'm not sure how to contact him.
polygot 7y ago
I'm very interested in this data; if there is a torrent, I would be able to seed it if needed.

Scarbutt 7y ago

Missing content for: https://boyter.org/posts/an-informal-survey-of-10-million-github-bitbucket-gitlab-projects/#the-most-complex-code-is-written-in-what-language https://boyter.org/posts/an-informal-survey-of-10-million-gi...

ojkelly 7y ago

As much as I love the confirmation that .yml is the correct extension, I’m most amazed by the fact that there’s over a TRILLION lines of code public on GitHub. That’s an astronomical number. And that’s only what we can all see. Can’t imagine how much more is private. Also, it look’s like 20% of code is comments. Which feels just about right.

taf2 7y ago
Surely copy & paste played a major role in this.
- ojkelly 7y ago
  Yep that and forking. But if you fork a repo and change a small thing, I’d it still a duplicate of the other repo? I don’t think the raw number is what’s so impressive, as much as the fact that there’s more code public on GitHub than I could comprehend in a lifetime.
- foobarqwertz 7y ago
  I am waiting for the results showing how much copy pasta is on the plate.
brazzledazzle 7y ago
People actually bike shed over .yaml vs .yml? If you thought tabs vs spaces was a useless debate...
- pixelbash 7y ago
  Never been comfortable with extension names longer than 3 letters long for some reason..
  - wazoox 7y ago
    MS-DOS. The reason is MS-DOS.

mirekrusin 7y ago

This is great. Btw. seeing "jquery" most-common filename 20,015,171 times - it probably skews js LoC (artificial LoC of dependencies).

jeremyjh 7y ago

This is a cool project and a great write-up. Some of the complexity numbers struck me as pretty off though. I know he caveats you can only compare files of the same language, but I took a quick look at the code[1] and at least as of now it looks like the complexity statements are just kind of copy/pasted, so keywords not seen in Java/C such as "match" are missing from Rust, OCaml, and Scala, "case" is missing from Haskell and Elixir etc, causing these languages to be much lower in complexity than they should be just based on standard control flow statements. [1] https://github.com/boyter/scc/blob/master/languages.json https://github.com/boyter/scc/blob/master/languages.json

boyter 7y ago
In all seriousness if you are a language expert for those languages please submit a PR or at least raise an issue saying what keywords should be in there. I am happy to include it into the next release should it produce more accurate numbers.

Groxx 7y ago

First I've heard of "Taco Bell Programming"[1]. I love it, this needs to become a normal part of my lexicon. Also heck yes, it completely mirrors my experience - a few dozen tools combine in fairly simple ways to make absurdly useful results. The problem is finding the useful N-dozen elemental tools you'll use. [1]: http://widgetsandshit.com/teddziuba/2010/10/taco-bell-programming.html http://widgetsandshit.com/teddziuba/2010/10/taco-bell-progra...

john-radio 7y ago
> I could have done the whole thing Taco Bell style if I had only manned up and broken out sed, but I pussied out and wrote some Python. And with one dumb word choice, I suddenly can't share this otherwise good piece with most audiences.
- homonculus1 7y ago
  Seriously? That's no harsher language than calling someone a dick. Do you have to keep an ultra-clinical PR image for some reason or are we really that puritanical?
  - john-radio 7y ago
    The networks that I belong to and benefit from are that "puritanical," if you want to choose a really stupid word for avoiding insulting half the population for no reason, yeah.
- fivefive55 7y ago
  The website it's linked on is called widgetsandshit.com. I think that's a bigger problem if you're that worried.
  - john-radio 7y ago
    Nothing offensive about shit.
- mhuffman 7y ago

tylerl 7y ago

Can't you do all this same research in just a few seconds for like pennies, just by sending a few sql queries to Google's BigQuery? I'm pretty sure we've had stories about that here. https://medium.com/google-cloud/github-on-bigquery-analyze-all-the-code-b3576fd2b150 https://medium.com/google-cloud/github-on-bigquery-analyze-a...

jmngomes 7y ago
I partially agree with you; unless the goal was to "do this in Go", the choice of tools seems odd/inneficient. Spark would have a been a simple option to do this kind of processing, with less lines of code, and could also run on "spare compute". Same goes for the "How does one process 10 million JSON files taking up just over 1 TB of disk space in an S3 bucket?": there are appropriate file formats for storing and querying big datasets, text/json is simply the least efficient option and likely the cause of the "$2.50 USD per query" number...

jtwaleson 7y ago

Haha, is this a coincidence or did you take inspiration from my 2016 blog post? "Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS" http://blog.waleson.com/2016/01/parsing-10tb-of-metadata-26m-domains.html http://blog.waleson.com/2016/01/parsing-10tb-of-metadata-26m... We used a lot of the same tech: python, golang, S3, 32 core machines. Anyway, nice read and good use of Hetzner.

chewxy 7y ago

A little bit of promo here: Ben will be giving a talk at GopherconAU later this year. - tickets are on sale now at https://gophercon.com.au https://gophercon.com.au

ran3824692 7y ago

plug https://www.softwareheritage.org/ https://www.softwareheritage.org/

trevyn 7y ago
Is this available to download in bulk?

lowsenberg 7y ago

> The front-end varnish box for instance is doing the square root of zero most of the time. Nicely written. :)

breakerbox 7y ago

I thought the most common file names had some peculiar results. > the filename “15” & “s15” made the top 50 most popular filenames Anyone know why? https://boyter.org/posts/an-informal-survey-of-10-million-github-bitbucket-gitlab-projects/#what-are-the-most-common-filenames https://boyter.org/posts/an-informal-survey-of-10-million-gi...

GrumpyNl 7y ago
I would like to know to, any insights?
- breakerbox 7y ago
  The only thing I’ve put together from light searching on GitHub might be to do with the fact that 15 and s15 can be used to describe school years (s for spring 15 semester) and a lot of people post assignments to GitHub. 15 is also a common number in coding problem sets that people post to GitHub. It’s a stretch but it might be something to do with 2015 being a year that has a lot of coursework commits, and 15 is a pretty low number (i.e. higher probability a student posts solutions to #1-15 than #1-19). The same goes for schoolwork. Many people do online MIT and Stanford courses that are from previous years ~2012-2016, but there has been less time elapsed for students to post answers from 2016,17,18,19) This is mostly conjecture, so I hope someone has a better answer!
  - WilliamEdward 7y ago
    advent of code also came out in 2015... so, maybe?
dmit 7y ago
s15 seems to be skewed by a bunch of JavaScript projects including Font Awesome, which stores the bathtub icon as s15.js for some reason (note that s1 through s14 are unused, so it's not just consecutive numeric ids). https://fontawesome.com/v4.7.0/icon/bath https://fontawesome.com/v4.7.0/icon/bath Plain 15 is trickier. GitHub's search is too fuzzy to see a pattern at a quick glance, but again it seems that JavaScript is the culprit with a disproportionately high number of filename:15 results compared to filename:14 or filename:16.

z3t4 7y ago

Nice to see that Hypecript is far from engulfing JavaScript. Hopefully the marketing team will not start to delete old JavaScript repos because of this. :P

deleted 7y ago

[deleted]

reacharavindh 7y ago

> I actually wrote a Python solution at first, but having to install the pip dependencies on my clean varnish box seemed like a bad idea and it keep breaking in odd ways which I didn’t feel like debugging. Amen! This is why I am learning Go at the moment and considering using it instead of Python for admin and data processing tasks on a fleet of servers. The single binary deployment makes it a lot easier for users to adopt. Python misses out on a lot of use because of the inability to do this. And No! I do not want to pip install a lot of stuff on the servers just to be able to run this script once. Heck, some of these servers don't even have access to public internet to be able to pip install whatever. Yes, I have looked into pyinstaller and Nuitka. They threw up some errors that were indicative of deeper issues that I didnt feel like a good use of my time to debug. I'd rather choose a language that has this as a priority/design goal instead.

deleted 7y ago
[deleted]
EdwardDiego 7y ago
Yeah, I was writing a Docker image built around Prometheus' jmx_exporter intended to be used as a mixin for our Java apps, and part of what I needed to do was provide a simple script to preprocess separate config files to produce the config used by jmx_exporter. My initial thought was Python, but it needed a couple of 3rd party dependencies, and there wasn't an overly clean _and_ simple way to copy the script in from the mixin and run locally. So I shrugged, rewrote my script in Go, and then used a multi-stage build to copy in the binary and nothing else. Ending up with a single statically linked binary was cool, even if Go does some stuff that made my eyebrows quirk a tad (I still can't believe that an idiomatic set in Go is map[T]struct{}...)
- twic 7y ago
  > I was writing a Docker image built around Prometheus' jmx_exporter intended to be used as a mixin for our Java apps, and part of what I needed to do was provide a simple script to preprocess separate config files to produce the config used by jmx_exporter Did you consider, and stop me if this suggestion is completely wild, Java?

gitgud 7y ago

One query that always interested me is how much duplication is in the Open-Source community. Like how many boiler-plate files are copied into projects. It seems generating a `react/angular` project produces many files, that almost never get changed, so it would be interesting to know the most duplicated files... I think this could also give valuable insight into how to make the language/frameworks better or simpler...

throwaway77384 7y ago

Glad to see Go Template make the list of languages with the most curse words in them. NOT Go, but Go Template!! A subset of Go is causing more anguish than all of Python and only a sliver less than all of Rust. Says it all really ;)

08-15 7y ago

Why would anyone store 1TB of data in highly redundant JSON? Stored as protobuf, I estimate it would be 8x smaller. A custom binary format would be smaller again. Not only is that smaller, which saves storage and transfer cost, it's also proportionally faster to process.

gilleain 7y ago

A couple of the large files are explainable: - The largest .c file is actually the CATH database, not sure why it has that extension - The large .cpp file is actually C++ but has a kind of 'data as code' approach defining bonds between atoms

tuananh 7y ago

when cpu utilization is high for a long period of time like this, dedicated servers always make more sense than cloud.