6 ms·

Koders: Search Over 3 Million Lines of Open Source Code Around the Web

49 points by dylangs1030 15y ago

qqqqqq 15y ago
I think you mean billion, not million.
aw3c2 15y ago
Searched for "i=i+1" and got many unrelated results. Tried "new Tree", got results for "Tree". Realised that quotes do not work. Not sure what use the search would be for me. "this sucks" kinda works. :)
- Ogre 15y ago
  Even more problematic, I tried searching for zmq_init, to see if it would be useful for finding how other people had used a specific API function, and it seems like it treats an underscore just like a space. Not one result on the first page contains "zmq_init" in the summary, it just matched "zmq" by itself for all of them. I'm sure some of them do contain zmq_init in the complete sources, but that's not what it's matching, and for a less common API function, it seems likely a regular google search would be more productive than this. Hopefully they can fix that, just having a bunch of source code indexed, or even available to be indexed if it turns out the current indexing is no good, is a good first step. But it seems like it needs work right now.
troygoode 15y ago
"We just finished migrating the entire code base from .NET to Java" Interesting. I'd love to hear more about the rationale behind this move. I can understand moving from .NET to a dynamic or functional programming language, but why move to what is effectively a peer?
- Too 15y ago
  I'm not part of the team but a qualified guess would be the hosting and/or license fees. And in either case, moving from .NET to Java would seem more rational than moving to a new functional language because they are so similar and wont require as much work. I wouldn't be surprised if there even are automatic converters.
- jnbiche 15y ago
  Could it be that a company dedicated to promoting, analyzing, integrating, and categorizing open source software wanted to use an open source solution themselves? And although the open nature of Java can certainly be questioned due to the manipulations of Sun (and now Oracle), it's certainly more open that .Net, at least since 2007.
- benatkin 15y ago
  My guess is first-class support for libraries like Lucene and Hadoop, and to avoid software purchases or cloud surcharges.
- consultutah 15y ago
  Classic indicator of a soon to be failed project: "We didn't like language X anymore do we rewrote the whole thing in language Y" There are always other "reasons", but they are usually just straw men. Users don't care what your site is written in.
Groxx 15y ago
Not ready for prime-time. Not even close. I wonder if they ever tested their query engine, it's hardly useful. No quotes? Queries ignore important characters like "_"? No roll-up of identical results? It also misses methods in its "(num) methods" results (or throws them into a totally different order) in several languages, and has some crazy match-ordering behavior. An example: I searched for "helper_method" (a Rails method), and found these (bracketed what was bolded as the search hits): class.Scheduler.php: // Holds [method]s for ... Editor.py: from Models import Editor[Helper], Controllers AutShare.h: // IShare [method]s and then I get over a dozen literally-identical files called "helper_method_example.rb" (tests with these results: require File.dirname(__FILE__) + '/spec_[helper]' describe "a context with [helper] a [method]" do def [helper_method] and the rest of the files contained this after those lines: "received call" end it "should make that method available to specs" do helper_method.should == "received call" end end There are about three pages of results like that, or the same wrapped in a `module HelperMethodExample`, with identical contents otherwise.
- alextgordon 15y ago
  It's been like that for years. Can't see it improving, now that they're going to have a monopoly on the market. I will happily pay money to anybody who can build a code search service that is actually functional.
  - dncrane 15y ago
    This isn't exactly the same thing, but I launched a search engine for programmers (that is, it doesn't ignore special characters) a few weeks ago: http://symbolhound.com/ http://symbolhound.com/ It doesn't search code repositories, but rather tutorials and question/answer sites like stackoverflow. Right now stackoverflow dominates the results but that should change as we index more sites.
  - boyter 15y ago
    Would you be interested in paying per search, paying for access or just general donations based on your experience? I ask as I am doing this but have not considered a business model for it as yet.
boyter 15y ago
I imagine this is on the front page due to Google code search going away. Since i'm currently working on a replacement this seems like as good a place as any to ask people what they are looking for. My list of required launch features are, 1. Regex search support. 2. Browse full file. 3. Filter search to single project. This is based on what I observed on the Reddit and HN discussions when it was announced. I can see from the below some people would like grouping of identical results which is fairly easy to add so no worries there. Any other thoughts?
- troels 15y ago
  Index by symbols. ctags would be a fine start.
  - boyter 15y ago
    I have been wondering if that's actually required. Take for example the following search print_r During the indexing process it would take print and r as separate words. Any search for "print_r" should break up the same way, and assuming the search is phrase heavy return any document with "print_r" near the top anyway.
- 110101001010100 15y ago
  I can envision a relatively easy way to implement this without writing lots of code. How big do you think the market is for internet-based "code search"? I think code search is very useful and I've come near to trying to construct a searchable database just for personal use. It's an itch I've been tempted to scratch. But it would require some time.
  - boyter 15y ago
    Big enough for several projects to already exist, koders (and a few others I have forgotten the name of), but not big enough for it to be worth maintaining for Google. No idea what number to put to that, but I would image its less then 5 million users. So far I have mostly everything working, its just pulling down enough repo's to make it worthwhile.
teyc 15y ago
How about this? http://searchco.de/ http://searchco.de/ (I'm not affiliated)
- boyter 15y ago
  Thanks for the link (I am the author). No code search in there yet, just API and snippets. I will soon have code search as per Google Code Search. I just need to finish pulling down several thousand more repos before pushing it live. For the moment however look at the following, http://codase.com/ http://codase.com/ http://opensearch.krugle.org/ http://opensearch.krugle.org/
jberets 15y ago
Thank you for your interest and comments. The Koders team is in the process of re-implementing the code behind Koders.com. Please don’t draw too many conclusions about the future functionality of Koders by experimenting with the current one. Here’s some feedback on a few of the comments. Why we are moving from .NET to Java technology is easily addressed: the .NET implementation was built by a company we acquired; we are a Java shop. We can invest in and expand the code search engine and site more effectively using technologies familiar to our developers. We haven’t yet upgraded Koders to the latest software version. But we’ve done quite a lot of work beyond just a vanilla language port. For example, search results will be filterable to a single (or multiple) projects as mentioned in one of the comments and by language(s) as mentioned in another. We have noted the feedback regarding duplicate results, underscores, quoted searches, etc. and will put these in our backlog. In preparation for the debut of the new Koders, we are currently doing scalability testing to ensure that performance of the re-launched site is up to par. We will index a lot more code than is currently there. Our test instance is now running with more than 7 billion LOCs in-house, about 2x the size of the current Koders.com index. If you’d like to download and try the Java-based implementation that will be deployed to power the new Koders, it’s at http://www.blackducksoftware.com/java_implementation_download/ http://www.blackducksoftware.com/java_implementation_downloa... . We look forward to making this a great code search resource. Again, thank you for your interest!!

49 points by dylangs1030 15y ago

I think you mean billion, not million.

aw3c2 15y ago

Searched for "i=i+1" and got many unrelated results. Tried "new Tree", got results for "Tree". Realised that quotes do not work. Not sure what use the search would be for me. "this sucks" kinda works. :)

Ogre 15y ago
Even more problematic, I tried searching for zmq_init, to see if it would be useful for finding how other people had used a specific API function, and it seems like it treats an underscore just like a space. Not one result on the first page contains "zmq_init" in the summary, it just matched "zmq" by itself for all of them. I'm sure some of them do contain zmq_init in the complete sources, but that's not what it's matching, and for a less common API function, it seems likely a regular google search would be more productive than this. Hopefully they can fix that, just having a bunch of source code indexed, or even available to be indexed if it turns out the current indexing is no good, is a good first step. But it seems like it needs work right now.

troygoode 15y ago

"We just finished migrating the entire code base from .NET to Java" Interesting. I'd love to hear more about the rationale behind this move. I can understand moving from .NET to a dynamic or functional programming language, but why move to what is effectively a peer?

Too 15y ago
I'm not part of the team but a qualified guess would be the hosting and/or license fees. And in either case, moving from .NET to Java would seem more rational than moving to a new functional language because they are so similar and wont require as much work. I wouldn't be surprised if there even are automatic converters.
jnbiche 15y ago
Could it be that a company dedicated to promoting, analyzing, integrating, and categorizing open source software wanted to use an open source solution themselves? And although the open nature of Java can certainly be questioned due to the manipulations of Sun (and now Oracle), it's certainly more open that .Net, at least since 2007.
benatkin 15y ago
My guess is first-class support for libraries like Lucene and Hadoop, and to avoid software purchases or cloud surcharges.
consultutah 15y ago
Classic indicator of a soon to be failed project: "We didn't like language X anymore do we rewrote the whole thing in language Y" There are always other "reasons", but they are usually just straw men. Users don't care what your site is written in.

Groxx 15y ago

Not ready for prime-time. Not even close. I wonder if they ever tested their query engine, it's hardly useful. No quotes? Queries ignore important characters like "_"? No roll-up of identical results? It also misses methods in its "(num) methods" results (or throws them into a totally different order) in several languages, and has some crazy match-ordering behavior. An example: I searched for "helper_method" (a Rails method), and found these (bracketed what was bolded as the search hits): class.Scheduler.php: // Holds [method]s for ... Editor.py: from Models import Editor[Helper], Controllers AutShare.h: // IShare [method]s and then I get over a dozen literally-identical files called "helper_method_example.rb" (tests with these results: require File.dirname(__FILE__) + '/spec_[helper]' describe "a context with [helper] a [method]" do def [helper_method] and the rest of the files contained this after those lines: "received call" end it "should make that method available to specs" do helper_method.should == "received call" end end There are about three pages of results like that, or the same wrapped in a `module HelperMethodExample`, with identical contents otherwise.

alextgordon 15y ago
It's been like that for years. Can't see it improving, now that they're going to have a monopoly on the market. I will happily pay money to anybody who can build a code search service that is actually functional.
- dncrane 15y ago
  This isn't exactly the same thing, but I launched a search engine for programmers (that is, it doesn't ignore special characters) a few weeks ago: http://symbolhound.com/ http://symbolhound.com/ It doesn't search code repositories, but rather tutorials and question/answer sites like stackoverflow. Right now stackoverflow dominates the results but that should change as we index more sites.
- boyter 15y ago
  Would you be interested in paying per search, paying for access or just general donations based on your experience? I ask as I am doing this but have not considered a business model for it as yet.

boyter 15y ago

I imagine this is on the front page due to Google code search going away. Since i'm currently working on a replacement this seems like as good a place as any to ask people what they are looking for. My list of required launch features are, 1. Regex search support. 2. Browse full file. 3. Filter search to single project. This is based on what I observed on the Reddit and HN discussions when it was announced. I can see from the below some people would like grouping of identical results which is fairly easy to add so no worries there. Any other thoughts?

troels 15y ago
Index by symbols. ctags would be a fine start.
- boyter 15y ago
  I have been wondering if that's actually required. Take for example the following search print_r During the indexing process it would take print and r as separate words. Any search for "print_r" should break up the same way, and assuming the search is phrase heavy return any document with "print_r" near the top anyway.
110101001010100 15y ago
I can envision a relatively easy way to implement this without writing lots of code. How big do you think the market is for internet-based "code search"? I think code search is very useful and I've come near to trying to construct a searchable database just for personal use. It's an itch I've been tempted to scratch. But it would require some time.
- boyter 15y ago
  Big enough for several projects to already exist, koders (and a few others I have forgotten the name of), but not big enough for it to be worth maintaining for Google. No idea what number to put to that, but I would image its less then 5 million users. So far I have mostly everything working, its just pulling down enough repo's to make it worthwhile.

teyc 15y ago

How about this? http://searchco.de/ http://searchco.de/ (I'm not affiliated)

boyter 15y ago
Thanks for the link (I am the author). No code search in there yet, just API and snippets. I will soon have code search as per Google Code Search. I just need to finish pulling down several thousand more repos before pushing it live. For the moment however look at the following, http://codase.com/ http://codase.com/ http://opensearch.krugle.org/ http://opensearch.krugle.org/

jberets 15y ago

Thank you for your interest and comments. The Koders team is in the process of re-implementing the code behind Koders.com. Please don’t draw too many conclusions about the future functionality of Koders by experimenting with the current one. Here’s some feedback on a few of the comments. Why we are moving from .NET to Java technology is easily addressed: the .NET implementation was built by a company we acquired; we are a Java shop. We can invest in and expand the code search engine and site more effectively using technologies familiar to our developers. We haven’t yet upgraded Koders to the latest software version. But we’ve done quite a lot of work beyond just a vanilla language port. For example, search results will be filterable to a single (or multiple) projects as mentioned in one of the comments and by language(s) as mentioned in another. We have noted the feedback regarding duplicate results, underscores, quoted searches, etc. and will put these in our backlog. In preparation for the debut of the new Koders, we are currently doing scalability testing to ensure that performance of the re-launched site is up to par. We will index a lot more code than is currently there. Our test instance is now running with more than 7 billion LOCs in-house, about 2x the size of the current Koders.com index. If you’d like to download and try the Java-based implementation that will be deployed to power the new Koders, it’s at http://www.blackducksoftware.com/java_implementation_download/ http://www.blackducksoftware.com/java_implementation_downloa... . We look forward to making this a great code search resource. Again, thank you for your interest!!