4 ms·

the joys of NLP ... this is a search index for Gitlab, that's used primarily to store code, not natural language. Looking at its competitor, Github's search en

the joys of NLP ... this is a search index for Gitlab, that's used primarily to store code, not natural language. Looking at its competitor, Github's search engine clearly has a base in NLP, and ditches many punctuation characters (https://help.github.com/articles/searching-code/ https://help.github.com/articles/searching-code/) - which are way more important in code than they are in English - for example, I can't search for code containing "$/" when it should have "\Z/" to match the end of a string. So, right now I have ~250 repos checked out from our Github Enterprise so I can search them offline. The queries I actually want to run are, pretty much, regexes. Those can be accelerated by trigram indexes like the one gitlab are using (see eg https://swtch.com/~rsc/regexp/regexp4.html https://swtch.com/~rsc/regexp/regexp4.html), but apparently this isn't implemented by say, Elastisearch (Whereas prefix matching can be made more efficient by preparing your data at index time, wildcard and regular expression matching can be done only at query time. https://www.elastic.co/guide/en/elasticsearch/guide/current/_wildcard_and_regexp_queries.html https://www.elastic.co/guide/en/elasticsearch/guide/current/...) I'd agree with you if this wasn't code. I've seen terrible text search in the DB, and I've built document management systems where we integrated real search engines. But those horses aren't for this course.

beachstartup 11y ago
sure, primarily code. except when code contains language, like in comments, and other related documentation that's stored in-repo. and of course, people are going to completely mis-use the repo to store ancillary human-readable binary documents, which will need to be converted, indexed, searched, updated, purged, etc, etc... since developers tag, reference, and document their code with comments, they're going to expect the full suite of NLP treatments with all indexed content, including code. which basically means: you'll have to tokenize out the special characterize for NLP, but retain them for literal code searches, thereby increasing the size of the inverted index for every permutation of the desired search criteria. maybe you can detect and filter out all the comments, and index them separately, or maybe have some kind of dual system where the NL indexing system exists separately from the code indexing system ... you see where this is going?
- bazzargh 11y ago
  they're going to expect the full suite of NLP treatments with all indexed content, including code. Nope. In 30 years of searching through code, I've never once felt the need for that. On the other hand, almost every time I use Github search, I find the limitations of a word-oriented index get in the way (looking for partial keywords to find code relevant to an api, etc. It's not just special characters).

beachstartup 11y ago

sure, primarily code. except when code contains language, like in comments, and other related documentation that's stored in-repo. and of course, people are going to completely mis-use the repo to store ancillary human-readable binary documents, which will need to be converted, indexed, searched, updated, purged, etc, etc... since developers tag, reference, and document their code with comments, they're going to expect the full suite of NLP treatments with all indexed content, including code. which basically means: you'll have to tokenize out the special characterize for NLP, but retain them for literal code searches, thereby increasing the size of the inverted index for every permutation of the desired search criteria. maybe you can detect and filter out all the comments, and index them separately, or maybe have some kind of dual system where the NL indexing system exists separately from the code indexing system ... you see where this is going?

bazzargh 11y ago
they're going to expect the full suite of NLP treatments with all indexed content, including code. Nope. In 30 years of searching through code, I've never once felt the need for that. On the other hand, almost every time I use Github search, I find the limitations of a word-oriented index get in the way (looking for partial keywords to find code relevant to an api, etc. It's not just special characters).