sure, primarily code. except when code contains language, like in comments, and other related documentation that's stored in-repo. and of course, people are g

sure, primarily code. except when code contains language, like in comments, and other related documentation that's stored in-repo. and of course, people are going to completely mis-use the repo to store ancillary human-readable binary documents, which will need to be converted, indexed, searched, updated, purged, etc, etc... since developers tag, reference, and document their code with comments, they're going to expect the full suite of NLP treatments with all indexed content, including code. which basically means: you'll have to tokenize out the special characterize for NLP, but retain them for literal code searches, thereby increasing the size of the inverted index for every permutation of the desired search criteria. maybe you can detect and filter out all the comments, and index them separately, or maybe have some kind of dual system where the NL indexing system exists separately from the code indexing system ... you see where this is going?