ok wow that actually seems to work pretty damn credibly[1], at least given the inputs that i could come up with. i did find it difficult to come up with queries

ok wow that actually seems to work pretty damn credibly[1], at least given the inputs that i could come up with. i did find it difficult to come up with queries that (1) clearly would rely more on the code embeddings than the description embeddings (which i'd assume are at least pretty decent on their own), while (2) actually making sense / having a sensical answer. the best i could come up with is describing what you want ~operationally, rather than with domain-specific (read: predictive) jargon. but while also not underspecifying it to the point where the query is nonsense[2]. but when i try the query "sort the operands in decreasing order", not only does it get a bunch of sorting functions[3], but i'll be damned if the top 2 results weren't: (1) a `swapOperands` function that takes two Comparator<T>s and returns a new one that invokes the comparison with the operands reversed and (2) a function that sorts a dequeue into decreasing order. obviously that's not the perfect query for telling the relative contributions apart because "operands" is kinda "jargon"-y by my earlier standard, but the results did (correctly) address the only reason i'd ever search for something like that: to remember if the default comparison operator is "A - B" or "B - A". if you change "operands" to "elements", the results do get a little "worse", but i'd argue that the query (1) is actually quite a bit more vague in that formulation and (2) is less likely to represent a code snippet that actually exists, if only because most comparison operators on individual "containee" types (~"element[ type]s") are parameterized by the sort direction [citation needed]. tl;dr - lmk if anyone thinks of particularly good ways to "fool" this and/or demarcate what kinds of things are and are not well-represented by the code embeddings; to my eye they definitely seem to be doing a decent job of... embedding the code... as code embeddings are wont to do. [1] not to "damn with faint praise"-- if you had asked me whether something like this paper would work well enough to be useful i... would have guessed "no" :) [2] e.g. queries like "do some machine learning on the user images" and "... on the training data" don't get very "good" results in terms of topic proximity, but there really isn't a sensible response anyway, so if anything that's a good sign-- garbage in, garbage out [3] which obviously to be expected, as P(<can't find sorting functions> | <paper published>) is (hopefully) pretty low...