3 ms·
After failing with three reasonable ideas, they solved the problem with an idea that previously would have seemed unlikely to work (get similarity to past known
by pama 2y ago
After failing with three reasonable ideas, they solved the problem with an idea that previously would have seemed unlikely to work (get similarity to past known nits and use a threshold of 3 similar nits above a certain cutoff similarity score for filtering). A lot of applications of ML have a similar hacky trial and error flavor like that. Intuition is often built in retrospect and may not transfer well to other domains. I would have guessed that finetuning would also work, but agree with the author that it would be more expensive and less portable across different models.
- dakshgupta 2y agoSince we posted this, two camps of people reached out: Classical ML people who recommended we try training a classifier, possibly on the embeddings. Fine tuning platforms that recommended we try their platform. The challenge there would be gathering enough data per customer to meaningfully capture their definition and standard for nit-pickiness.
- pama 2y agoWhat you use now is a simple KNN classifier, and if it works well enough, perhaps no need to go much further. If you need to squeeze out a couple additional percentage points maybe try a different simple and robust ML classifier (random forest, xgboost, or a simple two layer network). All these methods, including your current classifier, will get better with additional data and minor tuning in the future.
- dakshgupta 2y agoThank you, I will try this. I suspect we can extract some universal theory of nits and have a base filter to start with, and have it learn per-company preferences on top of that.
- keepingscore 2y agoYou should be able to do that already by taking all of your customers nit embeddings and averaging them to produce a point in space that represents the universal nit. Embeddings are really cool and the fact that they still work when averaging is one of their cool properties.
- dakshgupta 2y agoThis is a cool idea - I’ll try this and add it as an appendix to this post.
- olddustytrail 2y agoI don't think the prompt was reasonable. Nits are the eggs of headlice. Expecting the LLM to guess they meant superficial issues, rather than just spelling that out, is a bit weird. It's like saying "don't comment on elephants" when you mean don't comment on the obvious stuff.
- whitfin 2y agoYeah, even the title here says "nitpicky". I'm not sure if they tried "nitpicks" instead of "nits", but I don't know why you wouldn't...
- tayo42 2y agoDo llms struggle with other homonyms?
- Eisenstein 2y agoThey can, but I think the parent is saying that when you are crafting an instruction for someone or something to follow, you should be as direct and as clear as possible in order to remove the need for the actor to have to intuit a decision instead of act on certainty. I would start with something like: 'Avoid writing comments which only concern stylistic issues or which only contain criticisms considered pedantic or trivial.'