4 ms·

Wow, they managed to get an LLM (and a small one no less) that can acknowledge that it doesn't know details about obscure data structures > Alternatively, perh

by gpm 2y ago

Wow, they managed to get an LLM (and a small one no less) that can acknowledge that it doesn't know details about obscure data structures > Alternatively, perhaps using a wavelet tree or similar structure that can efficiently represent and query for subset membership. These structures are designed for range queries and could potentially handle this scenario better. > But I'm not very familiar with all the details of these data structures, so maybe I should look into other approaches. This is a few dozen lines in to a query asking DeepSeek-R1-Distill-Qwen-1.5B-GGUF:F16 to solve what I think is an impossible CS problem, "I need a datastructure that given a fairly large universe of elements (10s of thousands or millions) and a bunch of sets of those elements (10s of thousands or millions) of reason able size (up to roughly 100 elements in a set) can quickly find a list of subsets for a given set. " I'm also impressed that it immediately started thinking about tries and, which are the best solutions that I know of/stackoverflow came up with for basically the same problem (https://stackoverflow.com/questions/6512400/fast-data-structure-for-finding-strict-subsets-from-a-given-list https://stackoverflow.com/questions/6512400/fast-data-struct...). It didn't actually return anything using those, but then I wouldn't really expect it to since the solution using them isn't exactly "fast" just "maybe less slow". PS. If anyone knows an actually good solution to this, I'd appreciate knowing about it. I'm only mostly sure it's impossible.

throwaway314155 2y ago
FWIW, simply because the model claims it's "not very familiar" with something doesn't mean it's actually able to probe its own knowledge and gauge familiarity in any way at all. That it's correct about not knowing much about a fairly obscure data structure from advanced computer science has more to do with what the average person in its training data would likely say than an indicator of that type of reflectionof occurring. I agree that it happens to (likely) be right in this instance however and this output is in some ways refreshing compared to other models which appear (!!) to have overconfidence and plow right ahead.
- gpm 2y ago
  With my optimistic hat on maybe it realized "wavelet tree or similar structure that can efficiently represent and query for subset membership" doesn't actually describe wavelet tree and this was its way of backtracking. I.e. it might have learned to respond like this to a prior inconsistent series of tokens. But ya, I'm aware of the issue with them saying they don't know things they do know.

by gpm 2y ago

throwaway314155 2y ago

FWIW, simply because the model claims it's "not very familiar" with something doesn't mean it's actually able to probe its own knowledge and gauge familiarity in any way at all. That it's correct about not knowing much about a fairly obscure data structure from advanced computer science has more to do with what the average person in its training data would likely say than an indicator of that type of reflectionof occurring. I agree that it happens to (likely) be right in this instance however and this output is in some ways refreshing compared to other models which appear (!!) to have overconfidence and plow right ahead.

gpm 2y ago
With my optimistic hat on maybe it realized "wavelet tree or similar structure that can efficiently represent and query for subset membership" doesn't actually describe wavelet tree and this was its way of backtracking. I.e. it might have learned to respond like this to a prior inconsistent series of tokens. But ya, I'm aware of the issue with them saying they don't know things they do know.