3 ms·

When I see people evaluating LLMs in this way I can't help but think they are letting emotion get the better of them: > I am using the incorrect phrasing, but

When I see people evaluating LLMs in this way I can't help but think they are letting emotion get the better of them: > I am using the incorrect phrasing, but Ive been heavily using claude's "Project Folders" (paid account) - and when I put "context" files into the project folders, it will "forget" that the files are there - and Ill call it out when it switches back to boilerplate response language -- and it apologizes and says "You are correct I SHOULD HAVE BEEN using the project files for context. You should probably try to implement your own RAG system with free models locally (ollama, langchain, chromadb can do it, it's very straightforward) so you can understand the process a bit more under the hood. > How/why is it occurring mid conversation? I dunno, usually when something is retrieved it is added to context. But a key part of RAG is determining how to chunk up your content so that the prompt embedding matches up to information that is actually targeted and concise. So if RAG is behaving suboptimally the first thing to check is, are the input documents targeted and concise? If too much context is being stuffed into the prompt context, then the results will be poor. You can see this with even just very large prompts, the larger the context the worse the quality, despite model developers claiming ever larger context windows. I don't think inventing your own terms (Which therefore have extremely weak embeddings to match with the embeddings of the models training content) is the right way to go. If I chuck in what is a discernment lattice to gpt4o I get: > A "discernment lattice" isn't a widely recognized term in most common fields of study, but it can be interpreted in a few ways depending on context. Here's a breakdown of potential meanings and applications:.... So it's not really giving the model valuable tokens to work with.

samstave 2y ago
> You can see this with even just very large prompts, the larger the context the worse the quality, despite model developers claiming ever larger context windows. This is precicely the problem I am attempting to address. The point is that I want to be able to frame the way a particular complex iterative query discovery can be accomplished where I utilized already baked frames of how I want to address the problem, similar to how the Baseball workflow is on txtai - but in a more robust manner. https://github.com/neuml/txtai/blob/master/examples/42_Prompt_driven_search_with_LLMs.ipynb https://github.com/neuml/txtai/blob/master/examples/42_Promp... Where I can build particular query bots that can search properly based on the context where I dont have to spend time constructing the context in my prompt, I tell it to APPLY a context to the following prompt. So it limits the domain, formulates the domain to search. Start looking at how baseball players' Dossier stacks up: start building a context for player: https://i.imgur.com/2xvreAF.png https://i.imgur.com/2xvreAF.png Then I can take that idea, and start applying it to say public figures: https://i.imgur.com/pFODyms.png https://i.imgur.com/pFODyms.png Build a caged set of lenses, think of it like an agent - but when I told it to act like a PHD Chemist specializing in materials science as an expert... I then framed which space in that area to focus the response from - which was as an expert in PFTE. so, when I want to have it lookup aspects of say a bill passing, I can ask it "Looking at the congress-person lattice, what in their lattice is related to these various aspects from that bill/act/contract/bailout where you can see that when something occurred, these properties for this object were affected in such manner. Then you just have say trigger that will say "Whoever has this sheet balance out after this thing occurs" and I have refined how that information will be sought out - for example, it will learn where to grab the best pieces of information over time, as it can be asked "which sources had the best information for [property]" Then you say follow what MLOps was doing [0] for better understanding language used to define the same events. https://mlops.systems/posts/2024-06-25-evaluation-finetuning-manual-dataset.html https://mlops.systems/posts/2024-06-25-evaluation-finetuning... We can then see an article, and then look at whatever lattice we have defined to see who may be connected to that thing - and how. By developing the discernment model for that domain - you can have the ai evaluate the spaces of interest in various ways to then craft the lattice file - which then ideally keeps the FN thing focused so the context window can be more complex without superfluous token/memory cruft. So longer iterative can be made on the same subjects with room to context with less hallucination/forgetting. And the outputs of those can be fed to other workflows in txtai or succinctly wrapped in MagicLoop widgets for great justice.
Bluestein 2y ago
> don't think inventing your own terms (Which therefore have extremely weak embeddings to match with the embeddings of the models training content) Well put, in that, that was - in essence - what I was concerned with, with that term.-
- samstave 2y ago
  For me: in-head-logic is so FN weird when you attempt to speak it out loud. It makes the brilliant idiots and idiots brilliant! (what is best to capture logic?) --- Honestly guys - I am doing my best. But I do so with as an is-much informed place as I can... (this is why I always give benefit of the doubt ((I dont know the concept I amy be conveying)) HALP)
  - Bluestein 2y ago
    > (what is best to capture logic?) - Either formal logic notation or a state machine? (Depending on what you intend by 'logic' and 'capture' ...) PS. A Karnaugh map might be of help. As an added bonus, you get to simplify things.-

samstave 2y ago

> You can see this with even just very large prompts, the larger the context the worse the quality, despite model developers claiming ever larger context windows. This is precicely the problem I am attempting to address. The point is that I want to be able to frame the way a particular complex iterative query discovery can be accomplished where I utilized already baked frames of how I want to address the problem, similar to how the Baseball workflow is on txtai - but in a more robust manner. https://github.com/neuml/txtai/blob/master/examples/42_Prompt_driven_search_with_LLMs.ipynb https://github.com/neuml/txtai/blob/master/examples/42_Promp... Where I can build particular query bots that can search properly based on the context where I dont have to spend time constructing the context in my prompt, I tell it to APPLY a context to the following prompt. So it limits the domain, formulates the domain to search. Start looking at how baseball players' Dossier stacks up: start building a context for player: https://i.imgur.com/2xvreAF.png https://i.imgur.com/2xvreAF.png Then I can take that idea, and start applying it to say public figures: https://i.imgur.com/pFODyms.png https://i.imgur.com/pFODyms.png Build a caged set of lenses, think of it like an agent - but when I told it to act like a PHD Chemist specializing in materials science as an expert... I then framed which space in that area to focus the response from - which was as an expert in PFTE. so, when I want to have it lookup aspects of say a bill passing, I can ask it "Looking at the congress-person lattice, what in their lattice is related to these various aspects from that bill/act/contract/bailout where you can see that when something occurred, these properties for this object were affected in such manner. Then you just have say trigger that will say "Whoever has this sheet balance out after this thing occurs" and I have refined how that information will be sought out - for example, it will learn where to grab the best pieces of information over time, as it can be asked "which sources had the best information for [property]" Then you say follow what MLOps was doing [0] for better understanding language used to define the same events. https://mlops.systems/posts/2024-06-25-evaluation-finetuning-manual-dataset.html https://mlops.systems/posts/2024-06-25-evaluation-finetuning... We can then see an article, and then look at whatever lattice we have defined to see who may be connected to that thing - and how. By developing the discernment model for that domain - you can have the ai evaluate the spaces of interest in various ways to then craft the lattice file - which then ideally keeps the FN thing focused so the context window can be more complex without superfluous token/memory cruft. So longer iterative can be made on the same subjects with room to context with less hallucination/forgetting. And the outputs of those can be fed to other workflows in txtai or succinctly wrapped in MagicLoop widgets for great justice.

Bluestein 2y ago

> don't think inventing your own terms (Which therefore have extremely weak embeddings to match with the embeddings of the models training content) Well put, in that, that was - in essence - what I was concerned with, with that term.-

samstave 2y ago
For me: in-head-logic is so FN weird when you attempt to speak it out loud. It makes the brilliant idiots and idiots brilliant! (what is best to capture logic?) --- Honestly guys - I am doing my best. But I do so with as an is-much informed place as I can... (this is why I always give benefit of the doubt ((I dont know the concept I amy be conveying)) HALP)
- Bluestein 2y ago
  > (what is best to capture logic?) - Either formal logic notation or a state machine? (Depending on what you intend by 'logic' and 'capture' ...) PS. A Karnaugh map might be of help. As an added bonus, you get to simplify things.-