4 ms·

I've been building this for the last 6 months or so. I've basically got it working. The model is not the issue, the infra is. Keeping everything in context just

I've been building this for the last 6 months or so. I've basically got it working. The model is not the issue, the infra is. Keeping everything in context just isn't possible and LLMs, even Fable, don't mode switch well. To get around this I've built a database software that ingests as much digital information as possible, and annotates it, then creates timelines with resolution gradients (longer ago = less resolution) that it feeds to the LLM on every request. Then you have your cheap little MoE or ternary model just running in a loop, with an escalation pathway before it reaches the big expensive models. Currently it's doing things like reminding me to take allergy medication when I wake up because it's checked AQI or whatever, reminding me to stop at the market when I'm on my way to pick up the kids to get the cherry tomatoes I forgot, giving me heads up of what folks are expecting from me in certain meetings based on cross correlating email and calendar, etc. It's honestly the single most productive tool I've found for my ADHD.

nateb2022 2mo ago
Think of an LLM as a thesaurus, but for entire trains of thought rather than words. Your initial query yields something pertinent to the task at hand. But let it endlessly recurse and... you end up with something completely useless. People would do well to acquire at least a modest familiarity with what an LLM actually is. NLP is fascinating. So is entropy.
docjay 2mo ago
I’m curious why you don’t just use them like a Meeseeks box, rather than compressing and context stuffing into one. One only checks and categorizes your emails, another one for each category of email or even subcategory, one that only handles calendar additions, a different one to check it and notify you; you can go infinite with it. Hell, I’ll have one instance find a file and read it into the context of a different one because I don’t want a bunch of grep commands mucking up the context of the analysis. The find/read one exists for a few moments, as does the analysis one, and the ‘perform’ one is entirely different. I can run them all in parallel and use a queue if needed. I’m sure you have reasons for your setup though, so I’m curious how you landed on it.
- colingauvin 2mo ago
  I sort of do. For scheduled things I am using Goose and subagents per task. So I wake up at 6 am to a briefing that was composed by a ton of different agents. That works well in a narrow programmatic setting, but it breaks down in certain natural conversation context. For instance voice control of something like "text my sister that I'm running late" - self hosted models are terrible at figuring out who my sister is. I could annotate by hand but that is a never ending list that I'll always be one step behind. Another thing is just proactivity. For instance, if someone's birthday is coming up, the briefing may remind me a week early to get a card. But it will only do that if it's someone that I have a close relationship with, which it calculates by the magnitude of the cluster of conversation with them in embedded space. I'm not confident it's that most efficient way to do it, but it's quite a bit of fun.
  - docjay 2mo ago
    I was afraid my description sounded like agents, and it kind of is, but not like most implementations. Most use a “boss bot” to craft a prompt/system message and launch the model, and sometimes they redo it every time it launches the agent. It’s a low effort attempt that’s immediately flawed because it uses LLM output for LLM input. It can look like it’s working for some time, but the perplexity guarantees it’s a roll of the dice. That’s what eats away at these kinds of projects. “It was doing great until it rm’d prod.” Run the same exact prompt 100x in a single step test (one prompt, one response), hash the full responses, and you’ll see 10-50+ unique responses. The higher the unique the worse your prompt; focus on the system. A highly tuned system prompt will result in one response, even at a temperature of 1.0. Really. Once that’s done that’s the only thing it does and it’s the only one that does it and it never changes. Other LLMs that call ‘check_email()’ are unknowingly just passing a prompt to the specialized one. I use the API directly, craft a small Python script for the API call and task interface, then hyper-optimize the system/user prompt using test scenarios and automated loops. My system prompts rarely/never contain complete sentences, yet include all the tools/functions and requirements. Make your error messages user prompt instructions, not errors. That’s why “agent optimized” models exist. Chat models are primarily trained on conversational text, meaning the stackoverflow “How do I fix ‘too many levels of symbolic links’?” -> Explanation/resolution. It’s far less on “# ls broken_loop” -> “# ls: cannot access ‘broken_loop’: Too many levels of symbolic links” -> “# namei -l broken_loop” It’s not that the good ones are bad, but you’re leaning on the million training documents rather than the trillion. Anyway, go that route with your system. Think about it more like automating a factory floor rather than hiring interns. The least efficient methods, by definition, have the most room for improvement, which means they have the greatest reward potential, but for that one “eureka” moment. The path less taken is often interesting, but the ill-advised path still has fruit on the trees.

nateb2022 2mo ago

Think of an LLM as a thesaurus, but for entire trains of thought rather than words. Your initial query yields something pertinent to the task at hand. But let it endlessly recurse and... you end up with something completely useless. People would do well to acquire at least a modest familiarity with what an LLM actually is. NLP is fascinating. So is entropy.

docjay 2mo ago

I’m curious why you don’t just use them like a Meeseeks box, rather than compressing and context stuffing into one. One only checks and categorizes your emails, another one for each category of email or even subcategory, one that only handles calendar additions, a different one to check it and notify you; you can go infinite with it. Hell, I’ll have one instance find a file and read it into the context of a different one because I don’t want a bunch of grep commands mucking up the context of the analysis. The find/read one exists for a few moments, as does the analysis one, and the ‘perform’ one is entirely different. I can run them all in parallel and use a queue if needed. I’m sure you have reasons for your setup though, so I’m curious how you landed on it.

colingauvin 2mo ago
I sort of do. For scheduled things I am using Goose and subagents per task. So I wake up at 6 am to a briefing that was composed by a ton of different agents. That works well in a narrow programmatic setting, but it breaks down in certain natural conversation context. For instance voice control of something like "text my sister that I'm running late" - self hosted models are terrible at figuring out who my sister is. I could annotate by hand but that is a never ending list that I'll always be one step behind. Another thing is just proactivity. For instance, if someone's birthday is coming up, the briefing may remind me a week early to get a card. But it will only do that if it's someone that I have a close relationship with, which it calculates by the magnitude of the cluster of conversation with them in embedded space. I'm not confident it's that most efficient way to do it, but it's quite a bit of fun.
- docjay 2mo ago
  I was afraid my description sounded like agents, and it kind of is, but not like most implementations. Most use a “boss bot” to craft a prompt/system message and launch the model, and sometimes they redo it every time it launches the agent. It’s a low effort attempt that’s immediately flawed because it uses LLM output for LLM input. It can look like it’s working for some time, but the perplexity guarantees it’s a roll of the dice. That’s what eats away at these kinds of projects. “It was doing great until it rm’d prod.” Run the same exact prompt 100x in a single step test (one prompt, one response), hash the full responses, and you’ll see 10-50+ unique responses. The higher the unique the worse your prompt; focus on the system. A highly tuned system prompt will result in one response, even at a temperature of 1.0. Really. Once that’s done that’s the only thing it does and it’s the only one that does it and it never changes. Other LLMs that call ‘check_email()’ are unknowingly just passing a prompt to the specialized one. I use the API directly, craft a small Python script for the API call and task interface, then hyper-optimize the system/user prompt using test scenarios and automated loops. My system prompts rarely/never contain complete sentences, yet include all the tools/functions and requirements. Make your error messages user prompt instructions, not errors. That’s why “agent optimized” models exist. Chat models are primarily trained on conversational text, meaning the stackoverflow “How do I fix ‘too many levels of symbolic links’?” -> Explanation/resolution. It’s far less on “# ls broken_loop” -> “# ls: cannot access ‘broken_loop’: Too many levels of symbolic links” -> “# namei -l broken_loop” It’s not that the good ones are bad, but you’re leaning on the million training documents rather than the trillion. Anyway, go that route with your system. Think about it more like automating a factory floor rather than hiring interns. The least efficient methods, by definition, have the most room for improvement, which means they have the greatest reward potential, but for that one “eureka” moment. The path less taken is often interesting, but the ill-advised path still has fruit on the trees.