3 ms·

Do you have a solution for degradation in accuracy when compiling larger amounts of llm-produced text? I am also building LLM knowledge/memory systems and I've

Do you have a solution for degradation in accuracy when compiling larger amounts of llm-produced text? I am also building LLM knowledge/memory systems and I've been surprised how bad LLMs are, even SOTA models, at summarizing non-trivial input batches of text. They get things wrong, distort the underlying meaning or data, etc.

dariusmonsef 1mo ago
A constant challenge. Don't have a perfect solution for it yet, but importantly every change to any article logs who did it, what it did and the reasoning behind it. So I have enough data to work with as I continue to improve things.
embedding-shape 1mo ago
Divide and conquer essentially, is what I've found so far to work best. Split things into smaller and smaller chunks to independently be verified, double-check everything, then coalesce upwards with verified summarizations. Have benchmarks for every single task and sub-task that will happen everywhere a LLM is involved, so you can measure improvements. Takes a ton more effort and tokens in the system itself obviously, but if you're not paying per token, it seems to work pretty well, albeit feels slightly over-engineered already.
- cindyllm 1mo ago
  [dead]
- jayct 1mo ago
  > Have benchmarks for every single task and sub-task that will happen everywhere a LLM is involved, so you can measure improvements exactly. are there standard approaches or tools specifically for instrumenting LLM utilization, for applications rather than model development? i use claude code a bit. i relentlessly push it to keep its work ephemera in source controlled and discoverable locations, following a lot of ozbrain's motivation. i also have claude generate task logs for all spawns. the theory with the task logs is to try and build some standard measures so i can see whether a thing i do (writing skills, experimenting with third-party MCP tooling, etc) actually helps in a way that is more robust than gut. i have yet to turn that effort into something useful. i'm just starting to get enough data to try to do some diving. the problem is that this feels like the classic problem of "how do we measure productivity"? i was thinking very basic efficiency like "tokens in per tokens out" (maybe with a domain-specific value weighting function on the output tokens, e.g. code tokens > memory tokens > chat tokens) could be an possibility. the point being, seems like you'd want to become pretty sophisticated about measurement to sell a product like ozbrain, as a basis for continuous improvement across major versions. or convincing people to use it, quantitatively rather than qualitatively.
freemindcore 1mo ago
[flagged]

dariusmonsef 1mo ago

A constant challenge. Don't have a perfect solution for it yet, but importantly every change to any article logs who did it, what it did and the reasoning behind it. So I have enough data to work with as I continue to improve things.

embedding-shape 1mo ago

Divide and conquer essentially, is what I've found so far to work best. Split things into smaller and smaller chunks to independently be verified, double-check everything, then coalesce upwards with verified summarizations. Have benchmarks for every single task and sub-task that will happen everywhere a LLM is involved, so you can measure improvements. Takes a ton more effort and tokens in the system itself obviously, but if you're not paying per token, it seems to work pretty well, albeit feels slightly over-engineered already.

cindyllm 1mo ago
[dead]
jayct 1mo ago
> Have benchmarks for every single task and sub-task that will happen everywhere a LLM is involved, so you can measure improvements exactly. are there standard approaches or tools specifically for instrumenting LLM utilization, for applications rather than model development? i use claude code a bit. i relentlessly push it to keep its work ephemera in source controlled and discoverable locations, following a lot of ozbrain's motivation. i also have claude generate task logs for all spawns. the theory with the task logs is to try and build some standard measures so i can see whether a thing i do (writing skills, experimenting with third-party MCP tooling, etc) actually helps in a way that is more robust than gut. i have yet to turn that effort into something useful. i'm just starting to get enough data to try to do some diving. the problem is that this feels like the classic problem of "how do we measure productivity"? i was thinking very basic efficiency like "tokens in per tokens out" (maybe with a domain-specific value weighting function on the output tokens, e.g. code tokens > memory tokens > chat tokens) could be an possibility. the point being, seems like you'd want to become pretty sophisticated about measurement to sell a product like ozbrain, as a basis for continuous improvement across major versions. or convincing people to use it, quantitatively rather than qualitatively.

freemindcore 1mo ago

[flagged]