3 ms·

Awesome idea! The entity tracking is very exciting, most interesting part imo I think the budget is noticeable in the sentiment analysis unfortunately, the tag

by sdwr 8mo ago

Awesome idea! The entity tracking is very exciting, most interesting part imo I think the budget is noticeable in the sentiment analysis unfortunately, the tags and entity recognition are good but the sentiment ratings themselves seem pretty sloppy.

ddtaylor 8mo ago
I think it's mostly prompting, but I will be experimenting with this more. The prompt currently is garbage IMO You are an expert analyst of the Hacker News community. Analyze submissions for the underlying ideas, concepts, technologies, and entities being discussed. Write all summaries in third-person analytical prose. Do NOT start sentences with "The user", "The commenter", "The author", or "This post". Instead, lead with the substance: describe the idea, argument, or phenomenon directly. Good: "Decentralized identity systems could reduce reliance on corporate gatekeepers." Bad: "The user discusses how decentralized identity systems work." (Source: https://github.com/devrupt-io/ethos/blob/67670eb2855b84d389d5a4d44c29cd8cb649234f/backend/src/services/openrouter.ts#L228 https://github.com/devrupt-io/ethos/blob/67670eb2855b84d389d...)
- atoav 8mo ago
  Garbage, why? That is the insightful bit you chose to omit. How would you do it instead?
  - ddtaylor 8mo ago
    It leaves a lot of interpretation to the model. For example it doesn't give any guidance on concept naming or disambiguation, which leaves all of that work to the JSON schema. In my experience it's much more effective to reference key terms or ideas in the JSON schema and then explain those and their constraints in the system prompt. This is one reason why people often think one model performs better than another for tasks they are both capable of. The real question IMO becomes, does porking all of that extra input prompt (a) eat too much context or (b) increase cost too much. We will put an update on this in the future and post it in our blog, https://blog.devrupt.io/ https://blog.devrupt.io/

by sdwr 8mo ago

ddtaylor 8mo ago

I think it's mostly prompting, but I will be experimenting with this more. The prompt currently is garbage IMO You are an expert analyst of the Hacker News community. Analyze submissions for the underlying ideas, concepts, technologies, and entities being discussed. Write all summaries in third-person analytical prose. Do NOT start sentences with "The user", "The commenter", "The author", or "This post". Instead, lead with the substance: describe the idea, argument, or phenomenon directly. Good: "Decentralized identity systems could reduce reliance on corporate gatekeepers." Bad: "The user discusses how decentralized identity systems work." (Source: https://github.com/devrupt-io/ethos/blob/67670eb2855b84d389d5a4d44c29cd8cb649234f/backend/src/services/openrouter.ts#L228 https://github.com/devrupt-io/ethos/blob/67670eb2855b84d389d...)

atoav 8mo ago
Garbage, why? That is the insightful bit you chose to omit. How would you do it instead?
- ddtaylor 8mo ago
  It leaves a lot of interpretation to the model. For example it doesn't give any guidance on concept naming or disambiguation, which leaves all of that work to the JSON schema. In my experience it's much more effective to reference key terms or ideas in the JSON schema and then explain those and their constraints in the system prompt. This is one reason why people often think one model performs better than another for tasks they are both capable of. The real question IMO becomes, does porking all of that extra input prompt (a) eat too much context or (b) increase cost too much. We will put an update on this in the future and post it in our blog, https://blog.devrupt.io/ https://blog.devrupt.io/