3 ms·
It would be amazing if you: a) There were a test / eval suite to determine which model works best for what. It could be divided into a training suite and test
by bravura 2y ago
It would be amazing if you:
a) There were a test / eval suite to determine which model works best for what. It could be divided into a training suite and test suite. (Training tasks can be used for training, test tasks only for evaluation.) Possibly a combination of unit tests against known xpaths, and integration tests that are multi-step and end in a measurable result. I know the web is constantly changing, so I'm not 100% sure how this should work.
b) There were some sort of wiki, or perhaps another repo or discussion board, of community-generated prompt recipes for particular actions.
- gregpr07 2y agoA) we plan on thoroughly testing that with Mind2Web dataset. They have a very robust set of (persistant) selectors B) so, shadcn for prompts for web agents haha :) but I agree, that would be SICK! Just go to browseruse and get the prompt for your specific use case
- maggreenWAI 2y agoA) For Mind2Web: because there are multiple ways to reach a goal state - any thoughts how to evaluate if a task was successful? Should we let the LLM/ other LLM evaluate it?