dacharyc

4 ms·

1.
by dacharyc 6mo ago
I suspect this is a result of relevance-based retrieval. In my colleague's testing, they found that sometimes the content comes back out of order or not at all, depending on the implementation's interpretation of which chunks of c
2.
by dacharyc 6mo ago
Ah, good point - this was intended to be a bonus point for agents that do not use a working browser, to evaluate whether they understood and communicated that the content was missing. But it should be an either/or - not a missed point
3.
by dacharyc 6mo ago
Yeah, my colleague and I have been seeing in testing how much this is actually a problem in practice. It has been - surprising, and a little dismaying - how much this negatively impacts content retrieval and results in poor UX.
4.
by dacharyc 6mo ago
Hah, I actually originally had some stuff in the site that Claude Code's summarization agent (presumably Haiku) thought was prompt injection, and refused to give content to the foreground agent I was working with. I had to remove som
5.
by dacharyc 6mo ago
Hah, that's actually what drove me to try to create this to begin with. I've been writing a lot about these issues, and someone said to me: > It'd be nice to have a test harness: "Test my agent," to score them an
6.
by dacharyc 6mo ago
Yeah, good call, we're on the same page about that. I designed this tool (agentreadingtest.com) to raise awareness of these issues in a more general way, so people can point agents at it and see how it performs for them. Separately, I
7.
by dacharyc 6mo ago
Hey there - I'm the test author, and you've hit on one of the main points. For the summarization/relevance-based content return, this is a consideration for some of the agent platforms (although I've found others actua
8.
by dacharyc 6y ago
Actually, according to my count, there are 9 other companies in this space that offer deploy environments as a service, in addition to 6 platforms that also offer this functionality as part of the platform. I've been watching this spac
9.
by dacharyc 6y ago
There are a bunch of services that already do this in some permutation or another: Tugboat: https://tugboat.qa/ Squash: https://www.squash.io/ Dockup: https://getdockup.com/ Release: https