3 ms·

What if the most interesting finding ends up buried under a vague title? Aside from the "self-generated skills" aspect, there isn't much there that meaningfully

What if the most interesting finding ends up buried under a vague title? Aside from the "self-generated skills" aspect, there isn't much there that meaningfully warrants deeper discussion. I chose a title that directly reflects an interesting finding - something that offers substantial insight to the community. I think the rule should be applied with some nuance; in this case, being explicit is a net positive. I have no interest in linkbait, and I hope that's evident from my previous submissions

dang 8mo ago
Yes, I appreciate that, and yes there is room for nuance. But I think you went too far in this case, meaning that the delta between the article title and the submission title was too large. For example, the word "useless" appears nowhere in the article abstract nor in the article body. That's a big delta. I was starting to type out a longer explanation but I ran out of time - however, I probably would just be repeating things I've said many times before, for example here: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20influence%20titles&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... - perhaps some of that would be helpful. You're a fine HN contributor and obviously a genuine user and I hope I didn't come across as critical! From our side it's just standard HN moderation practice. The way we deal with titles has been stable for many years. It isn't entirely mechanical, there are many subtleties (back to the nuance thing) but the core rules have served the site really well. THe main thing we want to avoid is having the title field be a mini-genre where whoever makes the submission gets to put their spin on the article.
- mustaphah 8mo ago
  I'm fine with editing the title, and I see your point. In retrospect, I'd probably avoid "useless." While it's a fairly descriptive term for their finding, it probably carries a subjective tone.
xdotli 8mo ago
Thanks @dang for moderating! This is indeed not our original findings and this is a sub conclusion for an ablation we did to remove the confound of LLMs internal domain knowledge. Thanks for submitting for us @mustaphah here's a little bit more details on how we approach this: > I would frame the 'post-trajectory generated skills' as feedback-generated skills, so is Letta: https://www.letta.com/blog/skill-learning https://www.letta.com/blog/skill-learning. We haven't seen existing research or hypothesis debating whether the skills improvement might come from the skill prompt themselves activated knowledge in LLMs that can help itself. So that's why we added an ablation of 'pre-trajectory generated skills' because we have that hypothesis and this seems a very clean way to test it. Also it is very logical that feedback generated skills can help, because it most certainly contain the failure mode of agents on that specific tasks.
- mustaphah 8mo ago
  Yeah, I got your point when I read the paper. You're essentially controlling for "latent domain knowledge." I might have been a bit blunt with the title - sorry about that, but I still think it was a good title. From what I've observed, a lot of Skills on GitHub are just AI-generated without any feedback or deliberative refinement. Many thought those would still be valuable, but you've shown evidence otherwise.
  - xdotli 8mo ago
    no worries it's totally fine! there is indeed work needs to be done on the feedbacks generated skills. Thanks for helping us submitting on HackerNews. And for > a lot of Skills on GitHub are just AI-generated without any feedback or deliberative refinement. Many thought those would still be valuable, but you've shown evidence otherwise. we do find most skills on the internet to be useless, and thanks to the generosity of https://skillsmp.com/ https://skillsmp.com/ author, we were able to get all the meta data of the 99k skills indexed on his website. We did a lot of filtering and deduping and we discovered ~40k+ skills were relevant at the time we did the study.

dang 8mo ago

Yes, I appreciate that, and yes there is room for nuance. But I think you went too far in this case, meaning that the delta between the article title and the submission title was too large. For example, the word "useless" appears nowhere in the article abstract nor in the article body. That's a big delta. I was starting to type out a longer explanation but I ran out of time - however, I probably would just be repeating things I've said many times before, for example here: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20influence%20titles&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... - perhaps some of that would be helpful. You're a fine HN contributor and obviously a genuine user and I hope I didn't come across as critical! From our side it's just standard HN moderation practice. The way we deal with titles has been stable for many years. It isn't entirely mechanical, there are many subtleties (back to the nuance thing) but the core rules have served the site really well. THe main thing we want to avoid is having the title field be a mini-genre where whoever makes the submission gets to put their spin on the article.

mustaphah 8mo ago
I'm fine with editing the title, and I see your point. In retrospect, I'd probably avoid "useless." While it's a fairly descriptive term for their finding, it probably carries a subjective tone.

xdotli 8mo ago

Thanks @dang for moderating! This is indeed not our original findings and this is a sub conclusion for an ablation we did to remove the confound of LLMs internal domain knowledge. Thanks for submitting for us @mustaphah here's a little bit more details on how we approach this: > I would frame the 'post-trajectory generated skills' as feedback-generated skills, so is Letta: https://www.letta.com/blog/skill-learning https://www.letta.com/blog/skill-learning. We haven't seen existing research or hypothesis debating whether the skills improvement might come from the skill prompt themselves activated knowledge in LLMs that can help itself. So that's why we added an ablation of 'pre-trajectory generated skills' because we have that hypothesis and this seems a very clean way to test it. Also it is very logical that feedback generated skills can help, because it most certainly contain the failure mode of agents on that specific tasks.

mustaphah 8mo ago
Yeah, I got your point when I read the paper. You're essentially controlling for "latent domain knowledge." I might have been a bit blunt with the title - sorry about that, but I still think it was a good title. From what I've observed, a lot of Skills on GitHub are just AI-generated without any feedback or deliberative refinement. Many thought those would still be valuable, but you've shown evidence otherwise.
- xdotli 8mo ago
  no worries it's totally fine! there is indeed work needs to be done on the feedbacks generated skills. Thanks for helping us submitting on HackerNews. And for > a lot of Skills on GitHub are just AI-generated without any feedback or deliberative refinement. Many thought those would still be valuable, but you've shown evidence otherwise. we do find most skills on the internet to be useless, and thanks to the generosity of https://skillsmp.com/ https://skillsmp.com/ author, we were able to get all the meta data of the 99k skills indexed on his website. We did a lot of filtering and deduping and we discovered ~40k+ skills were relevant at the time we did the study.