Did exactly that, hacked together a small pipeline in Nushell with simonw/llm. Even with GPT-4 turbo and given direct guidelines on common spam heuristics, it s

Did exactly that, hacked together a small pipeline in Nushell with simonw/llm. Even with GPT-4 turbo and given direct guidelines on common spam heuristics, it seems to perform worse than a Bayesian BoW. Endless trails of questions with no relation to the title it describes as informative, presence of affiliate links often gets forgotten in a mass of tokens in their tracking parameters, relevance is consistently near 0.8 even if there's relationship between title and content, and as for insincerity, our favorite BS generator cannot for the life of it correctly recognize its own creations. Your ideas for metrics are good, but LLMs seem to be quite terrible at any of these. A simple set of heuristics and maybe a tiny language model for named entity detection and "vibe checking" would serve you much better. Also, a lot of the worst offenders seem to use the same Q&A +- conclusion structure, which Viktor from marginalia.nu wrote a simple heuristic for, which I recall he said did wonders for pruning it. Solving SEO spam is easy when you aren't the one being optimized against. What's left is scaling and information retrieval.