3 ms·

A quick smoke check takes just a few minutes. "Follow each link in this document. Read each link's contents against the contents in this document. Create a rep

A quick smoke check takes just a few minutes. "Follow each link in this document. Read each link's contents against the contents in this document. Create a report: for each link list a working hyperlink, whether it exists, what claim it supports, whether it supports or fails to support it, and why" If it returns a report claiming all correct? That's promising, but human verification is important. You've got a list of hyperlinks, and a list of claims; so you can click each with middle-mouse, Ctrl-F 'till you find the point, and close the tab when you do. If you find any discrepancies ? Your initial prompt was malformed and/or you picked the wrong LLM, the wrong human, or possibly all three. Whatever the way, the results are built on quicksand; you'll need to start over. If no sources are provided? Well now: "If there ain't no sources it never happened." Compare double-entry bookkeeping. It needs to all add up. If you're 1 cent off, that means something is broken. Idem if a single reference is off, it polluted the context. (This works for human-generated and hybrid documents too. Polluted reasoning is polluted reasoning. The process is what counts.)

deleted 5mo ago
[deleted]
throw310822 5mo ago
I would love to do it at scale on many online publications, and publish the results. That would teach 'em.
flail 5mo ago
A quick smoke test, then. Gemini 3, Thinking Mode. The article: https://techtrenches.dev/p/the-human-cost-of-10x-how-ai-is-physically https://techtrenches.dev/p/the-human-cost-of-10x-how-ai-is-p... The prompt: literally what you suggested. Gemini: The article focuses on the environmental and human labor costs of scaling Artificial Intelligence, specifically focusing on water usage, electricity, and "ghost work." Which is hilarious, since the article doesn't even mention the words "water" or "electricity." Gemini remains unfazed, reporting the links that are not in the article (some don't exist at all) to make the final ruling: "The Tech Trenches document is highly accurate in its citations." Now, I know. Had I used Claude Code with relevant skills, it would have done better. But would it be good?
- Kim_Bruning 5mo ago
  Interesting! Where did you apply it? Can you show your output in more detail? It's more like a small script, and it's supposed to extract urls and generate a table. Here's my result in Claude Web for comparison: https://claude.ai/public/artifacts/d76936f2-c97b-4bff-9205-23474b5970c5 https://claude.ai/public/artifacts/d76936f2-c97b-4bff-9205-2... Claude web finds a number of small discrepancies in the sources, which I manually crosschecked and seem consistent with a human mixing things up slightly. + I also tested in gemini 3 flash preview, which generates an actual table (twice). It doesn't flag any discrepancies, which is consistent with it being a weaker model. But the urls and claims are listed and line up, so you've got your verification table to work with. (it's a semantic formatting task, so that part would be hard to mess up) + Gemini 3.1 pro yields a fairly aggressive report. https://aistudio.google.com/app/prompts?state=%7B%22ids%22:%5B%221s8fhl_INrav1OTY47woC6VHnVEx8xBa4%22%5D,%22action%22:%22open%22,%22userId%22:%22112986444731006327669%22,%22resourceKeys%22:%7B%7D%7D&usp=sharing https://aistudio.google.com/app/prompts?state=%7B%22ids%22:%... + ChatGPT free (specific model not listed) needed 2 tries, didn't properly follow the prompt even then. I guess I got what I paid for, and I needed to download; https://vps.kimbruning.nl/productivity/Ai%20Productivity%20Article%20Link%20Audit%20Report.pdf https://vps.kimbruning.nl/productivity/Ai%20Productivity%20A... (pdf), https://vps.kimbruning.nl/productivity/ai_productivity_article_link_audit_report.md https://vps.kimbruning.nl/productivity/ai_productivity_artic... (md) + Kimi K2.6 instant: https://www.kimi.com/share/19e2cc40-d012-89bf-8000-00006267f6a7 https://www.kimi.com/share/19e2cc40-d012-89bf-8000-00006267f... + Summary of results. https://claude.ai/public/artifacts/10a42111-a0ee-42f3-b6d2-ab85f22a1a03 https://claude.ai/public/artifacts/10a42111-a0ee-42f3-b6d2-a... All of the models extracted the URLs into a table just fine, and that part at least is a lot easier than writing a perl script used to be in the '90s ;-) . The first part is the important bit so you as a human can "check your fucking sources". The second part the models handle variously, each does find discrepancies. None of them find all of them, but that makes sense: this is a fairly polished piece and it ideally shouldn't have discrepancies at all to begin with. So: it worked as a smoke check just fine in the above. Doing more than a quick smoke check obviously requires a somewhat more involved procedure.
- simianwords 5mo ago
  Gemini is famously bad at these things. Try using ChatGPT.

deleted 5mo ago

[deleted]

throw310822 5mo ago

I would love to do it at scale on many online publications, and publish the results. That would teach 'em.

flail 5mo ago

A quick smoke test, then. Gemini 3, Thinking Mode. The article: https://techtrenches.dev/p/the-human-cost-of-10x-how-ai-is-physically https://techtrenches.dev/p/the-human-cost-of-10x-how-ai-is-p... The prompt: literally what you suggested. Gemini: The article focuses on the environmental and human labor costs of scaling Artificial Intelligence, specifically focusing on water usage, electricity, and "ghost work." Which is hilarious, since the article doesn't even mention the words "water" or "electricity." Gemini remains unfazed, reporting the links that are not in the article (some don't exist at all) to make the final ruling: "The Tech Trenches document is highly accurate in its citations." Now, I know. Had I used Claude Code with relevant skills, it would have done better. But would it be good?

Kim_Bruning 5mo ago
Interesting! Where did you apply it? Can you show your output in more detail? It's more like a small script, and it's supposed to extract urls and generate a table. Here's my result in Claude Web for comparison: https://claude.ai/public/artifacts/d76936f2-c97b-4bff-9205-23474b5970c5 https://claude.ai/public/artifacts/d76936f2-c97b-4bff-9205-2... Claude web finds a number of small discrepancies in the sources, which I manually crosschecked and seem consistent with a human mixing things up slightly. + I also tested in gemini 3 flash preview, which generates an actual table (twice). It doesn't flag any discrepancies, which is consistent with it being a weaker model. But the urls and claims are listed and line up, so you've got your verification table to work with. (it's a semantic formatting task, so that part would be hard to mess up) + Gemini 3.1 pro yields a fairly aggressive report. https://aistudio.google.com/app/prompts?state=%7B%22ids%22:%5B%221s8fhl_INrav1OTY47woC6VHnVEx8xBa4%22%5D,%22action%22:%22open%22,%22userId%22:%22112986444731006327669%22,%22resourceKeys%22:%7B%7D%7D&usp=sharing https://aistudio.google.com/app/prompts?state=%7B%22ids%22:%... + ChatGPT free (specific model not listed) needed 2 tries, didn't properly follow the prompt even then. I guess I got what I paid for, and I needed to download; https://vps.kimbruning.nl/productivity/Ai%20Productivity%20Article%20Link%20Audit%20Report.pdf https://vps.kimbruning.nl/productivity/Ai%20Productivity%20A... (pdf), https://vps.kimbruning.nl/productivity/ai_productivity_article_link_audit_report.md https://vps.kimbruning.nl/productivity/ai_productivity_artic... (md) + Kimi K2.6 instant: https://www.kimi.com/share/19e2cc40-d012-89bf-8000-00006267f6a7 https://www.kimi.com/share/19e2cc40-d012-89bf-8000-00006267f... + Summary of results. https://claude.ai/public/artifacts/10a42111-a0ee-42f3-b6d2-ab85f22a1a03 https://claude.ai/public/artifacts/10a42111-a0ee-42f3-b6d2-a... All of the models extracted the URLs into a table just fine, and that part at least is a lot easier than writing a perl script used to be in the '90s ;-) . The first part is the important bit so you as a human can "check your fucking sources". The second part the models handle variously, each does find discrepancies. None of them find all of them, but that makes sense: this is a fairly polished piece and it ideally shouldn't have discrepancies at all to begin with. So: it worked as a smoke check just fine in the above. Doing more than a quick smoke check obviously requires a somewhat more involved procedure.
simianwords 5mo ago
Gemini is famously bad at these things. Try using ChatGPT.