4 ms·
Show HN: Bullshit Detector – agent skills that fact-check videos and articles
- skorniienko 2mo ago[flagged]
- skorniienko 2mo ago[flagged]
- cawingcrow 2mo agoThat's too much effort. Should have been a browser extension that uses ollama so it can run with local models or maybe even openrouter...
- skorniienko 2mo ago[flagged]
- skorniienko 2mo agoThat's fair. Install is a one time real friction and extension would be much simpler. But, there is a reason why it's not an extension. At least for now. Whole product isn't a UI or service, it's a set of skills for your AI agent of choice, which gets content from URL or whatever shared as a source, splits it into claims and does extensive web searches for every claim to compare it with statement from provided content. On local models constraint isn't a model itself - but search. Model don't judge from it's trained memory, so even local model will need a backend for search, otherwise it can't provide a verdict.
- khalic 2mo agoAwesome idea
- skorniienko 2mo agoThank you! Would love to hear your results if you'll have a chance to try it
- khalic 2mo agoI have an AI newsdesk i vibed together because I was losing too much time hunting for facts inside the torrent of misinformation, I'll try to hook it up to the writers
- skorniienko 2mo agoFantastic! Dying to hear the feedback. <3
- mysterydip 2mo agoWhat do you use as “ground truth”? The page says “independent sources”, and I’m sure there’s too many to list, but my question is how are they vetted as being truthful and how are two sources with opposite viewpoints reconciled?
- skorniienko 2mo agoThere is no whitelist source. It can't rate sources for truthfulness. When sources conflicts - skill drops verdict to misleading or unverifiable and both are linked. Can't pick a winner at the moment. That's probably the weakest part and still a judgement call for a model.
- gutechh 2mo agoMaybe you read them already, but wikipedia has a bunch of interesting pages about sourcing and truth like: https://en.wikipedia.org/wiki/Wikipedia:Verifiability,_not_truth https://en.wikipedia.org/wiki/Wikipedia:Verifiability,_not_t... and https://en.wikipedia.org/wiki/Wikipedia:Reliable_sources https://en.wikipedia.org/wiki/Wikipedia:Reliable_sources
- skorniienko 2mo agoI'll look at reliable-sources, and you are right "verifiability, not truth" is basically what I landed on too. Model doesn't get to decide what's true, it just needs to cite something. Will read reliable-sources page properly.
- desktopentree 2mo ago[flagged]
- simonw 2mo agoSomething I've found surprisingly effective is telling ChatGPT to "use credible sources" - you can then watch its thinking trace and see it do things like ruling out random blogs, considering media publications with a good reputation for fact checking, and double-checking information that seems unlikely.
- deleted 2mo ago[deleted]
- outime 2mo agoThe author is posting comments here (all flagged at this time) that are very clearly AI slop, which is quite ironic.
- skorniienko 2mo agoFair, drafted posts with claude, getting all my comments flagged. Thought that something is wrong with my account. Writing myself from here
- NetOpWibby 2mo agoBro, you don’t have to outsource your brain for basic communication on the internet. HN’s community has built-in bullshit detectors.
- skorniienko 2mo agoYou are right, no excuses. Will have few grammar errors but I'll own them. Cheers!
- MaxikCZ 2mo agoSounds like AI reply to me, lmao
- NetOpWibby 2mo agoSMH I don't get it
- N_Lens 2mo agoWho checks the checkers?
- skorniienko 2mo agoAt the moment - it's model's judgement upon search results for every claim from provided content. If search results returned inaccurate data, or were fabricated - it might affect a verdict for that claim, or highlight that it's unverifiable or contradictory. But there is a room for improvement, what do you suggest? Have a Judge agent which will check results?
- nhannht 2mo ago[flagged]
- qntmfred 2mo agome. I volunteer as ultimate arbiter of Truth.
- skorniienko 2mo agomaybe some community-driven and curated source blacklist could be implemented. It thing it would be too much for you doing that alone :)
- qntmfred 2mo agome.
- stoneman24 2mo agoTo paraphrase Monty Python “supreme executive power derives from a mandate from the masses, not from some farcical aquatic ceremony" (or authority based on a self declaration) In the age of ai, where you can flood the zone with easily generated content, even a democratic society might struggle to maintain an stable sense of shared reality and history
- receptopalak 2mo ago[flagged]
- ck2 2mo agoWashington Post gave up tracking Trump lies last term in 2021 because it became impossible by human hands with 21+ per day and over 30,000 in their database but with "AI" now it's possible not only to do non-stop but in REALTIME you could even just restrict the source of the check to the paper's own reporting the past fifty years * https://www.washingtonpost.com/graphics/politics/trump-claims-database/ https://www.washingtonpost.com/graphics/politics/trump-claim... I'd like to see that backfilled, all the way back to the "long form birth certificate" (remember that horror show)
- BugsJustFindMe 2mo agoI'm not sure AI helps for that particular task. If you just assume that 100% of everything he says is a lie you'll be 99% correct. Anyone who believes anything he says at this point is living in a constructed reality where facts never matter. Fact checking seems more useful for less pathological cases where it might make a difference.
- hunterpayne 2mo agoCool, now do Fauci... PS I've always assumed all politicians lie, I'm sure you don't think Pelosi or Bush Jr for example were some paragons of truth. Doesn't make it OK, but have some perspective.
- BugsJustFindMe 2mo ago> I've always assumed all politicians lie Everyone lies at various points to some degree or another, ranging from well-intentioned to malignant, and it takes tremendous sleight of hand to pretend that they're even close to comparable. > Cool, now do [1] Fauci... [1] Lied that PPE had no value for the public which he admitted later he did to try to preserve limited supplies for healthcare workers. Lied about the possibility of a lab leak having been the origin of COVID19. > [2] Bush Jr, [3] Pelosi [2] Lied about Iraq having WMDs and being connected to al-Qaeda to start a war. Built a torture program. [3] Used insider information to make money. Maybe also knew about the Bush torture program but didn't do anything to stop it if so. If you had to rank those three, what order would you place them in? > have some perspective. Indeed. Perspective: https://en.wikipedia.org/wiki/False_or_misleading_statements_by_Donald_Trump https://en.wikipedia.org/wiki/False_or_misleading_statements... > Doesn't make it OK, but Why the "but"? Are you justifying something? The difference between your framework and mine is that my framework statement ends with a period after "Doesn't make it OK."
- egeozcan 2mo agoCan someone please use this skill against the claims in its own repo?
- skorniienko 2mo agoDone, fixed 2 misleading lines in README. You can see example report in repo examples. Thanks for that, btw! At least someone questioned it :)
- egeozcan 2mo agoThat was an exceptionally fun to read slop :) Cool idea in any case!
- lynx97 2mo ago> Long output? Redirect to a file and read it from there: Hmm, smells AI generated. Why should an LLM (this is from a SKILL.md) care about $LINES?
- skorniienko 2mo agoDone, fixed in v0.4.1. Reason why agent reads from a file - it can do it in chunks or pass the path to subagent and keep it out from own context. I've tested it on 3hrs long interview YT video - worked smoothly.
- amelius 2mo agoFunny, I recently pasted output from Gemini into Claude, and it said it was total nonsense generated by an overconfident AI. Apparently the "AI-generated bullshit" detection is already part of LLMs.
- skorniienko 2mo agobut have Claude actually fact-checked it or just provided an opinion?
- LearnYouALisp 2mo ago"Lazy" evaluation
- amelius 2mo agoActually it was Fabel, and it totally tore apart Gemini's response.
- skorniienko 2mo agoWhat if you use OpenAI as a judge and check both responses? Not defending Gemini, just curious if it was that wrong, or Fable - that aggressive.
- leobg 2mo agoI don't even watch most videos anymore. YouTube app -> Share -> Gemini -> Summary. Only if that shows interesting content do I actually watch.
- skorniienko 2mo agoAnd you are not alone, that is one of the purposes why I am building this
- quard8 2mo agonice! my built-in BS detector: "comment "prompt" and i will send you a guide".
- skorniienko 2mo agothank you! comment "prompts" never works, right? Haven't seen single guide from it :D
- o10449366 2mo agogod i wish these low effort projects were banned from Show HN the entire premise of a "bullshit detector" that's entirely vibe-coded is laughable
- skorniienko 2mo agowhy it can't be vibe-coded? Would you be more pleased if it would be poorly hand-coded and claimed "No AI used during development"? What's the point? I don't mind that I've used AI to build this, and I'm open about it ;)
- o10449366 2mo ago[flagged]
- vivzkestrel 2mo ago- this is precisely the problem my current project solves - production grade app generator without AI :)
- h2aichat 2mo agoI like the idea! How can you let folks know they're being fed nonsense before they even finish the intro?
- skorniienko 2mo agocan't, at least with this design. Report itself takes minutes to proof-check. Realistic version is some kind of extension/plugin idea from earlier in a thread, but I'm not there yet
- alealvarezarg 2mo agoI think this would have a much bigger impact as short-form video content such as TikTok, Reels or Shorts. That's where most people consume this kind of content today, not in a .md file. What would really set it apart is adding fact-checking before publishing and letting it post directly to TikTok. That combination would make it something I'd actually use every day.
- skorniienko 2mo agoNot sure that I get your idea about short-form content. It already works with tiktok or yt shorts, especially if it has generated transcription/captions. Otherwise, skill will download video locally and ask your permission to run local models like whisper to transcribe video. Checking before you publish - that's interesting usecase I haven't thought about. Nothing stops you to fact-check your own content and decide what to do with it. Point is a report itself not more video content from it
- alealvarezarg 2mo ago[flagged]
- LearnYouALisp 2mo ago[If published] _as_
- madmaniak 2mo agoFor truth checking actual thinking and logic is required. It doesn't seem to be a proper tool for the task.
- skorniienko 2mo agoTrue, and it doesn't do logic. It decomposes content into claims and search for every claim citations. It catches a false premise, not a bad inference. Someone with valid reasoning from true facts to a wrong conclusion goes straight through.
- stevenalowe 2mo agothis - if trustworthy - would be great for social media and "news" sites
- skorniienko 2mo agoexactly, that is one of the use cases I see for these skills
- stevenalowe 2mo agostartup idea: a social media platform where illogical or false statements are rejected with an annotated refutation it would be a very quiet place...
- deleted 2mo ago[deleted]
- PlayerToo 2mo ago[flagged]