9 ms·

I worry about the ongoing ratcheting of the arms races that AI creates. It's nice that we have better tools to detect problems, but those tools are also readily

• explaininjs 3y ago
  Soon enough we’ll re-learn that you can’t trust anything except that what you can personally reproduce. Personally I’m glad. Too many years of people subscribing to the most bone headed theories and saying “look, it’s peer reviewed!”.
  • foota 3y ago
    While you shouldn't trust everything, trust is certainly still necessary since few people have the experience or resources to do independent scientific studies.
    • brutusborn 3y ago
      Of course you will still need to trust others to reproduce results in areas with which you are unfamiliar, but this is still better than implicitly trusting peer reviewers. Same thing with software security: if software is open source then I still need to trust someone else to determine if the source code is malicious, but it is better than implicitly trusting software because e.g. “a contractor has done a security audit.” Basically I don’t believe you can truly trust closed-anything, especially scientific analysis. You need _less_ trust when the thing is reproducible, which is the best you can ask for.
      • Guvante 3y ago
        Science is very reproducibile by design, half the goal of peer review is to ensure that. The problem is the incentive structure is not aligned with that and so reproduction is never done. I would instead argue that consensus is the best bar we have to judge science. If the majority of author papers align that means either there is a conspiracy or some fraction of things was reproducibile.
        • brutusborn 3y ago
          If that is the case, why did the replication crisis occur despite peer review? And why don’t most studies publish code or data? If you look closely most papers today are not reproducible. Consensus is a horrible way to judge science. The whole point of basing knowledge on empirical data is so that you don’t need to defer to authorities or consensus (nullius in verba). Consensus based science is just a synonym for Kuhnian paradigms (aka a scientific popularity contest). See Lakatos and Feyerabend for why this is not a logical or useful way to judge science. Imagine a field where the consensus methods are later found to be in error. In this case the only valuable thing to do is to challenge the consensus so the field adopts better methods. In the meantime, following the consensus guarantees following falsehoods because everyone is wrong in the same way. In many cases the minority heterodox views are the most correct.
          • Guvante 3y ago
            Replication crisis occurred because nobody replicated. It wasn't that you couldn't (although I slightly overgeneralized the ease of replication I will admit). Pretending that the tools we have aren't useful because they aren't perfect isn't valuable. Heliocentric models were good for science even if they were wrong. You aren't saying anything that moves the line towards a better situation just pointlessly complaining that it isn't good enough.
            • brutusborn 3y ago
              I think you should reread the original replication crisis paper, a huge proportion were simply not able to be replicated based on information in the original papers. I agree heliocentric models were useful, but if scientists had followed the consensus methods that produced heliocentrism then we would not have gone past it. It was heterodox theorists who resisted Catholic authority who progressed science. The Vatican are not good peer-reviewers. If you want a better solution look to Lakatos’ Methodology of Scientific Research Programmes. It is much better than the status quo of relying on peer review. It is rarely used due to inertia of the status quo, but methods like it will become essential as scientific failures mount. I don’t think my complaints are pointless, we need to take scientific failures very seriously because they eventually become policy failures and cause all kinds of suffering. Also, I’m not pretending that tools like peer review are useless, I’m arguing that they are useless and providing alternatives that have been developed over decades of epistemological progress.
      • wolverine876 3y ago
        How much research have you replicated?
        • brutusborn 3y ago
          A handful of papers. Lots more if you include high school and undergrad lab assignments. In my field replication usually involves a lot of guesswork since papers don't usually have enough information to be certain of their exact methods (no code or data unless the planets align or the authors are friendly) and yet peer reviewers happily apply their rubber stamp.
  • zarzavat 3y ago
    A large proportion of the population subscribes to one or more false conspiracy theories. On the other hand, man-made global warming is just barely accepted in the general population. We shall learn no such thing. People believe the papers they want to believe and don’t believe the papers they don’t want to believe. AI doesn’t change that equation because people don’t read papers to start with and journalists are easily fooled.
  • maxbond 3y ago
    > [Y]ou can’t trust anything except that what you can personally reproduce. Do you believe that matter is composed of atoms, or that stars are distant suns, or that black holes exist, or that the speed of light is 299,792,458 m/s? Have you reproduced any of those observations? Personally I don't really want to be confined to my own humble means. I'm just not that smart, life is much richer when I can learn from others. I'd rather stand on the shoulders of giants and inherit a body of knowledge from my society, accepting that it contains flaws and even outright lies that I will need to do my best to discard. There's a lot of space between blindly accepting all peer reviewed research and rejecting science altogether.
    • explaininjs 3y ago
      > Do you believe that matter is composed of atoms, or that stars are distant suns, or that black holes exist, or that the speed of light is 299,792,458 m/s? Not particularly. I don’t actively disbelieve them either. But whether or not any of those are true has zero measurable impact on my life, so it is of no consequence. I do however believe that exciting electricity in a periodic manner will cause a wave to emit that travels “really fucking fast” and can be received by a similar device a great distance away. And that such excitations can be received by antennae I have constructed which communicate with satellites in a precise manner that can allow me to locate myself on a coordinate plane. And that those calculations depend on the speed of light being as you say. So sure, it is likely to be as you say. Or at least isomorphic to that, for my purposes. How has your blind acceptance of these various other non-observed phenomena benefited you?
      • explaininjs 3y ago
        Not sure why this is downvoted. Is there some blaringly obvious life changing impact of holding faith in the common astronomer’s view of black holes / stars / etc that I am missing?
        • maxbond 3y ago
          I would wager it's because of the "blind acceptance" bit, which is rude and presumptive.
• civilized 3y ago
  I think it might stay the same as it's always been: perfectly executed fraud is impossible to detect, but fraudsters are lazy and leave clues.
  • corethree 3y ago
    Sounds like a problem that AI can solve! It can help both protect against and perpetrate fraud!
• pmarreck 3y ago
  Bad actors have always been able to take advantage of the same tools that good actors have. There is nothing new about this with regards to this technology iteration. You have to simply believe (which is fairly easily to validate) that there are far more good actors than bad. At least outside China and Mexico. I don't know what's happening in those countries. Systemic breakdown of morality perhaps. But that's a people problem, not an AI problem.
• robocat 3y ago
  > arms race I would guess it mostly isn't an arms race because it is not symmetric. There are significant asymmetries in tool costs and benefits, especially build versus SaaS. Any tool created would sell in different markets (selling to cheaters versus selling to verifiers). Also there is a time component for publications. A cheat could test their paper against a tool, only to find that the tool adds new detection capabilities in the future and the cheating is detected. Look at the markets for plagiarism tools and game cheats.
• dukeofdoom 3y ago
  I hope AI will find the nurses that hate their jobs and are harming their patients. There has to be a vast number, just going by the large percentage of other people in other professions that hate their jobs, and are doing it badly.
  • cgriswald 3y ago
    > I hope AI will find the nurses that hate their jobs and are harming their patients. I think it would be better to just find the nurses that are harming their patients irrespective of whether they enjoy it while they do it.
• ChatGTP 3y ago
  I thought the same thing, the next crisis will be "We've run the tool over it, it's all good, no further checking required", yet it fails to detect fraud. I know, I know, it's an improvement on humans until humans don't bother verifying anything anymore and it turns out we've been accepting fraudulent papers for 10 years.
• courseofaction 3y ago
  I agree that malicious actors will always be able to exploit holes in existing tools, it's an endless arms race. I worry as much about the use of these fears as a means to established centralised control of AI tools, rather than strengthening the institutions which are already supposed to deal with this kind of thing: Illegal things are already illegal, detect them and follow up. Consequences stop crime. For example, fake news is/was nothing new when it became a weaponised buzzword in the Trump era - lying has always existed, and social networks of trust plus legal consequences for malicious/dangerously negligent mistruths are a proven solution. The fact that these institutions might not be functioning properly (i.e. allowing the mass uncritical publication of mistruths for profit) is more dangerous than the technology that exploits those gaps.
• astrange 3y ago
  Remember that "AI" doesn't actually mean anything. In this case it seems to mean "perceptual hashing image search".
  • richk449 3y ago
    Yea, that’s super annoying! Just like the word car doesn’t mean anything. Last time someone threw out the buzzword car to me, it turned out they were actually talking about a 2008 four door Honda civic with four speed automatic in blue paint. I don’t know why they couldn’t have said that instead of trying to bs me by saying they were buckling their kid in the booster seat in their car.
• sacrosancty 3y ago
  [dead]
• km3r 3y ago
  Compared to malware, fraud has a lot more of a paper trail, and the AI tools of tomorrow can detect the fraud of today, limiting the damage the fraud can cause. Meanwhile malware can cause a cascading wave of damage very quickly.
• Leo_Germond 3y ago
  That's not quite the full picture you're painting here. First of, at the moment the effort is super asymetrical: it takes way more efforts to check a paper for image duplication than it takes to alter an existing image to add it to a paper. So the journals are losing against the counterfeiter as we speak. What's more, it is possible to run the detection tools long after publication, which makes it harder for someone to fake images because they run the risk of being detected later on with a more advanced version of the tooling. Typically malware devs try to erase all traces of their bad deeds and to run away undetected for this exact reason. But how do you hide an already published paper? Finally, it takes efforts to run the tools. If this effort is greater than the effort it takes to fake an image, which it is, then the cost may become prohibitive, so fakers will try to avoid faking+prechecking in favour of faking+hiding (by publishing in unchecked journals for example). That is until they group together to organize faking at scale (setting up and automating a tool like that is probably were the cost is, not running it). Faking at scale does exist, some countries practice it eagerly. So now the question becomes: do we turn a blank eye to such blatant misuse of research or do we try to take them red-handed and to ban/punish bad behavior? Not such a hard question since being on the offensive is the only scenario where we have a remote chance of stopping or at least reducing the influx of faked papers, you don't stop the mafia by ignoring them.
• nvm0n2 3y ago
  The tool in question isn't openly available, you have to apply to get access to it. So I don't think the arms race problem exists here.