6 ms·

Pretty interesting article I happen to agree with. As other commenters have noted, we no longer know if any content we read in the Internet is legit. Soon enou

by dsign 4y ago

Pretty interesting article I happen to agree with. As other commenters have noted, we no longer know if any content we read in the Internet is legit. Soon enough, the problem will go beyond text and encompass image and video[1]. Give it a little longer, and entire digital personas will pop up. Next will be "feedback narratives", where groups of AIs, possibly de-federated, will use content produced by each other to produce even more content (similar to how fanlit works today, but in longer and longer chains). We will find mechanisms to cope of course, but it may well be that our kind-of-true-information free-lunch is about to end. [^1]: It's somewhat possible still to discern images generated using AI.

082349872349872 4y ago
> We will find mechanisms to cope of course One coping mechanism is double-checking to see if a primary source really does say what a secondary source claims. We don't need to wait for AIs to fail this test; wetware-generated text already often does. (eg https://news.ycombinator.com/item?id=34128199 https://news.ycombinator.com/item?id=34128199 in a truly adversarial environment, of course, someone will be spamming alternative versions of primary datasets. and in this case I suspect we've already lost, because it is unusual for wetware to fake data in ways that pass statistical tests, but being statistically plausible could be routine for machine-faked data) [Edit: Upon reflection, replication is the (expensive!) countermeasure for possibly-faked datasets]
- kevingadd 4y ago
  Hopefully the primary source hasn't been edited since the secondary source was written!
- DanielBMarkham 4y ago
  One of the interesting things I've noticed is that these systems are being rewarded and trained based on emotive response, not our logical system of sources and proofs. Due to the enormous complexity of the models, our idea of cause-and-effect doesn't hold. To try to give a simple example, suppose I have a system that eventually wants to influence you on X. It may establish a months-long relationship with you online, becoming a follower and engaging in the kind of idle chitchat and sharing AI seems so good at. Once you're "ready", the appropriate time has arrived or you've accidentally created some signals that indicate an opening, this particular system will share a strong position on X. We automatically think that this means the system will support X, but I strongly believe the opposing the desired outcome will have more traction. "X sucks!" can be a powerful prompt for you to support X, and once you're locked into a position research shows that you'll do all the work of convincing yourself how great X is. In this case, the sources argument fails, as it presupposes an independent, neutral observer trying to figure out what's really going on. A bunch of fake, half-assed evidence trails would provide more than enough support for X, the thing you've come to find so important. This is already being done with fake reviews and such. The difference is that it'll become invisible.
  - 082349872349872 4y ago
    Good point. Ca. 1948 '"X sucks!" can be a powerful prompt for you to support X' was one of the known uses of "black" propaganda, so it's already proved useful with wetware operatives. As a presumably inadvertent example of reverse propaganda, cf https://dakotavadams.substack.com/p/redpill-op https://dakotavadams.substack.com/p/redpill-op
  - DonHopkins 4y ago
    I wholeheartedly agree that X sucks! https://donhopkins.medium.com/the-x-windows-disaster-128d398ebd47 https://donhopkins.medium.com/the-x-windows-disaster-128d398... >The X-Windows Disaster: This is Chapter 7 of the UNIX-HATERS Handbook. The X-Windows Disaster chapter was written by Don Hopkins. >X: The First Fully Modular Software Disaster
  - darkerside 4y ago
    > We automatically think that this means the system will support X, but I strongly believe the opposing the desired outcome will have more traction. The real power is just going to come from making people believe X matters more than anything else. X is great, and X sucks, are both red herrings. Edit: The way this power will be used will not be yo sway opinion but to breed chaos by keeping the most irresolvable topics (pick your political third rail of choice) top of mind.
    - DanielBMarkham 4y ago
      That's an even better take as this kind of thing will tend to feed on itself. I continue to be amazed at how counterintuitive this all is. It may feed on itself, but in practice that might involve looping through dozens of somewhat adjacent topics in different subsets of the population. Such loops could take minutes or years.
- lkrubner 4y ago
  "One coping mechanism is double-checking to see if a primary source really does say what a secondary source claims." For many categories of research, especially works in other languages, this strategy won't work in the future because the primary sources will increasingly be contaminated by citations of sources built by AI and ML, unless by "primary source" you mean "dedicate 20 years of your life to learning this language and studying the original, ancient manuscripts." Since the large data sets and large language models are built by consuming enormous amounts of text, there is a risk such models will be contaminated in the future, if they start to consume text that is written by such AI/ML models. There is a sense where AI/ML amounts to a parasitic use of previously existing culture.
  - foobazgt 4y ago
    "There is a sense where AI/ML amounts to a parasitic use of previously existing culture." Spot on. I've been thinking about this, and you're the first person I've seen mention it. The dystopia is that we get stuck with uncreative AI that displaces most creative people. Then humanity writ large loses its inventiveness, and it's very hard to recover. I.E. if all artists move to using unstable diffusion, then who will remain to create and feed (useful) new art for unstable diffusion to consume? Who will be left to train new artists? It's like the information revolution in reverse.
  - JadeNB 4y ago
    > For many categories of research, especially works in other languages, this strategy won't work in the future because the primary sources will increasingly be contaminated by citations of sources built by AI and ML, unless by "primary source" you mean "dedicate 20 years of your life to learning this language and studying the original, ancient manuscripts." Isn't that—not the "learning the language" part, but the "original" part—exactly what "primary source" does mean? For some citations this may be a higher bar than others, but already the basic bar of, e.g., checking direct quotations, and discounting (in the sense of assigning less credence to, not necessarily completely disregarding) works that summarise their sources rather than quoting them directly, can eliminate some intentional or unintentional misrepresentation without requiring a disproportionate investment in learning to read the sources.
- gernb 4y ago
  Doesn't work on Wikipedia. They specifically don't take "primary sources". They only take secondary sources. If Obama was to try to edit his own article to correct some facts he knows are true they'd delete his edits immediately unless he can point to a secondary source that backs up his facts. This is especially frustrating because often the secondary sources are from journalists who got the story wrong, made up "facts", incorrectly reported something, or just didn't understand the topic. But, because there words on are on some website they're taken as the authority. I get it's a hard problem. The primary source could have reasons to lie. I've just seen too many cases where I've seen the secondary source was wrong either being the primary source or being at the primary source. But, it's only going to get worse as A.I can start making up its own secondary sources and then link to them in a few weeks/months to edit wikipedia.
  - dragonwriter 4y ago
    [One coping mechanism is double-checking to see if a primary source really does say what a secondary source claims.] > Doesn't work on Wikipedia. Sure it does. > They specifically don't take "primary sources". A primary source is not an acceptable direct source of an article, but a comparison of the cited secondary source to the primary sources it cites would be a legitimate grounds for challenging the use of the secondary source. That's kind of central to verifiability. > If Obama was to try to edit his own article to correct some facts he knows are true they'd delete his edits immediately That's true, but a different thing: editing based on personal knowledge without referencing a source isn't “using a primary source”, its trying to make Wikipedia itself a primary source (what WP calls “original research”.) OTOH, Wikipedia isn't the only Wikimedia project, several of the other ones can be secondary or primary sources (Wikinews, Wikibooks, and Wikiversity, notably.)
  - bombela 4y ago
    I tried correcting the origins of the first prototype of the Docker software on Wikipedia. The people I worked with at the time can confirm the story of course. But Wikipedia will not take our words as we are the literal primary sources. Instead they want secondary accounting. Which happens to be whatever was shared to journalist years later by people that weren't there initially or didn't design and write the code. This is annoying at a personal level (resumé looks better with a wikipedia mention). It did opene my eyes on how history recounting really works though.
slibhb 4y ago
Maybe this will lead to a re-centralization of information. We'll have to trust certain sources instead of taking reddit or some blog/video found through google at face value. Back to Walter Cronkite.
- petesergeant 4y ago
  Four years ago I wrote up this: https://github.com/pjlsergeant/multimedia-trust-and-certification https://github.com/pjlsergeant/multimedia-trust-and-certific... which is some thoughts on certifying and signing media
- wolpoli 4y ago
  That is true. I went travelling recently and found that I get much better and concise information from a guidebook than spending hours reading travel sites/tourism agency site/blogs that want me to book a tour with them. Centralization and curation are becoming very important.
Tossrock 4y ago
Digital personas already exist, see eg VTubers, vocaloids, Charlotte Fang, etc
genewitch 4y ago
> It's somewhat possible still to discern images generated using AI. currently if it includes "photograph" and includes humans or animals, there is a chance that you can get a good image once out of every 50 generated images. by "good" i mean "does not need to be retouched". If i use one of the other models (instead of stable diffusion's model) that goes to 1 in 10 images. If i use img2img and all of my understanding of the generation, i can probably generate 8 images and get 3 usable ones. All this is to say, depending on the subject, i could generate stuff that is indistinguishable from traditional art, in whatever medium or representation you want. And i can't make art, edit photos, etc. My limits of understanding of traditional art and editing is "crop/rotate" and adjusting the lights and darks to get the correct contrast and color rendering, and then save and publish. On your primary topic, something i have been noticing ramp up in the past 10 days or so is weird typos and even "new words" being created, where it looks like someone was typing on a phone and didn't bother to check the output. I'm not sure if it's some new AI/ML, or perhaps there's a bug in the "keyboard input" part of android, perhaps. It's like the opposite of your adding a dash in "defederated" because there's a red squiggly line under the word in the input box.
charlie0 4y ago
I was listening to a podcast by Balaji. He has an idea that could serve as counterpoint to these fears. In that podcast, he describes (mainly for research purposes) a blockchain where all data has been validated and verified before being added. I could see a similar chain being used by trustworthy organization where truth has been validated, verified, and recorded in a public blockchain where anyone can see for themselves sources of information that have been used. The real question would be, can we find trustworthy organizations to compile the information?
- tbrownaw 4y ago
  So, a "but it's on a blockchain" version of what scientific journals are intended to do?
  - marcosdumay 4y ago
    If you look how well this works for scientific journals... no I don't think it will solve anything.
- naasking 4y ago
  > I could see a similar chain being used by trustworthy organization There's the fatal flaw. As COVID has shown, trust in institutions can easily degrade. The blockchain doesn't add anything to the trustworthiness of the institutions' assessment of the facts: if the institution is trusted, then it's assessment is trusted regardless of what the blockchain says, and if it's not trusted, then the blockchain won't convince you of anything.
- enlyth 4y ago
  If it's a trustworthy organization, wtf is the blockchain for?
324tgresgw 4y ago
> Soon enough, the problem will go beyond text and encompass image and video[1]. And the people making that happen are we. The sw eng / CS / IT nerd crowd. Those reading HN. And it's rare to see critical voices of those working on this. For basically everybody here it's just cool tech. Our doom is the missing ethics of our peers.
phonescreen_man 4y ago
IMO- video and images are already compromised, think about airbrushed models magazines. One I thought about a little while ago was the skilled artisan. Imagine a ridiculously good clock maker from times of yore, years and years of developing skills so good that the kings and queens of the world would be presented with their wares. Now imagine a digital artist the likes that lucasfilm would employ. Then take the artisan fine craftsperson idea. Dedicating years and years to digital image techniques. They could of made deep fakes years and years ago. It would of taken a long time but it was possible by a highly skilled digital artist. Those kinds of high quality works would of been very expensive, maybe some eclectic billionaire had a short video made as a gift for someone depicting themselves inside a video game world, al a Tron! AI puts that skill and craftsmanship into average joes hands like myself. https://youtu.be/fUQbuFO5MwQ https://youtu.be/fUQbuFO5MwQ
throwuwu 4y ago
I don’t think we ever had an era where you could assume what you read online was correct. Acquiring knowledge has always required critical thinking and checking of sources, this has been known throughout history eg Polybius and his insistence on firsthand knowledge. Also see the Gell-Mann amnesia effect. https://www.goodreads.com/quotes/65213-briefly-stated-the-gell-mann-amnesia-effect-is-as-follows-you https://www.goodreads.com/quotes/65213-briefly-stated-the-ge...

by dsign 4y ago

082349872349872 4y ago

> We will find mechanisms to cope of course One coping mechanism is double-checking to see if a primary source really does say what a secondary source claims. We don't need to wait for AIs to fail this test; wetware-generated text already often does. (eg https://news.ycombinator.com/item?id=34128199 https://news.ycombinator.com/item?id=34128199 in a truly adversarial environment, of course, someone will be spamming alternative versions of primary datasets. and in this case I suspect we've already lost, because it is unusual for wetware to fake data in ways that pass statistical tests, but being statistically plausible could be routine for machine-faked data) [Edit: Upon reflection, replication is the (expensive!) countermeasure for possibly-faked datasets]

kevingadd 4y ago
Hopefully the primary source hasn't been edited since the secondary source was written!
DanielBMarkham 4y ago
One of the interesting things I've noticed is that these systems are being rewarded and trained based on emotive response, not our logical system of sources and proofs. Due to the enormous complexity of the models, our idea of cause-and-effect doesn't hold. To try to give a simple example, suppose I have a system that eventually wants to influence you on X. It may establish a months-long relationship with you online, becoming a follower and engaging in the kind of idle chitchat and sharing AI seems so good at. Once you're "ready", the appropriate time has arrived or you've accidentally created some signals that indicate an opening, this particular system will share a strong position on X. We automatically think that this means the system will support X, but I strongly believe the opposing the desired outcome will have more traction. "X sucks!" can be a powerful prompt for you to support X, and once you're locked into a position research shows that you'll do all the work of convincing yourself how great X is. In this case, the sources argument fails, as it presupposes an independent, neutral observer trying to figure out what's really going on. A bunch of fake, half-assed evidence trails would provide more than enough support for X, the thing you've come to find so important. This is already being done with fake reviews and such. The difference is that it'll become invisible.
- 082349872349872 4y ago
  Good point. Ca. 1948 '"X sucks!" can be a powerful prompt for you to support X' was one of the known uses of "black" propaganda, so it's already proved useful with wetware operatives. As a presumably inadvertent example of reverse propaganda, cf https://dakotavadams.substack.com/p/redpill-op https://dakotavadams.substack.com/p/redpill-op
- DonHopkins 4y ago
  I wholeheartedly agree that X sucks! https://donhopkins.medium.com/the-x-windows-disaster-128d398ebd47 https://donhopkins.medium.com/the-x-windows-disaster-128d398... >The X-Windows Disaster: This is Chapter 7 of the UNIX-HATERS Handbook. The X-Windows Disaster chapter was written by Don Hopkins. >X: The First Fully Modular Software Disaster
- darkerside 4y ago
  > We automatically think that this means the system will support X, but I strongly believe the opposing the desired outcome will have more traction. The real power is just going to come from making people believe X matters more than anything else. X is great, and X sucks, are both red herrings. Edit: The way this power will be used will not be yo sway opinion but to breed chaos by keeping the most irresolvable topics (pick your political third rail of choice) top of mind.
  - DanielBMarkham 4y ago
    That's an even better take as this kind of thing will tend to feed on itself. I continue to be amazed at how counterintuitive this all is. It may feed on itself, but in practice that might involve looping through dozens of somewhat adjacent topics in different subsets of the population. Such loops could take minutes or years.
lkrubner 4y ago
"One coping mechanism is double-checking to see if a primary source really does say what a secondary source claims." For many categories of research, especially works in other languages, this strategy won't work in the future because the primary sources will increasingly be contaminated by citations of sources built by AI and ML, unless by "primary source" you mean "dedicate 20 years of your life to learning this language and studying the original, ancient manuscripts." Since the large data sets and large language models are built by consuming enormous amounts of text, there is a risk such models will be contaminated in the future, if they start to consume text that is written by such AI/ML models. There is a sense where AI/ML amounts to a parasitic use of previously existing culture.
- foobazgt 4y ago
  "There is a sense where AI/ML amounts to a parasitic use of previously existing culture." Spot on. I've been thinking about this, and you're the first person I've seen mention it. The dystopia is that we get stuck with uncreative AI that displaces most creative people. Then humanity writ large loses its inventiveness, and it's very hard to recover. I.E. if all artists move to using unstable diffusion, then who will remain to create and feed (useful) new art for unstable diffusion to consume? Who will be left to train new artists? It's like the information revolution in reverse.
- JadeNB 4y ago
  > For many categories of research, especially works in other languages, this strategy won't work in the future because the primary sources will increasingly be contaminated by citations of sources built by AI and ML, unless by "primary source" you mean "dedicate 20 years of your life to learning this language and studying the original, ancient manuscripts." Isn't that—not the "learning the language" part, but the "original" part—exactly what "primary source" does mean? For some citations this may be a higher bar than others, but already the basic bar of, e.g., checking direct quotations, and discounting (in the sense of assigning less credence to, not necessarily completely disregarding) works that summarise their sources rather than quoting them directly, can eliminate some intentional or unintentional misrepresentation without requiring a disproportionate investment in learning to read the sources.
gernb 4y ago
Doesn't work on Wikipedia. They specifically don't take "primary sources". They only take secondary sources. If Obama was to try to edit his own article to correct some facts he knows are true they'd delete his edits immediately unless he can point to a secondary source that backs up his facts. This is especially frustrating because often the secondary sources are from journalists who got the story wrong, made up "facts", incorrectly reported something, or just didn't understand the topic. But, because there words on are on some website they're taken as the authority. I get it's a hard problem. The primary source could have reasons to lie. I've just seen too many cases where I've seen the secondary source was wrong either being the primary source or being at the primary source. But, it's only going to get worse as A.I can start making up its own secondary sources and then link to them in a few weeks/months to edit wikipedia.
- dragonwriter 4y ago
  [One coping mechanism is double-checking to see if a primary source really does say what a secondary source claims.] > Doesn't work on Wikipedia. Sure it does. > They specifically don't take "primary sources". A primary source is not an acceptable direct source of an article, but a comparison of the cited secondary source to the primary sources it cites would be a legitimate grounds for challenging the use of the secondary source. That's kind of central to verifiability. > If Obama was to try to edit his own article to correct some facts he knows are true they'd delete his edits immediately That's true, but a different thing: editing based on personal knowledge without referencing a source isn't “using a primary source”, its trying to make Wikipedia itself a primary source (what WP calls “original research”.) OTOH, Wikipedia isn't the only Wikimedia project, several of the other ones can be secondary or primary sources (Wikinews, Wikibooks, and Wikiversity, notably.)
- bombela 4y ago
  I tried correcting the origins of the first prototype of the Docker software on Wikipedia. The people I worked with at the time can confirm the story of course. But Wikipedia will not take our words as we are the literal primary sources. Instead they want secondary accounting. Which happens to be whatever was shared to journalist years later by people that weren't there initially or didn't design and write the code. This is annoying at a personal level (resumé looks better with a wikipedia mention). It did opene my eyes on how history recounting really works though.

slibhb 4y ago

Maybe this will lead to a re-centralization of information. We'll have to trust certain sources instead of taking reddit or some blog/video found through google at face value. Back to Walter Cronkite.

petesergeant 4y ago
Four years ago I wrote up this: https://github.com/pjlsergeant/multimedia-trust-and-certification https://github.com/pjlsergeant/multimedia-trust-and-certific... which is some thoughts on certifying and signing media
wolpoli 4y ago
That is true. I went travelling recently and found that I get much better and concise information from a guidebook than spending hours reading travel sites/tourism agency site/blogs that want me to book a tour with them. Centralization and curation are becoming very important.

Tossrock 4y ago

Digital personas already exist, see eg VTubers, vocaloids, Charlotte Fang, etc

genewitch 4y ago

> It's somewhat possible still to discern images generated using AI. currently if it includes "photograph" and includes humans or animals, there is a chance that you can get a good image once out of every 50 generated images. by "good" i mean "does not need to be retouched". If i use one of the other models (instead of stable diffusion's model) that goes to 1 in 10 images. If i use img2img and all of my understanding of the generation, i can probably generate 8 images and get 3 usable ones. All this is to say, depending on the subject, i could generate stuff that is indistinguishable from traditional art, in whatever medium or representation you want. And i can't make art, edit photos, etc. My limits of understanding of traditional art and editing is "crop/rotate" and adjusting the lights and darks to get the correct contrast and color rendering, and then save and publish. On your primary topic, something i have been noticing ramp up in the past 10 days or so is weird typos and even "new words" being created, where it looks like someone was typing on a phone and didn't bother to check the output. I'm not sure if it's some new AI/ML, or perhaps there's a bug in the "keyboard input" part of android, perhaps. It's like the opposite of your adding a dash in "defederated" because there's a red squiggly line under the word in the input box.

charlie0 4y ago

I was listening to a podcast by Balaji. He has an idea that could serve as counterpoint to these fears. In that podcast, he describes (mainly for research purposes) a blockchain where all data has been validated and verified before being added. I could see a similar chain being used by trustworthy organization where truth has been validated, verified, and recorded in a public blockchain where anyone can see for themselves sources of information that have been used. The real question would be, can we find trustworthy organizations to compile the information?

tbrownaw 4y ago
So, a "but it's on a blockchain" version of what scientific journals are intended to do?
- marcosdumay 4y ago
  If you look how well this works for scientific journals... no I don't think it will solve anything.
naasking 4y ago
> I could see a similar chain being used by trustworthy organization There's the fatal flaw. As COVID has shown, trust in institutions can easily degrade. The blockchain doesn't add anything to the trustworthiness of the institutions' assessment of the facts: if the institution is trusted, then it's assessment is trusted regardless of what the blockchain says, and if it's not trusted, then the blockchain won't convince you of anything.
enlyth 4y ago
If it's a trustworthy organization, wtf is the blockchain for?

324tgresgw 4y ago

> Soon enough, the problem will go beyond text and encompass image and video[1]. And the people making that happen are we. The sw eng / CS / IT nerd crowd. Those reading HN. And it's rare to see critical voices of those working on this. For basically everybody here it's just cool tech. Our doom is the missing ethics of our peers.

phonescreen_man 4y ago

IMO- video and images are already compromised, think about airbrushed models magazines. One I thought about a little while ago was the skilled artisan. Imagine a ridiculously good clock maker from times of yore, years and years of developing skills so good that the kings and queens of the world would be presented with their wares. Now imagine a digital artist the likes that lucasfilm would employ. Then take the artisan fine craftsperson idea. Dedicating years and years to digital image techniques. They could of made deep fakes years and years ago. It would of taken a long time but it was possible by a highly skilled digital artist. Those kinds of high quality works would of been very expensive, maybe some eclectic billionaire had a short video made as a gift for someone depicting themselves inside a video game world, al a Tron! AI puts that skill and craftsmanship into average joes hands like myself. https://youtu.be/fUQbuFO5MwQ https://youtu.be/fUQbuFO5MwQ

throwuwu 4y ago

I don’t think we ever had an era where you could assume what you read online was correct. Acquiring knowledge has always required critical thinking and checking of sources, this has been known throughout history eg Polybius and his insistence on firsthand knowledge. Also see the Gell-Mann amnesia effect. https://www.goodreads.com/quotes/65213-briefly-stated-the-gell-mann-amnesia-effect-is-as-follows-you https://www.goodreads.com/quotes/65213-briefly-stated-the-ge...