2 ms·
I work in survey research and I'm rather appalled at how many people would rather survey a sample of AIs than a sample of people and claim they can come to some
by PaulHoule 1y ago
I work in survey research and I'm rather appalled at how many people would rather survey a sample of AIs than a sample of people and claim they can come to some valid conclusion as a result.
There are many ways AIs differ from real people and any conclusions you can draw from them are limited at best -- we've had enough bad experiments done with real people
https://en.wikipedia.org/wiki/Stanford_prison_experiment#Interpretation_and_reproducibility_of_results https://en.wikipedia.org/wiki/Stanford_prison_experiment#Int...
- jerf 1y agoAppalling. The entire question of "fixing social media", for any definition of "fixing", involves not just the initial reaction to some change but the second-and-greater-order effects. LLMs are point-in-time models and intrinsically can not be used for even guessing at second-order effects of a policy over time. This shouldn't have gotten past the proposal phase.
- reactordev 1y agoI trust your judgement more than Ars Technica. For us layman, the flaw of using AI trained on people for surveys is, human. Humans have a unique tendency to be spontaneous, wouldn’t you say? How would a focus group research team approach this when they’re bombarded by AI solutions that want their research funds?
- PaulHoule 1y agoThe worst problems with people these days seem to be they don’t pick up the phone. Probability-based polls are still pretty good about most things unless they involve Donald Trump —- it seems some Trump supporters either don’t pick up the phone or lie to pollsters. Some polls correct for this with aggressive weighting but how scientific it really is is up in the air.
- johnnyanmac 1y agoYeah. Sadly our phones for any unidentified number fell to spam. These days, if the message is important then they can leave a voicemail... And 90% of this voicemail reveal spam as well. >unless they involve Donald Trump A sense of shame perhaps. If you ask someone "how often do you brush your teeth" and compare it to more pragmatic testing, you see people have some sense of wanting to give the "right" answer. Even in a zero risk anonymous survey.
- add-sub-mul-div 1y ago"We trained a model on Twitter and Reddit content and were shocked to discover it generates a terrible community." It's so weird to live in a time when what you just said needs to be said.
- Mouvelie 1y agoGenuine question : are you scared for your job ? I see this tendency to use "synthetic personas" growing and frankly, having to explain why this sucks is insulting in itself. Decision makers are just not interested in having this kind of thought argument.
- PaulHoule 1y agoNot really. Sales is doing better than it ever has since I’ve been here. For one thing, AI folks want our data. Despite challenges in the industry, public opinion is more relevant than ever and the areas where we are really unsurpassed is (1) historical data and (2) the most usable web site, the latter one I am a part of.
- johnnyanmac 1y agoYes and mostly No. No, because I work in games and I've seen enough people thinking that a "good game" just needs pretty graphics and a facsimile of "fun" to know that AI can't ever simulate this. Mkst Humans can't even seem to do it consistently, on all organizational levels. But i have a footnote of "yes" because as you said, decision makers are just not interested in having this discussion about "focus on making fun games". So it will unfortunately affect my job in the short and even medium terms. Because so much of big money in games these days is in fact not focused on making a game, but on trying to either generate a gambling simulator, an engagement trap, or (you guessed it) AI hype. Both to try and claim you can just poof up assets, and to try and replace labor. Knowing this, I do have long term plans to break out into my own indie route.
- richardubright 1y agoWait what? Is there an article on this. That sounds absolutely insane.
- naravara 1y agoIt doesn’t surprise me if they found that the emergent behaviors didn’t change given their method. Modifying the simulation to make them behave differently would mean your rules have changed the model’s behavior to “jump tracks” into simulating a different sort of person who would generate different outputs. It’s not quite analogous to having the same Bob who likes fishing responding to different stimuli. Sort of like how Elon told Grok to be “unfathomably based” and stop caring about being PC” and suddenly it turned into a Neo-Nazi Chan-troll. Changing the inputs for an LLM isn’t taking a core identity and tweaking it, it’s completely altering the relationships between all the tokens it’s working with. I would assume there is so much in the corpus based on behavior optimized for the actual existing social media we have that the behavior of the bots is not going to change because the bot isn’t responding to incentives like a person would it’s mimicking the behavior it’s been trained on and if there isn’t enough training data of behavior under the different inputs you’re trying to test you’re not actually applying the “treatment” you would think you are.
- lispisok 1y ago>I work in survey research and I'm rather appalled at how many people would rather survey a sample of AIs than a sample of people and claim they can come to some valid conclusion as a result. A YC company just launched doing exactly that. https://news.ycombinator.com/item?id=44755654 https://news.ycombinator.com/item?id=44755654