5 ms·
We ran Anthropic’s interviews through structured LLM analysis
- jp8585 10mo agoAnthropic released 1,250 interviews about AI at work. Their headline: "predominantly positive sentiments." We ran the same interviews through structured LLM analysis, and the true story is a bit different. Key findings: • 85.7% have unresolved tensions (efficiency vs quality, convenience vs skill) • Creatives struggle MOST yet adopt FASTEST • Scientists have lowest anxiety despite lowest trust (see ai as a tool, plain and simple) • 52% of creatives frame AI through "authenticity" (using it makes them feel like a fraud) Same data, different lens. Full methodology at bottom of page. Analysis: https://www.playbookatlas.com/research/ai-adoption-explorer https://www.playbookatlas.com/research/ai-adoption-explorer Dataset: https://huggingface.co/datasets/Anthropic/AnthropicInterviewer https://huggingface.co/datasets/Anthropic/AnthropicInterview...
- Terretta 10mo ago“Not X. Not Y. Z.” – Are you not willing to edit the tropes out? Or maybe teach your LLM to fix itself. Starting rule set: https://en.wikipedia.org/wiki/Wikipedia:Signs_of_AI_writing https://en.wikipedia.org/wiki/Wikipedia:Signs_of_AI_writing
- hugh-avherald 10mo agoThat is not a definitive test.
- collinvandyck76 10mo agoNo, but the whole article felt definitively AI-generated.
- too_root 10mo agothe entire site feels ai-generated
- furyofantares 10mo agoAnd then you had it write your post even here on HN. C'mon.
- jp8585 10mo agoI guess all the interview quotes almost feel like those fake reviews on websites. They are all true excerpts, though. And we had a lot of fun creating all of those interactive bits. I get the sense there's a sort of ai content ptsd here, even some of the replies here are being flagged as ai, lol
- furyofantares 10mo agoIt's not PTSD, it's that I have no clue what you think of the results of your project here when even your comment on HN introducing it is an infodump that came out of an LLM. I can't tell what you think of the results, if you're skeptical of any of it, or if you think it's a smoking gun, or what. I don't know what parts of it you care more about than others. All I know is what the LLM thinks about the project.
- cmiles8 10mo agoThe story that’s solidifying is the tech is cool, it’s useful for certain things (eg, meeting note taking), but business have run a ton of “innovation lab” pilots that have returned little to no measurable value with leaders getting frustrated at the invested red ink. In short the substance isn't living up to the hype. Everywhere I look the adoption metrics and impact metrics are a tiny fraction of what was projected/expected. Yes tech keynotes have their shiny examples of “success” but the data at scale tells a very different story and that’s increasingly hard to brush under the carpet. Given the amount of financial engineering shenanigans and circular financing it’s unclear how much longer the present bonanza can continue before the financial and business reality playing out slams on the brakes.
- blindhippo 10mo agoIf anything, the AI bubble is reinforcing to me (and hopefully many more people) that the "markets" are anything but rational. None of the investments going on have followed any semblance of fundamentals - it's all pure instinct and chasing hype. I just hope it doesn't tear down the world for the 99% of us unable to actually reap any benefits from it. AI is basically a toy for 99% of us. It's a long long ways away from the productivity boost people love to claim to justify the sky high valuations. It will fade to being a background tech employed strategically I suspect - similar to other machine learning applications and this is exactly where it belongs. I'm forced to use it (literally, AI usage is now used as a talent review metric...) and frankly, it's maybe helped speed me up... 5-10%? I spend more time trying to get the tools to be useful than I would just doing the task myself. The only true benefit I've gotten has been unit test generation. Ask it to do any meaningful work on a mature code base and you're in for a wild ride. So there's my anecdotal "sentiment".
- dionian 10mo agoI multi task much more now that i can farm off small coding assignments to agents. i pay hndreds per month in tokens. for my role personally its been a massive paradigm shift.
- 10mo ago
- Lerc 10mo agoThe high usage and high anxiety tracks with what I have found from taking to artists IRL. There is a sense that any any public expression that is not wholly against AI will draw vilification from a section of the artistic community. There are a broad range of opinions but the expression seems to have been extremely chilled.
- huevosabio 10mo ago``` Creatives have the highest struggle scores and the highest adoption rates. ``` Here is my guess for the puzzle: creative work is subjective and full of scaffolding. AI can easily generate this subjective scaffolding to a "good enough" level so it can get used without much scrutiny. This is very attractive for a creative to use on a day to day basis. But, given the amount of content that wasn't created by the creative, the creative feels both a rejection of the work as foreign and a feeling of being replaced. The path is less stark in more objective fields because the quality is objective, so harder to just accept a merely plausible solution, and the scaffolding is just scaffolding so who cares if it does the job.
- ctoth 10mo agoPossible confound (seems important): "creatives" tend to have a certain political tribe, that political tribe is well-represented in places that have this precise type of authenticity/etc. language around AI use... Basically a good chunk of this could be measuring whether or not somebody is on Bluesky/is discourse-pilled... and there's no way to know from the study.
- layer8 10mo agoOne issue with AI for creatives is that it’s virtually impossible to get AI to create a specific vision you have in mind. It creates something, but you just have to accept whatever that is, you can only steer it very roughly. It can be useful for getting inspiration, but not for getting exact results. If AI was better suited for realizing one’s own creative vision and working in a detail-oriented fashion, creators would likely embrace it more.
- userbinator 10mo agoHaving tried some AI image generation, it feels more like gambling than work --- repeatedly submitting and hoping you get the result you wanted is extremely reminiscent of pulling a one-armed bandit hoping to win, except perhaps a bit cheaper. I can certainly understand a potential for addiction though.
- 10mo ago
- nphardon 10mo agoI'm a scientist and I mostly agree with the scientist part, but I am definitely collaborating with my bot, I don't view it as "just a tool". I know this because this morning I had to do a forced reboot and my VsCode wasn't connecting to our remote servers, it took like over 5 minutes after reboot to reload my bot chat, and from like minutes 3-5 I had the distinct feeling of losing a valuable colleague.
- gopher_space 10mo agoPersonification can build empathy up to a point, but the machine has no desires.
- nphardon 10mo agoI don't have illusions about whats going on on the other end, but we've done some deep collaborating and I 90% anthropomorphize it; much like how people on Star Trek TNG interact with Data.
- fragmede 10mo agoMine gets ashamed and embarrassed and goes and deletes the evidence (and my project folder! Good thing I've got backups.) when it fails. It also gets lazy and tells me to go stuff when it could do it, and I have to tell it to go do it instead of me having to go do it.
- nphardon 10mo agoThats wild! I have had nothing but consistent, stable experiences. It's possible they just take on the personalities of whoever theyre working with. So for me, it's become this like idealized version of a scientific collaborator. Also, I assume different models and versions have different personalities. As far as I can tell gpt-mini has no personality, whereas my claude sonnet 4.5 has a big one.
- malfist 10mo agoThis article is rife with unedited llm signals. This makes me question their methodology here. I want you believe what they found, but I don't trust this analysis. If they were this sloppy with the write up, how sloppy were they with the science?
- jp8585 10mo agoWe have a full page on the methodology we used! Let me know if you’d like access to the dataset we created for this. The aim was not to be scientific but to flush out some deeper meanings from these interviews that typical nlp techniques struggle with. Ps: Of course we used llm tools as a writing aid, I’d be willing to bet those “signals” probably come from my own writing though and my appreciation of Tom Wolfe. I’ve been told it can be “sloppy” sometimes.
- dcre 10mo agoThe bits that stand out to me are the non-question questions. “Their headline?” “Scientists are thriving. The workforce is managing. But creatives?” “The top trust destroyer?”
- userbinator 10mo agoWe have a full page on the methodology we used! Let me know if you’d like access to the dataset we created for this. I'm not sure if you realise that those two sentences sound like 100% verbatim LLM output, or am I actually replying to a bot and not a human.
- jp8585 10mo agoNow you are just being paranoid, lol. You have me wondering now why am I writing like an llm. (strawberry has 3 rs). Are you a bot? Shoot me an email (that we listed on the page) and let's have a zoom call
- userbinator 10mo ago<some sentence ending in an exclamation point!> <"let me know if you'd like"...> is a stereotypical ChatGPT response.
- WhyOhWhyQ 10mo agoAnother thing I might throw out there is that there are so many domains and niches out there that person A and person B are almost certainly having genuinely different experiences with the same tools. So when person A says "wow this is the best thing ever" and person B says "this thing is horrible" they might both be right.
- doug_durham 10mo agoIs this any different than the adoption of any technology. I think of the transition from practical effects to CGI in Hollywood. Anxiety levels of the creative model builders was sky high at the time. It worked itself out and now there are different jobs.
- WhyOhWhyQ 10mo agoAre they happy in their new jobs?
- doug_durham 10mo agoI presume many are. It's a different medium, but it's still creative. We got the "Mythbusters" show out of some of the model builders who didn't want to move to CGI.
- FuckButtons 10mo agoI think if you ask those specific people, you might find different answers. There are different jobs, sure, but not necessarily for those people.
- ursAxZA 10mo agoWhen railroads were built, canal operators were upset too.
- bronco21016 10mo agoI use AI coding almost daily. I’m able to move my repositories into context easily through the multitude of AI coding tools and I see a massive boost in productivity. I say this as a junior dev. Often the outputs are “almost” and I make the necessary fixes to get it the rest of the way there. To contrast with this, my org tried using a simple QA bot for internal docs and has been struggled to move anything beyond proof of concept. The proof of concepts have been awful. It answers maybe 60-70% of questions correctly. The major issue seems to be related to taking PDFs laced with images and poorly written explanations. To get decent performance from these RAG bots, a large FAQ has to be written for every question it gets wrong. Of course this is just my org so it can’t necessarily be extrapolated across industry. However, how often have people come across a new team and find there is little to no documentation, poorly written documentation, or outdated documentation? Where am I going with these two thoughts? Maybe the blocker to pushing more adoption within orgs is twofold, getting the correct context into the model and having decent context to start with. Extracting value from these things is going to require a heavy lift in data curation and developing the harnesses. So far most of that effort has gone into coding. It will take time for the nontechnical and technical to work together to move the rest of an org into these tools in my opinion. The big bet of course then is ROI and time to adoption vs current burn rates of the model providers.
- latentsea 10mo agoYep. There are a lot of things that apply equally to human engineering teams in terms of productivity. Poor documentation and information architecture is a thing I have seen time and time again, and is always something I put time into course correcting for because it makes performing cognitive work much easier. Same goes for poorly factored codebases. They make doing any work feel like wading through mud. Throughout my career I have done a lot of work on what I would call platform engineering and product re-engineering and it's always to course correct for how difficult an environment has become to work in. Agents are going to struggle with those same difficulties the way humans do too. You need to put work into making an environment productive to work in, and after having purposely switched my development workflow for the stuff I do outside of work to being "AI first on mobile", that's such a bandwidth constrained setup that it's really helping me to find all the things to optimise for to increase the batting average and minimise the back and forth.
- 000ooo000 10mo agoWhat kind of mindset do you need to have to trust anything a company like this has to say? A company riding the hype train, praying the bubble doesn't pop, desperately trying to even turn a profit? Would you believe cigarettes are healthy too?
- travisgriggs 10mo agoHow do I know this fine article wasn’t the result of “Create a web page infographic report that is convincing and boils down the essential truths of how people are feeling about AI in different professions and domains.. Include statistics and numbers and some rolling/animated sound bite quotes.”
- colonCapitalDee 10mo agoIt certainly reads like it
- jp8585 10mo agoWe actually have a full page on the methodology we used. We did over 20 experiments on the best way to digest this dataset and ended up settling on just using llms to review each interview in-depth. This helped us unlock some insights that would only be available if we paid someone to actually sit down and read every single interview. All other nlp approaches we used, like topic modeling, tfidf, and the roaring 2010s wordcloud,s were just too boring to even warrant a write-up. We tried bringing up as many raw excerpts to light as they make our point that much clearer. People are not simply "happy" about ai as Anthropic would have us believe, they are torn by it. It gives them a deep sense of angst and make them question the authenticity of their very existence.
- legsrunny 9mo ago``` Application error: a client-side exception has occurred while loading www.playbookatlas.com (see the browser console for more information). ``` hell yeah