17 ms·

I think I'm done thinking about GenAI for now

169 points by kaycebasques 1y ago

the_mitsuhiko 1y ago
I left this comment originally on lobsters: While I understand the fear, I don’t really share it. And if I where to go to the root of it, I think I really most take issue with this: > My experiences of genAI are all extremely bad, but that is barely even anecdata. Their experiences are neutral-to-positive. Little scientific data exists. How to resolve this? My experience is astonishingly positive. I would not have imagined how much of a help these tools have become. Deep research and similar tools alone have helped me navigate complex legal matters recently for my incorporation, they have uncovered really useful information that I just would not have found that quickly. First cursor, now Claude Code have really changed how I work. Especially since for the last month or so, I feel myself more and more in the position where I can do things while the machine works. It’s truly liberating and it gives me a lot of joy. So it’s not “neutral-to-positive” to me, it’s exhilarating. And that extends particularly to this part: > Despite this plethora of negative experiences, executives are aggressively mandating the use of AI. It looks like without such mandates, most people will not bother to use such tools, so the executives will need muscular policies to enforce its use. When I was at Sentry the adoption of AI happened by ICs before the company even put money behind it. In fact my memory is that if anything only at the point where an exceeding number of AI invoices showed up from IC expenses did we realize how widespread adoption has been. This was grounds up. For my non techy friends it’s even tricker because some of them work in companies that outright try to prevent the adoption of AI, but they are paying for it themselves to help them with the work. Some of them pay for the expensive ChatGPT package even! None of this should be disregarded, but it stands in a crass contrast to what this post says. That said, I understand where Glyph comes from and I appreciate that point. There is probably a lot of truth in the tail risks of all of that, and I share these. It just does not at all take away from my enjoyment and optimism at all.
- oulipo 1y ago
  Basically his article says: check the documented issue with the climate, the education, the destruction of trust in society, and your answer is "yes but it makes my nice cuddly programming job better I like it"... don't you understand the issue?
  - the_mitsuhiko 1y ago
    I responded to quite specific quotes which I took issues with, and I intentionally did not talk about the rest of it. I did not make a statement about the rest because it's too complex of a topic to which I do not have enough data to contribute meaningfully to the conversation.
    - haswell 1y ago
      > I did not make a statement about the rest because it's too complex of a topic to which I do not have enough data to contribute meaningfully to the conversation. > There is probably a lot of truth in the tail risks of all of that, and I share these. It just does not at all take away from my enjoyment and optimism at all. On the one hand, I get not expanding on things you don't feel you can contribute to meaningfully and respect that. On the other, isn't this a problem? While I agree with you and cannot fully understand the author's conclusion that it's all bad (I've also found value), I wrestle with the downsides on an ongoing basis and have found myself shying away from using these tools as a result, even though I know they are quite capable of some things. I think that if I only responded "Well, I personally find the tools valuable", that this is only a part of the picture and is a net negative to the overall conversation because the "real" conversation that needs to be had IMO is about those harms, not haggling over whether or not the thing can actually do stuff. There's quite a lot of pro-AI content and opinion floating around right now that amounts to: "Yeah, this thing is dangerous, threatens climate goals, is accelerating an education crisis, only exists because of mass theft, creates a massive category of new problems we don't yet know how to solve...but I'm getting value out of it". And the thing that keeps coming to mind is "Don't Look Up".
      - oulipo 1y ago
- bluefirebrand 1y ago
  > I can do things while the machine works I find this workflow makes my life extremely tedious, because reviewing and fixing whatever the machine produces is a slog for me I suppose it would be exhilarating if I just trusted the output but somehow I just can't bring myself to do that How do you reconcile this with your own work? Maybe you just skim the output? Or do you run it and test that way? Please don't tell me you just trust the AI to write automated tests or something...
  - the_mitsuhiko 1y ago
    > I find this workflow makes my life extremely tedious, because reviewing and fixing whatever the machine produces is a slog for me I don't actually mind the code review process, it's what I'm used to with open source contributions. Not infrequently did I look at a PR that people left me and decide to rewrite it entirely anyways because I did not quite appreciate the method they chose. That however did not mean the PR was useless, it might have provided the necessary input that made me go down that path ultimately. I also think that what can make code review enjoyable is not necessarily because the person on the other side learns something from it. I actually tend to think that this something that does not necessarily happen in PR review, but in 1:1 conversations in person or though more synchronous collaboration. So really my answer is that I do not mind code review. That said there is an important part here: > How do you reconcile this with your own work? Maybe you just skim the output? Or do you run it and test that way? What makes agentic workflows work is good tooling and fast iteration cyclces. That is also exactly what a human needs for a PR review to be actually thorough. I cannot accept a change when I do not know that it exhibits the intended behavior, and to validate that behavior I usually need a test case. Do I need to run the test case myself? Not necessarily because the agent / CI already does for me. However I do sometimes run it with verbose output to also see the intermediate outputs that might otherwise be hidden from me to give me more confidence. It really is hard to describe, but when you put the necessary effort in to make an agentic workflow work, it's really enjoyable. But it does require some upfront investment for it to work. > Please don't tell me you just trust the AI to write automated tests or something... I do in fact trust the AI a lot to write tests. They have become incredibly good at taking certain requirements definitions and derive something that can be mostly trusted. It does not always work, but you get a pretty good feeling for if it will or not. At the end of the day many of the tasks we do are basic CRUD or data transformation tasks.
- vouaobrasil 1y ago
  > Deep research and similar tools alone have helped me navigate complex legal matters recently for my incorporation, On the flipside, it will make it even easier for corporations to use the legal system in their favour and corporations will more easily and effectively use GenAI against individuals even if both have it due to the nature of corporations and their ability to invest in better tools than the individual. So it's just an arms race/prisoner's dilemma and while it provides incremental advantages, it makes everything worse because the system becomes more complex and more oppressive.
- dyauspitr 1y ago
  I don’t understand these people that say they are getting no value from LLM. I’m getting value from them probably every other hour.
  - dingnuts 1y ago
    Simple, I bother to subtract all the time LLMs have wasted by giving me fake leads and lying to me from the time it saves when it successfully generates a few lines of code
    - jessoteric 1y ago
      This right here. Every time I talk to someone claiming 2-4x multipliers, after a few rounds of conversation they eventually admit "yeah you have to learn when to stop when it's no longer productive, you get a feel for when it's veering into wasting your time". It can't be both. It can't be 2-4x multiplier _and_ be wasting your time to the extent that you have to "get a feel for when it has been wasting your time".
    - solarwindy 1y ago
      Having intermittently given in to the hype and tried whatever is supposedly latest and greatest, it without fail goes like this. I’ve even on occasion deceived myself into thinking I’ve finally hit on the right way to use these tools, only to realize in retrospect that under a complete accounting, the time spent to reach that productive mode of working in no way generalizes, so must be paid effectively every time.
  - recursive 1y ago
    Sometimes I try to do stuff with AI. Results are mixed. Sometimes it's somewhat helpful. Sometimes it's a complete waste of time. Maybe that means I'm holding it wrong. In the end, it's probably a wash for me. Maybe slightly positive. I don't understand these people that are saying they are getting huge value from LLMs. I haven't put a ton of effort into figuring out how to "hold it right" because stuff is changing so fast still. I've had bad experiences with rabbit holes like this where the true believers will tell me the pot of gold is always right around the next corner. If and when I can get some positive value out it using whatever's easily available on the default, then I might investigate deeper. My experience mostly consists of GPT and Copilot, as provided by my employer. I'm part of a pilot program to evaluate Copilot. My feedback is that it's slightly positive in agggregate, but individual results are very mixed. It's not worth much to me.
- cdavid 1y ago
  The issue about executive mandate is likely coming from the context of large corporations. It creates fatigue, even though the underlying technology can be used very effectively to do "real work". It becomes hard for people to really see where the tech is valuable (reduce cost to test ideas, accelerate getting into a new area, etc.) vs where it is just a BS generator. Those are typical dysfunctions in larger companies w/ weak leadership. They magnified by a few factors: AI is indistinguishable from magic for non tech leadership, demos that can be slapped quickly but that don't actually work w/o actual investments (which was what leadership wanted to avoid in the first place), and ofc the promise to reduce costs. This happens in parallel to people using it to do their own work in a more bottom up manner. My anecdotal observation is that it is overused for low-value/high visibility work. E.g. it replaces "writing as proof of work" by reducing the cost to write bland, low information, documents used by middle management, which increases bureaucratic load.
- jmsdnns 1y ago
  I'm curious to understand the nature of what you're working on better. Whether or not AI performs well is influenced both by the work you're doing and how experienced you are in it. AI performs better when the work is closer to mainstream work than novel work. It also performs better with lower level instructions, eg. being more specific. As for experience, there are two things: people with less experience get a bigger lift than those with lots of experience, or people wiht lots of experience get a lift by having AI doing the work they don't want to do, which is often unittests and comments or writing the bobdyllionth api endpoint. I read your post the other day and appreciated the optimism, but I wasn't able to work out what kind of work you've been doing since leaving Sentry.
starkparker 1y ago
> The process of coding with an “agentic” LLM appears to be the process of carefully distilling all the worst parts of code review, and removing and discarding all of its benefits. This has been my experience as well, especially since the only space I have to work with agents are on C++ projects where they flat-out spiral into an increasingly dire loop of creating memory leaks, identifying the memory leaks they created, and then creating more memory leaks to fix them. There are probably some fields or languages where these agents are more effective-I've had more luck with small tasks in JS and Python for sure. But I've burned a full work week trying and falling to get Claude to add some missing destructors to a working but leaky C++ project. At one point I let it run fully unattended on the repo in a VM for four hours with the goal of adding a destructor to a class. It added nearly 2k broken LOC that couldn't compile because its first fix added an undeclared destructor, and its response to that failure was to do the same thing to _every class in the project_. Every time saying "I now know what the problem is" as it then created a new problem. If LLMs could just lack confidence in themselves for even a moment and state that they don't know what the problem is but it's willing to throw spaghetti at the wall to find out, I could respect that more than it marching through with absolute confidence that its garbage code did anything but break the project.
- bcrosby95 1y ago
  I tried using them for a game server, written in C, as a test case and I came to the same conclusions. It did pretty well at making something that was buggy but worked, but increasingly continually fell apart around the 3k lines mark, which really isn't much code. It's not like these are esoteric languages, so I'm not sure why they (in my experience) have such a big problem with them. My current guess is that C and C++ aren't so mono domain. The languages its good at seems to be languages used predominantly in the web world, and most of the open code out there is for web. I'm curious if anyone has tried to use it for C# in Unity or a bespoke game in Rust and how well it does there.
  - starkparker 1y ago
    I'm OK with an LLM not knowing how to do something. The part of working with it that I can't get past is its full-throated confidence in its actions. It'll break down, step by step, why its actions would solve the problem, then attempts to build it and can't recognize how or why its changes made the problem worse.
deleted 1y ago
[deleted]
firesteelrain 1y ago
This commentary so far seems to reflect around code. It really helps with brainstorming and I have used it to - Plan my garden - Identify plant issues - Help with planting tips - General research/better google - Systems Engineering - etc Maybe the code generation isnt great but it is really good in a lot of areas.
- asdff 1y ago
  There are ideas that emerge. One is the idea of gel mann amesia being the source of a lot of people complaining about genai code gen, yet people fall into the logical fallacy of trusting a misleading source of information they don't know much about even if they are a domain expert in another area and themselves demand higher quality information to conclude anything in that domain. The other question is, would you have fared any worse without gen ai? Personally I find old school sources more comprehensive. I once picked up a book for free at my garden center, it contained all information about tomatoes including color photos of most every possible deficiency, pest, and disease. If something goes wrong with my tomatoes in a 30 second trip to my bookshelf I have access to high quality information. Meanwhile, what is the genAI trained on? Is my book in the training set? Maybe it is. But maybe its also polluted with the blogspam slop of the last 15 years on the internet, and that isn't easy to parse out without having a book like mine on hand, or better yet the horticultural training the authors behind that book I have on hand have had. Trusting my book is therefore trusting the highest quality output from the corpus of all humanity. And it is like that with most other things. For most things you might encounter in life, there is a high quality source of information or book for low cost or free available. Why reach into the dark and pull out words from the sack when you can get the real deal so easily?
  - firesteelrain 1y ago
    > blogspam slop of the last 15 years on the internet ChatGPT provides sources and so does Gemini now. For example, today, I was doing some brainstorming prompts and found this regarding ideal soil mixes and settled on Soil Mix #3 because I am going to be using Jungle Growth https://mamasnowcooks.com/gardening/my-potting-mix-recipes/ https://mamasnowcooks.com/gardening/my-potting-mix-recipes/
vouaobrasil 1y ago
> the model could be created with ethically, legally, voluntarily sourced training data There is no such thing as ethical AI, because "voluntary" usually means voluntary without the participants really understanding what they are making, which is just another tool in the arms race of increasingly sophisticated AI models – which will largely be needed just to "one up" the other guy. "Ethical" AI is like forced pit-fighting where we ask if we can find willing volunteers to fight for the death for a chance for their freedom. It's sickening.
Game_Ender 1y ago
I am not sure there is too much value for this article for the typical hacker news conversation on LLM based tooling. Here we generally focus on if the tooling is effective, and can it be used make software quicker or more cheaply. The problem is the author is opposed using the cutting edge models on privacy and ethics grounds. So they say: > I have woefully little experience with these tools. > I do not want to be using the cloud versions of these models with their potentially hideous energy demands; I’d like to use a local model. But there is obviously not a nicely composed way to use local models like this. > The models and tools that people are raving about are the big, expensive, harmful ones. If I proved to myself yet again that a small model with bad tools was unpleasant to use, I wouldn’t really be addressing my opponents’ views. Then without having any real practical experience with the cutting edge tooling they predict: > As I have written about before, I believe the mania will end. There will then be a crash, and a “winter”. But, as I may not have stressed sufficiently, this crash will be the biggest of its kind — so big, that it is arguably not of a kind at all. The level of investment in these technologies is bananas and the possibility that the investors will recoup their investment seems close to zero. I think a more accurate take is this will be like self driving, huge investments, many more losers thank winners, and it will take longer than all the boosters think. But in the end we did get actual self driving cars, but this time it's with LLMs it is something that anyone can use by clicking a link vs. waiting for lots of cars to be built and deployed.
- recursive 1y ago
  > Here we generally focus on if the tooling is effective That's one of the things we sometimes focus on. > But in the end we did get actual self driving cars Kind of. Barely. You still can't buy a car without a steering wheel. If they were self-driving, that would be a waste of space and resources.
empath75 1y ago
There is a split between people who think of labor as an end in itself and people who think of a labor as a means of achieving some other end and people in the former category are going to always dislike any kind of automation. Even though I've written a bunch of code, I don't really enjoy writing code. I enjoy building systems and solving problems. If I never have to write another line of code to do that, I couldn't be happier. Other people have invested a lot of their time and their self image in becoming a good "computer programmer", and they are going to have a really hard time interacting with AI programmers or really any other kind of automation that removes the part of the job that they actually enjoy doing. Really, it's not much difference between that and musicians that got mad at djs for just playing other people's music, who then got mad at djs that use ableton live instead of learning to beat match. Do you think the _process_ of making music is the important part, or the final sounds that are produced? Just like DJ's and collage artists can be inventive and express ideas using work created by other people, people who use AI to code can still express creative ideas through how they use it and combine it and manipulate it.
- seadan83 1y ago
  > Do you think the _process_ of making music is the important part, or the final sounds that are produced? It is both, for different reasons. Not caring for the process is the path of the mid-wit.
  - thoroughburro 1y ago
    Ultimately, you’re arguing for generalism over specialism. Any specialist must be ignoring part of the larger process. “All specialists are midwits” is obviously false. Alternatively, even “most expert musicians are their own luthiers”… maybe more obviously false?
    - seadan83 1y ago
      I'm interested for you to elaborate a bit more! I'd like to see more clearly the connection of my statements and the argument for generalist over specialist. Overall, I think I'm arguing that you cannot become an expert without continually improving the process; and to do that you need to continually understand and study the process at progressively deeper levels. If a person cares nothing for the process, just the outcome - they therefore cannot achieve mastery (I'm like 85% convinced this is true). If we assume a specialist is also an expert, then by definition specialists are not midwits - I agree. An expert musician I would suspect should know how the different qualities of the instrument they play affect the outcome. Whether they can actually construct an instrument is a different question, but they ought to know the differences. Can you also be an expert musician and not know, nor care about the differences in instrument construction quality and material? Can you be an expert on a piano and not know what makes for a good piano? I think the most exact analogy would be a musician who does not care about how an instrument is played - they don't study it, they don't care, just do something that makes for a nice sound, and that is all they desire. I don't think a person can achieve mastery in that manner, to achieve mastery - the 'how' does matter; the master should know everything about the different 'hows' and the trade-offs; and that is what makes them a master.
ednite 1y ago
Hats off to the author for expressing this so candidly, I really get where they’re coming from. While I tend to lean more toward the side of AI progression, it’s clearly an inevitable part of society now. I do worry about the long-term impact. I also feel that constant pressure to keep up, adapt, and integrate AI into everything feels like it’s already contributing to fatigue and burnout. Thoughtful post.
- dingnuts 1y ago
  > it’s clearly an inevitable part of society now. this won't be true until the financials are squared up. Right now there is a lot of funky math propping up these companies. If the bet doesn't pay off with real productivity gains the whole AI industry will disappear the way the first generation of AI assistants did ten years ago. Remember Echo?
  - ednite 1y ago
    You make a solid point, the financials will ultimately decide how far this goes, as with most things in business. That said, I don’t think it quite echoes Echo (that pun might be intentional) or maybe it does, financially speaking. For me, this wave feels different: deeper integration, broader scale, and real-time utility. I agree, if the returns don’t hold up, a major correction isn’t off the table. It reminds me a bit of the smartphone era, rapid adoption, strong dependence. I guess, the difference is, that phones had clear monetization paths. With AI, we’re still betting. The bigger question for me is, if this all collapses, what happens to the workflows and investments already built around it?
  - mholm 1y ago
    There's funky math propping up the training of new AI. That much is certainly true. But just running frontier models on hardware is very doable, and is already capable of absolutely changing the software landscape in the next decade, even if nobody ever trains a model again.
triceratops 1y ago
GenAI may not be done thinking about you though. (Sorry for the low-effort response. But it's kinda true)
energy123 1y ago
> I have woefully little experience with these tools. I’ve tried them out a little bit, and almost every single time the result has been a disaster that has not made me curious to push further. Yet, I keep hearing from all over the industry that I should. Well this is the issue. The author assumed that these tools require no skill to use properly.
CuriouslyC 1y ago
"Despite this plethora of negative experiences, executives are aggressively mandating the use of AI6. It looks like without such mandates, most people will not bother to use such tools, so the executives will need muscular policies to enforce its use. Being forced to sit and argue with a robot while it struggles and fails to produce a working output, while you have to rewrite the code at the end anyway, is incredibly demoralizing. This is the kind of activity that activates every single major cause of burnout at once. But, at least in that scenario, the thing ultimately doesn’t work, so there’s a hope that after a very stressful six month pilot program, you can go to management with a pile of meticulously collected evidence, and shut the whole thing down." The counterpoint to this is that _SOME_ people are able to achieve force multiplication (even at the highest levels of skill, it's not just a juniors-only phenomenon), and _THAT_ is what is driving management adoption mandates. They see that 2-4x increases in productivity are possible under the correct circumstances, and they're basically passing down mandates for the rank and file to get with the program and figure out how to reproduce those results, or find another job.
- seadan83 1y ago
  Burning the furniture in your home is a force multiplier for your homes heating system. Get with the program, or find a new home.
  - CuriouslyC 1y ago
    If your argument is that agentic coding workflows will retard the skill development of junior and midrange engineers I'm not entirely in disagreement with you, that's a problem that has to be solved in relation to how we use gen AI across every domain.
    - seadan83 1y ago
      Skill retardation is actually beyond the point. I'm largely raising a counter example for why the following (rough paraphrasing) is not sound: "SOME people figured out how to use these tools to go 2x to 4x faster, you do the same, or you're fired!". Let's say "n" is the sum complexity of a system. While some developers can take an approach that yields a development output of: (1.5 * log n), the AI tools might have a development output of: (4 * log n)^4/n. That is, initially more & faster, but eventually a lot less and slower. The parable of the soviet beef farmer comes to mind: In this parable, the USSR mandated its beef farmers increase beef output YoY by 20%, every year. The first year, the heroic farmer improved the health of their livestock, bought a few extra cows and hit their target. The next year, to achieve 20% YoY, the farmer cuts every corner and maximizes every efficiency, they even exchange all their possessions to buy some black market cows. The third year, the farmer can't make the 20% increase, they slaughter almost all of their herd. The fourth year, the farmer has essentially no herd, they can't come close to their last years output - let alone increase it. So far short of quota, the heroic beef farmer then shot himself. (side-note: Which is also analagous to people not raising their skill levels too, but not my main point - I'm more thinking about how development slows down relative to the complexity and size of a software system. The 'not-increasing skills' angle is arguably there too. The main point is short term trade-offs to achieve goals rather than selecting long term and sustainable targets, and the relationship of those decisions to a blind demand to increase output) So, instead of working on the insulation of the home, instead of upgrading the heating system, to heat the home faster we burn the furniture. It works.. to a point. Like, what happens when you run out of furniture, or the house catches fire? Seemingly that will be a problem for Q2 of next year, for now, we are moving faster!! I think this ties into the programming industry quite heavily from the perspective where managers often want things to work just long enough for them to be promoted. Doesn't have to work well for years, doesn't have to have the support tools needed for that, nope - just long enough that they can get the quarterly reward and then move on to not worry about the support mess left behind. To boot too, the feedback cycle for whether something was a good idea in software or not is slow, oftentimes years. AI tools have not been out for a long time, just a couple years themselves, it'll be another few before we see what happens when a system is grown to 5M lines through mostly AI tooling and the codebase itself is 10 years old - will that system be too brittle to update? FWIW, I'm of the point of view that quality, time and cost are not an iron triangle - it is not a choose two situation. Instead, quality is a requirement for low cost and low time. You cannot move quickly when quality is low (from my experience, the slowdown of low quality can manifest quickly too - on the order of hours. A shortcut taken now can reduce velocity just even later that same day). Thus, mandates from management to move 2x to 4x faster, when it's not clear that AI tools actually deliver 2x to 4x benefits over the longer term (perhaps not even in the shorter term), feels a lot like the soviet beef farmer parable, or burning furniture to stay warm.
Noumenon72 1y ago
> I found the experience of reading this commit history and imagining myself using such a tool — without exaggeration — nauseating. The commit history[1] looks like a totally normal commit history on page 1, but I clicked farther down and found commits like > Ask Claude to fix bug with token exchange callback when reusing refresh token. > As explained in the readme, when we rotate the refresh token, we still allow the previous refresh token to be used again, in case the new token is lost due to an error. However, in the previous commit, we hadn't properly updated the wrapped key to handle this case. > Claude also found that the behavior when `grantProps` was returend without `tokenProps` was confusing -- you'd expect the access token to have the new props, not the old. I agreed and had Claude update it. > Claude Code transcript: https://claude-workerd-transcript.pages.dev/oauth-provider-token-exchange-callback2 https://claude-workerd-transcript.pages.dev/oauth-provider-t... It seems no different than "run prettier on the code" or "re-run code gen". I don't think it's fair to pick on this specific repo if your objection is just "I don't like reviewing code written by a tool that can't improve, regardless of what the commits look like". I'd call that "misanthropomorphizing". 1. https://github.com/cloudflare/workers-oauth-provider/commits/main/ https://github.com/cloudflare/workers-oauth-provider/commits...
lsy 1y ago
Part of the reason the debate over usefulness is so exhausting is because there is an element of randomness, anecdote, and special pleading built into the use of the tool. One person has a good experience in their context, one person has a bad experience in their context. One person sees a bad result and throws up their hands and says it sucks, another tweaks their prompt until they get a good result and says it works. In addition there's an element of personality: One person might think it's more meaningful to compose a party invitation to friends personally, another asks ChatGPT to write it to make sure they don't miss anything. As someone who is on the skeptic side, I of course don't appreciate the "holding it wrong" discourse, which I think is rude and dismissive of people and their right to choose how they do things. But at the end of the day it's just a counterweight to the corresponding discourse on the skeptic side which is essentially "gross, you used AI". What's worse though is that institutions are forcing us to use this type of tool without consent, either through mandates or through integrations without opt-outs. To me it doesn't meet the bar for a revolutionary technology if it has to be propagated forcibly or with social scolding.
wagwang 1y ago
Its hard to tell if genai is just hopelessly complicated or if there are people with really accurate takes that are drowned out by the noise.
AIorNot 1y ago
1. the hype is really insane so I get the fatigue, but the tech is insane as well, and the promise for the (long term) future is insane.. 2. People mostly hate new things (human nature) 3. its fun watching nerds complain about AI after they spent the past 50 years automating other peoples careers.. turnabout is fair play? All the nerdsniping around GenAI and tech taking away from programming is funny. 4. Remember its a tool, not a replacement (yet), get used to it.. we're not supposed to be luddites. use it and optimize - its our job to convince managers and executives that we are masters over the tool and not vice versa
mjburgess 1y ago
I can at least say why offering a "theory of AI" in the dective sense is very difficult: AI development is an anti-inductive process. Anti-inductive processes are those, like e.g., fraud, which change because you have measured them. Once a certain sort of fraud is made illegal and difficult, most move on to a different kind. AI models are at-base memorisers, with a little structured generalisation around the region of memory. They are not structure learners who happen to memorize. This makes creating a theory of an AI model kinda impossible because the "representations" they learn are so-called "entangled", another way of saying: garbage. The AI model's representation of language does not match language structure, rather each representation is a mixture of what we can evidently see as several competing structural features. But the model has no access to this structure, because its literally not a property of the data but of the data generating process. Now this seems like "a good theory" of AI in the monkish sense, but a problem arises: for each fragile boundary that benchmarking and testing shows, the AI companies collect data on these failures and retrain the model. This is, in a sense, anti-inductive fraud: companies seek to find out how they are found out, and extend the model to cover those regions. But models never actually gain the structural capabilities their behaviour is taken to imply. So, of course, this would drive any theorist insane. By the time you've figured out gpt3.5's fakery, all the openai chatbot data theyve collected has made gpt4 -- and all paper has been covered over all the edges. Until your fingers are cut next time on the now dark fragile boundary of openai's deception. I can give very good reasons, independent of any particular AI model why: the manifold hypothesis is wrong, AI models are memorizers + structure, why they dont find latent structure but combine statistical heurstics, why their representations are not "entangled" such that they can be "disentangled" but necessarily model-unknowable blends of expert-known scientific joints -- and so on. But none of this knowledge can become useful in the face of a billion-dollar training company competing against me, anti-inductively, to apply any of this to any given model. Perhaps then, I suppose, we can look at the behaviour these systems induce in their users and its downstream effects. This is what OP here does, throws hands up at a theory and says only: however this is working, it cannot be good. This is, of course, what we do with fraud, cults, and many other systems of deception. We say: we're not going to argue with the conspiracy theoriest, they are specialists at coming up with ever more elbobate adaptions to the self-deception. Instead, we observe by proxy, that they behave pathological;y -- that they are broken in otehr ways. I think that's a fair approach, and at least one that allows the author to proceed without a theory.
grafelic 1y ago
For me the problem with GenAI is that it erodes away what I enjoy in my job and in my free time. I enjoy learning and eventually understanding methods to solve problems. I enjoy applying these methods in my mind, before trying them on the computer. I am not interested in getting the solutions to problems served to me without understanding why it works. I feel, yes it is an emotion, that GenAI is unhealthy for the progress of mankind, since it encourages a lazy mind, stifles curiosity and will lead to inbred information filling our digital spaces. I am looking forward to my future career as a gardener (although with a smidge of sadness) when AI has sucked out all creativity, ingeniuity and enjoyment from my field of work.
kentonv 1y ago
The author goes into great detail about how he looked at my commit log[0] where I used AI, and he found it "nauseating" and concluded he'd never want to work that way. I'm certainly not going to tell anyone that they're wrong if they try AI and don't like it! But this guy... did not try it? He looked at a commit log, tried to imagine what my experience was like, and then decided he didn't like that? And then he wrote about it? Folks, it's really not that hard to actually try it. There is no learning curve. You just run the terminal app in your repo and you ask it to do things. Please, I beg you, before you go write walls of text about how much you hate the thing, actually try it, so that you actually have some idea what you're talking about. Six months ago, I myself imagined that I would hate AI-assisted coding! Then I tried it. I found out a lot of things that surprised me, and it turns out I don't hate it as much as I thought. [0] https://github.com/cloudflare/workers-oauth-provider/commits/main/?after=a6e3e06c2642e0fd4c185374753201cffc21ce8a+104 https://github.com/cloudflare/workers-oauth-provider/commits... (link to oldest commits so you can browse in order; newer commits are not as interesting)
- mwcampbell 1y ago
  He explained why he hasn't tried it in the section "The Challenge". Also see footnote 21, which says that what he called his "quirky constraints" in the main text are really bedrock ethical concerns. I respect him a great deal for not compromising on these. Edit to add: Full disclosure: I've been a fan of his writing and work for a long time anyway, and am a supporter on Patreon.
  - kentonv 1y ago
    Yes, the ethical debate is fair and worth having, and I don't think someone necessarily has to try it to be able to debate the ethics. I obviously don't quite agree with his opinions here but I respect them. (Though I would say, before I really tried it, I had believed that AI does a lot more "plagiarism" -- lifting of entire passages or snippets. Actually using it has made clearer to me that it really is more like someone who has learned by reading, and then applies those learnings, at least most of the time.) I just don't like it when people argue that it doesn't work or doesn't do what they want without having actually tried it... There's too much echo chamber of people who don't want AI to work, and so convince each other that it doesn't, and repeat each other's arguments, all without ever taking a step outside their comfort zones. (Again... this was me, 6 months ago.)
malwrar 1y ago
I’m near term AI-skeptical, but I think most of the capability & cost-related problems will be solved by improvements to current model architectures. My personal canary for the start of this will be when we start seeing successful architectures beyond (roughly) “tokens -> transformer blocks -> linear transformation to probability distribution”. I wanted to engage with a few of the author’s specific points of skepticism though (hope you don’t mind, if you’re reading this) since I’m not sure if I agree with them. > Energy Usage This has always seemed like a fake problem to me, and mostly just a way to rally environmentalists around protesting genai (and bitcoin and…). The costs (time & capital) associated with nuclear power are mostly regulatory; in the past decade China has routinely built numerous modern reactors while the west has been actively scaling nuclear power down. I have yet to see a good case for not simply scaling up power generation to meet demand. This is assuming that AI power demands remain constant and even require gigawatt-scale power sources to train & operate. For all we know, one paper from one researcher anywhere in the world could show us that current model architectures are orders of magnitude inefficient and make current-standard frontier model training doable in a basement. > The Educational Impact A few years ago I came across some cool computer vision videos on youtube and became inspired to try reading the papers and understanding more. The papers were surprisingly short (only a few dozen pages usually), but contained jargon and special symbols that I found impossible to google. I eventually read more papers and even some books, and eventually discovered that if I had simply read e.g. Hartley & Zisserman’s Multiple View Geometry (commonly just referenced as “MVG”, another mystery I had to figure out) I would have had the hidden knowledge necessary to understand much of the math. If I had simply read earlier papers, I’d be familiar with the fact that much of the content is just slight additions on previous works rather than fully greenfield ideas. I’m not rich enough or successful enough to access the formal education that would have probably provided me the resources to learn this. Youtube videos from practitioners in the field felt more like insecure nerds gatekeeping their field with obtuse lessons. It took me about a year to produce anything useful without just using opencv. When I tried asking the same questions I had as a beginner, chatgpt was able to answer everything I was confused on, answer my dumb followup questions, and even produce demo code so I could explore the concepts. That experience has made it an irreplaceable educational tool for me. Im sure more people will cheat, but college is already an overvalued competence signal anyways. > The Invasion of Privacy Temporary problem, someone will produce a usable hardware equivalent that makes it similarly easy to use genai w/o the privacy implications. This is a segment begging for someone to provide a solution, I think most people with the resources currently to solve it just want to be the AI gatekeeer and thus the service model for this tech. > The Stealing We won’t find common ground, I dont believe in ownership of information and think copyright is largely immoral. > The Fatigue This section seems to largely revolve around no clear obvious tooling route to simply using the tech (complete empathy there, Im currently building my own tooling around it), along with not having a clear mental model of how these models work. I had the same anxiety when chatgpt first blew my mind and just decided to learn how they work. They’re dirt fucking simple, shitty non-learned computer vision systems are orders of magnitude more complex. You seem like a cool person, hmu my profile name @ gmail if you want to talk about it! I like 3b1b’s videos if you want a shorter version, karpathy is overhyped imo but also cool if you want to build a toy model line by line,

169 points by kaycebasques 1y ago

I left this comment originally on lobsters: While I understand the fear, I don’t really share it. And if I where to go to the root of it, I think I really most take issue with this: > My experiences of genAI are all extremely bad, but that is barely even anecdata. Their experiences are neutral-to-positive. Little scientific data exists. How to resolve this? My experience is astonishingly positive. I would not have imagined how much of a help these tools have become. Deep research and similar tools alone have helped me navigate complex legal matters recently for my incorporation, they have uncovered really useful information that I just would not have found that quickly. First cursor, now Claude Code have really changed how I work. Especially since for the last month or so, I feel myself more and more in the position where I can do things while the machine works. It’s truly liberating and it gives me a lot of joy. So it’s not “neutral-to-positive” to me, it’s exhilarating. And that extends particularly to this part: > Despite this plethora of negative experiences, executives are aggressively mandating the use of AI. It looks like without such mandates, most people will not bother to use such tools, so the executives will need muscular policies to enforce its use. When I was at Sentry the adoption of AI happened by ICs before the company even put money behind it. In fact my memory is that if anything only at the point where an exceeding number of AI invoices showed up from IC expenses did we realize how widespread adoption has been. This was grounds up. For my non techy friends it’s even tricker because some of them work in companies that outright try to prevent the adoption of AI, but they are paying for it themselves to help them with the work. Some of them pay for the expensive ChatGPT package even! None of this should be disregarded, but it stands in a crass contrast to what this post says. That said, I understand where Glyph comes from and I appreciate that point. There is probably a lot of truth in the tail risks of all of that, and I share these. It just does not at all take away from my enjoyment and optimism at all.

oulipo 1y ago
Basically his article says: check the documented issue with the climate, the education, the destruction of trust in society, and your answer is "yes but it makes my nice cuddly programming job better I like it"... don't you understand the issue?
- the_mitsuhiko 1y ago
  I responded to quite specific quotes which I took issues with, and I intentionally did not talk about the rest of it. I did not make a statement about the rest because it's too complex of a topic to which I do not have enough data to contribute meaningfully to the conversation.
  - haswell 1y ago
    > I did not make a statement about the rest because it's too complex of a topic to which I do not have enough data to contribute meaningfully to the conversation. > There is probably a lot of truth in the tail risks of all of that, and I share these. It just does not at all take away from my enjoyment and optimism at all. On the one hand, I get not expanding on things you don't feel you can contribute to meaningfully and respect that. On the other, isn't this a problem? While I agree with you and cannot fully understand the author's conclusion that it's all bad (I've also found value), I wrestle with the downsides on an ongoing basis and have found myself shying away from using these tools as a result, even though I know they are quite capable of some things. I think that if I only responded "Well, I personally find the tools valuable", that this is only a part of the picture and is a net negative to the overall conversation because the "real" conversation that needs to be had IMO is about those harms, not haggling over whether or not the thing can actually do stuff. There's quite a lot of pro-AI content and opinion floating around right now that amounts to: "Yeah, this thing is dangerous, threatens climate goals, is accelerating an education crisis, only exists because of mass theft, creates a massive category of new problems we don't yet know how to solve...but I'm getting value out of it". And the thing that keeps coming to mind is "Don't Look Up".
    - oulipo 1y ago
bluefirebrand 1y ago
> I can do things while the machine works I find this workflow makes my life extremely tedious, because reviewing and fixing whatever the machine produces is a slog for me I suppose it would be exhilarating if I just trusted the output but somehow I just can't bring myself to do that How do you reconcile this with your own work? Maybe you just skim the output? Or do you run it and test that way? Please don't tell me you just trust the AI to write automated tests or something...
- the_mitsuhiko 1y ago
  > I find this workflow makes my life extremely tedious, because reviewing and fixing whatever the machine produces is a slog for me I don't actually mind the code review process, it's what I'm used to with open source contributions. Not infrequently did I look at a PR that people left me and decide to rewrite it entirely anyways because I did not quite appreciate the method they chose. That however did not mean the PR was useless, it might have provided the necessary input that made me go down that path ultimately. I also think that what can make code review enjoyable is not necessarily because the person on the other side learns something from it. I actually tend to think that this something that does not necessarily happen in PR review, but in 1:1 conversations in person or though more synchronous collaboration. So really my answer is that I do not mind code review. That said there is an important part here: > How do you reconcile this with your own work? Maybe you just skim the output? Or do you run it and test that way? What makes agentic workflows work is good tooling and fast iteration cyclces. That is also exactly what a human needs for a PR review to be actually thorough. I cannot accept a change when I do not know that it exhibits the intended behavior, and to validate that behavior I usually need a test case. Do I need to run the test case myself? Not necessarily because the agent / CI already does for me. However I do sometimes run it with verbose output to also see the intermediate outputs that might otherwise be hidden from me to give me more confidence. It really is hard to describe, but when you put the necessary effort in to make an agentic workflow work, it's really enjoyable. But it does require some upfront investment for it to work. > Please don't tell me you just trust the AI to write automated tests or something... I do in fact trust the AI a lot to write tests. They have become incredibly good at taking certain requirements definitions and derive something that can be mostly trusted. It does not always work, but you get a pretty good feeling for if it will or not. At the end of the day many of the tasks we do are basic CRUD or data transformation tasks.
vouaobrasil 1y ago
> Deep research and similar tools alone have helped me navigate complex legal matters recently for my incorporation, On the flipside, it will make it even easier for corporations to use the legal system in their favour and corporations will more easily and effectively use GenAI against individuals even if both have it due to the nature of corporations and their ability to invest in better tools than the individual. So it's just an arms race/prisoner's dilemma and while it provides incremental advantages, it makes everything worse because the system becomes more complex and more oppressive.
dyauspitr 1y ago
I don’t understand these people that say they are getting no value from LLM. I’m getting value from them probably every other hour.
- dingnuts 1y ago
  Simple, I bother to subtract all the time LLMs have wasted by giving me fake leads and lying to me from the time it saves when it successfully generates a few lines of code
  - jessoteric 1y ago
    This right here. Every time I talk to someone claiming 2-4x multipliers, after a few rounds of conversation they eventually admit "yeah you have to learn when to stop when it's no longer productive, you get a feel for when it's veering into wasting your time". It can't be both. It can't be 2-4x multiplier _and_ be wasting your time to the extent that you have to "get a feel for when it has been wasting your time".
  - solarwindy 1y ago
    Having intermittently given in to the hype and tried whatever is supposedly latest and greatest, it without fail goes like this. I’ve even on occasion deceived myself into thinking I’ve finally hit on the right way to use these tools, only to realize in retrospect that under a complete accounting, the time spent to reach that productive mode of working in no way generalizes, so must be paid effectively every time.
- recursive 1y ago
  Sometimes I try to do stuff with AI. Results are mixed. Sometimes it's somewhat helpful. Sometimes it's a complete waste of time. Maybe that means I'm holding it wrong. In the end, it's probably a wash for me. Maybe slightly positive. I don't understand these people that are saying they are getting huge value from LLMs. I haven't put a ton of effort into figuring out how to "hold it right" because stuff is changing so fast still. I've had bad experiences with rabbit holes like this where the true believers will tell me the pot of gold is always right around the next corner. If and when I can get some positive value out it using whatever's easily available on the default, then I might investigate deeper. My experience mostly consists of GPT and Copilot, as provided by my employer. I'm part of a pilot program to evaluate Copilot. My feedback is that it's slightly positive in agggregate, but individual results are very mixed. It's not worth much to me.
cdavid 1y ago
The issue about executive mandate is likely coming from the context of large corporations. It creates fatigue, even though the underlying technology can be used very effectively to do "real work". It becomes hard for people to really see where the tech is valuable (reduce cost to test ideas, accelerate getting into a new area, etc.) vs where it is just a BS generator. Those are typical dysfunctions in larger companies w/ weak leadership. They magnified by a few factors: AI is indistinguishable from magic for non tech leadership, demos that can be slapped quickly but that don't actually work w/o actual investments (which was what leadership wanted to avoid in the first place), and ofc the promise to reduce costs. This happens in parallel to people using it to do their own work in a more bottom up manner. My anecdotal observation is that it is overused for low-value/high visibility work. E.g. it replaces "writing as proof of work" by reducing the cost to write bland, low information, documents used by middle management, which increases bureaucratic load.
jmsdnns 1y ago
I'm curious to understand the nature of what you're working on better. Whether or not AI performs well is influenced both by the work you're doing and how experienced you are in it. AI performs better when the work is closer to mainstream work than novel work. It also performs better with lower level instructions, eg. being more specific. As for experience, there are two things: people with less experience get a bigger lift than those with lots of experience, or people wiht lots of experience get a lift by having AI doing the work they don't want to do, which is often unittests and comments or writing the bobdyllionth api endpoint. I read your post the other day and appreciated the optimism, but I wasn't able to work out what kind of work you've been doing since leaving Sentry.

starkparker 1y ago

> The process of coding with an “agentic” LLM appears to be the process of carefully distilling all the worst parts of code review, and removing and discarding all of its benefits. This has been my experience as well, especially since the only space I have to work with agents are on C++ projects where they flat-out spiral into an increasingly dire loop of creating memory leaks, identifying the memory leaks they created, and then creating more memory leaks to fix them. There are probably some fields or languages where these agents are more effective-I've had more luck with small tasks in JS and Python for sure. But I've burned a full work week trying and falling to get Claude to add some missing destructors to a working but leaky C++ project. At one point I let it run fully unattended on the repo in a VM for four hours with the goal of adding a destructor to a class. It added nearly 2k broken LOC that couldn't compile because its first fix added an undeclared destructor, and its response to that failure was to do the same thing to _every class in the project_. Every time saying "I now know what the problem is" as it then created a new problem. If LLMs could just lack confidence in themselves for even a moment and state that they don't know what the problem is but it's willing to throw spaghetti at the wall to find out, I could respect that more than it marching through with absolute confidence that its garbage code did anything but break the project.

bcrosby95 1y ago
I tried using them for a game server, written in C, as a test case and I came to the same conclusions. It did pretty well at making something that was buggy but worked, but increasingly continually fell apart around the 3k lines mark, which really isn't much code. It's not like these are esoteric languages, so I'm not sure why they (in my experience) have such a big problem with them. My current guess is that C and C++ aren't so mono domain. The languages its good at seems to be languages used predominantly in the web world, and most of the open code out there is for web. I'm curious if anyone has tried to use it for C# in Unity or a bespoke game in Rust and how well it does there.
- starkparker 1y ago
  I'm OK with an LLM not knowing how to do something. The part of working with it that I can't get past is its full-throated confidence in its actions. It'll break down, step by step, why its actions would solve the problem, then attempts to build it and can't recognize how or why its changes made the problem worse.

deleted 1y ago

[deleted]

firesteelrain 1y ago

This commentary so far seems to reflect around code. It really helps with brainstorming and I have used it to - Plan my garden - Identify plant issues - Help with planting tips - General research/better google - Systems Engineering - etc Maybe the code generation isnt great but it is really good in a lot of areas.

asdff 1y ago
There are ideas that emerge. One is the idea of gel mann amesia being the source of a lot of people complaining about genai code gen, yet people fall into the logical fallacy of trusting a misleading source of information they don't know much about even if they are a domain expert in another area and themselves demand higher quality information to conclude anything in that domain. The other question is, would you have fared any worse without gen ai? Personally I find old school sources more comprehensive. I once picked up a book for free at my garden center, it contained all information about tomatoes including color photos of most every possible deficiency, pest, and disease. If something goes wrong with my tomatoes in a 30 second trip to my bookshelf I have access to high quality information. Meanwhile, what is the genAI trained on? Is my book in the training set? Maybe it is. But maybe its also polluted with the blogspam slop of the last 15 years on the internet, and that isn't easy to parse out without having a book like mine on hand, or better yet the horticultural training the authors behind that book I have on hand have had. Trusting my book is therefore trusting the highest quality output from the corpus of all humanity. And it is like that with most other things. For most things you might encounter in life, there is a high quality source of information or book for low cost or free available. Why reach into the dark and pull out words from the sack when you can get the real deal so easily?
- firesteelrain 1y ago
  > blogspam slop of the last 15 years on the internet ChatGPT provides sources and so does Gemini now. For example, today, I was doing some brainstorming prompts and found this regarding ideal soil mixes and settled on Soil Mix #3 because I am going to be using Jungle Growth https://mamasnowcooks.com/gardening/my-potting-mix-recipes/ https://mamasnowcooks.com/gardening/my-potting-mix-recipes/

vouaobrasil 1y ago

> the model could be created with ethically, legally, voluntarily sourced training data There is no such thing as ethical AI, because "voluntary" usually means voluntary without the participants really understanding what they are making, which is just another tool in the arms race of increasingly sophisticated AI models – which will largely be needed just to "one up" the other guy. "Ethical" AI is like forced pit-fighting where we ask if we can find willing volunteers to fight for the death for a chance for their freedom. It's sickening.

Game_Ender 1y ago

I am not sure there is too much value for this article for the typical hacker news conversation on LLM based tooling. Here we generally focus on if the tooling is effective, and can it be used make software quicker or more cheaply. The problem is the author is opposed using the cutting edge models on privacy and ethics grounds. So they say: > I have woefully little experience with these tools. > I do not want to be using the cloud versions of these models with their potentially hideous energy demands; I’d like to use a local model. But there is obviously not a nicely composed way to use local models like this. > The models and tools that people are raving about are the big, expensive, harmful ones. If I proved to myself yet again that a small model with bad tools was unpleasant to use, I wouldn’t really be addressing my opponents’ views. Then without having any real practical experience with the cutting edge tooling they predict: > As I have written about before, I believe the mania will end. There will then be a crash, and a “winter”. But, as I may not have stressed sufficiently, this crash will be the biggest of its kind — so big, that it is arguably not of a kind at all. The level of investment in these technologies is bananas and the possibility that the investors will recoup their investment seems close to zero. I think a more accurate take is this will be like self driving, huge investments, many more losers thank winners, and it will take longer than all the boosters think. But in the end we did get actual self driving cars, but this time it's with LLMs it is something that anyone can use by clicking a link vs. waiting for lots of cars to be built and deployed.

recursive 1y ago
> Here we generally focus on if the tooling is effective That's one of the things we sometimes focus on. > But in the end we did get actual self driving cars Kind of. Barely. You still can't buy a car without a steering wheel. If they were self-driving, that would be a waste of space and resources.

empath75 1y ago

There is a split between people who think of labor as an end in itself and people who think of a labor as a means of achieving some other end and people in the former category are going to always dislike any kind of automation. Even though I've written a bunch of code, I don't really enjoy writing code. I enjoy building systems and solving problems. If I never have to write another line of code to do that, I couldn't be happier. Other people have invested a lot of their time and their self image in becoming a good "computer programmer", and they are going to have a really hard time interacting with AI programmers or really any other kind of automation that removes the part of the job that they actually enjoy doing. Really, it's not much difference between that and musicians that got mad at djs for just playing other people's music, who then got mad at djs that use ableton live instead of learning to beat match. Do you think the _process_ of making music is the important part, or the final sounds that are produced? Just like DJ's and collage artists can be inventive and express ideas using work created by other people, people who use AI to code can still express creative ideas through how they use it and combine it and manipulate it.

seadan83 1y ago
> Do you think the _process_ of making music is the important part, or the final sounds that are produced? It is both, for different reasons. Not caring for the process is the path of the mid-wit.
- thoroughburro 1y ago
  Ultimately, you’re arguing for generalism over specialism. Any specialist must be ignoring part of the larger process. “All specialists are midwits” is obviously false. Alternatively, even “most expert musicians are their own luthiers”… maybe more obviously false?
  - seadan83 1y ago
    I'm interested for you to elaborate a bit more! I'd like to see more clearly the connection of my statements and the argument for generalist over specialist. Overall, I think I'm arguing that you cannot become an expert without continually improving the process; and to do that you need to continually understand and study the process at progressively deeper levels. If a person cares nothing for the process, just the outcome - they therefore cannot achieve mastery (I'm like 85% convinced this is true). If we assume a specialist is also an expert, then by definition specialists are not midwits - I agree. An expert musician I would suspect should know how the different qualities of the instrument they play affect the outcome. Whether they can actually construct an instrument is a different question, but they ought to know the differences. Can you also be an expert musician and not know, nor care about the differences in instrument construction quality and material? Can you be an expert on a piano and not know what makes for a good piano? I think the most exact analogy would be a musician who does not care about how an instrument is played - they don't study it, they don't care, just do something that makes for a nice sound, and that is all they desire. I don't think a person can achieve mastery in that manner, to achieve mastery - the 'how' does matter; the master should know everything about the different 'hows' and the trade-offs; and that is what makes them a master.

ednite 1y ago

Hats off to the author for expressing this so candidly, I really get where they’re coming from. While I tend to lean more toward the side of AI progression, it’s clearly an inevitable part of society now. I do worry about the long-term impact. I also feel that constant pressure to keep up, adapt, and integrate AI into everything feels like it’s already contributing to fatigue and burnout. Thoughtful post.

dingnuts 1y ago
> it’s clearly an inevitable part of society now. this won't be true until the financials are squared up. Right now there is a lot of funky math propping up these companies. If the bet doesn't pay off with real productivity gains the whole AI industry will disappear the way the first generation of AI assistants did ten years ago. Remember Echo?
- ednite 1y ago
  You make a solid point, the financials will ultimately decide how far this goes, as with most things in business. That said, I don’t think it quite echoes Echo (that pun might be intentional) or maybe it does, financially speaking. For me, this wave feels different: deeper integration, broader scale, and real-time utility. I agree, if the returns don’t hold up, a major correction isn’t off the table. It reminds me a bit of the smartphone era, rapid adoption, strong dependence. I guess, the difference is, that phones had clear monetization paths. With AI, we’re still betting. The bigger question for me is, if this all collapses, what happens to the workflows and investments already built around it?
- mholm 1y ago
  There's funky math propping up the training of new AI. That much is certainly true. But just running frontier models on hardware is very doable, and is already capable of absolutely changing the software landscape in the next decade, even if nobody ever trains a model again.

triceratops 1y ago

GenAI may not be done thinking about you though. (Sorry for the low-effort response. But it's kinda true)

energy123 1y ago

> I have woefully little experience with these tools. I’ve tried them out a little bit, and almost every single time the result has been a disaster that has not made me curious to push further. Yet, I keep hearing from all over the industry that I should. Well this is the issue. The author assumed that these tools require no skill to use properly.

CuriouslyC 1y ago

"Despite this plethora of negative experiences, executives are aggressively mandating the use of AI6. It looks like without such mandates, most people will not bother to use such tools, so the executives will need muscular policies to enforce its use. Being forced to sit and argue with a robot while it struggles and fails to produce a working output, while you have to rewrite the code at the end anyway, is incredibly demoralizing. This is the kind of activity that activates every single major cause of burnout at once. But, at least in that scenario, the thing ultimately doesn’t work, so there’s a hope that after a very stressful six month pilot program, you can go to management with a pile of meticulously collected evidence, and shut the whole thing down." The counterpoint to this is that _SOME_ people are able to achieve force multiplication (even at the highest levels of skill, it's not just a juniors-only phenomenon), and _THAT_ is what is driving management adoption mandates. They see that 2-4x increases in productivity are possible under the correct circumstances, and they're basically passing down mandates for the rank and file to get with the program and figure out how to reproduce those results, or find another job.

seadan83 1y ago
Burning the furniture in your home is a force multiplier for your homes heating system. Get with the program, or find a new home.
- CuriouslyC 1y ago
  If your argument is that agentic coding workflows will retard the skill development of junior and midrange engineers I'm not entirely in disagreement with you, that's a problem that has to be solved in relation to how we use gen AI across every domain.
  - seadan83 1y ago
    Skill retardation is actually beyond the point. I'm largely raising a counter example for why the following (rough paraphrasing) is not sound: "SOME people figured out how to use these tools to go 2x to 4x faster, you do the same, or you're fired!". Let's say "n" is the sum complexity of a system. While some developers can take an approach that yields a development output of: (1.5 * log n), the AI tools might have a development output of: (4 * log n)^4/n. That is, initially more & faster, but eventually a lot less and slower. The parable of the soviet beef farmer comes to mind: In this parable, the USSR mandated its beef farmers increase beef output YoY by 20%, every year. The first year, the heroic farmer improved the health of their livestock, bought a few extra cows and hit their target. The next year, to achieve 20% YoY, the farmer cuts every corner and maximizes every efficiency, they even exchange all their possessions to buy some black market cows. The third year, the farmer can't make the 20% increase, they slaughter almost all of their herd. The fourth year, the farmer has essentially no herd, they can't come close to their last years output - let alone increase it. So far short of quota, the heroic beef farmer then shot himself. (side-note: Which is also analagous to people not raising their skill levels too, but not my main point - I'm more thinking about how development slows down relative to the complexity and size of a software system. The 'not-increasing skills' angle is arguably there too. The main point is short term trade-offs to achieve goals rather than selecting long term and sustainable targets, and the relationship of those decisions to a blind demand to increase output) So, instead of working on the insulation of the home, instead of upgrading the heating system, to heat the home faster we burn the furniture. It works.. to a point. Like, what happens when you run out of furniture, or the house catches fire? Seemingly that will be a problem for Q2 of next year, for now, we are moving faster!! I think this ties into the programming industry quite heavily from the perspective where managers often want things to work just long enough for them to be promoted. Doesn't have to work well for years, doesn't have to have the support tools needed for that, nope - just long enough that they can get the quarterly reward and then move on to not worry about the support mess left behind. To boot too, the feedback cycle for whether something was a good idea in software or not is slow, oftentimes years. AI tools have not been out for a long time, just a couple years themselves, it'll be another few before we see what happens when a system is grown to 5M lines through mostly AI tooling and the codebase itself is 10 years old - will that system be too brittle to update? FWIW, I'm of the point of view that quality, time and cost are not an iron triangle - it is not a choose two situation. Instead, quality is a requirement for low cost and low time. You cannot move quickly when quality is low (from my experience, the slowdown of low quality can manifest quickly too - on the order of hours. A shortcut taken now can reduce velocity just even later that same day). Thus, mandates from management to move 2x to 4x faster, when it's not clear that AI tools actually deliver 2x to 4x benefits over the longer term (perhaps not even in the shorter term), feels a lot like the soviet beef farmer parable, or burning furniture to stay warm.

Noumenon72 1y ago

> I found the experience of reading this commit history and imagining myself using such a tool — without exaggeration — nauseating. The commit history[1] looks like a totally normal commit history on page 1, but I clicked farther down and found commits like > Ask Claude to fix bug with token exchange callback when reusing refresh token. > As explained in the readme, when we rotate the refresh token, we still allow the previous refresh token to be used again, in case the new token is lost due to an error. However, in the previous commit, we hadn't properly updated the wrapped key to handle this case. > Claude also found that the behavior when `grantProps` was returend without `tokenProps` was confusing -- you'd expect the access token to have the new props, not the old. I agreed and had Claude update it. > Claude Code transcript: https://claude-workerd-transcript.pages.dev/oauth-provider-token-exchange-callback2 https://claude-workerd-transcript.pages.dev/oauth-provider-t... It seems no different than "run prettier on the code" or "re-run code gen". I don't think it's fair to pick on this specific repo if your objection is just "I don't like reviewing code written by a tool that can't improve, regardless of what the commits look like". I'd call that "misanthropomorphizing". 1. https://github.com/cloudflare/workers-oauth-provider/commits/main/ https://github.com/cloudflare/workers-oauth-provider/commits...

lsy 1y ago

Part of the reason the debate over usefulness is so exhausting is because there is an element of randomness, anecdote, and special pleading built into the use of the tool. One person has a good experience in their context, one person has a bad experience in their context. One person sees a bad result and throws up their hands and says it sucks, another tweaks their prompt until they get a good result and says it works. In addition there's an element of personality: One person might think it's more meaningful to compose a party invitation to friends personally, another asks ChatGPT to write it to make sure they don't miss anything. As someone who is on the skeptic side, I of course don't appreciate the "holding it wrong" discourse, which I think is rude and dismissive of people and their right to choose how they do things. But at the end of the day it's just a counterweight to the corresponding discourse on the skeptic side which is essentially "gross, you used AI". What's worse though is that institutions are forcing us to use this type of tool without consent, either through mandates or through integrations without opt-outs. To me it doesn't meet the bar for a revolutionary technology if it has to be propagated forcibly or with social scolding.

wagwang 1y ago

Its hard to tell if genai is just hopelessly complicated or if there are people with really accurate takes that are drowned out by the noise.

AIorNot 1y ago

1. the hype is really insane so I get the fatigue, but the tech is insane as well, and the promise for the (long term) future is insane.. 2. People mostly hate new things (human nature) 3. its fun watching nerds complain about AI after they spent the past 50 years automating other peoples careers.. turnabout is fair play? All the nerdsniping around GenAI and tech taking away from programming is funny. 4. Remember its a tool, not a replacement (yet), get used to it.. we're not supposed to be luddites. use it and optimize - its our job to convince managers and executives that we are masters over the tool and not vice versa

mjburgess 1y ago

I can at least say why offering a "theory of AI" in the dective sense is very difficult: AI development is an anti-inductive process. Anti-inductive processes are those, like e.g., fraud, which change because you have measured them. Once a certain sort of fraud is made illegal and difficult, most move on to a different kind. AI models are at-base memorisers, with a little structured generalisation around the region of memory. They are not structure learners who happen to memorize. This makes creating a theory of an AI model kinda impossible because the "representations" they learn are so-called "entangled", another way of saying: garbage. The AI model's representation of language does not match language structure, rather each representation is a mixture of what we can evidently see as several competing structural features. But the model has no access to this structure, because its literally not a property of the data but of the data generating process. Now this seems like "a good theory" of AI in the monkish sense, but a problem arises: for each fragile boundary that benchmarking and testing shows, the AI companies collect data on these failures and retrain the model. This is, in a sense, anti-inductive fraud: companies seek to find out how they are found out, and extend the model to cover those regions. But models never actually gain the structural capabilities their behaviour is taken to imply. So, of course, this would drive any theorist insane. By the time you've figured out gpt3.5's fakery, all the openai chatbot data theyve collected has made gpt4 -- and all paper has been covered over all the edges. Until your fingers are cut next time on the now dark fragile boundary of openai's deception. I can give very good reasons, independent of any particular AI model why: the manifold hypothesis is wrong, AI models are memorizers + structure, why they dont find latent structure but combine statistical heurstics, why their representations are not "entangled" such that they can be "disentangled" but necessarily model-unknowable blends of expert-known scientific joints -- and so on. But none of this knowledge can become useful in the face of a billion-dollar training company competing against me, anti-inductively, to apply any of this to any given model. Perhaps then, I suppose, we can look at the behaviour these systems induce in their users and its downstream effects. This is what OP here does, throws hands up at a theory and says only: however this is working, it cannot be good. This is, of course, what we do with fraud, cults, and many other systems of deception. We say: we're not going to argue with the conspiracy theoriest, they are specialists at coming up with ever more elbobate adaptions to the self-deception. Instead, we observe by proxy, that they behave pathological;y -- that they are broken in otehr ways. I think that's a fair approach, and at least one that allows the author to proceed without a theory.

grafelic 1y ago

For me the problem with GenAI is that it erodes away what I enjoy in my job and in my free time. I enjoy learning and eventually understanding methods to solve problems. I enjoy applying these methods in my mind, before trying them on the computer. I am not interested in getting the solutions to problems served to me without understanding why it works. I feel, yes it is an emotion, that GenAI is unhealthy for the progress of mankind, since it encourages a lazy mind, stifles curiosity and will lead to inbred information filling our digital spaces. I am looking forward to my future career as a gardener (although with a smidge of sadness) when AI has sucked out all creativity, ingeniuity and enjoyment from my field of work.

kentonv 1y ago

The author goes into great detail about how he looked at my commit log[0] where I used AI, and he found it "nauseating" and concluded he'd never want to work that way. I'm certainly not going to tell anyone that they're wrong if they try AI and don't like it! But this guy... did not try it? He looked at a commit log, tried to imagine what my experience was like, and then decided he didn't like that? And then he wrote about it? Folks, it's really not that hard to actually try it. There is no learning curve. You just run the terminal app in your repo and you ask it to do things. Please, I beg you, before you go write walls of text about how much you hate the thing, actually try it, so that you actually have some idea what you're talking about. Six months ago, I myself imagined that I would hate AI-assisted coding! Then I tried it. I found out a lot of things that surprised me, and it turns out I don't hate it as much as I thought. [0] https://github.com/cloudflare/workers-oauth-provider/commits/main/?after=a6e3e06c2642e0fd4c185374753201cffc21ce8a+104 https://github.com/cloudflare/workers-oauth-provider/commits... (link to oldest commits so you can browse in order; newer commits are not as interesting)

mwcampbell 1y ago
He explained why he hasn't tried it in the section "The Challenge". Also see footnote 21, which says that what he called his "quirky constraints" in the main text are really bedrock ethical concerns. I respect him a great deal for not compromising on these. Edit to add: Full disclosure: I've been a fan of his writing and work for a long time anyway, and am a supporter on Patreon.
- kentonv 1y ago
  Yes, the ethical debate is fair and worth having, and I don't think someone necessarily has to try it to be able to debate the ethics. I obviously don't quite agree with his opinions here but I respect them. (Though I would say, before I really tried it, I had believed that AI does a lot more "plagiarism" -- lifting of entire passages or snippets. Actually using it has made clearer to me that it really is more like someone who has learned by reading, and then applies those learnings, at least most of the time.) I just don't like it when people argue that it doesn't work or doesn't do what they want without having actually tried it... There's too much echo chamber of people who don't want AI to work, and so convince each other that it doesn't, and repeat each other's arguments, all without ever taking a step outside their comfort zones. (Again... this was me, 6 months ago.)

malwrar 1y ago

I’m near term AI-skeptical, but I think most of the capability & cost-related problems will be solved by improvements to current model architectures. My personal canary for the start of this will be when we start seeing successful architectures beyond (roughly) “tokens -> transformer blocks -> linear transformation to probability distribution”. I wanted to engage with a few of the author’s specific points of skepticism though (hope you don’t mind, if you’re reading this) since I’m not sure if I agree with them. > Energy Usage This has always seemed like a fake problem to me, and mostly just a way to rally environmentalists around protesting genai (and bitcoin and…). The costs (time & capital) associated with nuclear power are mostly regulatory; in the past decade China has routinely built numerous modern reactors while the west has been actively scaling nuclear power down. I have yet to see a good case for not simply scaling up power generation to meet demand. This is assuming that AI power demands remain constant and even require gigawatt-scale power sources to train & operate. For all we know, one paper from one researcher anywhere in the world could show us that current model architectures are orders of magnitude inefficient and make current-standard frontier model training doable in a basement. > The Educational Impact A few years ago I came across some cool computer vision videos on youtube and became inspired to try reading the papers and understanding more. The papers were surprisingly short (only a few dozen pages usually), but contained jargon and special symbols that I found impossible to google. I eventually read more papers and even some books, and eventually discovered that if I had simply read e.g. Hartley & Zisserman’s Multiple View Geometry (commonly just referenced as “MVG”, another mystery I had to figure out) I would have had the hidden knowledge necessary to understand much of the math. If I had simply read earlier papers, I’d be familiar with the fact that much of the content is just slight additions on previous works rather than fully greenfield ideas. I’m not rich enough or successful enough to access the formal education that would have probably provided me the resources to learn this. Youtube videos from practitioners in the field felt more like insecure nerds gatekeeping their field with obtuse lessons. It took me about a year to produce anything useful without just using opencv. When I tried asking the same questions I had as a beginner, chatgpt was able to answer everything I was confused on, answer my dumb followup questions, and even produce demo code so I could explore the concepts. That experience has made it an irreplaceable educational tool for me. Im sure more people will cheat, but college is already an overvalued competence signal anyways. > The Invasion of Privacy Temporary problem, someone will produce a usable hardware equivalent that makes it similarly easy to use genai w/o the privacy implications. This is a segment begging for someone to provide a solution, I think most people with the resources currently to solve it just want to be the AI gatekeeer and thus the service model for this tech. > The Stealing We won’t find common ground, I dont believe in ownership of information and think copyright is largely immoral. > The Fatigue This section seems to largely revolve around no clear obvious tooling route to simply using the tech (complete empathy there, Im currently building my own tooling around it), along with not having a clear mental model of how these models work. I had the same anxiety when chatgpt first blew my mind and just decided to learn how they work. They’re dirt fucking simple, shitty non-learned computer vision systems are orders of magnitude more complex. You seem like a cool person, hmu my profile name @ gmail if you want to talk about it! I like 3b1b’s videos if you want a shorter version, karpathy is overhyped imo but also cool if you want to build a toy model line by line,