6 ms·
I feel like we get one of these articles that addresses valid AI criticisms with poor arguments every week and at this point I’m ready to write a boilerplate re
by ofjcihen 1y ago
I feel like we get one of these articles that addresses valid AI criticisms with poor arguments every week and at this point I’m ready to write a boilerplate response because I already know what they’re going to say.
Interns don’t cost 20 bucks a month but training users in the specifics of your org is important.
Knowing what is important or pointless comes with understanding the skill set.
- mrkurt 1y agoPLEASE write your response. We'll publish it on the Fly.io blog. Unedited. If you want.
- kubb 1y agoMaybe make a video of how you're vibecoding a valuable project in an existing codebase, and how agents are saving you time by running your tools in a loop.
- metaltyphoon 1y agoSeriously… thats the one thing I never see being posted? Is it because Agent mode will take 30-40 minutes to just bookstrap a project and create some file?
- csallen 1y agoIt takes like 2-6 minutes to do that, depending on the scope of the project
- sensanaty 1y agoWell no, the reality of this workflow is the farcica, abject failures unleashed on the Dotnet codebase a week ago. https://news.ycombinator.com/item?id=44050152 https://news.ycombinator.com/item?id=44050152
- andrepd 1y agoSo they can cherry pick the 1 out of 10 times that it actually performs in an impressive manner? That's the essence of most AI demos/"benchmarks" I've seen. Testing for myself has always yielded unimpressive results. Maybe I'm just unlucky?
- kubb 1y agoLivestream would be fair.
- ofjcihen 1y agoI’m uninterested in giving you content. In particular because of your past behavior. Thanks for the offer though.
- tptacek 1y agoKurt, how dare you.
- ofjcihen 1y agoYou wouldn’t happen to work for fly.io as well, would you? Edit: Nm, thought I remembered your UN and see on your profile that you do.
- grzm 1y agoYes. And the author of the submission.
- Mofpofjis 1y agoOMG I didn't notice. Way to burn down a huge amount of respect stemming from past cryptography work in just one blog post.
- dogecoinbase 1y agoit's irksome to see tptacek having his patio11 moment for sure.
- tptacek 1y agoWhat does this even mean? (I should be so lucky.)
- throwaway314155 1y ago> past behavior Do go on.
- csallen 1y agoCan you direct me somewhere with superior counterarguments? I'm quite curious
- briandrupieski 1y ago> with poor arguments every week This roughly matches my experience too, but I don't think it applies to this one. It has a few novel things that were new ideas to me and I'm glad I read it. > I’m ready to write a boilerplate response because I already know what they’re going to say If you have one that addresses what this one talks about I'd be interested in reading it.
- slg 1y ago>> with poor arguments every week >This roughly matches my experience too, but I don't think it applies to this one. I'm not so sure. The argument that any good programming language would inherently eliminate the concern for hallucinations seems like a pretty weak argument to me.
- ofjcihen 1y agoIt’s a confusing one for sure. To be honest I’m not sure where the logic for that claim comes from. Maybe an abundance of documentation is the assumption? Either way, being dismissive of one of LLMs major flaws and blaming it on the language doesn’t seem like the way to make that argument.
- simonw 1y agoWhy does that seem weak to you? It seems obviously true to me: code hallucinations are where the LLM outputs code with incorrect details - syntax errors, incorrect class methods, invalid imports etc. If you have a strong linter in a loop those mistakes can be automatically detected and passed back into the LLM to get fixed. Surely that's a solution to hallucinations? It won't catch other types of logic error, but I would classify those as bugs, not hallucinations.
- ofjcihen 1y agoA good example of where a linter wouldn’t work is when the LLM has you import a package that doesn’t exist.
- kubb 1y agoThere's also the reverse genre: valid criticism of absolutely strawman arguments that nobody makes.
- tptacek 1y agoWhich of the arguments in this post hasn't occurred on HN in the past month or so?
- sethev 1y agoOr in this very thread, for that matter.
- kubb 1y ago"I tried copilot 2 years ago and I didn’t like it." Great article BTW, it’s amazing that you’re now blaming developers smarter than you for lack of LLM adoption, as if it weren’t enough for the technology to be useful to become widespread. Try to deal with „an agent takes 3 minutes to make a small transformation to my codebase and it takes me another 5 to figure out why it changed what it did only to realize that it was the wrong approach and redo it by hand, which took another 7 minutes” in your next one.
- mountainriver 1y agoI feel the opposite, and pretty much every metric we have shows basically linear improvement of these models over time. The criticisms I hear are almost always gotchas, and when confronted with the benchmarks they either don’t actually know how they are built or don’t want to contribute to them. They just want to complain or seem like a contrarian from what I can tell. Are LLMs perfect? Absolutely not. Do we have metrics to tell us how good they are? Yes I’ve found very few critics that actually understand ML on a deep level. For instance Gary Marcus didn’t know what a test train split was. Unfortunately, rage bait like this makes money
- attemptone 1y ago>I feel the opposite, and pretty much every metric we have shows basically linear improvement of these models over time. Wait, what kind of metric are you talking about? When I did my masters in 2023 SOTA models where trying to push the boundaries by minuscule amounts. And sometimes blatantly changing the way they measure "success" to beat the previous SOTA
- mountainriver 1y agoAlmost every single major benchmark, and yes progress is incremental but it adds up, this has always been the case
- attemptone 1y agoWe were talking about linear improvements and I have yet to see it
- mountainriver 1y agocheck the benchmarks or make one of your own
- attemptone 1y agoI checked the BlEU-Score and Perplexity of popular models and both have stagnated around 2021. As a disclaimer this was a cursory check and I didn't dive into the details of how individuals scores were evaluated.
- calf 1y agoWhat valid AI criticisms? Most criticisms of AI are not very deep nor founded in complexity theoretic arguments, whereas Yann LeCun himself gave an excellent 1 slide explanation of the limits of LLMs. Most AI criticisms are low quality arguments.
- therealpygon 1y ago“Valid” criticism rarely come from the people barely capable of understanding the difference between AI and LLMs, and using them interchangeably.