20 ms·

AI isn’t good enough

172 points by MaysonL 3y ago

ericksoa 3y ago
When I see VCs make this argument, all I hear is "oh shit, we're late to the party" and doing something like this as a desperate attempt to bring valuations down.
ryanklee 3y ago
"We are quickly reaching the limits of current AI" So many words in this piece, but nothing to back up this primary assertion.
- deleted 3y ago
  [deleted]
- system2 3y ago
  For what it's worth, I can actually tell the hype around AI died down a little. 8-10 months ago you'd hear AI from grandmas and their interest in playing with gpt. That's gone. From programmer's perspective, it is still useless. I can't still get chatgpt to give me a useful ffmpeg or a basic php database script. I don't know who is really using ai for what, other than crappy chatbots. They are saying JSON works perfectly with chatgpt (gpt4) and still out of 10 queries I get 1 or 2 messed up. I can't depend on it yet. Image generation is still terrible and useless, can't use for programming, text-to-speech neural still sucks big time, not sure who is doing what with it. Almost feels like crypto-hype.
  - dontupvoteme 3y ago
    Really? I have a lot of success with bash and ffmpeg scripts. I don't even try to write them myself these days. I don't do JSON stuff much but either function calling or grammar based sampling should be able to force correct formatting if the issue is syntax. ImageGen seemed like a solved problem to me with stable diffusion about 6 months ago or so, between LoRAs and the tons of models availble on sites like civitas.. SOTA Text to speech/voice cloning is scary good, but it's only available via SaaS like 11labs (perhaps just them). One interesting weekend they allowed anyone to clone a voice (required only maybe 10-60s of audio) to say anything I made Bob Ross read Neuromancer using 60 seconds of fuzzy audio from youtube, but 4chan made something like 20k fake audio recordings on vocaroo of everything horrible and hilarious you can think of. Then they shut that off for good reason on Monday. There are some cool applications there. I watched an hour video about the talkshow wars between Leno and Conan and i couldn't figure out what the voice was but it sounded like a smart old guy, apparently it was a weird clone of Biden. Which means anyone can now make YouTube content with any kind of voice. Which is bad for impersonation, but good if, say, you don't like the sound of your voice and want to go for a different one (maybe an old timey radio broadcaster) What definitely has become shittier is the quality of gpt4. But that's not an AI issue, that's OpenAI being OpenAI.
    - ozgung 3y ago
      Same. GPT-4 completely replaced my googling/stackoverflow routine. I generates a customized solution for my specific needs with step by step instructions. I think it works in a smarter (but slower) mode if the question is non-trivial. With pdf plugins it helps me reading and understanding recent papers. It generates an excellent summary and answers all my questions regarding any advanced concepts in the paper. I believe it's already smarter than me. I don't know if it can replace me in the mid-term but it definitely increases my productivity.
  - SanderNL 3y ago
    > Image generation is still terrible and useless Image generation is thriving and now an indispensible part of the creative process. The tools are fresh and still evolving, but the impact even in this early stage is ridiculous. You can easily take rough sketches and have it spit out fully fleshed out colored prototypes. Depending on your perspective it's either an insane productivity-multiplier or scary as f-ck or both. Look at https://civitai.com https://civitai.com for a taste. Look around. You really believe this is terrible and useless? Most of these images came straight from SD with some minor tweaking, usually upscaling. I think it's quite something. It's not just MidJourney, it's the entire ecosystem surrounding models like SD, ControlNet, etc. Some artists are swinging their fists, others shrug, but all are affected by it whether they like it or not and the damn thing is just getting started. SD came out in 2022. As a young aspiring artist I can imagine it's hard to not feel like this whole thing is a massive punch in the gut. Like an experienced and highly-skilled artist, it's easy to shrug this off when you're an accomplished programmer. It can't replace me, right? Well, true. It's a gradual thing, but the impact it has on junior programmers is undeniable. What it means for our branch if leagues of young people lose interest in the meticulous acquiring of technical and often arcane knowledge through "manual programming" is anybody's guess. Maybe it's a good thing, maybe it's not. > Text-to-speech neural still sucks big time This is not my experience at all. I am actually practicing French with it and it can pick up my weird accent. It's ridiculous how good it has become in such a short time. EDIT: Oh, sorry, I read it wrong. But in that case, the same thing applies. 11labs is insane and getting better. It fools many, many people on Youtube. The Bark project is also scary at times. This is just getting started. > I can't still get chatgpt to give me a useful ffmpeg or a basic php database script. I believe these issues are at least for a significant fraction caused by the severely castrated and surprisingly lacking UI efforts from OpenAI. Just free-form "talking" with an LLM might be a good way to get grandma to use it, but as a professional programmer (or professional anything, really) it's bordering on stupid. The prompts and settings you use determine 90% of your success. OpenAI's interface gives you nothing. They just recently introduced something like "profiles", but it's amateurish at best. It makes me feel like programming without IDE (and Vi/Emacs).
  - msp26 3y ago
    Making good tools with language models takes time. There are nuances that aren't immediately visible without working on them. Unless there's a mismatch in the schema/messages, function calling is very reliable at providing JSON.
- dontupvoteme 3y ago
  This, how absurd. Do they think the future is always going to be in "just add more parameters"? It's obviously going to be smaller, efficient and such. Even GPT4 itself is supposedly along these lines with MoE
- _xivi 3y ago
  > nothing to back up this primary assertion. Well, he's making a prediction, a speculation, a claim. If it was as simple as `x->y` it would no longer be valuable.
  - Philpax 3y ago
    Sure, but even then you're expected to back up your speculation. There doesn't seem to be anything within the article to support the claim that we're at the tail end of development; just assumptions that today's limitations will apply tomorrow.
petermcneeley 3y ago
"too few people for all the jobs, for the most part" - Are wages going up? "But almost everywhere else needs people. Badly. Across retail, restaurants, manufacturing, trades, and on and on, companies are struggling to hire. " - Wow sounds like an exciting star trek future that I cant wait to be part of! "labor became more expensive than capital" - So wages must be going up right? "In essence, the authors show that for automation to have widespread benefits it must deliver high productivity gains to more than compensate for the high displacement of workers." - the reason why this sentence makes no sense is because it is written in two frames at the same time. One of a planned economy and the other of capitalism. "We are in a middle zone, however, with AI able to displace huge numbers of workers quickly, but not provide compensatory and broader productivity benefits." - But what about the horses? I tried my best to find the central thesis of this rambling multi-tone work. I think it is difficult because it was written with too many different frames.
- PaulDavisThe1st 3y ago
  Capital's willingness to ride out labor shortages without increasing wages seems to have become substantially greater in the last couple of decades. TIRED: labor shortage -> wages rise! WIRED: labor shortage -> y'all work harder for maybe a bit of overtime or maybe not, and we'll just absorb the complaints about customer service and product quality. Those damn employees aren't getting a penny more!
  - wilg 3y ago
    Wages are rising: https://www.axios.com/2023/07/12/real-wage-gains-inflation https://www.axios.com/2023/07/12/real-wage-gains-inflation
    - tuatoru 3y ago
      Just catching up to years of undershooting inflation. In real terms still worse than 1969.
      - 3y ago
YeBanKo 3y ago
With few prompts I just wrote a small script with a bunch of openssl commands. Saved me an hour. And it works. And it even explained it. There is AGI and then there is “practically agi”. The latter has been achieved with LLMs.
jongjong 3y ago
As impressive as AI is and what it can do. I've felt the same way in my last job. It was almost there; it was almost good enough such that we could almost ship the project but we didn't. GPT3.5 was not good enough, GPT4 seemed like it was good enough in terms of accuracy but it was too slow and we were getting throttled like crazy. I lost my job over it. When I was worried about AI taking my job, I didn't picture it like this.
- jongjong 3y ago
  Our use case was that we needed to consider information from thousands of pages of documents to answer customer questions. I let GPT decide where to look based on content tables of the documents (thankfully each section was not more than ~500 words) and only fetched information from the relevant sections to let GPT answer the user's question. GPT3.5 was fast enough to be usable but often inaccurate; it would ignore some information or get confused about simple mathematical comparisons in the user's questions like 'greater than versus less than'. GPT4 was good and potentially fast enough but the API throttling was too restrictive and it was not possible to feed it less information and still get a good, nuanced answer every time. Actually the trick to get GPT to find the information based on table of contents worked really well in narrowing down the information for the second-stage summary (to answer the user's question) but our use case nevertheless required a lot of information (often from multiple sections) with a lot of grey areas and nuance and the limited amount of throughout provided by GPT4 didn't cut it. It was frustrating that, aside from accuracy issues (which depended on the complexity of the question), it often seemed to work very well and fast with GPT3.5 but the exact same code did not meet usability expectations when we switched over to GPT4 due to slow speed (though the accuracy was awesome). I only spent 1 month and a half on this project but management didn't have the patience to wait for Open AI to remove GPT4 throttling and because they couldn't see it working (at the speed they wanted), they had no reason to believe me that my solution was designed correctly... That's startups I guess. Anyway I still think there is a lot of hope for this technology and I'm confident that the problem can and will be solved. It was just unfortunate timing for me. Being an early adopter isn't always a good thing.
  - ta988 3y ago
borissk 3y ago
Very interesting article. IMHO a key factor in how the world changes in the next 10-20 years will be how far the current wave of AI, based on neural networks can go. If scientists and engineers manage to implement autonomous cars and robots that can do basic human tasks like clean, wait and take care of the sick and elderly we will wake up in a brand new world. It will put a lot of stress on the basics of society, such as democracy and having to work for a salary.
- tuatoru 3y ago
  The real world has many trillions of parameters. Also, doing stuff in the real world is all about sensors. Progress on integrating millions of sensors into a robot has been slow.
  - borissk 3y ago
    Don't think a robot needs millions of sensors. 90% of the information that a human uses comes through the eyes - so just 2 sensors. Really in terms of doing most types of work all a robot would need is vision and hearing.
    - klipt 3y ago
      Q: But if the robot has no nose, how will it smell? A: Awful!
    - csomar 3y ago
      Humans have an insane number of sensors all working with each other. The eyes are not just 2 sensors and a blind person can still function fairly well though with limitations.
  - dontupvoteme 3y ago
    >The real world has many trillions of parameters. But at any given time you're probably only going to require some millions or double digit billions, unless you're a nation state or something like that.
  - 3y ago
michaelmrose 3y ago
> But almost everywhere else needs people. Badly. Across retail, restaurants, manufacturing, trades, and on and on, companies are struggling to hire. And this brings us back to the cross-country trip observation and that “signing bonus” sign in a McDonald’s window. For jobs that pay just enough to not qualify for food stamps or Medicaid but not enough to buy food medicine and rent. Previously it was more possible to earn little enough to qualify for benefits and live if not well but the numbers have become misaligned and don't account for differences between expensive and cheaper parts of states. Given present concerns the situation doesn't change appreciably unless you go all in on automation implement UBI or both.
ambrozk 3y ago
This entire piece is based on one massive, unsupported assertion, which is that LLM progress will cease. Or, as the author puts it, "we are at the tail end of the first wave of large language model-based AI... [it] ends somewhere in the next year or two with the kinds of limits people are running up against." I want to know only one thing, which is what gives him the confidence necessary to say that. If that one statement is untrue, the thesis of the piece completely fails, and I do not know how any one person alive today can be certain that that statement is true. Has Paul Kedrovsky or Eric Norlin spent $150M training a 2T parameter model that no one's heard about? Do they have access to classified OpenAI docs which state that GPT5 exists already, and it's no better than GPT4? Without this sort of information, "LLMs are not going to get smart enough for widespread practical use" is an unsubstantiated bet, and nothing more.
- user3939382 3y ago
  I asked ChatGPT how to add a JSDoc type to a Vue 2 prop and it gave me a wrong answer. There have been several times where I’ve asked it questions and it sprinkles in well disguised misinformation. These tools are impressive but they definitely have limitations.
  - golol 3y ago
    ChatGPT (3.5) is not state of the art
  - seanthemon 3y ago
    Curious, how does it do when you give it examples / use playground.openai? I find that no examples often leads to the same result as a befuddled junior, but with examples often it gains confidence. Also I find playground to give me much better code snippets than chat sometimes
    - cbozeman 3y ago
      You've identified the problem inherent in these models right now, though. There's a lot of "extraneous" information and details that appear useless and unrelated, but that long-time developers have tucked away in their brains (or really anyone who has done something at a high level for a long time), that turns out to be incredibly useful; generally these people don't require examples - they just know what the right answer is, because they've been exposed to a variety of problems over a long career or lifespan. That's where LLMs need to be to be truly "useful". I think if we get them to that point, we'll really have something useful on our hands.
paulpauper 3y ago
The absence of human workers has become a limiting factor on economic growth in the U.S. the problem is real GDP growth is flat or negative due to high inflation. having more labor would make this worse, if anything. job creation tends to be inflationary. from 2010-2021 saw strong economic growth and low inflation, but now things suddenly changed due to too much inflation, not a lack of growth.
- WillPostForFood 3y ago
  The jobs are already created, unfilled. The lack of workers causes wages to go up which adds to inflation (e.g., the $1000 McDonalds bonus). But if you add more workers, you increase productivity, and reduce wage inflation.
atleastoptimal 3y ago
"we are already at the tail end of the current wave of AI." Is that really true? Generative AI spending this year just so far is 4x greater than last year, and the results of that spending haven't been released yet. How can they conclude this? Most "predictions" about the future I've seen assume that ChatGPT will stay the same and not improve for the next 10 years, or only marginally. Why they make this assumption, I don't know, it seems to go against every past trend we've seen with machine learning scaling. There are even people who say LLMs have reached their limit because we've exhausted all the text data that exists, assuming that the only way for LLMs to improve is to blindly feed it more static data and that no other sources of improvements exist.
- sharts 3y ago
  I think they mean tail end of LLM innovation? As in, all the discovery has been done and so all that is left is just computing power and tweaks rather than breakthroughs. I believe the OpenAI folks have already said something to that effect a while ago as well. For end users, the results of tweaks/computational power will seem like "breakthrough" to them but it's fundamentally not.
- nottorp 3y ago
  > Generative AI spending this year just so far is 4x greater than last year I'm sure they spent a lot of money looking for gold in the long rushes long before the gold was all gone too.
- albertzeyer 3y ago
  Also, if we add videos, images, audio, whatever other data we have, there is still quite some orders of magnitude more data available.
Madmallard 3y ago
GPT4 has been beyond our expectations by a mile. What in the world is this person even on?
nmca 3y ago
This is nuanced and interesting, despite most likely being completely wrong - as others have mentioned, it's predicted on LLMs being at the top of the sigmoid and they just ain't. Also, this goal-post-moving did crack me up rather: > Sure, it is terrific at using its statistical models to come up with textual passages that read better than the average human’s writing, but that’s not a particularly high hurdle.
dontupvoteme 3y ago
I am completely bewildered by the responses here. GPT-3 is so good it could write code, or it could emulate an entire TV show to the point of lawsuit, or simulate an eternal debate between Herzog and Zizek. You could translate Bulgarian to Sindarin for heaven's sake. And all of this comes at a 50% tax rate or so because of the alignment tax that OAI opted for. (which doesn't even work because anyone who puts effort in can jailbreak it scarily well) The real solution is not these monstrosities of the internet mushed together and taxed like a Belgian billionaire. It's models and frameworks specific to what you want in a given query. It was less than 24 hours ago that we finally got a local model that can do code generation well, and we also know that Meta has a far better one that they are holding back. There is no reason that for any query you should be restricted to a single inference run on a single fixed model, and there is no reason that we shouldn't perform a bunch of non-LLM processing on the output before the user sees it. Switch between multiple models, fine-tune, or use LoRAs based on the query -- Python plots? Load llama-coding-python-plots. You like Plotly? Add -Plotly. Run the code in a sandbox (with hard kills on resources) and regenerate it if it does not meet standards. You're working in linguistics? Switch to a chat model that's fine-tuned in the literature and codebase of that field. There are a trillion ways to improve things; we're basically at the cavemen-banging-rocks-together point. Hell, I'm a sleep deprived ESL and i just invoked the monstrosity to fix my grammar and spelling for this post. What a bloody waste of CO2. If for nothing but to keep the Earth afloat we shouldn't be using these huge closed source SaaS things unless we need them. Repurpose them for SETI@Home or processing human genetics for healthcare, or something else that benefits mankind
seydor 3y ago
Just hook the LLMs to a robot arm and watch another, much larger wave of AI innovation
arisAlexis 3y ago
>We are bumping against many of its >limits He keeps saying that as a fact with absolutely no proof and based his whole argument on it.
- tacticalDonut 3y ago
  Yeah, it's frustrating that he doesn't call out the specific limits.
- bamboozled 3y ago
  Conversely, if you don't know what the limits are yourself (and you don't) you wouldn't know either way. So I'm not quite sure if you'e added anything here unfortunately.
  - arisAlexis 3y ago
    I haven't written an article asserting my argument with certainty. He did. Big difference.
    - bamboozled 3y ago
      We’ll you have a opinion either way.
      - arisAlexis 3y ago
        My opinion is that we don't know the limits. This is essentially no opinion because of no data. Stating a limit without data is an opinion thought and a wrong one.
jillesvangurp 3y ago
By the time AIs stop hallucinating, it will be effectively super human (i.e. much better) because humans hallucinate all the time. And some people are barely coherent. We're holding AIs to a much higher standard than ourselves. And we move the goal posts all the time as well. Let's deconstruct that title. "AI isn't good enough". Good enough for what? Great example of moving the goal posts. Because anytime it fails to do whatever, it's not good enough. But what about all the stuff it is good enough for that people are actually using it for already? It's passing tests at a level most humans only manage very briefly after lots of intensive preparation. The knowledge slips away quickly after that. The way I see it, there's a long, rapidly growing list of stuff that AIs have nailed already and a list of things where it is clearly struggling. That list is shrinking. Self driving gets cited a lot as something where AIs is failing. It's hard to say that in a world where multiple companies are now operating self driving vehicles commercially in several cities across the US, China, and soon Europe. Are they perfect? No. Do they drive better than the average jerk on the road? Definitely; not even close. Most people simply suck at driving. That's why traffic deaths are so common. Most of those deaths are caused by somebody with a drivers license proving that they shouldn't have received one. Not quite a solved problem but at this point the question of when these things will be on the road in my city is more a matter of when than if. It already works in other cities. I don't see what's so special in mine that it couldn't be replicated here. In other words, I expect the number of cities with autonomous vehicles to start exploding over the next few years. And who knows, Tesla might hit the good enough mark at some point as well. Again, the barrier is very low because humans aren't held to the same standards. They'll give drivers licenses to just about anyone.
- ChatGTP 3y ago
  "Guy calls humans flawed why claiming the same flawed species will build superhuman intelligence...", pretty good for a bunch of dumb LLMs in "meat bags"...
  - arrow7000 3y ago
    You're pointing out the best that humanity is capable of. He's pointing out that people judge LLMs by their worst performances. You're not disagreeing with the parent comment.
Havoc 3y ago
I don't think the assertion that we've reached the end of current wave of AI is correct. It hasn't even been rolled out to anything yet. Once it is fully integrated into os/browser/corporate workspace and the MBA's have thrown LLMs at every business problem they can think of then perhaps but right now it hasn't even left high tech circles
- layer8 3y ago
  > It hasn't even been rolled out to anything yet. The question is why that is, after almost 9 months of ChatGPT API availability.
fullstackchris 3y ago
AI won't work to reeingineer society, just like the last 100 "latest things" didnt. There is a massive undercurrent that always goes in the opposite direction of technology advancement: tradition! You need new people growing up and living in a new paradigm to get these changes, you cant just ask (or expect) people who have lived their life one way to change completely to some other way of living
manuhortet 3y ago
Great article. Although I am not sure if we really are reaching the value limits of the current LLMs, as the author mentions. Some fields as legal litigation are being disrupted as much as call centers have, but the change is happening in a much slower fashion, for many reasons. The productivity gain in these fields is potentially high, but so is human displacement. Relevant fact to consider
Ekaros 3y ago
How far are we from a point you could leave AI to prompt something itself. That is not just respond to something, but actively seek it. Let's say use this place. Could it without being ordered read articles, comments and add it own comments? Maybe search internet by itself for relevant content and post it? Maybe try to earn maximal amount of upvotes? And all this without there being any bit of code poking it to do it?
- firtoz 3y ago
  I was going to ask "how would it know what's a good post" but it's embarrassingly easy to train the AI on interesting or not content based on upvote information...
  - Hoasi 3y ago
    > but it's embarrassingly easy to train the AI on interesting or not content based on upvote information... And yet, the best topics and answers do not necessarily correlate with the most upvoted.
    - firtoz 3y ago
      How would you qualify the best?
      - Hoasi 3y ago
        We can't all agree on what best means. It can't be objective since we are humans. HN guidelines and moderation are doing a decent job of promoting interestingness in general. I don't think upvotes are a reliable indicator, but maybe they are as close to good enough as a system can be. Depending on timing, you see great topics flying under the radar. Likewise, "poor" posts regularly prove to be quite popular. But then again, this is also subjective.
        firtoz 3y ago
        I guess if we can track moderator actions we can automate them too ;)
countWSS 3y ago
Data-centric token-guessing will be outperformed by more complex architectures, LLMs are too inefficient and their hardware requirements are disproportionate for the utility. Building a sand castle of "LLM apis" that rely on plugging their deficiences is a fundamentally bad decision that hides the underlying waste of resources. What will be great if the input data it was fed was more structured and labeled like images: imaging a "Book, 17th century, reliablity:54%, contains geographic errors". Instead this "attention is all you need" approach treats garbage data equal to top material.
abdussamit 3y ago
I haven't read the article (like many here) and I don't get why the writer would write this. LLMs/AI are not a stage where they can replace people or their jobs (broadly speaking), they still have to be taught and engineered to solve your problems. For example, if I enter a prompt "Write a Javascript function that does XYZ for me.", and don't get a desired result, it's unfair for me to say "ChatGPT bad, AI bad". At this moment, you have to work with the limits, use better prompts, use ChatGPT as a guide and not as your personal robot. With that approach, I feel lots of revolutionary content is incoming in products. Use the tool better to get the most out of it.
asu_thomas 3y ago
> There is a persistent structural imbalance in the U.S. workforce: too few people for all the jobs Lies. If the minimum wage increased proportional to inflation, none of these "jobs" would exist. The McDonalds sign-on bonus is nothing but a loan, just like all sign-on bonuses with a retention requirement. Such arrangements merely require that minimum wage is low enough to partition that compensation out of the base wage, a modern American financial industry to efficiently enforce the contracts, and a society with no remaining semblance of dignity in sight.
w_t_payne 3y ago
We're just not that good at managing AI yet. Current agent-based systems are little more than toys ... but this situation is improving very rapidly.
version_five 3y ago
Labor shortage it BS. It's rhetoric to increase immigration so we can continue sustaining a slave class and don't have to pay people properly. Canada's been very successful at this.
speak_plainly 3y ago
“Historically, when this has happened—labor became more expensive than capital—economies have responded with automation, so we should expect that again today.” Foolish assertion. Immigration has always been the stop-gap solution for expensive labor to the point that it’s built in as an essential feature of liberalism.
philipwhiuk 3y ago
We don't need better LLMs to eliminate all McDonalds workers. In fact there's no language model needed there at all. The job just needs better robotics.
- naasking 3y ago
  I think you're underestimating how LLMs can help here: https://www.deepmind.com/blog/rt-2-new-model-translates-vision-and-language-into-action https://www.deepmind.com/blog/rt-2-new-model-translates-visi...
ironborn123 3y ago
Rather than asserting that current LLMs are at their tail end, or that AI isnt good enough, it is much more instructive to check what are the bottlenecks or constraints to further progress, and what would help remove these bottlenecks. They can largely be divided into 3 buckets 1) Compute constraint - Currently large companies using expensive nvidia chips do most of the heavylifting of training good models. Although chips will improve over time, and competition like Intel/AMD will bring down prices, this is a slow process. But what could be a faster breakthrough is training using distributed computing over millions of consumer GPUs. There are already efforts in that direction (eg. petals/swarm parallelism for finetuning/full training, but the eastern europe/russian guys developing them dont seem to have enough resources). 2) Data constraint - If you just rely on human generated text data, you will soon exhaust this resource (maybe GPT4 has already). But the Tinystories dataset generated from GPT4 shows if we can have SOTA models generate more data (and especially on niche topics that appear less frequently in human generated data), and have deterministic/AI filters to segregate the good and bad quality data thus generated, data quantity would not be an issue any longer. Also, multimodal data is expected (with the right model architectures) to be more efficient at training world grokking SOTA models than single modal data and here we have massive amounts of online video data to tap into. 3) Architectural knowledge constraint - This may be the most difficult of all, figuring out what is the next big scalable architecture after Transformers. Either we keep trying newer ideas (like the stanford hazy research group does), and hope something sticks, or we get SOTA models few years down the line to do this ideation part for us.
tlogan 3y ago
I do not think it is fair to use AI in support call centers as an example. These call centers are not there to help customers (cancel subscription, refund, change something, etc.) and AI is perfectly tuned for that. Or at least they have procedures and AI is perfectly train to follow these. In other words, AI in support centers is designed and works to follow companies procedures and rules—which may not always align with customer expectations.
- manicennui 3y ago
  So is a simple form on a website in most cases. It would also be cheaper for the company and less frustrating for the customer. No one wants this "AI" chatbot bullshit except for companies selling "AI".
  - tlogan 3y ago
    Or that customer wanting to cancel spend hours talking to the bot so they finally give up and hopefully forget. Having a simple form to cancel/refund/change/lost item/sent to wrong address/etc. is easier way to do and handle. But companies do not want that so … chatbot to rescue.
xpe 3y ago
I’m so done with writing that has an implicit but uncommunicated quantitative model. (AI costs are too high? Relative to what? Show us!) Almost all analytical / business / industry writing should quantified. I hope for better. We all should. It is within reach. Yes, I'm grumpy on this topic, but this doesn't make it unwarranted; it is a consequence of wanting better for a decade or longer. At this point, if you claim to be a business analyst and you aren't sharing a simulation, logic, or code, I'm just not going to take you seriously.
wildermuthn 3y ago
The most interesting part of this article isn’t its assertion that AI innovation is about to slow, but rather that current LLMs can only produce “so-so automation” — that although LLMs can replace many jobs, LLMs aren’t good enough to offset the cost of people losing such jobs. True as of today, but the article is overlooking multi-modal LLMs. Once AI can see, we’ll go from “wow, GPT4 is pretty good at helping me to my job” to “uh oh, GPT4-Vision can do my job better than I can”. This is especially true if the model can read text off a screen. At that point, it’s likely to replace many workers who stare at screens for a few hours a day and pretend to work the remainder. The model won’t take coffee breaks, or long lunches, or take the afternoon off to drive their kid to the dentist. If you’re a worker whose work depends upon looking at a screen, and mostly involves moving data from one place to another with light transformations and interpretations, you’re going to be in trouble by the end of the decade.
- tempodox 3y ago
  And since everyone who has been writing has been replaced by machines as well, most of those texts will be machine-generated by then. Machines reading and processing texts written by machines would be the logical consequence in this scenario, but I can't help finding the whole idea deeply absurd.
intrasight 3y ago
> too few people for all the jobs, for the most part. Change that qualifier to "for a minority part" and it would be correct. There are plenty of workers - old and young and everything in between - who would gladly work if a) you allowed them to and/or b) you paid them enough.
Nemi 3y ago
I hope beyond hope that we continue to iterate on the current GPT-4 model. But statements made by Altman himself make me leery that it might not happen. That, coupled with the fact that it is just so expensive to run make me wonder if this whole thing is sustainable. Take this into consideration - NVDA’s massive profit in the last couple of quarters is someone else’s expense. When no one is making money on AI, can this continue? I absolutely think companies can make money using AI that people will pay for, but I am not convinced that companies can make money supplying AI services. I see it as almost a commodity type service, like internet access or something. There is no clear path to profitability, and that is ok to some extent, but with the massive costs this could come crashing to a halt very quickly.
zoogeny 3y ago
When I read articles like this I can't help but question my own intuitions since they are so opposed to this author's thinking. Firstly, his suggestion on the worker shortage and the anecdotal $1000 sign on bonus at some local McDonalds. These seem more tailored to a narrative interpretation of the state of the world that is formed to the author's own bias. I look at my own small town homeless and drug addicted population and consider the changes in the last 5-10 years on that front. I consider the insane rise in the cost of home ownership and the massive rise in rental costs locally. I just don't see $1000 signing bonuses at fast food restaurants as a sign of some latent demand for labor. It feels like the author is using a single anecdote and a narrow reading of a few graphs to support a world view that I don't see existing in my own experience. Then he says: "We are quickly reaching the limits of current AI". You mean the AI that surpassed most informed peoples expectations on AI capability less than one year ago? I feel like I'm in 1996 reading an article about the Internet where someone gives some list of reasons why the Internet is reaching its limits. And just as an aside, this author uses the term "explosive" especially in the context of "explosive growth" over and over. It may be that his target market just isn't me. The narrative and analysis probably hits home for a particular set of capitalist minded investor class individuals. Perhaps they really do see North America as having a huge labor shortage and that we just need the "right" kinds of automation so we don't all feel so bad about the continuing displacement of workers. As long as they keep seeing "explosive growth" in the value of their assets.
jy1 3y ago
People often conflate AI progress with hardware progress. If we had hardware as flexible, robust, and "cheap" as the human body, we'd solve a lot of these problems such as retail, trades etc. The problem isn't software. It's hardware.
rldjbpin 3y ago
disclaimer: i am in the middle of the journey into learning this subject, and have at least done finetuning on existing models. some may still consider my take as misinformed. one thing i am not a big fan of in this discourse is that how the community makes up jargon to describe relatively simple things on one hand (a la "embeddings"), and while come up with misnomers like "hallucinations" to describe otherwise blunt things. it is contributing to miss the forest for the trees in this discussion. while i agree it is quite impressive to see what the state of the art can seemingly achieve on surface, the way we approach designing these models are fundamentally not aligned with what we want to do with its result. while some media for generative machine learning produce relatively harmless results (e.g. images and audio), it is the text where i have a lot of reservations. we train these models to essentially turn noise into something that humans can pass as acceptable (for the given prompt). Sadly, however, people outside (and sadly "inside") this space claim it to be the word of God. i still strongly believe that a hybrid system is going to be the only way we can achieve the most from the current approach. almost a decade ago we were already good at parsing through sentence structure and "understanding" what the user writes. what the output is can easily be done with some hardcoding in logic with some noise thrown in by the models. however, it should only be treated with that level of seriousness. i might be shouting into a void, but it is imperitive that we stop taking what these models output as production-ready code, or as a starting point even! otherwise, the only thing you achieve from this is to train the models running the chat services further.

172 points by MaysonL 3y ago

When I see VCs make this argument, all I hear is "oh shit, we're late to the party" and doing something like this as a desperate attempt to bring valuations down.

ryanklee 3y ago

"We are quickly reaching the limits of current AI" So many words in this piece, but nothing to back up this primary assertion.

deleted 3y ago
[deleted]
system2 3y ago
For what it's worth, I can actually tell the hype around AI died down a little. 8-10 months ago you'd hear AI from grandmas and their interest in playing with gpt. That's gone. From programmer's perspective, it is still useless. I can't still get chatgpt to give me a useful ffmpeg or a basic php database script. I don't know who is really using ai for what, other than crappy chatbots. They are saying JSON works perfectly with chatgpt (gpt4) and still out of 10 queries I get 1 or 2 messed up. I can't depend on it yet. Image generation is still terrible and useless, can't use for programming, text-to-speech neural still sucks big time, not sure who is doing what with it. Almost feels like crypto-hype.
- dontupvoteme 3y ago
  Really? I have a lot of success with bash and ffmpeg scripts. I don't even try to write them myself these days. I don't do JSON stuff much but either function calling or grammar based sampling should be able to force correct formatting if the issue is syntax. ImageGen seemed like a solved problem to me with stable diffusion about 6 months ago or so, between LoRAs and the tons of models availble on sites like civitas.. SOTA Text to speech/voice cloning is scary good, but it's only available via SaaS like 11labs (perhaps just them). One interesting weekend they allowed anyone to clone a voice (required only maybe 10-60s of audio) to say anything I made Bob Ross read Neuromancer using 60 seconds of fuzzy audio from youtube, but 4chan made something like 20k fake audio recordings on vocaroo of everything horrible and hilarious you can think of. Then they shut that off for good reason on Monday. There are some cool applications there. I watched an hour video about the talkshow wars between Leno and Conan and i couldn't figure out what the voice was but it sounded like a smart old guy, apparently it was a weird clone of Biden. Which means anyone can now make YouTube content with any kind of voice. Which is bad for impersonation, but good if, say, you don't like the sound of your voice and want to go for a different one (maybe an old timey radio broadcaster) What definitely has become shittier is the quality of gpt4. But that's not an AI issue, that's OpenAI being OpenAI.
  - ozgung 3y ago
    Same. GPT-4 completely replaced my googling/stackoverflow routine. I generates a customized solution for my specific needs with step by step instructions. I think it works in a smarter (but slower) mode if the question is non-trivial. With pdf plugins it helps me reading and understanding recent papers. It generates an excellent summary and answers all my questions regarding any advanced concepts in the paper. I believe it's already smarter than me. I don't know if it can replace me in the mid-term but it definitely increases my productivity.
- SanderNL 3y ago
  > Image generation is still terrible and useless Image generation is thriving and now an indispensible part of the creative process. The tools are fresh and still evolving, but the impact even in this early stage is ridiculous. You can easily take rough sketches and have it spit out fully fleshed out colored prototypes. Depending on your perspective it's either an insane productivity-multiplier or scary as f-ck or both. Look at https://civitai.com https://civitai.com for a taste. Look around. You really believe this is terrible and useless? Most of these images came straight from SD with some minor tweaking, usually upscaling. I think it's quite something. It's not just MidJourney, it's the entire ecosystem surrounding models like SD, ControlNet, etc. Some artists are swinging their fists, others shrug, but all are affected by it whether they like it or not and the damn thing is just getting started. SD came out in 2022. As a young aspiring artist I can imagine it's hard to not feel like this whole thing is a massive punch in the gut. Like an experienced and highly-skilled artist, it's easy to shrug this off when you're an accomplished programmer. It can't replace me, right? Well, true. It's a gradual thing, but the impact it has on junior programmers is undeniable. What it means for our branch if leagues of young people lose interest in the meticulous acquiring of technical and often arcane knowledge through "manual programming" is anybody's guess. Maybe it's a good thing, maybe it's not. > Text-to-speech neural still sucks big time This is not my experience at all. I am actually practicing French with it and it can pick up my weird accent. It's ridiculous how good it has become in such a short time. EDIT: Oh, sorry, I read it wrong. But in that case, the same thing applies. 11labs is insane and getting better. It fools many, many people on Youtube. The Bark project is also scary at times. This is just getting started. > I can't still get chatgpt to give me a useful ffmpeg or a basic php database script. I believe these issues are at least for a significant fraction caused by the severely castrated and surprisingly lacking UI efforts from OpenAI. Just free-form "talking" with an LLM might be a good way to get grandma to use it, but as a professional programmer (or professional anything, really) it's bordering on stupid. The prompts and settings you use determine 90% of your success. OpenAI's interface gives you nothing. They just recently introduced something like "profiles", but it's amateurish at best. It makes me feel like programming without IDE (and Vi/Emacs).
- msp26 3y ago
  Making good tools with language models takes time. There are nuances that aren't immediately visible without working on them. Unless there's a mismatch in the schema/messages, function calling is very reliable at providing JSON.
dontupvoteme 3y ago
This, how absurd. Do they think the future is always going to be in "just add more parameters"? It's obviously going to be smaller, efficient and such. Even GPT4 itself is supposedly along these lines with MoE
_xivi 3y ago
> nothing to back up this primary assertion. Well, he's making a prediction, a speculation, a claim. If it was as simple as `x->y` it would no longer be valuable.
- Philpax 3y ago
  Sure, but even then you're expected to back up your speculation. There doesn't seem to be anything within the article to support the claim that we're at the tail end of development; just assumptions that today's limitations will apply tomorrow.

petermcneeley 3y ago

"too few people for all the jobs, for the most part" - Are wages going up? "But almost everywhere else needs people. Badly. Across retail, restaurants, manufacturing, trades, and on and on, companies are struggling to hire. " - Wow sounds like an exciting star trek future that I cant wait to be part of! "labor became more expensive than capital" - So wages must be going up right? "In essence, the authors show that for automation to have widespread benefits it must deliver high productivity gains to more than compensate for the high displacement of workers." - the reason why this sentence makes no sense is because it is written in two frames at the same time. One of a planned economy and the other of capitalism. "We are in a middle zone, however, with AI able to displace huge numbers of workers quickly, but not provide compensatory and broader productivity benefits." - But what about the horses? I tried my best to find the central thesis of this rambling multi-tone work. I think it is difficult because it was written with too many different frames.

PaulDavisThe1st 3y ago
Capital's willingness to ride out labor shortages without increasing wages seems to have become substantially greater in the last couple of decades. TIRED: labor shortage -> wages rise! WIRED: labor shortage -> y'all work harder for maybe a bit of overtime or maybe not, and we'll just absorb the complaints about customer service and product quality. Those damn employees aren't getting a penny more!
- wilg 3y ago
  Wages are rising: https://www.axios.com/2023/07/12/real-wage-gains-inflation https://www.axios.com/2023/07/12/real-wage-gains-inflation
  - tuatoru 3y ago
    Just catching up to years of undershooting inflation. In real terms still worse than 1969.
    - 3y ago

YeBanKo 3y ago

With few prompts I just wrote a small script with a bunch of openssl commands. Saved me an hour. And it works. And it even explained it. There is AGI and then there is “practically agi”. The latter has been achieved with LLMs.

jongjong 3y ago

As impressive as AI is and what it can do. I've felt the same way in my last job. It was almost there; it was almost good enough such that we could almost ship the project but we didn't. GPT3.5 was not good enough, GPT4 seemed like it was good enough in terms of accuracy but it was too slow and we were getting throttled like crazy. I lost my job over it. When I was worried about AI taking my job, I didn't picture it like this.

jongjong 3y ago
Our use case was that we needed to consider information from thousands of pages of documents to answer customer questions. I let GPT decide where to look based on content tables of the documents (thankfully each section was not more than ~500 words) and only fetched information from the relevant sections to let GPT answer the user's question. GPT3.5 was fast enough to be usable but often inaccurate; it would ignore some information or get confused about simple mathematical comparisons in the user's questions like 'greater than versus less than'. GPT4 was good and potentially fast enough but the API throttling was too restrictive and it was not possible to feed it less information and still get a good, nuanced answer every time. Actually the trick to get GPT to find the information based on table of contents worked really well in narrowing down the information for the second-stage summary (to answer the user's question) but our use case nevertheless required a lot of information (often from multiple sections) with a lot of grey areas and nuance and the limited amount of throughout provided by GPT4 didn't cut it. It was frustrating that, aside from accuracy issues (which depended on the complexity of the question), it often seemed to work very well and fast with GPT3.5 but the exact same code did not meet usability expectations when we switched over to GPT4 due to slow speed (though the accuracy was awesome). I only spent 1 month and a half on this project but management didn't have the patience to wait for Open AI to remove GPT4 throttling and because they couldn't see it working (at the speed they wanted), they had no reason to believe me that my solution was designed correctly... That's startups I guess. Anyway I still think there is a lot of hope for this technology and I'm confident that the problem can and will be solved. It was just unfortunate timing for me. Being an early adopter isn't always a good thing.
- ta988 3y ago

borissk 3y ago

Very interesting article. IMHO a key factor in how the world changes in the next 10-20 years will be how far the current wave of AI, based on neural networks can go. If scientists and engineers manage to implement autonomous cars and robots that can do basic human tasks like clean, wait and take care of the sick and elderly we will wake up in a brand new world. It will put a lot of stress on the basics of society, such as democracy and having to work for a salary.

tuatoru 3y ago
The real world has many trillions of parameters. Also, doing stuff in the real world is all about sensors. Progress on integrating millions of sensors into a robot has been slow.
- borissk 3y ago
  Don't think a robot needs millions of sensors. 90% of the information that a human uses comes through the eyes - so just 2 sensors. Really in terms of doing most types of work all a robot would need is vision and hearing.
  - klipt 3y ago
    Q: But if the robot has no nose, how will it smell? A: Awful!
  - csomar 3y ago
    Humans have an insane number of sensors all working with each other. The eyes are not just 2 sensors and a blind person can still function fairly well though with limitations.
- dontupvoteme 3y ago
  >The real world has many trillions of parameters. But at any given time you're probably only going to require some millions or double digit billions, unless you're a nation state or something like that.
- 3y ago

michaelmrose 3y ago

> But almost everywhere else needs people. Badly. Across retail, restaurants, manufacturing, trades, and on and on, companies are struggling to hire. And this brings us back to the cross-country trip observation and that “signing bonus” sign in a McDonald’s window. For jobs that pay just enough to not qualify for food stamps or Medicaid but not enough to buy food medicine and rent. Previously it was more possible to earn little enough to qualify for benefits and live if not well but the numbers have become misaligned and don't account for differences between expensive and cheaper parts of states. Given present concerns the situation doesn't change appreciably unless you go all in on automation implement UBI or both.

ambrozk 3y ago

This entire piece is based on one massive, unsupported assertion, which is that LLM progress will cease. Or, as the author puts it, "we are at the tail end of the first wave of large language model-based AI... [it] ends somewhere in the next year or two with the kinds of limits people are running up against." I want to know only one thing, which is what gives him the confidence necessary to say that. If that one statement is untrue, the thesis of the piece completely fails, and I do not know how any one person alive today can be certain that that statement is true. Has Paul Kedrovsky or Eric Norlin spent $150M training a 2T parameter model that no one's heard about? Do they have access to classified OpenAI docs which state that GPT5 exists already, and it's no better than GPT4? Without this sort of information, "LLMs are not going to get smart enough for widespread practical use" is an unsubstantiated bet, and nothing more.

user3939382 3y ago
I asked ChatGPT how to add a JSDoc type to a Vue 2 prop and it gave me a wrong answer. There have been several times where I’ve asked it questions and it sprinkles in well disguised misinformation. These tools are impressive but they definitely have limitations.
- golol 3y ago
  ChatGPT (3.5) is not state of the art
- seanthemon 3y ago
  Curious, how does it do when you give it examples / use playground.openai? I find that no examples often leads to the same result as a befuddled junior, but with examples often it gains confidence. Also I find playground to give me much better code snippets than chat sometimes
  - cbozeman 3y ago
    You've identified the problem inherent in these models right now, though. There's a lot of "extraneous" information and details that appear useless and unrelated, but that long-time developers have tucked away in their brains (or really anyone who has done something at a high level for a long time), that turns out to be incredibly useful; generally these people don't require examples - they just know what the right answer is, because they've been exposed to a variety of problems over a long career or lifespan. That's where LLMs need to be to be truly "useful". I think if we get them to that point, we'll really have something useful on our hands.

paulpauper 3y ago

The absence of human workers has become a limiting factor on economic growth in the U.S. the problem is real GDP growth is flat or negative due to high inflation. having more labor would make this worse, if anything. job creation tends to be inflationary. from 2010-2021 saw strong economic growth and low inflation, but now things suddenly changed due to too much inflation, not a lack of growth.

WillPostForFood 3y ago
The jobs are already created, unfilled. The lack of workers causes wages to go up which adds to inflation (e.g., the $1000 McDonalds bonus). But if you add more workers, you increase productivity, and reduce wage inflation.

atleastoptimal 3y ago

"we are already at the tail end of the current wave of AI." Is that really true? Generative AI spending this year just so far is 4x greater than last year, and the results of that spending haven't been released yet. How can they conclude this? Most "predictions" about the future I've seen assume that ChatGPT will stay the same and not improve for the next 10 years, or only marginally. Why they make this assumption, I don't know, it seems to go against every past trend we've seen with machine learning scaling. There are even people who say LLMs have reached their limit because we've exhausted all the text data that exists, assuming that the only way for LLMs to improve is to blindly feed it more static data and that no other sources of improvements exist.

sharts 3y ago
I think they mean tail end of LLM innovation? As in, all the discovery has been done and so all that is left is just computing power and tweaks rather than breakthroughs. I believe the OpenAI folks have already said something to that effect a while ago as well. For end users, the results of tweaks/computational power will seem like "breakthrough" to them but it's fundamentally not.
nottorp 3y ago
> Generative AI spending this year just so far is 4x greater than last year I'm sure they spent a lot of money looking for gold in the long rushes long before the gold was all gone too.
albertzeyer 3y ago
Also, if we add videos, images, audio, whatever other data we have, there is still quite some orders of magnitude more data available.

Madmallard 3y ago

GPT4 has been beyond our expectations by a mile. What in the world is this person even on?

nmca 3y ago

This is nuanced and interesting, despite most likely being completely wrong - as others have mentioned, it's predicted on LLMs being at the top of the sigmoid and they just ain't. Also, this goal-post-moving did crack me up rather: > Sure, it is terrific at using its statistical models to come up with textual passages that read better than the average human’s writing, but that’s not a particularly high hurdle.

dontupvoteme 3y ago

I am completely bewildered by the responses here. GPT-3 is so good it could write code, or it could emulate an entire TV show to the point of lawsuit, or simulate an eternal debate between Herzog and Zizek. You could translate Bulgarian to Sindarin for heaven's sake. And all of this comes at a 50% tax rate or so because of the alignment tax that OAI opted for. (which doesn't even work because anyone who puts effort in can jailbreak it scarily well) The real solution is not these monstrosities of the internet mushed together and taxed like a Belgian billionaire. It's models and frameworks specific to what you want in a given query. It was less than 24 hours ago that we finally got a local model that can do code generation well, and we also know that Meta has a far better one that they are holding back. There is no reason that for any query you should be restricted to a single inference run on a single fixed model, and there is no reason that we shouldn't perform a bunch of non-LLM processing on the output before the user sees it. Switch between multiple models, fine-tune, or use LoRAs based on the query -- Python plots? Load llama-coding-python-plots. You like Plotly? Add -Plotly. Run the code in a sandbox (with hard kills on resources) and regenerate it if it does not meet standards. You're working in linguistics? Switch to a chat model that's fine-tuned in the literature and codebase of that field. There are a trillion ways to improve things; we're basically at the cavemen-banging-rocks-together point. Hell, I'm a sleep deprived ESL and i just invoked the monstrosity to fix my grammar and spelling for this post. What a bloody waste of CO2. If for nothing but to keep the Earth afloat we shouldn't be using these huge closed source SaaS things unless we need them. Repurpose them for SETI@Home or processing human genetics for healthcare, or something else that benefits mankind

seydor 3y ago

Just hook the LLMs to a robot arm and watch another, much larger wave of AI innovation

arisAlexis 3y ago

>We are bumping against many of its >limits He keeps saying that as a fact with absolutely no proof and based his whole argument on it.

tacticalDonut 3y ago
Yeah, it's frustrating that he doesn't call out the specific limits.
bamboozled 3y ago
Conversely, if you don't know what the limits are yourself (and you don't) you wouldn't know either way. So I'm not quite sure if you'e added anything here unfortunately.
- arisAlexis 3y ago
  I haven't written an article asserting my argument with certainty. He did. Big difference.
  - bamboozled 3y ago
    We’ll you have a opinion either way.
    - arisAlexis 3y ago
      My opinion is that we don't know the limits. This is essentially no opinion because of no data. Stating a limit without data is an opinion thought and a wrong one.

jillesvangurp 3y ago

By the time AIs stop hallucinating, it will be effectively super human (i.e. much better) because humans hallucinate all the time. And some people are barely coherent. We're holding AIs to a much higher standard than ourselves. And we move the goal posts all the time as well. Let's deconstruct that title. "AI isn't good enough". Good enough for what? Great example of moving the goal posts. Because anytime it fails to do whatever, it's not good enough. But what about all the stuff it is good enough for that people are actually using it for already? It's passing tests at a level most humans only manage very briefly after lots of intensive preparation. The knowledge slips away quickly after that. The way I see it, there's a long, rapidly growing list of stuff that AIs have nailed already and a list of things where it is clearly struggling. That list is shrinking. Self driving gets cited a lot as something where AIs is failing. It's hard to say that in a world where multiple companies are now operating self driving vehicles commercially in several cities across the US, China, and soon Europe. Are they perfect? No. Do they drive better than the average jerk on the road? Definitely; not even close. Most people simply suck at driving. That's why traffic deaths are so common. Most of those deaths are caused by somebody with a drivers license proving that they shouldn't have received one. Not quite a solved problem but at this point the question of when these things will be on the road in my city is more a matter of when than if. It already works in other cities. I don't see what's so special in mine that it couldn't be replicated here. In other words, I expect the number of cities with autonomous vehicles to start exploding over the next few years. And who knows, Tesla might hit the good enough mark at some point as well. Again, the barrier is very low because humans aren't held to the same standards. They'll give drivers licenses to just about anyone.

ChatGTP 3y ago
"Guy calls humans flawed why claiming the same flawed species will build superhuman intelligence...", pretty good for a bunch of dumb LLMs in "meat bags"...
- arrow7000 3y ago
  You're pointing out the best that humanity is capable of. He's pointing out that people judge LLMs by their worst performances. You're not disagreeing with the parent comment.

Havoc 3y ago

I don't think the assertion that we've reached the end of current wave of AI is correct. It hasn't even been rolled out to anything yet. Once it is fully integrated into os/browser/corporate workspace and the MBA's have thrown LLMs at every business problem they can think of then perhaps but right now it hasn't even left high tech circles

layer8 3y ago
> It hasn't even been rolled out to anything yet. The question is why that is, after almost 9 months of ChatGPT API availability.

fullstackchris 3y ago

AI won't work to reeingineer society, just like the last 100 "latest things" didnt. There is a massive undercurrent that always goes in the opposite direction of technology advancement: tradition! You need new people growing up and living in a new paradigm to get these changes, you cant just ask (or expect) people who have lived their life one way to change completely to some other way of living

manuhortet 3y ago

Great article. Although I am not sure if we really are reaching the value limits of the current LLMs, as the author mentions. Some fields as legal litigation are being disrupted as much as call centers have, but the change is happening in a much slower fashion, for many reasons. The productivity gain in these fields is potentially high, but so is human displacement. Relevant fact to consider

Ekaros 3y ago

How far are we from a point you could leave AI to prompt something itself. That is not just respond to something, but actively seek it. Let's say use this place. Could it without being ordered read articles, comments and add it own comments? Maybe search internet by itself for relevant content and post it? Maybe try to earn maximal amount of upvotes? And all this without there being any bit of code poking it to do it?

firtoz 3y ago
I was going to ask "how would it know what's a good post" but it's embarrassingly easy to train the AI on interesting or not content based on upvote information...
- Hoasi 3y ago
  > but it's embarrassingly easy to train the AI on interesting or not content based on upvote information... And yet, the best topics and answers do not necessarily correlate with the most upvoted.
  - firtoz 3y ago
    How would you qualify the best?
    - Hoasi 3y ago
      We can't all agree on what best means. It can't be objective since we are humans. HN guidelines and moderation are doing a decent job of promoting interestingness in general. I don't think upvotes are a reliable indicator, but maybe they are as close to good enough as a system can be. Depending on timing, you see great topics flying under the radar. Likewise, "poor" posts regularly prove to be quite popular. But then again, this is also subjective.
      - firtoz 3y ago
        I guess if we can track moderator actions we can automate them too ;)

countWSS 3y ago

Data-centric token-guessing will be outperformed by more complex architectures, LLMs are too inefficient and their hardware requirements are disproportionate for the utility. Building a sand castle of "LLM apis" that rely on plugging their deficiences is a fundamentally bad decision that hides the underlying waste of resources. What will be great if the input data it was fed was more structured and labeled like images: imaging a "Book, 17th century, reliablity:54%, contains geographic errors". Instead this "attention is all you need" approach treats garbage data equal to top material.

abdussamit 3y ago

I haven't read the article (like many here) and I don't get why the writer would write this. LLMs/AI are not a stage where they can replace people or their jobs (broadly speaking), they still have to be taught and engineered to solve your problems. For example, if I enter a prompt "Write a Javascript function that does XYZ for me.", and don't get a desired result, it's unfair for me to say "ChatGPT bad, AI bad". At this moment, you have to work with the limits, use better prompts, use ChatGPT as a guide and not as your personal robot. With that approach, I feel lots of revolutionary content is incoming in products. Use the tool better to get the most out of it.

asu_thomas 3y ago

> There is a persistent structural imbalance in the U.S. workforce: too few people for all the jobs Lies. If the minimum wage increased proportional to inflation, none of these "jobs" would exist. The McDonalds sign-on bonus is nothing but a loan, just like all sign-on bonuses with a retention requirement. Such arrangements merely require that minimum wage is low enough to partition that compensation out of the base wage, a modern American financial industry to efficiently enforce the contracts, and a society with no remaining semblance of dignity in sight.

w_t_payne 3y ago

We're just not that good at managing AI yet. Current agent-based systems are little more than toys ... but this situation is improving very rapidly.

version_five 3y ago

Labor shortage it BS. It's rhetoric to increase immigration so we can continue sustaining a slave class and don't have to pay people properly. Canada's been very successful at this.

speak_plainly 3y ago

“Historically, when this has happened—labor became more expensive than capital—economies have responded with automation, so we should expect that again today.” Foolish assertion. Immigration has always been the stop-gap solution for expensive labor to the point that it’s built in as an essential feature of liberalism.

philipwhiuk 3y ago

We don't need better LLMs to eliminate all McDonalds workers. In fact there's no language model needed there at all. The job just needs better robotics.

naasking 3y ago
I think you're underestimating how LLMs can help here: https://www.deepmind.com/blog/rt-2-new-model-translates-vision-and-language-into-action https://www.deepmind.com/blog/rt-2-new-model-translates-visi...

ironborn123 3y ago

Rather than asserting that current LLMs are at their tail end, or that AI isnt good enough, it is much more instructive to check what are the bottlenecks or constraints to further progress, and what would help remove these bottlenecks. They can largely be divided into 3 buckets 1) Compute constraint - Currently large companies using expensive nvidia chips do most of the heavylifting of training good models. Although chips will improve over time, and competition like Intel/AMD will bring down prices, this is a slow process. But what could be a faster breakthrough is training using distributed computing over millions of consumer GPUs. There are already efforts in that direction (eg. petals/swarm parallelism for finetuning/full training, but the eastern europe/russian guys developing them dont seem to have enough resources). 2) Data constraint - If you just rely on human generated text data, you will soon exhaust this resource (maybe GPT4 has already). But the Tinystories dataset generated from GPT4 shows if we can have SOTA models generate more data (and especially on niche topics that appear less frequently in human generated data), and have deterministic/AI filters to segregate the good and bad quality data thus generated, data quantity would not be an issue any longer. Also, multimodal data is expected (with the right model architectures) to be more efficient at training world grokking SOTA models than single modal data and here we have massive amounts of online video data to tap into. 3) Architectural knowledge constraint - This may be the most difficult of all, figuring out what is the next big scalable architecture after Transformers. Either we keep trying newer ideas (like the stanford hazy research group does), and hope something sticks, or we get SOTA models few years down the line to do this ideation part for us.

tlogan 3y ago

I do not think it is fair to use AI in support call centers as an example. These call centers are not there to help customers (cancel subscription, refund, change something, etc.) and AI is perfectly tuned for that. Or at least they have procedures and AI is perfectly train to follow these. In other words, AI in support centers is designed and works to follow companies procedures and rules—which may not always align with customer expectations.

manicennui 3y ago
So is a simple form on a website in most cases. It would also be cheaper for the company and less frustrating for the customer. No one wants this "AI" chatbot bullshit except for companies selling "AI".
- tlogan 3y ago
  Or that customer wanting to cancel spend hours talking to the bot so they finally give up and hopefully forget. Having a simple form to cancel/refund/change/lost item/sent to wrong address/etc. is easier way to do and handle. But companies do not want that so … chatbot to rescue.

xpe 3y ago

I’m so done with writing that has an implicit but uncommunicated quantitative model. (AI costs are too high? Relative to what? Show us!) Almost all analytical / business / industry writing should quantified. I hope for better. We all should. It is within reach. Yes, I'm grumpy on this topic, but this doesn't make it unwarranted; it is a consequence of wanting better for a decade or longer. At this point, if you claim to be a business analyst and you aren't sharing a simulation, logic, or code, I'm just not going to take you seriously.

wildermuthn 3y ago

The most interesting part of this article isn’t its assertion that AI innovation is about to slow, but rather that current LLMs can only produce “so-so automation” — that although LLMs can replace many jobs, LLMs aren’t good enough to offset the cost of people losing such jobs. True as of today, but the article is overlooking multi-modal LLMs. Once AI can see, we’ll go from “wow, GPT4 is pretty good at helping me to my job” to “uh oh, GPT4-Vision can do my job better than I can”. This is especially true if the model can read text off a screen. At that point, it’s likely to replace many workers who stare at screens for a few hours a day and pretend to work the remainder. The model won’t take coffee breaks, or long lunches, or take the afternoon off to drive their kid to the dentist. If you’re a worker whose work depends upon looking at a screen, and mostly involves moving data from one place to another with light transformations and interpretations, you’re going to be in trouble by the end of the decade.

tempodox 3y ago
And since everyone who has been writing has been replaced by machines as well, most of those texts will be machine-generated by then. Machines reading and processing texts written by machines would be the logical consequence in this scenario, but I can't help finding the whole idea deeply absurd.

intrasight 3y ago

> too few people for all the jobs, for the most part. Change that qualifier to "for a minority part" and it would be correct. There are plenty of workers - old and young and everything in between - who would gladly work if a) you allowed them to and/or b) you paid them enough.

Nemi 3y ago

I hope beyond hope that we continue to iterate on the current GPT-4 model. But statements made by Altman himself make me leery that it might not happen. That, coupled with the fact that it is just so expensive to run make me wonder if this whole thing is sustainable. Take this into consideration - NVDA’s massive profit in the last couple of quarters is someone else’s expense. When no one is making money on AI, can this continue? I absolutely think companies can make money using AI that people will pay for, but I am not convinced that companies can make money supplying AI services. I see it as almost a commodity type service, like internet access or something. There is no clear path to profitability, and that is ok to some extent, but with the massive costs this could come crashing to a halt very quickly.

zoogeny 3y ago

When I read articles like this I can't help but question my own intuitions since they are so opposed to this author's thinking. Firstly, his suggestion on the worker shortage and the anecdotal $1000 sign on bonus at some local McDonalds. These seem more tailored to a narrative interpretation of the state of the world that is formed to the author's own bias. I look at my own small town homeless and drug addicted population and consider the changes in the last 5-10 years on that front. I consider the insane rise in the cost of home ownership and the massive rise in rental costs locally. I just don't see $1000 signing bonuses at fast food restaurants as a sign of some latent demand for labor. It feels like the author is using a single anecdote and a narrow reading of a few graphs to support a world view that I don't see existing in my own experience. Then he says: "We are quickly reaching the limits of current AI". You mean the AI that surpassed most informed peoples expectations on AI capability less than one year ago? I feel like I'm in 1996 reading an article about the Internet where someone gives some list of reasons why the Internet is reaching its limits. And just as an aside, this author uses the term "explosive" especially in the context of "explosive growth" over and over. It may be that his target market just isn't me. The narrative and analysis probably hits home for a particular set of capitalist minded investor class individuals. Perhaps they really do see North America as having a huge labor shortage and that we just need the "right" kinds of automation so we don't all feel so bad about the continuing displacement of workers. As long as they keep seeing "explosive growth" in the value of their assets.

jy1 3y ago

People often conflate AI progress with hardware progress. If we had hardware as flexible, robust, and "cheap" as the human body, we'd solve a lot of these problems such as retail, trades etc. The problem isn't software. It's hardware.

rldjbpin 3y ago

disclaimer: i am in the middle of the journey into learning this subject, and have at least done finetuning on existing models. some may still consider my take as misinformed. one thing i am not a big fan of in this discourse is that how the community makes up jargon to describe relatively simple things on one hand (a la "embeddings"), and while come up with misnomers like "hallucinations" to describe otherwise blunt things. it is contributing to miss the forest for the trees in this discussion. while i agree it is quite impressive to see what the state of the art can seemingly achieve on surface, the way we approach designing these models are fundamentally not aligned with what we want to do with its result. while some media for generative machine learning produce relatively harmless results (e.g. images and audio), it is the text where i have a lot of reservations. we train these models to essentially turn noise into something that humans can pass as acceptable (for the given prompt). Sadly, however, people outside (and sadly "inside") this space claim it to be the word of God. i still strongly believe that a hybrid system is going to be the only way we can achieve the most from the current approach. almost a decade ago we were already good at parsing through sentence structure and "understanding" what the user writes. what the output is can easily be done with some hardcoding in logic with some noise thrown in by the models. however, it should only be treated with that level of seriousness. i might be shouting into a void, but it is imperitive that we stop taking what these models output as production-ready code, or as a starting point even! otherwise, the only thing you achieve from this is to train the models running the chat services further.