4 ms·
Early Days of AI
- mdp2021 3y ago> it is worth thinking of this as an entirely new era In this case, we could considered the «Early Days of AI» as not having happened yet. It is absurd to forget a past that worked to celebrate a present that largely does not, to sensible understanding of the goal. Tools must be reliable. > and discontinuity from the past Let us hope this is a bump on the road, a phase, better if organic and eventually productive of something good.
- eladgil 3y agoDefinitely not my intention to forgot or denigrate the past. Obviously all this exists due to deep learning and prior architectures. What I have been running into is many people and companies are interpreting this as "just more of the same" for prior ML waves, when really this is an entirely new capability set. To the (bad) analogy on cars versus planes - both have wheels and can drive on the ground, but planes open up an entirely new dimension / capability set that can transform transportation, logistics, defense and other areas that cars were important, but different enough in.
- simonw 3y agoI'm finding the comparison to the previous wave of ML absolutely fascinating. I tinkered with ML for a few years, and it took a LOT of work to get anything useful out of it at all. Now with LLMs I can literally type out a problem in English and there's a reasonably good chance I'll get a useful result! It really does feel like an entirely new set of capabilities to me.
- andy99 3y agoThat's the big difference in this round. Before you had to have the ML expertise and the expertise to understand the implication of say a MNIST classifier example. Now anyone can "get" it because you're prompting and getting inference back in English. Underneath the fundamentals aren't all that different though, it has the same novelty factor and the same limitations apply.
- simonw 3y agoI think the fundamentals are radically different, just due to the ease of applying this stuff. I used to be able to train and deploy a ML model to help solve a problem... if I put aside a full week to get that done. Now I tinker with LLMs five minutes at a time, or maybe for a full hour if I have something harder - and get useful results. I use them on a daily basis.
- npsomaratna 3y agoExactly. LLMs are useful out of the box—and that enables tons of use-cases that just weren't economical (or even possible) with older models.
- hulitu 3y agoThe only "entirely new dimension / capability set" that i saw at this new AI hype is its use in propaganda. Microsoft is not capable of basic testing of its software so i don't see what AI will bring to the table besides more bugs.
- gumby 3y agoI think in five years, LLMs will be like expert systems: largely considered not really "AI" but sitting around in the back end of all sorts of random systems.
- eladgil 3y agoThe bar for what is "AI" keeps moving. For example plane autopilots would be "AI" in the 1980s, the ability for a machine to win at chess, go, and other games etc.
- deleted 3y ago[deleted]
- calderknight 3y agoI don't think those things have ever been widely considered AI. It may have widely thought that those things would require AI, though.
- beambot 3y agoChess is featured in Peter Norvig's "Artificial Intelligence: A Modern Approach" dating back to the 1st edition (1995) and at least up until the 3rd edition (2009). Algorithms such as alpha-beta pruning were definitely considered AI at the time.
- calderknight 3y agoYeah AI has multiple denotations, those things have never been considered "real" AI tho
- deleted 3y ago[deleted]
- intelkishan 3y agoIt's even mentioned in the recent 4th edition(2020).
- galaxytachyon 3y agoWhile I largely agree with this blog, I would like to point out one thing that often come up in HN: an inherent weakness or flaw in the current AI architecture that prevent it from widespread adoption. Right now it is either scaling hardware or hallucination. But there can be something deeper that we yet to see in public, for example, the inability to adapt to some specific, but critical, reasoning logic. I don't see the post address this possibility even though it is very likely. Microsoft promised AI powered Office a while ago and we are still waiting. GPT4 is supposed to be able to look at images and solve problems but we still haven't seen that yet. Something is preventing these big companies from implementing these features and this is supposed to be a solved problem. How can we be sure that there is no serious roadblocks in the future that plunge the field into another AI winter?
- alooPotato 3y agoIt's been like 4 months
- galaxytachyon 3y agoIn term of AI research and development, that was a long time ago. Microsoft is also pivoting hard into AI. They treat it as a cornerstone technology and I am sure they are not skimping on the cost of implementing it into their main products. Yet the only thing they have is Bing. Mind you they have early access to this tech. Bing uses GPT4 before the paper for GPT4 even came out so it is not like they only had it for 4 months.
- taneq 3y agoIn terms of designing, engineering, implementing, testing, documenting and shipping a new suite of products, four months is nothing.
- joe_the_user 3y agoI think this view, that present day AI is something absolutely new, is actually the dominant view. I don't believe the "absolutely new" view is very enlightening very often. Notably, it seems like "the dawn of a new era of tech" offers little insight to the process of change (but much hype). Even something like the explosion of the Internet is usefully compared to earlier technologies and what gave it's uniqueness wasn't incomparability but an explosion of scale.
- joewferrara 3y agoThe author has more experience in the field than me, so gotta defer to him for the most part, and while I generally agree with the post, I disagree strongly with one point. The author frames this new era of AI about using transformer models (and diffusion models) but transformer models have been around for a while and have been useful from before GPT3, the model the author claims as the starting point for this new AI. BERT is a transformer model that came out in 2018 and is a very useful transformer style model, which showed the promise of transformers before GPT3. Edit: Going back through the post, the author’s slide has Transformers labeled as 2017, so he is aware of the history and he’s just emphasizing that GPT3 was the first transformer model that he thinks had something interesting and related to the current AI explosion. I think BERT style models would be worth a mention in the post as the first transformer models found to be widely useful.
- whimsicalism 3y agoFrom the article, > The biggest inklings that something interesting was afoot came kicked with GPT-3 launching in June 2020 Yes, I agree with you that if you were in this field, this is quite late to the realization. Most of my grad seminar in NLP in 2018 imagined ChatGPT-style tech would be possible as "language modeling is essentially world modeling."
- eladgil 3y ago100% agree the theory on AI is old and actually dates back to the early days of "cybernetics". But the real difference is at what point do we considered it sufficiently reduced to practice? I chose GPT-3 but undoubtedly people can point to earlier examples as glimpses of what was coming for sure.
- abatilo 3y agoThe highlight of the fact that large org adoption is slow feels really valuable here. Especially when we just recently had all of the articles that claimed that ChatGPT was "over" when it appears that this is mostly because of things like summer vacations with students. The adoption and understanding of these products and the risk involved with hallucinations is something that will take time to understand. I recently joined one of the LLM provider companies and watching these phases over the next few years will be really interesting. Especially combined with what's going on with regulation and the like. Random aside, hi Elad! I think you're reading some of these comments. I just left Color after ~2.5 years. I hope to get to formally introduce myself to you one day.
- vasilipupkin 3y agoML in general has been in production in many places/companies for a while now. Specifically GPt-4 is useful as a coding assistant and a reference tool. However, it's hard to know whether what it's telling you is accurate or fake and if you want to be thorough you need to double check it. So, it's already useful but in a limited way and it remains to be seen how much better it's likely to get in the near future. But let's not confuse that in general with AI or Machine Learning which is already used heavily in lots of places. the specific type of architecture that gpt-4 uses for next word prediction is not the only possible architecture and is not what's used for many real world tasks. There a lot of different problems being addressed by ML and next word prediction is just one of them, although quite important.
- eladgil 3y agoAgreed on ML being in production in a lot of places and has been quite valuable, particularly for large incumbents. I write about this a bit here: https://blog.eladgil.com/p/ai-startup-vs-incumbent-value https://blog.eladgil.com/p/ai-startup-vs-incumbent-value I think the difference this time is the types of capabilities provided by transformers vs prior waves of AI are sufficiently different to allow many more types of startups to emerge, as well as big changes in some types of enterprise software by incumbents - in ways that were not enabled by pre-existing ML approaches.
- vasilipupkin 3y agoThe jury is still out on how useful these additional capabilities provided by transformers are. The question is what is the degree to which it’s possible to reduce the frequency and severity of hallucinations. If that degree is limited without major changes in architecture or major new breakthroughs, then usefulness of gpt-4 style models will be limited. And we just don’t know the answer yet. So far, usefulness of gpt-4 is real but extremely limited. Another issue is this approach means models are costly to train and don’t easily incorporate latest info about the world. In short, it’s way too early to hype this up.
- dvt 3y ago> There is enormous potential for this new wave of tech to impact humanity. For example, Google’s MedPaLM2 model outperforms human physicians to such a strong degree that having medical experts RLHF the model makes it worse (!). Wow, what a ridiculously disingenuous cherry-picked claim. If you actually read the paper you'll find this gem: "However, for one of the axes, including inaccurate or irrelevant information, Med-PaLM 2 answers were not as favorable as physician answers." Typical AI hype blog post donning the HN front page. At this point, I'm ready to put on my tinfoil hat and say that a16z, etc. is heavily pushing all these narratives because the next round of investments for the great majority of AI startups will almost certainly be the bagholder round.
- rafaelero 3y agoMaybe. But then look at how fast accuracy is improving. I would bet the next iteration will be better in all relevant metrics.
- ericfr11 3y agoFelt the same: the Internet is now full of "I saw it coming, and can predict the future" type of blog article, where you could replace AI by block chain (or any new technology - superconductors?) and write the same article. Very bland and not worth of HN time...
- tinco 3y agoThe author picks an example of an aspect where Med-Palm2 is impressive, and then your counter argument is to find another cherry picked example where the performance is not satisfactory. Great discourse.. There is nothing disingenuous about the post. Is your argument literally that there is not enormous potential for impact with this technology? And you're going to go with the argument that there's a single axis on which an ML system underperforms a frigging doctor? In what universe is an ML system (slightly) underperforming a human doctor on a single axis not one of the most impressive things to have happened in the past 50 years?
- phillipcarter 3y agoWhat’s really special about this new era of ML is its accessibility. Every business had problems with probabilistic solutions. ML is the way to do that. But the barrier to entry has been so high for so long. And so you had to be in big tech or a highly specialized shop to play. Now all you need is an API key and one line of code to call the most powerful models in the world.
- foob 3y agoI agree, but it's worth mentioning explicitly that the main driver of accessibility has been generalizability. The fact that LLMs are so effective at zero-shot and few-shot learning tasks is what made out-of-the-box API access practical for a wide range of use cases. Products like SageMaker tried to automate away some of the complexity of training bespoke models, but they're still way more complicated than using an off-the-shelf model.
- moffkalast 3y ago> Google’s MedPaLM2 model outperforms human physicians to such a strong degree that having medical experts RLHF the model makes it worse That's genuinely hilarious. I was wondering why it's not deployed yet, but apparently it is being rolled out: https://www.theverge.com/2023/7/8/23788265/google-med-palm-2-mayo-clinic-chatbot-bard-chatgpt https://www.theverge.com/2023/7/8/23788265/google-med-palm-2...
- RivieraKid 3y agoI hope there will be something like GPT for real world interaction / fine motor skills / robotics. Activities like cleaning or walking require the sort of intelligence that deep nets are able to mimic. The problem is how to train the model, what should be the task during training, what's the alternative to predicting the next word.
- rdli 3y agoAnother parallel that I see is “cloud”. Salesforce was one of the first (if not the first) real successful cloud companies. But it took a really long time (and it still is going on) for cloud-first to be the default in enterprises. And along the way, there was a ton of new technologies that were (re)invented: VMs and then containers etc at the infra level, new app architectures like AJAX (holy smokes was this amazing when it first came out), etc. Similarly I think we’re in for a wild ride on AI and figuring out its implications. There’s a ton of obvious use cases today but I’m really interested in the ones that aren’t obvious right now.
- lovegrenoble 3y agoI'm not convinced that we should view Transformers and diffusion models as an entirely new era of AI. While they certainly represent advancements, it's crucial to acknowledge the incremental nature of progress in this field.
- SrslyJosh 3y ago[flagged]