8 ms·

Having worked in the space on-and-off since my PhD, the pace is not "overwhelming". You just started paying attention to it. ChatGPT and Stable Diffusion are g

• version_five 4y ago
  Yes thank you, I find myself in the funny position of going from what would popularly be called AI optimist to AI pessimist over a short time without changing any of my views. The fact that we've found a new mode of ML that's 95% impressive doesn't solve the problem with the other 5% which is the same thing that's happened with the other modes. We're all just still in the honeymoon phase. I feel bad in a sense for the serious ML people working in this area, because there was never even a real period to reflect about how cool it is before having to be the grownups in the room and push back on absurd expectations.
  • chrbr 4y ago
    What sort of expectations are you seeing that you'd call absurd, out of curiosity? No agenda here, just also trying to sort through hype and reality.
    • mjr00 4y ago
      Any expectation that we're going to see mass unemployment, or entire career paths made obsolete, is based on extremely inflated expectations of what LLMs could maybe do in the future. It has a very similar feeling to the mid-2010s claims that Blockchain would displace bank settlements/SWIFT, completely change the supply chain, and make governments change how land/property ownership was tracked.
      • version_five 4y ago
        Same was true for "deep learning" 4-5 years ago. There's a demand for consulting companies to come out with "future of work" reports about how technology will impact jobs. Everyone knows it's flying-car style futurism and not serious, but the media picks it up as reality. Last cycle, there were all kinds of reports on jobs that deep learning would make obsolete, truck drivers and radiologitst being the flagship cases. This time it's something else. It almost feels comforting to know your job is one of the predicted obsolete ones, because of how wrong these things are.
      • stemlord 4y ago
        That's maybe a bad comparison because blockchain was practically useful to no one whereas chatgpt, copilot, et al are legitimately starting to become useful tools.
      • baq 4y ago
        I’m seeing at least applied NLP positions going up in smoke… research is probably still going to be done, but as far as business is concerned, GPT ate NLP whole and didn’t even spit a single bone out.
• TeMPOraL 4y ago
  > Having worked in the space on-and-off since my PhD, the pace is not "overwhelming". You just started paying attention to it. Really? Maybe I too only recently "started paying attention to it", but it's way above my other go-to example of churn, which is JavaScript ecosystem - and that has a benefit of mostly being fashion and faux-innovation. There's nothing comparable I can think of that's been advancing as fast in recent years as AI/ML.
• muyuu 4y ago
  what can be seen as overwhelming is the pace of news hitting the mainstream and that you might be expected to have seen if you're active in the industry I'm annoyed by the large amount of minor projects that are interesting but belong in specialised news hitting front pages of large sites, and then people expecting you to have checked them out just because of it
• poksn 4y ago
  Just because we're still a ways off from AGI, doesn't mean that the space isn't developing rapidly right now. I'm not sure autonomous driving is a fair comparable. It's pretty crazy how much better AI offerings are today than they were a year ago. Non-Tech companies are now actually using it (and paying for it), which is a night and day difference from where it used to be.
  • electrondood 4y ago
    Advances in AI are informing vehicle autonomy. Tesla is effectively developing an LLM where the road system is the domain language. Recent innovations in NeRFs are enabling volumetric object detection with a single frame from multiple cameras, etc.
• jasfi 4y ago
  ChatGPT was a tipping point.
• tryitnow 4y ago
  Agreed. But I think you're referring to "AI innovation" and the author (and others) are referring to "AI product innovation," i.e. how AI is actually showing up in people's everyday life - and that pace is absolutely overwhelming.
• baq 4y ago
  I don’t know. Progress may be linear for you but for mere mortals the jump from ‘useless but interesting toy’ to ‘indeed capable bullshit generator’ to ‘holy shit this thing run in a loop does my work for me’ was anything but linear.
  • disgruntledphd2 4y ago
    And it turns out that the really valuable part (the ability to converse) doesn't even need that much data. That has been the most surprising part to me.
  • robopsychology 4y ago
    How are you running it in a loop? I thought about using GPT to generate code that passes a description & test code could be interesting
    • baq 4y ago
      Langchain or just feed it what it generates manually. I particularly like the ai-as-ceo publicity stunts or attempts at hooking it up to a bug tracker to manage the work and then execute it… no links on mobile but should be easy to google
      • robopsychology 4y ago
        Thanks!
      • gremlinsinc 4y ago
        I'm working on building a CEO-aaS for fun, more than profit, though that too... Something mere mortals (not langchain experts) can just input maybe some competitor info, api docs, and benefits/features and then the CEO bot basically can help begin building a company around the tech/offering/business idea etc.... Whether it's a SaaS or a marketing agency, or landscaping business -- it'll give you the best items to work on to expand your business so you can just focus on that, and not have to think about what to do to build your business.
  • 4y ago
• famouswaffles 4y ago
  GPT-2 was four years ago. The pace has indeed been enourmous. The fact that the general public has only been paying attention recently really doesn't mean much
  • timr 4y ago
    Meh. The difference between GPT-2 and GPT-4 is not as revolutionary as you're making it seem. The ability to process images and text as input is impressive, and also an evolutionary combination of things that were happening separately before.
    • crop_rotation 4y ago
      It is revolutionary for many. GPT2 was just another "looks cool" thing, GPT4 is just so jaw dropping that I am scared what GPT5 would look like.
      • famouswaffles 4y ago
        Yeah lol. I didn't want to argue further because hey it's his/her opinion. But saying GPT-2 to 4 in 4 years isn't revolutionary just seems contrarian to me. Bespoke NLP of any kind is out the door with a single model. That's fucking crazy.
        • timr 4y ago
          Yours is also just an opinion. But having a baseline understanding of what came before is important to having an informed opinion. To that end: GPT is not the first tech to combine "NLP" into a single model. For example, from 2019 [1]. This stuff is evolutionary. [1] https://pub.towardsai.net/unified-language-model-pre-training-for-natural-language-understanding-and-generation-f87dc226aa2 https://pub.towardsai.net/unified-language-model-pre-trainin...
          • famouswaffles 4y ago
            You linked a paper that came months after the release of GPT-2. It is also a modified transformer, the basis of most LLMs today and an architecture that was released only two years prior. and this is without mentioning that NLP is far from the only things LLMs do/excel at. Frankly, nothing you have said so far has made me think your opinion is more informed than mine.
• UncleOxidant 4y ago
  > the pace is not "overwhelming". You just started paying attention to it. I've been paying some attention since 2015 when I worked a neural net gig. And the pace does seem to be increasing. Just in a small corner of LLMs we see LLaMa and then alpaca and then the LORA fine tuned version of the LLaMa params and then the various effects of different levels of quantization - allowing some of these models to run on things like RPis. More people are working on this stuff now meaning that the pace is going faster. Yeah, most of it is incremental, but that doesn't mean that it isn't difficult to keep up with where the best results are being found.
• deleted 4y ago
  [deleted]
• RC_ITR 4y ago
  It’s kind of wild to hear someone say this and it leads me to believe your expertise is outside the world of transformers. In 2014, we were training encoder/decoder models on maybe a billion tokens, mostly limited by models architected around time steps. Today, we are training encoder only models on trillions of tokens, mostly hindered by eminently solvable stability problems (i.e. can we make enough parallel compute for our infinitely parallel models). Maybe that 10 years of progress feels the same as going from LSTM (1997-ish) to decoder attention (2014-ish) over 20 or so years, but it doesn’t to me.
  • adamsmith143 4y ago
    Not only not in the world of transformers but I find it unlikely it's in modern ML at all. Deepmind has been putting out shocking work in Deep RL which is as impressive if not as hyped as anything from OpenAI.
• lettergram 4y ago
  Having worked in the generative model space for years I agree. People don’t realize how accurate NLP, generative and similar systems are expected to be
• pessimizer 4y ago
  If you told me that LLMs should be trusted to drive cars or operate any other weaponry, I'd tell you it's never going to happen. But they're not: the jobs they'll be doing are the liberal arts/management jobs that have a tremendous fudge factor, close is more than good enough, and failure has few consequences[*]. If an LLM doesn't give me what I want, I can tweak it and try again; absolutely no sweat. Difficult to do after a fatal car accident. edit: and the ability to construct intention from input has to make them ideal for interfaces (from an ignorant layman's pov.) Is anybody using generative AI to create interfaces for prosthetic limbs or other peripherals? Search engines offering them in order to interface with the internet seems like the least interesting thing to do with them, although (as I mentioned above) with artistic/creative output it's difficult to define failure.
• deleted 4y ago
  [deleted]
• imwithstoopid 4y ago
  > Newsflash: it's 2023. We still drive. ...but self-driving is still making incredible progress lots of techies seem stuck in a mindset of "it doesn't work today, so it never will!"
• blueorange8 4y ago
  I agree with you that the "amazing" part is the query understanding. There are two really big things I'm excited about: 1. It really understands what I am asking. What my meaning is. Gpt-4 is way way better even than gpt-3.5 at this. This is something truly different that a computer has never been able to do and (hopefully) will only get better at it. This really truly changes everything in how we interact with computers. We shouldn't be downplaying this. This is real. Today. At the same time a lot of this turning "understanding" into useful output comes at a huge amount of work on the RLHF side so we shouldn't necessarily extrapolate into the future too much just based on the current progress of openai. It's still very difficult work. 2. Everyone is now paying attention to LLM's. When the world starts paying attention to things and engineers start hacking things together, big companies start pouring money into ai, maybe Google wakes up for once, students start choosing ai as their career field, etc etc - amazing things happen. AI just went from niche to something that everyone is using and thinking about.
  • deleted 4y ago
    [deleted]
• comte7092 4y ago
  In popular consciousness, there seems to exist this blind optimism of “it will only get better/faster/etc from here”. Which may be true, but as we saw with semiconductors, there’s always going to be an asymptote. And we don’t really know where that is until we hit it. Are we in 1970 semiconductor territory with LLMs, or are we in 2020 semiconductor territory? Ultimately, arguing about where we will be in ten years is somewhat futile.
• surgical_fire 4y ago
  This is the kind of sober take that is sadly missing from these discussions. Reading all the people LARPing about how ChatGPT can solve everything that humans did until months ago is amusing, but gets tiresome pretty quickly. And I say that as someone enthusiastic enough about it to pay for premium acess to the thing. Has been an amazing programming assistant, I use it as a web search on steroids. But I think the advancements on the medium term will be only refinements over what is already here. It will be quite a while for another leap.
• throwaway292939 4y ago
  > The output is still (very pretty) garbage >10% of the time, and that's the danger zone...the Level 4 of autonomous thinking. What use cases do you have for it? Isn't it the other way? I'd argue that the output is garbage <10% of the time, and it's those times you have to be a little careful for. What else explains the massive adoption if it isn't providing a decent amount of utility? Overall I agree with the sentiment though. It's like people underestimating how long it takes to complete the last 10% of a software project.
  • timr 4y ago
    > What use cases do you have for it? Honestly, I don't know yet. Maybe it's a better version of those useless support chatbots that were ubiquitous during the last AI hype cycle? Beyond that, you start to get into questions like "how much better is this technology than competing technology X, at benchmarks I care about?". But those questions don't capture the hive mind in quite the same way. > I'd argue that the output is garbage <10% of the time, and it's those times you have to be a little careful for. I don't know where the "bullshit threshold" is for a given task, but I've already seen articles claiming that programmers will be obsolete, citing GPT-generated code that was incorrect. So, you know...YMMV. > What else explains the massive adoption if it isn't providing a decent amount of utility? Hmmmm...I forgot the word. Begins with "H"...hyphen? Hippy? Hippo? Hyperbola?