3 ms·

What was the big “unlock” that allowed so much progress this past year? I ask as a noob in this area.

• mlboss 3y ago
  Stable diffusion open source release and llama release
  • alberth 3y ago
    But what technically allowed for so much progress? There’s been open source AI/ML for 20+ years. Nothing comes close to the massive milestones over the past year.
    • Chabsff 3y ago
      Public availability of large transformer-based foundation models trained at great expense, which is what OP is referring to, is definitely unprecedented.
    • kmeisthax 3y ago
      Attention, transformers, diffusion. Prior image synthesis techniques - i.e. GANs - had problems that made it difficult to scale them up, whereas the current techniques seem to have no limit other than the amount of RAM in your GPU.
    • jasonjmcghee 3y ago
      People figuring out how to train and scale newer architectures (like transfomers) effectively, to be wildly larger than ever before. Take AlexNet - the major "oh shit" moment in image classification. It had an absolutely mind-blowing number of parameters at a whopping 62 million. Holy shit, what a large network, right? Absolutely unprecedented. Now, for language models, anything under 1B parameters is a toy that barely works. Stable diffusion has around 1B or so - or the early models did, I'm sure they're larger now. A whole lot of smart people had to do a bunch of cool stuff to be able to keep networks working at all at that size. Many, many times over the years, people have tried to make larger networks, which fail to converge (read: learn to do something useful) in all sorts of crazy ways. At this size, it's also expensive to train these things from scratch, and takes a shit-ton of data, so research/discovery of new things is slow and difficult. But, we kind of climbed over a cliff, and now things are absolutely taking off in all the fields around this kind of stuff. Take a look at XTTSv2 for example, a leading open source text-to-speech model. It uses multiple models in its architecture, but one of them is GPT. There are a few key models that are still being used in a bunch of different modalities like CLIP, U-Net, GPT, etc. or similar variants. When they were released / made available, people jumped on them and started experimenting.
      • dragonwriter 3y ago
        > Stable diffusion has around 1B or so - or the early models did, I'm sure they're larger now. SDXL is 6.6 billion.
    • mschuster91 3y ago
      > But what technically allowed for so much progress? The availability of GPU compute time. Up until the Russian invasion into Ukraine, interest rates were low AF so everyone and their dog thought it would be a cool idea to mine one or another sort of shitcoin. Once rising interest rates killed that business model for good, miners dumped their GPUs on the open market, and an awful lot of cloud computing capacity suddenly went free.
    • fragmede 3y ago
      the Transformers are all you need paper from Google, which may end up being a larger contribution to society than Google search, is foundational. Emad Mostaque and his investment in stable diffusion, and his decision to release it to the world. I'm sure there are others, but those are the two that stick out to me.
• 4death4 3y ago
  I think these are the main drivers behind the progress: - Unsupervised learning techniques, e.g. transformers and diffusion models. You need unsupervised techniques in order to utilize enough data. There have been other unsupervised techniques in the past, e.g. GANs, but they don't work as well. - Massive amounts of training data. - The belief that training these models will produce something valuable. It costs between hundreds of thousands to millions of dollars to train these models. The people doing the training need to believe they're going to get something interesting out at the end. More and more people and teams are starting to see training a large model as something worth pursuing. - Better GPUs, which enables training larger models. - Honestly the fall of crypto probably also contributed, because miners were eating a lot of GPU time.
  • mkaic 3y ago
    I don't think transformers or diffusion models are inherently "unsupervised", especially not the way they're used in Stable Diffusion and related models (which are very much trained in a supervised fashion). I agree with the rest of your points though.
    • ebalit 3y ago
      Generative methods have usually been considered unsupervised. You're right that conditional generation start to blur the lines though.
      • n2d4 3y ago
        "Generative AI" is a misnomer; it's not the same kind of "generative" as the G in GAN. While you're right about GANs, diffusion models as transformers as transformers are most commonly trained with supervised learning.
        • ebalit 3y ago
          I disagree. Diffusion models are trained to generate the probability distribution of their training dataset, like other generative models (GAN, VAE, etc). The fact that the architecture is a Transformer (or a CNN with attention like in Stable Diffusion) is orthogonal to the generative vs discriminative divide. Unsupervised is a confusing term as there is always an underlying loss being optimized and working as a supervision signal, even for good old kmeans. But generative models are generally considered to be part of unsupervised methods.
• Cyphase 3y ago
  One factor is that Stable Diffusion and ChatGPT were released within 3 months of each other – August 22, 2022 and November 3, 2022, respectively. That brought a lot of attention and excitement to the field. More excitement, more people, more work being done, more progress. Of course those two releases didn't fall out of the sky.
  • JCharante 3y ago
    Dalle 2 also went viral around the same time
• marricks 3y ago
  I mean, you probably didn't pay much attention to battery capacity before phones, laptops, and electric cars, right? Battery capacity has probably increased though at some rate before you paid attention. It's just when something actually becomes relevant that we notice. Not that more advances don't happen with sustained hype, just there's some sort of tipping point involving usefulness based either on improvement of the thing in question or it's utility elsewhere.
• throwaway290 3y ago
  MS subsidizing it with 10 billions USD and (un)healthy contempt towards copyright.
• moritonal 3y ago
  Where do you want to start? The Internet collection and structuring the world's knowledge into a few key repositories? The focus on GPUs in gaming and then the crypto market creating a suite of libraries dedicated to hard scaling math. Or then the miniaturization and focus on energy efficiency due to phones making scaled training cost-effective. Finally the papers released by Google and co which didn't seem to recognise quite how easy it would be to build and replicate upon. Nothing was unlocked apart from a lot of people suddenly noticed how doable all this already was.
• password54321 3y ago
  There has been massive progress in ML every year since 2013, partly due to better GPUs and lots of training data. Many are only taking notice now that it is in products but it wasn't that long ago there was skepticism on HN even when software like Codex existed in 2021.