3 ms·

So, I came up with a pretty decent neural net from scratch about 20 years ago - it ran in the browser in Flash. It basically had a 10x10 bitmap input and an out

So, I came up with a pretty decent neural net from scratch about 20 years ago - it ran in the browser in Flash. It basically had a 10x10 bitmap input and an output of the same size, and lots of "neurons" in between that strengthened or weakened their connections based on feedback from the end result. And at a certain point they randomly mutated how they processed the input. I don't see anything wildly different now, other than scale and youth and the hubris that accompanies those things.

soulofmischief 3y ago
You're describing genetic programming and a very simple neural net, which is cool. However, the utility of transformer models should not be discounted, and if that interested you 20 years ago, you would be blown away by what's possible today.
quickthrower2 3y ago
Except the emergent properties at scale? At some point you go from making word like sentences, upping the neurons/architecture you get real sounding sentences and then upping again with RLHF loops you get impressive emergent intelligence and ability to solve tasks that were not forseen. It is a rare bird that’s not impressed with 2020s AI.
- nwienert 3y ago
  > emergent intelligence and ability to solve tasks that were not forseen What's your best examples of this? Some of the most impressive examples I've seen ended up being likely in the dataset, or very close to being so. I've yet to see something where it definitely wasn't approximately in the dataset and was solved in a way that seemed to use some sort of novel process, but open to being wrong.
  - soulofmischief 3y ago
    A good example is the 100s of conversation histories I have with GPT-4 where it does everything from help me code entirely novel and original ideas, or develop more abstract ideas. Every single day, I get immense use out of modern language models. Even if an output is similar to something it's already processed, that's fine! Such is the nature of synthesis.
    - noduerme 3y ago
      That's anything at scale. Emergence isn't a sole feature of NNs. NNs are to emergent behavior what crypto is to cash; hyping an enormous waste of resources with the promise to solve every problem, when any given problem has already been solved more elegantly. If you don't believe me about NNs, look at the caloric burden of the human brain, for fuck sake.
      - quickthrower2 3y ago
        I agree the energy cost is concerning. And we are lucky we don't have unlimited coal, unlimited power and unlimited GPUs because we'd hit 4 degrees warming by Christmas with everyone trying it out. The human brain is a salient point because often we are using AI so that the human brain can do less. Get this GPU to RTFM instead of the human. The human time is more valuable. All the while making the human brain probably less effective (compare someone who learns another language vs. someone who speaks it through an AI translator only). I hold both points of view that AI is both marvelous, but also concerning in terms of energy use. To nitpick - in " NNs are to emergent behavior what crypto is to cash " applies more to large language models. Simpler NNs for easy tasks that don't consume much power wouldn't apply (that might be like a VISA card?)
      - soulofmischief 3y ago
        I'm sorry, what is your argument? Is it that this behavior is the result of any system at scale? That is undeniably preposterous. Is it that the human brain is more efficient? At energy usage, sure, but for me to find an individual who is capable enough to assist me in the manner GPT does, at the speed and level of breadth and depth that it does, would be next to impossible. If I did, their required compensation would be astronomical. What are you arguing for or against? Are you aware that these systems will, like all previous computationally intensive systems, become drastically more efficient over time?
    - lossolo 3y ago
      > entirely novel and original ideas They are not novel if there is an equivalent pattern in the training dataset. I guess you are not really trying anything that isn't available already on github or google in some form. If you think you do then please show an example of "entirely novel and original idea", that GPT-4 developed for you. I had at least 4 cases in which ChatGPT failed to produce correct solution (after pushing it for hours to correct itself in many ways) in an actual novel problem (solution not longer than 200 lines of code) for which there was no solution on Google or github. But you can't blame statistical model that was trained to create the most probable outcomes based on it's training data.
  - quickthrower2 3y ago
    Maybe not novel novel but you can get it to write code in an application’s automation language and assist users using that application but with it’s general intelligence too (so it can figure out what the user intends, what to do in the app and generate the code to do that stuff). With a good UI that passes and executes automation code automatically, you now have magic in your app.
  - deleted 3y ago
    [deleted]
dhoe 3y ago
As a fellow old person, the way I think about it is that every time I have a thought like that it's because the neural networks inside my head have stopped being updated and are resulting in wildly outdated pattern matching. "So this car thing is just like a horse but this time with circular legs? Nothing new under the sun, I swear".
- noduerme 3y ago
  Nah. It really isn't new. Some of the neologisms take a moment to understand, but they all refer to the same ideas. "Inference" is when you show it stuff and it shows you results. "Tokens" are the matrix of bitmaps or whatever you show it, turned into a "vector" which is a set of bleeps and boops like what you send over a modem. "transformers" are just squeezing and scaling your tokens. It's fucking bog simple, stupid script kiddie shit. Inferring it or getting the inference to do what you want, anyway. Or whatever these, uh, "data scientists" with an online certificate actually do. Apparently, sit there and write plain English at a billion-dollar cluster and wait for astonishing answers at 300bps.
visarga 3y ago
You don't see the intermediate steps from a 10x10 neural net to LLMs? Like, a whole decade of ML: better optimisers, better init, residual connections, tokenisation and token embeddings, training with large batches over thousands of machines, the attention mechanism, causal masking, flash attention and other memory optimisations, and even having the foresight to train on the totality of web text. Not seeing the intermediate steps doesn't mean they are not essential and needed. If you still believe a toy 10x10 fully connected net is the same with current models (bar scaling), then what is you opinion on MLP-Mixer? That was an "MLP is all you need" moment but it didn't lead to adoption.
laurent_du 3y ago
So you wrote a toy script 20 years ago and somehow think you were doing just the same thing as OpenAI now? And you thought writing this down was a good flex? Reminds me of the old dudes in the gym who come to you to tell you how they used to bench four plates when they were young. In their mind, they are badasses. In their mind only.
- noduerme 3y ago
  no, I'm saying that lacking a modern GPU then, the results of Microsoft doing the same thing at massive scale were already fairly obvious. That there has been no major innovation. That what you think of as amazing is actually banal. BTW, awesome "flex" about how much time you spend at the gym, you sound like a guy who knows what he's talking about ;)
lpapez 3y ago
Wow, the hubris in this comment... Almost as much as in the infamous Dropbox comment: https://news.ycombinator.com/item?id=9224 https://news.ycombinator.com/item?id=9224 You too might be a Hubris News celebrity one day.

soulofmischief 3y ago

You're describing genetic programming and a very simple neural net, which is cool. However, the utility of transformer models should not be discounted, and if that interested you 20 years ago, you would be blown away by what's possible today.

quickthrower2 3y ago

Except the emergent properties at scale? At some point you go from making word like sentences, upping the neurons/architecture you get real sounding sentences and then upping again with RLHF loops you get impressive emergent intelligence and ability to solve tasks that were not forseen. It is a rare bird that’s not impressed with 2020s AI.

nwienert 3y ago
> emergent intelligence and ability to solve tasks that were not forseen What's your best examples of this? Some of the most impressive examples I've seen ended up being likely in the dataset, or very close to being so. I've yet to see something where it definitely wasn't approximately in the dataset and was solved in a way that seemed to use some sort of novel process, but open to being wrong.
- soulofmischief 3y ago
  A good example is the 100s of conversation histories I have with GPT-4 where it does everything from help me code entirely novel and original ideas, or develop more abstract ideas. Every single day, I get immense use out of modern language models. Even if an output is similar to something it's already processed, that's fine! Such is the nature of synthesis.
  - noduerme 3y ago
    That's anything at scale. Emergence isn't a sole feature of NNs. NNs are to emergent behavior what crypto is to cash; hyping an enormous waste of resources with the promise to solve every problem, when any given problem has already been solved more elegantly. If you don't believe me about NNs, look at the caloric burden of the human brain, for fuck sake.
    - quickthrower2 3y ago
      I agree the energy cost is concerning. And we are lucky we don't have unlimited coal, unlimited power and unlimited GPUs because we'd hit 4 degrees warming by Christmas with everyone trying it out. The human brain is a salient point because often we are using AI so that the human brain can do less. Get this GPU to RTFM instead of the human. The human time is more valuable. All the while making the human brain probably less effective (compare someone who learns another language vs. someone who speaks it through an AI translator only). I hold both points of view that AI is both marvelous, but also concerning in terms of energy use. To nitpick - in " NNs are to emergent behavior what crypto is to cash " applies more to large language models. Simpler NNs for easy tasks that don't consume much power wouldn't apply (that might be like a VISA card?)
    - soulofmischief 3y ago
      I'm sorry, what is your argument? Is it that this behavior is the result of any system at scale? That is undeniably preposterous. Is it that the human brain is more efficient? At energy usage, sure, but for me to find an individual who is capable enough to assist me in the manner GPT does, at the speed and level of breadth and depth that it does, would be next to impossible. If I did, their required compensation would be astronomical. What are you arguing for or against? Are you aware that these systems will, like all previous computationally intensive systems, become drastically more efficient over time?
  - lossolo 3y ago
    > entirely novel and original ideas They are not novel if there is an equivalent pattern in the training dataset. I guess you are not really trying anything that isn't available already on github or google in some form. If you think you do then please show an example of "entirely novel and original idea", that GPT-4 developed for you. I had at least 4 cases in which ChatGPT failed to produce correct solution (after pushing it for hours to correct itself in many ways) in an actual novel problem (solution not longer than 200 lines of code) for which there was no solution on Google or github. But you can't blame statistical model that was trained to create the most probable outcomes based on it's training data.
- quickthrower2 3y ago
  Maybe not novel novel but you can get it to write code in an application’s automation language and assist users using that application but with it’s general intelligence too (so it can figure out what the user intends, what to do in the app and generate the code to do that stuff). With a good UI that passes and executes automation code automatically, you now have magic in your app.
- deleted 3y ago
  [deleted]

dhoe 3y ago

As a fellow old person, the way I think about it is that every time I have a thought like that it's because the neural networks inside my head have stopped being updated and are resulting in wildly outdated pattern matching. "So this car thing is just like a horse but this time with circular legs? Nothing new under the sun, I swear".

noduerme 3y ago
Nah. It really isn't new. Some of the neologisms take a moment to understand, but they all refer to the same ideas. "Inference" is when you show it stuff and it shows you results. "Tokens" are the matrix of bitmaps or whatever you show it, turned into a "vector" which is a set of bleeps and boops like what you send over a modem. "transformers" are just squeezing and scaling your tokens. It's fucking bog simple, stupid script kiddie shit. Inferring it or getting the inference to do what you want, anyway. Or whatever these, uh, "data scientists" with an online certificate actually do. Apparently, sit there and write plain English at a billion-dollar cluster and wait for astonishing answers at 300bps.

visarga 3y ago

You don't see the intermediate steps from a 10x10 neural net to LLMs? Like, a whole decade of ML: better optimisers, better init, residual connections, tokenisation and token embeddings, training with large batches over thousands of machines, the attention mechanism, causal masking, flash attention and other memory optimisations, and even having the foresight to train on the totality of web text. Not seeing the intermediate steps doesn't mean they are not essential and needed. If you still believe a toy 10x10 fully connected net is the same with current models (bar scaling), then what is you opinion on MLP-Mixer? That was an "MLP is all you need" moment but it didn't lead to adoption.

laurent_du 3y ago

So you wrote a toy script 20 years ago and somehow think you were doing just the same thing as OpenAI now? And you thought writing this down was a good flex? Reminds me of the old dudes in the gym who come to you to tell you how they used to bench four plates when they were young. In their mind, they are badasses. In their mind only.

noduerme 3y ago
no, I'm saying that lacking a modern GPU then, the results of Microsoft doing the same thing at massive scale were already fairly obvious. That there has been no major innovation. That what you think of as amazing is actually banal. BTW, awesome "flex" about how much time you spend at the gym, you sound like a guy who knows what he's talking about ;)

lpapez 3y ago

Wow, the hubris in this comment... Almost as much as in the infamous Dropbox comment: https://news.ycombinator.com/item?id=9224 https://news.ycombinator.com/item?id=9224 You too might be a Hubris News celebrity one day.