7 ms·

There's nothing intrinsically nuclear about ML and it includes a bunch of simple approaches too. The advice to spend your limited time and attention on outdate

by jstx1 3y ago

There's nothing intrinsically nuclear about ML and it includes a bunch of simple approaches too. The advice to spend your limited time and attention on outdated approaches seems counterproductive. The things in this book aren't just old - they ended up being a dead end in research. So if it's 2023 and you have 20 hours to learn something new, you can do much better than this book.

djtango 3y ago
I don't deny that AI from the lisp days is out of favour today but I recall Neural Networks sat in a niche corner of maths mostly dormant for quite a while too... I'm not an expert in either but am confident that progress is non linear. Are there any ideas that you think are definitely bad (or even possibly good) from the lisp days?
- brabel 3y ago
  Try using neural networks on a computer from 30 years ago, or even 10 years ago, and you'll quickly realize why they were not feasible for almost any problem when this book was written.
  - eggy 3y ago
    I was studying evolutionary computation back in the 80s. Yes, you did not have the large ANNs you have today, but you also broke down your problem into manageable, computable bits (no pun intended). Working through Koza's book on Genetic Programming in the 90s, or Mark Watson's book on ANNS, chaos theory, and a bunch of other stuff, was very illuminating. I particularly liked the artificial life stuff out of the Santa Fe Institute. I still have a VHS from the 2nd proceedings. I could have rode the ML/DL wave to a high salary job, but I took another path building real-world objects and machines, and I am very happy I did. I remember when WordPerfect was popular and people spoke of careers in wordprocessing. I always saw it as evolution of the cube farm, and of course, it is. Now I have friends who work in industry with ML/DL, and they joke that it's the same but with a lot more things to track and keep running. I like GOFAI for the simple matter that sometimes 90% is good enough, and 125% is a waste of time, water in the blender. When robotics was taking off I saw the two schools of thought: Rodney Brooke's subsumption architecture making it simple to create emergent behavior, and Mark Tilden's analog circuits and discrete components. I found Tilden's work more interesting. It spoke to me about continuous or biological-like processes. I am now into researching neurocomputing and following the developments in designing neurocomputing hardware. I was so happy to have received free tickets to watch Terminator 2 back in the day, but now I feel like I have a front-row seat to it!
    - myth_drannon 3y ago
      Do you remember the names of topics from Sant Fe's "artifical life stuff"? Was it agent-based modeling? They have a bunch of courses on https://www.complexityexplorer.org https://www.complexityexplorer.org, but it's mostly Game Theory, Chaos, Complexity...
      - eggy 3y ago
        Look at the table of contents here: https://www.amazon.com/Artificial-INSTITUTE-SCIENCES-COMPLEXITY-PROCEEDINGS/dp/0201525712?asin=0201525712&revisionId=&format=4&depth=1 https://www.amazon.com/Artificial-INSTITUTE-SCIENCES-COMPLEX...
      - mistrial9 3y ago
        plenty of game programming has ingested "agents with actions" kind of programming directions; I believe that procedural code with some understanding of state-machines, is all that "artificial life" content was.. minus some details.
- kmavm 3y ago
  It's "out of favor" because it completely failed as a research program. Let's not equivocate about this; it's nice to understand heuristic search, and there was a time when things like compilation were poorly understood enough to seem like AI. But as a path towards machines that succeed at cognitive tasks, these approaches are like climbing taller and taller trees in the hopes of getting to the moon.
codr7 3y ago
Compared to any other approach to AI we've come up with, neural networks are way more complicated. Besides, they're not the final solution to anything. Learning your history is the only way to avoid repeating the same mistakes.
- jstx1 3y ago
  You should really know that there's more to ML than neural networks. Those are the simpler approaches I was referring to (like linear models, ensemble models etc). They are 1. Machine learning. 2. Not neural networks. 3. Not in Norvig's book. 4. Still useful and relevant.
  - lukego 3y ago
    In fairness they are almost certainly in Norvig's other book...
  - codr7 3y ago
    TIL; can't remember ever hearing about them before, where are they used? But it seems like we finally agree on something, simpler approaches to AI that predate neural networks are still potentially useful. Because you're not going to claim that ML is the only useful kind of AI, are you?
    - deleted 3y ago
      [deleted]
bmitc 3y ago
Outdated in what sense?
phtrivier 3y ago
> So if it's 2023 and you have 20 hours to learn something new, you can do much better than this book. Practically, which other book / ressource should someone with little time check out first ?
- bcrosby95 3y ago
  People seem to say Norvig's "other" book is still relevant. I assume that's "Artificial Intelligence: a Modern Approach". My version is from when I went to school 20 years ago. I assume it's been greatly updated over the past couple of decades. I wonder if it's worth taking a spin through the new edition.
  - Jtsummers 3y ago
    It's up to a 4th edition now (I had the 2nd when I was in school). Here are the full contents for the current edition: http://aima.cs.berkeley.edu/contents.html http://aima.cs.berkeley.edu/contents.html Chapters 19 on are going to be the biggest additions from the earlier editions.
    - phtrivier 3y ago
      The 4rth edition is from 2019. My understanding is that the world of AI has changed about a quadrillion times since those days, right ;) ? (Interesting that AI is finally catching up with javascript frameworks.) My personnal goal is to find some time to dig into https://course.fast.ai/ https://course.fast.ai/ , assuming it's not terribly outdated, either.
      - Jtsummers 3y ago
        > The 4rth edition is from 2019. My understanding is that the world of AI has changed about a quadrillion times since those days, right ;) ? The world isn't moving that fast. Transformers and LLMs are built on neural networks and lots of data and fast computers. You could jump straight to that point, but even the course you've pointed to starts off with more foundational ANN topics before getting to transformers. Much of which is at least in the TOC for the current edition of AIMA. Ought to be complementary texts. Also, only fools ignore history, "classical" AI and topics also covered in the book are still applicable. ANNs aren't going to solve all the world's problems. Other techniques that fall under the category of "AI" are still applicable and very effective for a large number of real-world problems (and much more efficient than LLMs).
dreamcompiler 3y ago
This attitude is not just ignorant; it's dangerous. I'm seeing rampant use of ML now for problems we already know how to solve in much simpler ways: linear control theory, bayesian statistics, Kalman filters, etc. "Oh hey, no need to study those old, dry topics. Just throw a bunch of training data at this GPU-bound black box and it will probably work." That's right, it will probably work. Until it doesn't. And then you won't be able to debug it. More important: You won't be able to predict when the system will fail, because it's a black box. And if it's controlling a high-consequence system, when it fails people could die. The moral is that if your problem falls into one of the already known easy-to-solve domains, you should use the old techniques. It will probably need at least 1/10^6 the CPU resources as an ML approach and you'll be able to characterize its failure regimes in advance.
- jstx1 3y ago
  You're arguing with a point that I haven't made. > I'm seeing rampant use of ML now for problems we already know how to solve in much simpler ways: linear control theory, bayesian statistics, Kalman filters, etc. How many of these techniques are in the book in the original post? I'm not saying the we should throw ML at everything, I'm saying the Norvig's book isn't useful in 2023.
  - medo-bear 3y ago
    Norvig's book is extremely useful for a whole different reason. It remains one of the best advanced books on general software design. Don't get distracted by the title
  - abecedarius 3y ago
    Those are all in his other textbook -- presumably he'd agree that AIMA is much more up to date than PAIP. I do think PAIP is still great for programmers wanting to improve their craft.
    - Jtsummers 3y ago
      > presumably he'd agree that AIMA is much more up to date than PAIP. Yes, he would. He's written as much in the past. https://norvig.com/Lisp-retro.html https://norvig.com/Lisp-retro.html
chaxor 3y ago
Neural networks were known as being "always second best" for many decades not that long ago... Its very possible (highly likely even) that elements of gofai end up being implemented into some of the upcoming RL/GNN combo based architectures. I highly doubt that the transformer will be the end-all-be-all for generating representations. At the very least, many of those 'in-the-know' around these GNNs realize that sheaf-NNs are much more expressive and can yield far better general results if improved properly for long range dependences - perhaps with a performer or longformer -like addition. Ultimately, some of the best researchers in the field (Velockovic is one of the best, and heavily focused on dynamic programming for example) are not just focused on the transformer or any of the hype around it right now. In order to improve, you have to look elsewhere. Understanding old methods is typically a great resource to draw that inspiration / algorithm from.
- abecedarius 3y ago
  Do you have a favorite overview of sheaf NNs? First I've heard of them, though I've read a bit about GNNs.

by jstx1 3y ago

djtango 3y ago

I don't deny that AI from the lisp days is out of favour today but I recall Neural Networks sat in a niche corner of maths mostly dormant for quite a while too... I'm not an expert in either but am confident that progress is non linear. Are there any ideas that you think are definitely bad (or even possibly good) from the lisp days?

brabel 3y ago
Try using neural networks on a computer from 30 years ago, or even 10 years ago, and you'll quickly realize why they were not feasible for almost any problem when this book was written.
- eggy 3y ago
  I was studying evolutionary computation back in the 80s. Yes, you did not have the large ANNs you have today, but you also broke down your problem into manageable, computable bits (no pun intended). Working through Koza's book on Genetic Programming in the 90s, or Mark Watson's book on ANNS, chaos theory, and a bunch of other stuff, was very illuminating. I particularly liked the artificial life stuff out of the Santa Fe Institute. I still have a VHS from the 2nd proceedings. I could have rode the ML/DL wave to a high salary job, but I took another path building real-world objects and machines, and I am very happy I did. I remember when WordPerfect was popular and people spoke of careers in wordprocessing. I always saw it as evolution of the cube farm, and of course, it is. Now I have friends who work in industry with ML/DL, and they joke that it's the same but with a lot more things to track and keep running. I like GOFAI for the simple matter that sometimes 90% is good enough, and 125% is a waste of time, water in the blender. When robotics was taking off I saw the two schools of thought: Rodney Brooke's subsumption architecture making it simple to create emergent behavior, and Mark Tilden's analog circuits and discrete components. I found Tilden's work more interesting. It spoke to me about continuous or biological-like processes. I am now into researching neurocomputing and following the developments in designing neurocomputing hardware. I was so happy to have received free tickets to watch Terminator 2 back in the day, but now I feel like I have a front-row seat to it!
  - myth_drannon 3y ago
    Do you remember the names of topics from Sant Fe's "artifical life stuff"? Was it agent-based modeling? They have a bunch of courses on https://www.complexityexplorer.org https://www.complexityexplorer.org, but it's mostly Game Theory, Chaos, Complexity...
    - eggy 3y ago
      Look at the table of contents here: https://www.amazon.com/Artificial-INSTITUTE-SCIENCES-COMPLEXITY-PROCEEDINGS/dp/0201525712?asin=0201525712&revisionId=&format=4&depth=1 https://www.amazon.com/Artificial-INSTITUTE-SCIENCES-COMPLEX...
    - mistrial9 3y ago
      plenty of game programming has ingested "agents with actions" kind of programming directions; I believe that procedural code with some understanding of state-machines, is all that "artificial life" content was.. minus some details.
kmavm 3y ago
It's "out of favor" because it completely failed as a research program. Let's not equivocate about this; it's nice to understand heuristic search, and there was a time when things like compilation were poorly understood enough to seem like AI. But as a path towards machines that succeed at cognitive tasks, these approaches are like climbing taller and taller trees in the hopes of getting to the moon.

codr7 3y ago

Compared to any other approach to AI we've come up with, neural networks are way more complicated. Besides, they're not the final solution to anything. Learning your history is the only way to avoid repeating the same mistakes.

jstx1 3y ago
You should really know that there's more to ML than neural networks. Those are the simpler approaches I was referring to (like linear models, ensemble models etc). They are 1. Machine learning. 2. Not neural networks. 3. Not in Norvig's book. 4. Still useful and relevant.
- lukego 3y ago
  In fairness they are almost certainly in Norvig's other book...
- codr7 3y ago
  TIL; can't remember ever hearing about them before, where are they used? But it seems like we finally agree on something, simpler approaches to AI that predate neural networks are still potentially useful. Because you're not going to claim that ML is the only useful kind of AI, are you?
  - deleted 3y ago
    [deleted]

bmitc 3y ago

Outdated in what sense?

phtrivier 3y ago

> So if it's 2023 and you have 20 hours to learn something new, you can do much better than this book. Practically, which other book / ressource should someone with little time check out first ?

bcrosby95 3y ago
People seem to say Norvig's "other" book is still relevant. I assume that's "Artificial Intelligence: a Modern Approach". My version is from when I went to school 20 years ago. I assume it's been greatly updated over the past couple of decades. I wonder if it's worth taking a spin through the new edition.
- Jtsummers 3y ago
  It's up to a 4th edition now (I had the 2nd when I was in school). Here are the full contents for the current edition: http://aima.cs.berkeley.edu/contents.html http://aima.cs.berkeley.edu/contents.html Chapters 19 on are going to be the biggest additions from the earlier editions.
  - phtrivier 3y ago
    The 4rth edition is from 2019. My understanding is that the world of AI has changed about a quadrillion times since those days, right ;) ? (Interesting that AI is finally catching up with javascript frameworks.) My personnal goal is to find some time to dig into https://course.fast.ai/ https://course.fast.ai/ , assuming it's not terribly outdated, either.
    - Jtsummers 3y ago
      > The 4rth edition is from 2019. My understanding is that the world of AI has changed about a quadrillion times since those days, right ;) ? The world isn't moving that fast. Transformers and LLMs are built on neural networks and lots of data and fast computers. You could jump straight to that point, but even the course you've pointed to starts off with more foundational ANN topics before getting to transformers. Much of which is at least in the TOC for the current edition of AIMA. Ought to be complementary texts. Also, only fools ignore history, "classical" AI and topics also covered in the book are still applicable. ANNs aren't going to solve all the world's problems. Other techniques that fall under the category of "AI" are still applicable and very effective for a large number of real-world problems (and much more efficient than LLMs).

dreamcompiler 3y ago

This attitude is not just ignorant; it's dangerous. I'm seeing rampant use of ML now for problems we already know how to solve in much simpler ways: linear control theory, bayesian statistics, Kalman filters, etc. "Oh hey, no need to study those old, dry topics. Just throw a bunch of training data at this GPU-bound black box and it will probably work." That's right, it will probably work. Until it doesn't. And then you won't be able to debug it. More important: You won't be able to predict when the system will fail, because it's a black box. And if it's controlling a high-consequence system, when it fails people could die. The moral is that if your problem falls into one of the already known easy-to-solve domains, you should use the old techniques. It will probably need at least 1/10^6 the CPU resources as an ML approach and you'll be able to characterize its failure regimes in advance.

jstx1 3y ago
You're arguing with a point that I haven't made. > I'm seeing rampant use of ML now for problems we already know how to solve in much simpler ways: linear control theory, bayesian statistics, Kalman filters, etc. How many of these techniques are in the book in the original post? I'm not saying the we should throw ML at everything, I'm saying the Norvig's book isn't useful in 2023.
- medo-bear 3y ago
  Norvig's book is extremely useful for a whole different reason. It remains one of the best advanced books on general software design. Don't get distracted by the title
- abecedarius 3y ago
  Those are all in his other textbook -- presumably he'd agree that AIMA is much more up to date than PAIP. I do think PAIP is still great for programmers wanting to improve their craft.
  - Jtsummers 3y ago
    > presumably he'd agree that AIMA is much more up to date than PAIP. Yes, he would. He's written as much in the past. https://norvig.com/Lisp-retro.html https://norvig.com/Lisp-retro.html

chaxor 3y ago

Neural networks were known as being "always second best" for many decades not that long ago... Its very possible (highly likely even) that elements of gofai end up being implemented into some of the upcoming RL/GNN combo based architectures. I highly doubt that the transformer will be the end-all-be-all for generating representations. At the very least, many of those 'in-the-know' around these GNNs realize that sheaf-NNs are much more expressive and can yield far better general results if improved properly for long range dependences - perhaps with a performer or longformer -like addition. Ultimately, some of the best researchers in the field (Velockovic is one of the best, and heavily focused on dynamic programming for example) are not just focused on the transformer or any of the hype around it right now. In order to improve, you have to look elsewhere. Understanding old methods is typically a great resource to draw that inspiration / algorithm from.

abecedarius 3y ago
Do you have a favorite overview of sheaf NNs? First I've heard of them, though I've read a bit about GNNs.