4 ms·

I think neural nets are just a subset of machine learning techniques. I wonder what would have happened if we poured the same amount of money, talent and hardw

I think neural nets are just a subset of machine learning techniques. I wonder what would have happened if we poured the same amount of money, talent and hardware into SVMs, random forests, KNN, etc. I don't say that transformers, LLMs, deep learning and other great things that happened in the neural network space aren't very valuable, because they are. But I think in the future we should also study other options which might be better suited than neural networks for some classes of problems. Can a very large and expensive LLM do sentiment analysis or classification? Yes, it can. But so can simple SVMs and KNN and sometimes even better. I saw some YouTube coders doing calls to OpenAI's o1 model for some very simple classification tasks. That isn't the best tool for the job.

ldjkfkdsjnv 2y ago
This is such a terrible opinion, im so tired of reading the LLM deniers
Meloniko 2y ago
And based on what though do you think that? I think neural networks are fundamental and we will focus/experiment a lot more with architecture, layers and other parts involved but emerging features arise through size
mentalgear 2y ago
KANs (Kolmogorov-Arnold Networks) are one example of a promising exploration pathway to real AGI, with the advantage of full explain-ability.
- astrange 2y ago
  "Explainable" is a strong word. As a simple example, if you ask a question and part of the answer is directly quoted from a book from memory, that text is not computed/reasoned by the AI and so doesn't have an "explanation". But I also suspect that any AGI would necessarily produce answers it can't explain. That's called intuition.
  - diffeomorphism 2y ago
    Why? If I ask you what the height of the Empire State Building is, then a reference is a great, explainable answer.
    - astrange 2y ago
      It wouldn't be a reference; "explanation" for an LLM means it tells you which of its neurons were used to create the answer, ie what internal computations it did and which parts of the input it read. Their architecture isn't capable of referencing things. What you'd get is an explanation saying "it quoted this verbatim", or possibly "the top neuron is used to output the word 'State' after the word 'Empire'". You can try out a system here: https://monitor.transluce.org/dashboard/chat https://monitor.transluce.org/dashboard/chat Of course the AI could incorporate web search, but then what if the explanation is just "it did a web search and that was the first result"? It seems pretty difficult to recursively make every external tool also explainable…
      - Retric 2y ago
        LLM’s are not the only possible option here. When talking about AGI none of what we are doing is currently that promising. The search is for something that can write an essay, drive a car, and cook lunch so we need something new.
        Vampiero 2y ago
        When people talk about explainability I immediately think of Prolog. A Prolog query is explainable precisely because, by construction, it itself is the explanation. And you can go step by step and understand how you got a particular result, inspecting each variable binding and predicate call site in the process. Despite all the billions being thrown at modern ML, no one has managed to create a model that does something like what Prolog does with its simple recursive backtracking. So the moral of the story is that you can 100% trust the result of a Prolog query, but you can't ever trust the output of an LLM. Given that, which technology would you rather use to build software on which lives depend on? And which of the two methods is more "artificially intelligent"?
- yathaid 2y ago
  Neural networks can encode any computable function. KANs have no advantage in terms of computability. Why are they a promising pathway? Also, the splines in KANs are no more "explainable" than the matrix weights. Sure, we can assign importance to a node, but so what? It has no more meaning than anything else.
trhway 2y ago
>I wonder what would have happened if we poured the same amount of money, talent and hardware into SVMs, random forests, KNN, etc. people did that to horses. No car resulted from it, just slightly better horses. >I saw some YouTube coders doing calls to OpenAI's o1 model for some very simple classification tasks. That isn't the best tool for the job. This "not best tool" is just there for the coders to call while the "simple SVMs and KNN" would require coding and training by those coders for the specific task they have at hand.
- guappa 2y ago
  [citation needed]
matusp 2y ago
Deep learning is easy to adapt to various domains, use cases, training criteria. Other approaches do not have the flexibility of combining arbitrary layers and subnetworks and then training them with arbitrary loss functions. The depth in deep learning is also pretty important, as it allows the model to create hierarchical representations of the inputs.
- f1shy 2y ago
  But is very hard to validate for important or critical applications
jasode 2y ago
>I wonder what would have happened if we poured the same amount of money, talent and hardware into SVMs, random forests, KNN, etc. But that's backwards from how new techniques and progress is made. What actually happens is somebody (maybe a student at a university) has an insight or new idea for an algorithm that's near $0 cost to implement a proof-of concept. Then everybody else notices the improvement and then extra millions/billions get directed toward it. New ideas -- that didn't cost much at the start -- ATTRACT the follow on billions in investments. This timeline of tech progress in computer science is the opposite from other disciplines such as materials science or bio-medical fields. Trying to discover the next super-alloy or cancer drug all requires expensive experiments. Manipulating atoms & molecules requires very expensive specialized equipment. In contrast, computer science experiments can be cheap. You just need a clever insight. An example of that was the 2012 AlexNet image recognition algorithm that blew all the other approaches out of the water. Alex Krizhevsky had an new insight on a convolutional neural network to run on CUDA. He bought 2 NVIDIA cards (GTX580 3GB GPU) from Amazon. It didn't require NASA levels of investment at the start to implement his idea. Once everybody else noticed his superior results, the billions began pouring in to iterate/refine on CNNs. Both the "attention mechanism" and the refinement of "transformer architecture" were also cheap to prove out at a very small scale. In 2014, Jakob Uszkoreit thought about an "attention mechanism" instead of RNN and LSTM for machine translation. It didn't cost billions to come up with that idea. Yes, ChatGPT-the-product cost billions but the "attention mechanism algorithm" did not. >into SVMs, random forests, KNN, etc. If anyone has found an unknown insight into SVM, KNN, etc that everybody else in the industry has overlooked, they can do cheap experiments to prove it. E.g. The entire Wikipedia text download is currently only ~25GB. Run the new SVM classification idea on that corpus. Very low cost experiments in computer science algorithms can still be done in the proverbial "home garage".
- FrustratedMonky 2y ago
  "$0 cost to implement a proof-of concept" This falls apart for breakthroughs that are not zero cost to do a proof-of concept. Think that is what the parent is rereferring . That other technologies might have more potential, but would take money to build out.
- 2y ago
edude03 2y ago
Transformers were made for machine translation - someone had the insight that when going from one language to another the context mattered such that the tokens that came before would bias which ones came after. It just so happened that transformers we more performant on other tasks, and at the time you could demonstrate the improvement on a small scale.
f1shy 2y ago
> neural nets are just a subset of machine learning techniques. Fact by definition
dr_dshiv 2y ago
The best tool for the job is, I’d argue, the one that does the job most reliably for the least amount of money. When you consider how little expertise or data you need to use openai offerings, I’d be surprised if sentiment analysis using classical ML methods are actually better (unless you are an expert and have a good dataset).
jensgk 2y ago
> I wonder what would have happened if we poured the same amount of money, talent and hardware into SVMs, random forests, KNN, etc. From my perspective, that is actually what happened between the mid-90s to 2015. Neural netowrks were dead in that period, but any other ML method was very, very hot.
netdevnet 2y ago
You are supposed to call it AI now. The word "machine learning" is for GOFAI 2nd gen only. Once all investors have been money drained and the next AI winter begins, then you will be allowed to call it Machine Learning

ldjkfkdsjnv 2y ago

This is such a terrible opinion, im so tired of reading the LLM deniers

Meloniko 2y ago

And based on what though do you think that? I think neural networks are fundamental and we will focus/experiment a lot more with architecture, layers and other parts involved but emerging features arise through size

mentalgear 2y ago

KANs (Kolmogorov-Arnold Networks) are one example of a promising exploration pathway to real AGI, with the advantage of full explain-ability.

astrange 2y ago
"Explainable" is a strong word. As a simple example, if you ask a question and part of the answer is directly quoted from a book from memory, that text is not computed/reasoned by the AI and so doesn't have an "explanation". But I also suspect that any AGI would necessarily produce answers it can't explain. That's called intuition.
- diffeomorphism 2y ago
  Why? If I ask you what the height of the Empire State Building is, then a reference is a great, explainable answer.
  - astrange 2y ago
    It wouldn't be a reference; "explanation" for an LLM means it tells you which of its neurons were used to create the answer, ie what internal computations it did and which parts of the input it read. Their architecture isn't capable of referencing things. What you'd get is an explanation saying "it quoted this verbatim", or possibly "the top neuron is used to output the word 'State' after the word 'Empire'". You can try out a system here: https://monitor.transluce.org/dashboard/chat https://monitor.transluce.org/dashboard/chat Of course the AI could incorporate web search, but then what if the explanation is just "it did a web search and that was the first result"? It seems pretty difficult to recursively make every external tool also explainable…
    - Retric 2y ago
      LLM’s are not the only possible option here. When talking about AGI none of what we are doing is currently that promising. The search is for something that can write an essay, drive a car, and cook lunch so we need something new.
      - Vampiero 2y ago
        When people talk about explainability I immediately think of Prolog. A Prolog query is explainable precisely because, by construction, it itself is the explanation. And you can go step by step and understand how you got a particular result, inspecting each variable binding and predicate call site in the process. Despite all the billions being thrown at modern ML, no one has managed to create a model that does something like what Prolog does with its simple recursive backtracking. So the moral of the story is that you can 100% trust the result of a Prolog query, but you can't ever trust the output of an LLM. Given that, which technology would you rather use to build software on which lives depend on? And which of the two methods is more "artificially intelligent"?
yathaid 2y ago
Neural networks can encode any computable function. KANs have no advantage in terms of computability. Why are they a promising pathway? Also, the splines in KANs are no more "explainable" than the matrix weights. Sure, we can assign importance to a node, but so what? It has no more meaning than anything else.

trhway 2y ago

>I wonder what would have happened if we poured the same amount of money, talent and hardware into SVMs, random forests, KNN, etc. people did that to horses. No car resulted from it, just slightly better horses. >I saw some YouTube coders doing calls to OpenAI's o1 model for some very simple classification tasks. That isn't the best tool for the job. This "not best tool" is just there for the coders to call while the "simple SVMs and KNN" would require coding and training by those coders for the specific task they have at hand.

guappa 2y ago
[citation needed]

matusp 2y ago

Deep learning is easy to adapt to various domains, use cases, training criteria. Other approaches do not have the flexibility of combining arbitrary layers and subnetworks and then training them with arbitrary loss functions. The depth in deep learning is also pretty important, as it allows the model to create hierarchical representations of the inputs.

f1shy 2y ago
But is very hard to validate for important or critical applications

jasode 2y ago

>I wonder what would have happened if we poured the same amount of money, talent and hardware into SVMs, random forests, KNN, etc. But that's backwards from how new techniques and progress is made. What actually happens is somebody (maybe a student at a university) has an insight or new idea for an algorithm that's near $0 cost to implement a proof-of concept. Then everybody else notices the improvement and then extra millions/billions get directed toward it. New ideas -- that didn't cost much at the start -- ATTRACT the follow on billions in investments. This timeline of tech progress in computer science is the opposite from other disciplines such as materials science or bio-medical fields. Trying to discover the next super-alloy or cancer drug all requires expensive experiments. Manipulating atoms & molecules requires very expensive specialized equipment. In contrast, computer science experiments can be cheap. You just need a clever insight. An example of that was the 2012 AlexNet image recognition algorithm that blew all the other approaches out of the water. Alex Krizhevsky had an new insight on a convolutional neural network to run on CUDA. He bought 2 NVIDIA cards (GTX580 3GB GPU) from Amazon. It didn't require NASA levels of investment at the start to implement his idea. Once everybody else noticed his superior results, the billions began pouring in to iterate/refine on CNNs. Both the "attention mechanism" and the refinement of "transformer architecture" were also cheap to prove out at a very small scale. In 2014, Jakob Uszkoreit thought about an "attention mechanism" instead of RNN and LSTM for machine translation. It didn't cost billions to come up with that idea. Yes, ChatGPT-the-product cost billions but the "attention mechanism algorithm" did not. >into SVMs, random forests, KNN, etc. If anyone has found an unknown insight into SVM, KNN, etc that everybody else in the industry has overlooked, they can do cheap experiments to prove it. E.g. The entire Wikipedia text download is currently only ~25GB. Run the new SVM classification idea on that corpus. Very low cost experiments in computer science algorithms can still be done in the proverbial "home garage".

FrustratedMonky 2y ago
"$0 cost to implement a proof-of concept" This falls apart for breakthroughs that are not zero cost to do a proof-of concept. Think that is what the parent is rereferring . That other technologies might have more potential, but would take money to build out.
2y ago

edude03 2y ago

Transformers were made for machine translation - someone had the insight that when going from one language to another the context mattered such that the tokens that came before would bias which ones came after. It just so happened that transformers we more performant on other tasks, and at the time you could demonstrate the improvement on a small scale.

f1shy 2y ago

> neural nets are just a subset of machine learning techniques. Fact by definition

dr_dshiv 2y ago

The best tool for the job is, I’d argue, the one that does the job most reliably for the least amount of money. When you consider how little expertise or data you need to use openai offerings, I’d be surprised if sentiment analysis using classical ML methods are actually better (unless you are an expert and have a good dataset).

jensgk 2y ago

> I wonder what would have happened if we poured the same amount of money, talent and hardware into SVMs, random forests, KNN, etc. From my perspective, that is actually what happened between the mid-90s to 2015. Neural netowrks were dead in that period, but any other ML method was very, very hot.

netdevnet 2y ago

You are supposed to call it AI now. The word "machine learning" is for GOFAI 2nd gen only. Once all investors have been money drained and the next AI winter begins, then you will be allowed to call it Machine Learning