5 ms·
We're in an early and explosive growth stage of AI where well-established statistical knowledge is having an unreasonable effect when combined with computing po
by roenxi 7y ago
We're in an early and explosive growth stage of AI where well-established statistical knowledge is having an unreasonable effect when combined with computing power. I've yet to see any AI platform that is mindbending vs doing basic math with a multivariate normal distribution. The eyewatering stuff is the number of Hz of computing power being thrown into simulating Go games, etc, etc.
Assuming that among 1.4 billion people there are a few good coder/statistician people and using supercomputers [0] as a rough proxy for available computing power, it isn't obvious the US is even going to inconvenience the Chinese military. Presumably they are going to have a parallel AI effort anyway given that they have been investing in the area.
[0] https://en.wikipedia.org/wiki/TOP500#Top_countries https://en.wikipedia.org/wiki/TOP500#Top_countries
- toxik 7y agoNone of the convnets are Gaussian. It was one of the big reasons convnets came to be at all, to model highly kurtotic distributions like natural images. Chinese convnet research is absolutely state of the art already, indeed.
- yorwba 7y agoYou can do basic math to a multivariate normal distribution to approximate other distributions. In classic feedforward networks, that's achieved by using nonlinear activation functions. Very wide networks turn into Gaussian processes in the limit [0]. There are also approaches explicitly embracing the "basic math on multivariate normal distributions" framework using normalizing flows [1]. [0] https://arxiv.org/abs/1711.00165 https://arxiv.org/abs/1711.00165 [1] https://arxiv.org/abs/1505.05770 https://arxiv.org/abs/1505.05770
- jahewson 7y agoI feel like this is kinda cheating though - “Gaussian in the limit” is not the same as “Gaussian”.
- sgt101 7y agoI'd be interested in how you would do NLP tasks such as those now done with transformers like gtp2 and Bert by working with multi variate normal distributions.
- roenxi 7y agoMaybe I didn't make myself clear enough. There was a time when a software engineer didn't get basic stuff. A time when languages like C were developed where there wasn't an associative data structure baked in for example. It wasn't because associative data structures are a secret tech that requires great insight to uncover. It is because the field was new and people hadn't cottoned on to how basic and important having access to hash-maps is. Times moved on. Now basically all modern languages have hash-maps as a basic data type. 'AI' is in that early phase where the engineering world is still getting excited over stuff that will basic practice eventually. BERT and GTP2 are signs of how much computer power Google/etc's researchers have access to, not signs that the architectures are fundamentally complicated or somehow hard to work out if you live in China. AlphaGo for instance was breathtaking as a standalone project, but not hard to implement.
- sgt101 7y agoI spent at least five years trying to use statistical ml and mlps to do NLP on social media comments from about 2003. Nothing like a transformer (or an rnn even) occurred to me. I have a belief that someone in the USSR worked out a way of doing fluid dynamics that has enabled the Russians to develop hypersonics and super cavitation. This is probably rather straightforward - in the style of NS - of you know the principles. No one in the West ( or China) knows those principles, so Western torpedos and reentry vehicles are rather poor Vs Russian ones. Once you grasp how something works the fact that it's rather easy to apply in comparison to the process of getting the insight shouldn't detract from the value of the insight .
- roenxi 7y agoThe wiki page on RNN says the early groundwork was done in 1986 and the LSTM was a 1997 innovation. If they didn't occur to you in 2003 that doesn't imply much, they are not suprising concepts. The surprise was that in the mid-2000s suddenly GPU became so powerful that LSTM could be used to achieve interesting results. The story here isn't the models, it is the computers running the models.
- mihaaly 7y agoI sense that they will hurt themselves more with the imposed additional bureaucracy, hindering business and collaboration - the latter being a key aspect in efficient research, not to mention multinational companies already living on the area. Also, how effective could this regulation be with so much knowledge and open source code already disseminated in the field?
- Maro 7y ago"I've yet to see any AI platform that is mindbending" 1. Siri / Alexa and similar for voice recognition and doing basic tasks. 2. Face recognition for uploaded pictures on Facebook. 3. Lots of people use FaceID on iPhones. 4. Tesla and other SDC systems: yes, it's not good enough for general use, but the fact that it mostly works in California is pretty cool.. "Mindbending" is subjective, but these all use DNNs, and are used by millions to billions of people every day. So it's incorrect to suggest that all DNN use-cases could be replaced with "doing basic math with a multivariate normal distribution".
- arpa 7y ago1, 2 and 3 are absolutely mundane. One could argue that if anything, they are disappointing: i remember passable speech recognition running on pentium 2 with 32 megs of ram (dragon naturally speaking was first released in 1997). "Natural" language parsers were around starting with the first text adventures (80s as far as I remember). Considering the computing resources big G and big A have, the performance of this "AI" is mediocre at best. Facial feature extraction is not rocket science either, it dates back to at least 1993, so 386/486 with 4-8 megs of RAM. You want mindbending and scary? Mindbending: deepfakes. Scary: automated ai-based law enforcement.
- useful 7y agoimo, working memory at scale via improvements to more obscure methods like a DNC is what will be the next leap and make todays 'AI' seem mundane. Most AI today like deepfakes involve giving the model all memories available via input.
- arpa 7y agoScaling SHRDLU in a similar manner would also rock, though.
- pp34 7y agoThey are Dumb and they are just Reacting out of some "fuck we have to do something" instinct driven by jobless fucks like Peter Thiel/Graham Allison/Kai-Fu Lee constant rhetoric about AI and falling behind and how its going to effect everyone. This is Fear based Decision Making 101. All it leads to is more absurd outcomes such as Endless Wars, Huge Monopolies, More consolidation of power and resources in the hand of few therefore more inequality. These people and this thinking style would have more credibility if they had stopped Wars, reduced inequality, disrupted monopoly and oligopolies. They have not done that. The can't imagine a Chinese AI team and American AI team working together to solve problems in humane way. They can't imagine constructing orgs that push that through. They can't imagine punishing their own who cross lines out of fear that the other side wont. When we allow Fear based thinking to dominate decision making Imagination dies. Outcomes are consistently shit. And way below the potential of what people collaborating and communicating across artificial bullshit boundaries are capable off. Pick a side and don't back Fear based Decsion makers in your org. These guys hold back progress, are the reason climate change research is hidden, endless Wars keep getting funded and monopolies cling to power way past their expiry date. How can it be the age of information and knowledge when fear wins?
- hanniabu 7y ago> All it leads to is more absurd outcomes such as Endless Wars, Huge Monopolies, More consolidation of power and resources in the hand of few therefore more inequality This is going to persist regardless of this decision
- DataWorker 7y agoThe thing you fear most is fear itself. But you shouldn’t. Fear is human and probably not going to stop being a thing any time soon, despite your demands.