14 ms·
The Great AI Reckoning
- hwers 5y agoI used to be a huge sceptic of this cycle of deep learning. But not anymore. GANs are incredible and there's a huge amount of low hanging fruit left to pluck. I definitely don't believe AI will repeat it's history (history doesn't have to cycle).
- bilater 5y agoCan you elaborate on some of the use cases of GANs? I'm particularly interested in generative content but the output has been pretty meh so far.
- hwers 5y agoYeah I was totally anti hype when everyone was excited about style transfer and stuff like that. I found most of the VQGAN+CLIP way overhyped (sure it's abstract and funky but you get bored of that look after 10 images - it's like that deep dream filter from 5 years ago, shouldn't we have progressed more by now) but lately CLIP guided diffusion has blown me away. I literally thought it was all deception and cherry picking until I tried it myself[1] and realized the amazing outputs from https://twitter.com/RiversHaveWings https://twitter.com/RiversHaveWings are real. [1] https://colab.research.google.com/drive/1QBsaDAZv8np29FPbvjffbE1eytoJcsgA https://colab.research.google.com/drive/1QBsaDAZv8np29FPbvjf...
- user_named 5y agoWhat do you mean by real? The images clearly exist because you can see them.
- kragen 5y agoThey mean "not cherry-picked or otherwise faked"; that is, that the images shown on that Twitter account are really typical of the images produced by CLIP guided diffusion, not, say, hand-painted.
- bilater 5y agoYeah it's cool but I agree it's still not quite mind blowing or even very useful. Same with GPT-3. Hopefully we get there soon.
- areoform 5y agoSomething that I'm fascinated with is the question, would other models show equivalent results if given the same amount of compute? I looked through the list of solvers for the protein folding challenges and there were other deep learning, neural network, and classical machine learning approaches on there. Even some hybrid ones! But none of the participants had even a fraction of the compute power that AlphaFold had behind it. Some of the entries were small university teams. Others were powered by the computers some professor had in their closet (!). Most of the teams were dramatically under-powered as compared to AlphaFold. How much did this influence the final result? What would the other results look like if they'd been on equal footing? Would they have been closer? It's a genuine question.
- WithinReason 5y agoThe advantage of DeepMind from more compute resources is cumulative, you can develop better models if you have more compute. So the question really is: If academia and other research institutions had DeepMind's resources, would their output match DeepMind's?
- gradys 5y agoProbably not. It's not clear how to scale other methods to make use of so much compute. They top out sooner, with fewer parameters and less compute. One way to look at why deep learning is having the impact it does is that unlike other ML methods, it's actually capable of making use of so much compute. It gives us modular ways to add more and more parameters and still fit them effectively.
- WanderPanda 5y agoWhat always astonishes me that deep learning seems to work on human timescales. For other problems (even if they are polynomial in complexity) we get into infeasible runtimes when we increase the problems complexity by 10x. With deep learning the fuzzy, approximative nature seems to help to grasp the gist of the 10x problem and somehow allows us to reach 95% of the solution in just e.g. 2x the runtime. Heuristics might play in the same league, but the development time kind of scales with the problem, while in deep learning I would put it in the linear or log basket.
- m_ke 5y agoThe recent advances in deep learning are good enough to keep industry busy for the next 20 years. Each year has brought us substantial improvements and if you look past the popular benchmarks there's no sign of things slowing down.
- antupis 5y agoI think it is that we are going to hit soon local maximum where modern NN architectures with better vertical integration and better hardware are going to create lots of value but other hand doesn't bring use anywhere near AGI.
- m_ke 5y agoNo sane person believed that deep learning would get us to AGI withing the next few decades. Anyone saying so is doing it to fool VCs out of their money.
- short_sells_poo 5y agoI'd like to agree with you but just 1 year ago every AI related thread here were chock full of people breathlessly predicting AGI within years... and presumably HN is a well informed community! It's often not about expertise or sanity, but AGI captures people's imagination to a huge extent and as evolution has conditioned us to do, we ascribe certain behaviors and desires to systems which only exhibit those by random chance.
- m_ke 5y agoSorry to break it to you but most of HN is not well informed. Look for some threads in your area of expertise for proof of that. Only people selling AGI are marketers and futurists.
- 0xFACEFEED 5y agoAs a layman I wonder when we're going to start bearing the fruit of all this effort in DL. In the average person's day to day life I mean. I don't rely on DL driven systems and I'm not even a skeptic. I want it to work. But I can't rely on these systems in the same way that I rely on my computer/phone, a light bulb, or a refrigerator. Is that ever going to change?
- deleted 5y ago[deleted]
- cowanon22 5y agoThe truth is that artificial intelligence and machine learning today are still aspirational titles - the current tech simple does not learn or reason in any way that is similar to human or even biological reasoning. Data mining and probabilistic pattern recognition are much more accurate descriptions, but don't sound as exciting. It's definitely possible that true AI will one day exist, but it may be anywhere from 5 to 1000 years away. I suspect the current approaches will not resemble the final form when it comes to AI.
- debacle 5y agoI think the calculus was that you didn't need "true" AI for a self-driving car. That still might be accurate, just maybe not in the near term. It may be controversial, but I think that humanity's hubris is the biggest barrier towards developing more effective AI.
- HenryKissinger 5y agoThe future is littered with The Onion headlines. "AI researchers drop silicon, turn to biological computers" "AI researchers discover AI biological computers not so artificial after all" "AI researchers to allow AI biological computers to grow neurons and grey matter" "AI researchers realize AI biological computers need oxygen and organic nutrients, not electricity" "Shortages of neurology textbooks as AI researchers switching careers to neurologists" "AI researchers turn to live human farms to grow brains" "AI researchers realize brains in a vat don't work, need sensory inputs, to farm human heads instead of just the brain" "AI researchers realize heads in a vat don't work, need the rest of the body" "'Why spend billions on cutting edge quantum processing units when a normal human will do the job?' asks machine learning pioneer" "AI researchers give up, say human brains are better than supercomputers for a fraction of the cost" "AI research community spent $635,920 billion and 537 years, has produced a human, says GoogleSoft-Zon report" The end product of AI research will be Homo Sapiens.
- dr1337 5y agoDon't be surprised in the coming months to read 6 out of your 11 Onion headlines coming out as real scientific work.
- jasonsync 5y agoWe once utilized an ad agency that relied heavily on "AI" to pre-test ad creative. The agency presented regional branding campaign creative with our national flag flying at half mast. The AI predicted success. The ad agency stood by the AI. Certainly would have generated clicks. But the ad agency lost a customer. Not sure if the AI would have predicted that!
- monkeydust 5y agoThere's so much BS around use if AI/ML in the enterprise space right now. It's totally self perpetuating. My sales team keep asking me to add AI to X product. Doesn't matter if it's not required, or even makes sense they ask because the competition is doing it and they get asked by our clients/prospects in an expecting tone, if we offer some AI on our products. There are places we do offer it where it genuinely adds value but this 'sprinkle AI on everything for the sake of it' needs to die.
- fzingle 5y ago> My sales team keep asking me to add AI to X product. Doesn't matter if it's not required, or even makes sense they ask because the competition is doing it and they get asked by our clients/prospects in an expecting tone, if we offer some AI on our products. It could be worse, they might be asking you to put it on a blockchain :P
- supperburg 5y agoImagine every program that fits within 10TB. It’s a finite number of programs. Inside there is GTP-3, 2 and 1. Every GAN. Tesla FSD. It’s everything. What this entire industry is really about at its core is finding programs in that set that behave in intelligent ways. That’s all. Recently we’ve had success in letting the computer dig through programs automatically until it finds a good one. So overnight we went from testing maybe hundreds per year to billions per month or something like that. So obviously things have been getting weird. Imagine how many we’ve dug up so far out of the total set. It’s an infinitesimal percentage. We haven’t even scratched the surface. Imagine the set. What’s inside? Is there something inside that we will regret?
- metalliqaz 5y ago>Is there something inside that we will regret? Unambiguously yes. 10TB is a lot of space. That's large enough for a program that does nothing but show 8 hours of HD footage of your kids being tortured and eaten. It's also more than enough to define a program that would reliably precipitate a global thermonuclear war if connected to the Internet.
- jeremysalwen 5y agoThat's a very interesting way of thinking about things, but I think unfortunately the search techniques we are using (gradient descent with certain datasets/self-supervision tasks) limit us to exploring a very small subset of that 10TB of possible programs. A search strategy that could actually find the optimal 10TB program to accomplish some task would actually be a superintelligence beyond anything we have ever thought of creating using present day AI techniques.
- 6gvONxR4sf7o 5y agoI'd bet people outside the field are going to be disillusioned before long as the marketing BS falls apart. Even years ago, I was already getting questions from friends that surprised me in how detached their perceptions are from the reality of state of the art ML. Eventually, they'll realize that no, we're not building superhuman AI today or soon. However, there's a ton of cool shit happening and tons of stuff already powering industry $$$. Some of this works because of mismatched expectations, like companies who are trying to sell magic. They will fail as their customers stop buying their products that fail spectacularly 2% of the time, but really can't fail 2% of the time. But there's tons of room in the market for stuff that fails spectacularly 2% of the time. For that reason, I don't see a "real" reckoning coming. People's expectations are too high, sure. But the reality is still Good Enough™ for a lot of problems and getting better.
- rscho 5y agoThe main issue is that people have unrealistic expectations in domains that are personally relevant, medicine being an iconic example. So, there there might still be a 'reckoning' of sorts, even if Walmart applies AI successfully behind the scenes to power non-critical user-facing stuff.
- 6gvONxR4sf7o 5y agoOh for sure. I guess I could have worded it better (not just scare quoted "real"), but it seems like a public expectations reckoning will happen, but not a reckoning that really affects practitioners who aren't 'script-kiddies' for lack of a better term. (side note, does anyone know a less insulting way to refer to the group people call 'script-kiddies' that still gets the same point across?)
- CityOfThrowaway 5y agoTo your side note: “technicians” is a reasonable label here. They know how to use a technique, but aren’t themselves capable of creating one.
- 5y ago
- woopwoop 5y agoAll of their arguments for why deep learning is reaching its limits seem to me to be arguments for why its future is bright. The fact that we have no even halfway plausible theory of why deep models generalize well, or why training them via sgd works, or even what the effect of depth is in a neural network, is exciting, not discouraging. I do not believe these are impossible problems to solve (well maybe the generalization one, but even there surely more can be said than the current state of knowledge which is basically a shrug). And even partial solutions should yield big practical steps forward.
- m_ke 5y agosome good insights on depth: https://arxiv.org/abs/2106.09647 https://arxiv.org/abs/2106.09647
- lifekaizen 5y agoSeems to assume carbon emissions from electricity will stay constant, yet that is likely to fall which reduces the impact in one dimension (cost remains).[0][1] [0]Diminishing Returns https://spectrum.ieee.org/deep-learning-computational-cost https://spectrum.ieee.org/deep-learning-computational-cost [1] renewables and non-renewables in $.05~.15/kWh range https://www.forbes.com/sites/dominicdudley/2019/05/29/renewable-energy-costs-tumble https://www.forbes.com/sites/dominicdudley/2019/05/29/renewa...
- version_five 5y agoCurrently, the cost of ML R&D has a floor based on what advertising companies are willing to pay people to work in the space. This actually has a huge upside, as most of the tools that have made ML so accessible (pytorch, tensorflow, all the research advances) are coming out of these companies. But it has the downside that if I want to get someone to work on my ML problem I have to compete with what google and fb can pay. A consequence, I guess, is that there are lots of unexplored / underexplored problems waiting to be tackled, and there are tools around that can make it happen. If there is a reckoning in the advertising space, there will be lots of other applications to focus on.
- airstrike 5y agoWell, one issue is that Google and FB can finance said research because they make so much money off ads to begin with, and by doing so essentially provide enough funding to keep the field going So this reckoning in the advertising space would only be a net positive for society if others came in to fill that funding gap and threw enough money at researchers to keep the field afloat in a similar fashion
- carom 5y agoAm I the only one who just sees links to other articles? Checked on desktop and mobile.
- pineconewarrior 5y agoSame, even with content blockers off!
- haihaibye 5y agoIt's a special report issue, with lots of articles on the topic of AI
- photochemsyn 5y agoPersonally, I'm won't think of it as 'artificial intelligence' until an AI system can teach me to speak a human language as well as a native human speaker of that language could. Seems a long ways off. All it appears to be now is a collection of very sophisticated pattern matching algorithms that build their own opaque internal sorting/matching/classification algorithms based on the data they're trained on, as I understand it. This is of course incredibly useful in some domains, but it's not really 'intelligence'. And, they can't do math very well: > "For example, Hendrycks and his colleagues trained an AI on hundreds of thousands of math problems with step-by-step solutions. However, when tested on 12,500 problems from high school math competitions, "it only got something like 5 percent accuracy," he says. In comparison, a three-time International Mathematical Olympiad gold medalist attained 90 percent success on such problems." 1. https://spectrum.ieee.org/ai-failures https://spectrum.ieee.org/ai-failures
- quotemstr 5y ago> AI system can teach me to speak a human language as well as a native human speaker of that language could. Seems a long ways off. And when such a system appears, you'll claim that it's still not AI --- just a fancy pattern matching trick --- and say that it's not real AI until some other arbitrary benchmark is met. "AI" is just what machine learning can't quite do yet. > collection of very sophisticated pattern matching algorithms What do you think human brains are? Humans are Turing machines as well --- all physical computers are. We process inputs, match them against internal state, and generate outputs. You can't criticize AI on the grounds that it's "pattern matching": everything is pattern matching. You can't look at the output of GPT-3 and tell me that it's some kind of dumb regular expression here. You just can't. https://www.gwern.net/GPT-3 https://www.gwern.net/GPT-3
- neatze 5y agoExcept, human brain changes on second by second bases, it is highly distributed and concurrent system with unimaginable redundancy, furthermore brain capability of learning/adaptability from just from one or few examples/tries does not compare to any AI models; what aspect of GPT-3 you comparing to human brain, specifically ?
- varelse 5y agoThere are still plenty of targets to hit with unitasking AI like we have that could each have their imagenet moment. But there is a shortage of domain experts who understand both those targets and AI. That creates opportunity for domain experts who learn AI. Less so the other way around because these domains are generally more complicated than AI and lack the unending tsunami of online courses to learn the details.
- gumby 5y agoSadly these days the word for “program” is “AI”
- kragen 5y agoWe're back to "Giant Electronic Brains"!
- gibsonf1 5y agoRight now Machine Learning with "neural" networks is all A and no I.
- sabhiram 5y agoWe get better and better at building pattern matchers that carefully oscillate between overfit and under-parametrized. Funny how we spend so much time and effort to mimic something that grows free by the billions :)
- worik 5y agoI am very disappointed that I do not have a self driving car. Looks like I never will get one. Too hard, it is.
- Veedrac 5y agoThe article “An Inconvenient Truth About AI > AI won't surpass human intelligence anytime soon” doesn't even once mention or justify the claim in its title. The article “Deep Learning's Diminishing Returns” was discussed a week ago, here: https://news.ycombinator.com/item?id=28646256 https://news.ycombinator.com/item?id=28646256.
- dr_dshiv 5y agoSo long as researchers are willing to reify AI as a thing, the collective delusion will continue. I'm specifically looking at researchers focused on the ethics of AI; they set up AI as a thing more than other researchers. AI is not a thing. It's a field of research. Know thyself.
- qualudeheart 5y agoAI would be more effective were liner algebra libraries to be rewritten in Rust. The performance would increase by half an order of magnitude and the AI safety problem would be solved by use of the borrow checker.
- lumost 5y agoMy two cents, over the last decade I’ve observed a hype cycle where DS type roles become more prominent than software in organizations followed by a decline. Businesses pay for results, applying 1000 slightly different variations of the same techniques on the same dataset produces very little return. Businesses take note, don’t give raises etc and the DS team fades. Eventually someone tries the new hot thing like Deep Nets and sees a large gain in a core business metric with relatively little effort. As every team is going through the same hype cycle DS salaries spike.