4 ms·

I’m tired of Hinton over promising. It’s not going to do everything and he’s promoting an irresponsible position. Has there really been a huge breakthrough sin

I’m tired of Hinton over promising. It’s not going to do everything and he’s promoting an irresponsible position. Has there really been a huge breakthrough since the initial wave of CNNs? I guess transformers/attention, but I don’t consider GPT-3 to solve any problem at all.

brutus1213 6y ago
I'm generally skeptical on research and work in the field. However, the progress I've seen on GANs/deep fakes, natural language understanding, multi-modal, video understanding/activity recognition .. it is unbelievable how fast the field is moving. This may not result in new widgets next CES but may be game changing to society in a decade. In academic research, GPT-3 and in general language models are applicable to a host of downstream tasks across different topics. Edit .. reworded since it seemed insulting (unintentional).
- Filligree 6y ago
  Having spent the last two months writing a novel, with GPT-3's help, I take issue with the claim that it's useless. It's anything but useless. Sure, I do 80-85% of the work, but that remaining 15% is the portion that would otherwise have me stuck for days. In practice, it's a 10-50x improvement in writing speed.
  - a_wild_dandan 6y ago
    What's your process for using GPT-3 to aid your writing?
    - Filligree 6y ago
      See sibling comment.
  - the_jeremy 6y ago
    Can you elaborate on how you use it? Do you have it generate next paragraphs when you're stuck, or something else?
    - Filligree 6y ago
      I've elaborated significantly here: https://forums.sufficientvelocity.com/threads/generic-pawn-templates-a-gpt-3-quest.70063/ https://forums.sufficientvelocity.com/threads/generic-pawn-t... If that doesn't answer your questions, feel free to ask some in the thread. I don't check comment replies here very often. (Note: The first ten-or-so updates were purely educational, and it's only the last few where I've been trying for high quality text. If you just want to see how well that can work, then start at the end.)
- pdimitar 6y ago
  > This may not result in new widgets next CES but may be game changing to society in a decade. And I am tired of hearing that ever since the 90s. Clearly, deep fakes changed society. Made people more paranoid and less believing of what they see. Is that a positive change though? Meanwhile, you still have to manually load dishes in the dishwasher, and robot vacuum cleaners stumble and block on the easiest of obstacles. Call me a cynic but AI work needs to get out of its comfy well-funded bro-club corner and start to seriously try and solve real-actual-physical-world problems. It's all well and good that scientist X can model a text paragraph with this or that NN but when will he solve a car driving itself? And don't get me wrong. I know science can take a long time to make a breakthrough. It's not adhering to the same laws as everyday work -- I am aware and I am clamouring for science. But I don't feel the AI area is even heading in any direction at all. Could be wrong though.
  - bonoboTP 6y ago
    There is a lot of work going on in many directions. Believe me, whenever the general public notices some "obvious" shortcomings or holes or missing research, there is usually already years of effort in the making and heaps of papers exploring it. It's just that there is a huge delay to media and popular consciousness. If you just read the "AI" comment sections on HN it may seem everybody is just training a cat/dog classifier in Keras on ImageNet and nothing new is going on. In reality there's tons of work in one-shot/few-shot learning, unsupervised, self-supervised methods, combining modalities, quantifying uncertainty, robustness to attacks, etc. Yes, robotics is still not at the stage where you could get a human-like dexterous dishwashing/T-shirt folding machine. But people are working on it as well. However, not all companies need it. The "comfy well-funded bro-club corner" (why so much envy in the phrasing?) probably does not need physical robots, they just do data analysis, like if you need to process videos uploaded to Youtube, or you program an intelligent tool for photo or video editing software, that's also important and has nothing to do with robotic arms. I think it's the fault of the media that "AI" appears as this single conceptual blur, when it's actually tons of different applications. It is capable of a lot more than 1-2 decades ago. But it's not AGI, and not everyone needs to work on human-like/conversational AGI in a humanoid body. Image analysis, machine translation, speech recognition, all these work today at least to some extent and just a short time ago they just did not work at all outside extremely carefully crafted cases in prototypes that fell apart immediately when the researcher wasn't there to keep it from collapsing (I mean this metaphorically: the predictions were extremely bad outside the scenario and dataset it was crafted for).
- JHonaker 6y ago
  I mean, don’t just take my word for it. Here’s Yann LeCun [1]. I will concede that I forgot about DeepFakes. How’d I do that? But still, cool now we can live in a dystopian future. The point that I was trying to make still stands. Assuming that more data and more compute will solve all of our problems is a ridiculous position. The field might be moving incredibly fast, but most of the work I see is not great. Everyone is hyped. People claiming that we can replace all of statistics and machine learning with pure deep learning models forget about how hard it is to actually get these things working in the first place. Do you have 100 TPUs to run full blast for a week? I mean consider the series of Nature papers a few years ago. Google says, “I can predict earthquakes using NNs!” Somebody comes along and says, “I thought about the problem for a little bit and beat you with logistic regression.” Self-driving cars have been just around the corner for the last 10 years. Just like with the rest of stats and ML, it’s not the place you have a lot of data that’ll kill you. It’s the edge cases. [1] https://m.facebook.com/yann.lecun/posts/10157253205637143?notif_id=1603803722095314&notif_t=story_reshare&ref=notif https://m.facebook.com/yann.lecun/posts/10157253205637143?no...
  - bonoboTP 6y ago
    Be careful, the narrative that you need tons of hardware to do anything is pushed by companies selling licenses to cloud APIs and cloud training platforms. In reality people routinely publish new results at AI conferences while only having access to a few GPUs. Not everything is about hardware. There are thousands and thousands of AI papers. Not all of them are about scaling up some model like GPT-3. Yes, there is a lot of hype, a lot of misuse of ML by people who don't understand it. Yes, in some cases it makes no sense to use a deep neural net if logistic regression (which is a single-layer NN) will work. This is generally the case when your features are already very processed, curated and meaningful as in your earthquake example. But you will never solve e.g. image or speech understanding with logistic regression. Perception on high-dimensional, complex, messy real-world signals is where deep nets excel. And they have unparalleled performance there, they blow everything out of the water. It's hard to overemphasize this. The last decade has turned many applications from does-not-work into works. A zero-one transition. Not 10% better, but it was hopeless and now it is "quite nice". Still not divine perfection, but it's far from just being hype. It can be hard for outsiders to appreciate this, but before 2010 or so, everything in computer vision was very fragile for real-world applicability. The algorithms only worked in one specific setting, like one particular assembly line, with all parameters hand tuned in the pipeline. You needed a lot of experience and intuition to configure your feature extraction pipeline and the regression method and it was still very fragile. Much of research was only demonstrated on toy examples, synthetic data, or only on a few images. The shift to huge datasets is also something from the last 10-15 years. Before that people cherry-picked example, where their hand-crafted algorithm with hand-tuned parameters finally produced something resembling an okay result. I'm only exaggerating a little. These things really didn't work. Many of the concepts were already in place, but it still didn't work.
mcilai 6y ago
Overpromising? Underpromising, if anything. I cannot understand the point of view that denies the _overwhelming_ progress that has taken place in AI. And to deny GPT-3’s mega advance — we now have a system that can achieve state of the art or near state of the art requiring a negligible number of training cases (i.e., few shot learning) — seems wrong to me.

brutus1213 6y ago

I'm generally skeptical on research and work in the field. However, the progress I've seen on GANs/deep fakes, natural language understanding, multi-modal, video understanding/activity recognition .. it is unbelievable how fast the field is moving. This may not result in new widgets next CES but may be game changing to society in a decade. In academic research, GPT-3 and in general language models are applicable to a host of downstream tasks across different topics. Edit .. reworded since it seemed insulting (unintentional).

Filligree 6y ago
Having spent the last two months writing a novel, with GPT-3's help, I take issue with the claim that it's useless. It's anything but useless. Sure, I do 80-85% of the work, but that remaining 15% is the portion that would otherwise have me stuck for days. In practice, it's a 10-50x improvement in writing speed.
- a_wild_dandan 6y ago
  What's your process for using GPT-3 to aid your writing?
  - Filligree 6y ago
    See sibling comment.
- the_jeremy 6y ago
  Can you elaborate on how you use it? Do you have it generate next paragraphs when you're stuck, or something else?
  - Filligree 6y ago
    I've elaborated significantly here: https://forums.sufficientvelocity.com/threads/generic-pawn-templates-a-gpt-3-quest.70063/ https://forums.sufficientvelocity.com/threads/generic-pawn-t... If that doesn't answer your questions, feel free to ask some in the thread. I don't check comment replies here very often. (Note: The first ten-or-so updates were purely educational, and it's only the last few where I've been trying for high quality text. If you just want to see how well that can work, then start at the end.)
pdimitar 6y ago
> This may not result in new widgets next CES but may be game changing to society in a decade. And I am tired of hearing that ever since the 90s. Clearly, deep fakes changed society. Made people more paranoid and less believing of what they see. Is that a positive change though? Meanwhile, you still have to manually load dishes in the dishwasher, and robot vacuum cleaners stumble and block on the easiest of obstacles. Call me a cynic but AI work needs to get out of its comfy well-funded bro-club corner and start to seriously try and solve real-actual-physical-world problems. It's all well and good that scientist X can model a text paragraph with this or that NN but when will he solve a car driving itself? And don't get me wrong. I know science can take a long time to make a breakthrough. It's not adhering to the same laws as everyday work -- I am aware and I am clamouring for science. But I don't feel the AI area is even heading in any direction at all. Could be wrong though.
- bonoboTP 6y ago
  There is a lot of work going on in many directions. Believe me, whenever the general public notices some "obvious" shortcomings or holes or missing research, there is usually already years of effort in the making and heaps of papers exploring it. It's just that there is a huge delay to media and popular consciousness. If you just read the "AI" comment sections on HN it may seem everybody is just training a cat/dog classifier in Keras on ImageNet and nothing new is going on. In reality there's tons of work in one-shot/few-shot learning, unsupervised, self-supervised methods, combining modalities, quantifying uncertainty, robustness to attacks, etc. Yes, robotics is still not at the stage where you could get a human-like dexterous dishwashing/T-shirt folding machine. But people are working on it as well. However, not all companies need it. The "comfy well-funded bro-club corner" (why so much envy in the phrasing?) probably does not need physical robots, they just do data analysis, like if you need to process videos uploaded to Youtube, or you program an intelligent tool for photo or video editing software, that's also important and has nothing to do with robotic arms. I think it's the fault of the media that "AI" appears as this single conceptual blur, when it's actually tons of different applications. It is capable of a lot more than 1-2 decades ago. But it's not AGI, and not everyone needs to work on human-like/conversational AGI in a humanoid body. Image analysis, machine translation, speech recognition, all these work today at least to some extent and just a short time ago they just did not work at all outside extremely carefully crafted cases in prototypes that fell apart immediately when the researcher wasn't there to keep it from collapsing (I mean this metaphorically: the predictions were extremely bad outside the scenario and dataset it was crafted for).
JHonaker 6y ago
I mean, don’t just take my word for it. Here’s Yann LeCun [1]. I will concede that I forgot about DeepFakes. How’d I do that? But still, cool now we can live in a dystopian future. The point that I was trying to make still stands. Assuming that more data and more compute will solve all of our problems is a ridiculous position. The field might be moving incredibly fast, but most of the work I see is not great. Everyone is hyped. People claiming that we can replace all of statistics and machine learning with pure deep learning models forget about how hard it is to actually get these things working in the first place. Do you have 100 TPUs to run full blast for a week? I mean consider the series of Nature papers a few years ago. Google says, “I can predict earthquakes using NNs!” Somebody comes along and says, “I thought about the problem for a little bit and beat you with logistic regression.” Self-driving cars have been just around the corner for the last 10 years. Just like with the rest of stats and ML, it’s not the place you have a lot of data that’ll kill you. It’s the edge cases. [1] https://m.facebook.com/yann.lecun/posts/10157253205637143?notif_id=1603803722095314&notif_t=story_reshare&ref=notif https://m.facebook.com/yann.lecun/posts/10157253205637143?no...
- bonoboTP 6y ago
  Be careful, the narrative that you need tons of hardware to do anything is pushed by companies selling licenses to cloud APIs and cloud training platforms. In reality people routinely publish new results at AI conferences while only having access to a few GPUs. Not everything is about hardware. There are thousands and thousands of AI papers. Not all of them are about scaling up some model like GPT-3. Yes, there is a lot of hype, a lot of misuse of ML by people who don't understand it. Yes, in some cases it makes no sense to use a deep neural net if logistic regression (which is a single-layer NN) will work. This is generally the case when your features are already very processed, curated and meaningful as in your earthquake example. But you will never solve e.g. image or speech understanding with logistic regression. Perception on high-dimensional, complex, messy real-world signals is where deep nets excel. And they have unparalleled performance there, they blow everything out of the water. It's hard to overemphasize this. The last decade has turned many applications from does-not-work into works. A zero-one transition. Not 10% better, but it was hopeless and now it is "quite nice". Still not divine perfection, but it's far from just being hype. It can be hard for outsiders to appreciate this, but before 2010 or so, everything in computer vision was very fragile for real-world applicability. The algorithms only worked in one specific setting, like one particular assembly line, with all parameters hand tuned in the pipeline. You needed a lot of experience and intuition to configure your feature extraction pipeline and the regression method and it was still very fragile. Much of research was only demonstrated on toy examples, synthetic data, or only on a few images. The shift to huge datasets is also something from the last 10-15 years. Before that people cherry-picked example, where their hand-crafted algorithm with hand-tuned parameters finally produced something resembling an okay result. I'm only exaggerating a little. These things really didn't work. Many of the concepts were already in place, but it still didn't work.

mcilai 6y ago

Overpromising? Underpromising, if anything. I cannot understand the point of view that denies the _overwhelming_ progress that has taken place in AI. And to deny GPT-3’s mega advance — we now have a system that can achieve state of the art or near state of the art requiring a negligible number of training cases (i.e., few shot learning) — seems wrong to me.