8 ms·

People are vastly underestimating the changes that are about to come from NLP. The basic ideas of how to get language models working are just about in place. Tr

People are vastly underestimating the changes that are about to come from NLP. The basic ideas of how to get language models working are just about in place. Transformer networks, and recent innovations like GPT-2, googles reformer model, etc are precursors to the real machine learning boom. Machine learning as we have known it, has been stuck as an optimization tool, and used for computer vision here and there. NLP, and with it, the ability to create, synthesize, and understand content, will change the internet. More than that, I think NLP will unlock new ways of interacting with computers. Computers will be able to handle the ambiguity of human language, transcending their rigid “only do exactly what you tell them” models of the world. Edit: Adding this to give more technical context. I think most people don’t know where the line is currently between what possible, and what’s not, but also what we are on the cusp of. And we are on the cusp of a lot. A quick explanation of one area is here: Basically, transformer models are the best for NLP. They use something called attention based mechanisms, which allows the model to draw correlations between pieces of text/tokens that are far apart. The issue is that this is an O(n^2) operation. So the model is bounded by the context window, which is currently mostly at 512 tokens, and is thus, bounded in how much it can understand. Recent innovations, and further study, will broaden the context window, and thus unlock better reading comprehension and context understanding. For instance, the ability to answer a question using a piece of text is mostly stuck at just finding one paragraph. The future will see models that can find multiple different paragraphs, understand how they relate, pull the relevant information, and synthesize it. This sounds like a minor step forwards, but its important. This will unlock better conversational abilities, but also, better ways to understand how different pieces of textual information relate. The scattershot of information across the internet can go away. Computers can better understand context to act on human intention through language, unlocking the ability to handle ambiguity. This will change the internet. Again to empathize, these models only started showing up in 2017! The progress has been rapid.

gzu 7y ago
I can't wait for the day we see "deep-dream" styled literary works.
- jandrese 7y ago
  There have been plenty of "AI written" textual works, like the AI Dungeon for a D&D game, or the AI Recipe Generator that attempts to make something that looks like a cooking recipe. For the most part they aren't successful because the "AI" isn't smart enough to have a goal in mind, so they end up just monkey-cheesing everything. Pasting together snippets in ways that are usually grammatically correct but make no sense.
  - okigan 7y ago
    any books/papers that address "goal" oriented NLP or hybrid based discussion?
    - jandrese 7y ago
      The ability to set your own goals and task yourself to achieve them is the essence of AI. Not "AI" as we know it today, but Sci-fi AI where it's a machine person.
      - joe_the_user 7y ago
        Indeed, but that kind of describes the gulf between current language processing and present AI. Present AI generates tokens that seem to have meaning or seem like an appropriate response to a statement but where it become evident after 2-3 paragraphs, there's no substantial relation to either underlying meaning or underlying goals. Part of this is "underlying meaning" is an intuitive way to describe things but whatever is underlying here is more tenuous than a classical logic/GOFAI model of the world but more "solid" than a long, clever stream of associations.
      - sgt101 7y ago
        Is it? It's pretty clear that this kind of activity isn't that common in humans - mostly we pick up our goals from cues and drivers in our environment and society.
      - okigan 7y ago
K0SM0S 7y ago
If what you claim is true, it's not just the internet that will be changed by NLP but all of civilization!
m0zg 7y ago
I think you're the one who overestimates how much this will affect NLP. I'd say bulk of what was possible to deliver with this is already here, the subsequent changes will be incremental. The cold hard truth about statistical (and by extension, deep) NLP is that it's just a fancy way of counting numbers mostly. The only way to get to _real_ language understanding is AGI, and _nobody_ is working on that. You fundamentally cannot interact comfortably with a human if your system does not have probabilistic, contextualized cognition, and can't incorporate knowledge about the world.
- corporateslave5 7y ago
  I respectfully disagree. I’ve been in the field full time for a few years, I watch the state of the art closely. It’s hard to see the thought/theoretical progression of deep transformer models by just reading posts here and there. I’m not saying these NLP methods will be some kind of AI, just that they will produce products, content, and ways of interacting with the world that are categorically different from what we have seen in the past. For instance, question and answering tasks have only recently been able to: Find an answer in a text document that spans multiple non contiguous paragraphs Understand context across a whole book. The context window of current nlp is stuck at 512 tokens, mostly because of computational complexity. This has been broken just recently by the reformer model. Which is a primitive, early way to get around the computation costs of attention mechanisms. Just wait. The ideas are there. They just take time to refine.
  - skywhopper 7y ago
    Since you seem to be observing this closely, I'm curious what steps are being taken to identify and avoid bias in the results generated by these systems?
  - p1esk 7y ago
    What exactly do you expect from these Transformer-based models? This particular one is underwhelming because it provides a tiny improvement over the previous largest one (from Nvidia) with more than double the size. The fundamental limitation of these "optimization tools" as you call them - they don't have any common sense, and any way to query an external source of information (e.g. wikipedia), or ask a human to clarify. Another big problem is we don't have any way to do quality filtering on the outputs. From my experiments with GPT-2, it produces one interesting paragraph of text out of 20 - if you squint at it really hard. And most of those 20 don't make much sense at all. So no, the existing ideas are definitely not enough. Maybe some novel hybrid of symbolic AI with statistical optimization will lead to a breakthrough. This one does not strike me as anything other than "let's use moar weights!!"
lopmotr 7y ago
Can you explain your reasoning? It's easy to imagine a new invention or idea will revolutionize everything because it's never been done before and feels powerful, but even if it works, it might not. It sounds like you believe NLP will enable more general voice control of computers than Siri/Alexa/etc. But will that really be much more significant than people expect? Google is already pretty good at understanding ambiguous queries. It's eliminated the need to know where to look for information. Or are you talking about the computer writing code or doing business itself?
- codingslave 7y ago
  Replying on my laptop. Basically, transformer models are the best for NLP. They use something called attention based mechanisms, which allows the model to draw correlations between pieces of text/tokens that are far apart. The issue is that this is an O(n^2) operation. So the model is bounded by the context window, which is currently mostly at 512 tokens, and is thus, bounded in how much it can understand. Recent innovations, and further study, will broaden the context window, and thus unlock better reading comprehension and context understanding. For instance, the ability to answer a question using a piece of text is mostly stuck at just finding one paragraph. The future will see models that can find multiple different paragraphs, understand how they relate, pull the relevant information, and synthesize it. This sounds like a minor step forwards, but its important. This will unlock better conversational abilities, but also, better ways to understand how different pieces of textual information relate. The scattershot of information across the internet can go away. Computers can better understand context to act on human intention through language, unlocking the ability to handle ambiguity. This will change the internet.
  - skywhopper 7y ago
    If this is really where the researchers think these tools are headed (and I don't really doubt you on that point), then this is incredibly dangerous stuff. No matter how good your system is, the impact of implicit, unintentional, and non-targeted bias is huge on the sorts of content these systems will produce. But expose it to the levels of intentional manipulation present on the Internet of today, and these models don't stand a chance of producing something that safely does what you claim. I'm happy to be wrong about this, but I'm not seeing any discussion about the safety and security of using these systems. And if it's not even being discussed, we can be sure nothing's actually being done about it. Selling promises of active-agent computers interpreting human intent and summarizing information from the Internet without addressing this concern is irresponsible at this point.
skywhopper 7y ago
Can you explain more about what you mean by "able to handle the ambiguity of human language"? Such a claim goes far beyond language processing itself. Such a capability would have to involve cultural and interpersonal dynamics, domain knowledge, and definitive agency on the part of the computer. And all of _that_ is just to interpret sincere, honest attempts at communication. Can it safely and appropriately handle humor, irony, and sarcasm? What about coordinated malicious attacks?
heavyarms 7y ago
I would agree with you to a certain extent, but I still think there is a big missing component to make this a reality. Larger and more accurate general language models are great, but to enable use cases other than categorization, translation, summarization, etc., there will almost certainly have to a be contextualized knowledge graph layer. This is basically what I assume you mean when you say "the ability to create, synthesize, and understand content." The way I see it, transformer-based general language models will be the first and last step of a NLP system. I other words, they will do the raw processing of the input and will do the "take this output and put it in natural language based on this context" portion. Automating the part in the middle, the part that actually understands what the text means and can do logic with it, there's still no progress on that AFAIK. This is similar to how computer vision models work great in most cases, but to build a self-driving car you still need all the other components that do path planning, predicting what the cars around you will do based on the state of the environment, etc.
- codingslave 7y ago
  Replying from my laptop. I agree, there needs to be a way to represent relationships between information. I personally don't think knowledge graphs will be the ones to do it, not because they dont work, but because of how imperfect they are, in the data quality sense. See this paper here: "DIFFERENTIABLE REASONING OVER A VIRTUAL KNOWLEDGE BASE" https://openreview.net/pdf?id=SJxstlHFPH https://openreview.net/pdf?id=SJxstlHFPH Which is a recent effort, among many, by google research to build a model that can view a document as a knowledge graph, instead of explicitly tying pieces of the document to the graph, the idea to is to create a graph from the document. This is paper is a bit different from that, they do input a knowledge graph for training, but I think the idea and track of where they are headed has a ton of room to evolve. The trick is that transformer models have unlocked the ability to understand the text, so all of this "quasi knowledge graph extraction" that i was just explaining, is only recently possible! There's no research on it, because the baseline understanding of tokens has been too primitive. This is why there is so much room to grow, BERT has unlocked new methods, it can be used as a base for a ton of new NLP. Just to emphasize again, I'm not saying what I outlined above will be a good way to do it, just that ideas like this could only be tested recently. There's a million new ways to spin this problem.
the8472 7y ago
> More than that, I think NLP will unlock new ways of interacting with computers. Computers will be able to handle the ambiguity of human language, transcending their rigid “only do exactly what you tell them” models of the world. Such as google's duplex?
- codingslave 7y ago
  Yes, but duplex doesn't work and was too early.
deleted 7y ago
[deleted]
Tenoke 7y ago
> mostly at 512 token GPT uses 1024 tokens context window which does work out to a fair amount (given the massive vocab of 50k+ which means a token can be more than a word), though of course it's pretty limited. Google's recent Reformer[0] allows you to do attention much more cheaply and I'm currently training a Reformer model that isn't quite as big but has a context of ~64k tokens (though a much smaller vocab). I'm not completely sure if this is the solution but it looks like a step in that direction and so far the model is doing pretty good (I also plan to post the weights when I'm finished, though I am not sure if Google don't just plan to do that themselves). I am somewhat disappointed they went with just 1024 for this model, too though. 0. https://ai.googleblog.com/2020/01/reformer-efficient-transformer.html https://ai.googleblog.com/2020/01/reformer-efficient-transfo...
- corporateslave5 7y ago
  Yep, the number has been expanding. I just picked 512 because Bert uses that. But yeah 50k and beyond unlocks new ways of using context. I can’t wait to see how they optimize and improve the reformer model. I expect there to be an improvement like there was for BERT -> Albert
joe_the_user 7y ago
Computers will be able to handle the ambiguity of human language, transcending their rigid “only do exactly what you tell them” models of the world. So, are reasonable examples now of these models allowing semantic context? So, far, what I have seen is generated text where the lack of understanding takes three paragraphs to become obvious rather than one. Human language is this marvelous framework involving symbols associating with other symbols as well as to well-known and vaguely-guessed facts about the world. Human relations are very robust and, for example, two people can have a longish conversations where at the end, they realize they're talking about two different people (or different days or events). But in those circumstances, they can correct and adjust. "Solid" understanding is there but it's under a lot of layers of social cues and protocols and multiple meanings.
- mumblemumble 7y ago
  > So, are reasonable examples now of these models allowing semantic context? This is about where I am stuck. I'll start believing that we truly are on the cusp of a revolution as soon as I see Google Translate reliably knowing when to translate "home" into French as "domicile", "foyer", something those lines, or as "accueil." Right now it seems to very frequently choose "accueil", which is generally wrong, except when you're talking about websites and software user interfaces. That it's biased so strongly toward that error speaks volumes about how critical semantics are to sorting out natural language, and also about how bad current NLP systems are at dealing with semantics.
  - joe_the_user 7y ago
    Syntax and semantics were developed for human language, yet it's much easier to puzzle the difference in a computer language than in human language. With syntax and semantics so wrapped together, however, it kind of seems like you can go a long way with just capturing syntax, rhythm, word choice and etc. Which is to say the semantic side can be even worse than it seems, ie, nonexistent.
    - deleted 7y ago
      [deleted]
lostcolony 7y ago
>More than that, I think NLP will unlock new ways of interacting with computers. Computers will be able to handle the ambiguity of human language, transcending their rigid “only do exactly what you tell them” models of the world. As exciting as this sounds, I can't help but feel that given -we- haven't figured out how to handle the ambiguity of human language, I'm not convinced a computer attempting to is really markedly better for many use cases than requiring exactness. But operating at a human level of 'understanding', and being broadly accessible, may be enough to change the world. Hopefully for the better.
KKKKkkkk1 7y ago
Feels like I’ve heard this tune before. Actually, I think it was first played in the 1950s.
empath75 7y ago
I wonder if you could have like a higher order transformer — you first generate a series of prompts and then expand on the prompts one at a time.
Someone 7y ago
”People are vastly underestimating the changes that are about to come from NLP.” Maybe, maybe not. IMO, we don’t know what problem we have to solve to get what you describe. We also don’t have a metric as to how far we are along the path towards that goal (do we need 100B parameters? A trillion?), nor do we have any idea as to whether the current approach can get us there.
jaggednad 7y ago
I think you’re right that we are on the cusp or real natural language understanding. It’s an incredible moment. But we are also kinda far from these huge models being in every home because of the enormous amount of compute they require. Even running inference with these things is kinda expensive
tastroder 7y ago
> Basically, transformer models are the best for NLP. Yes, this year. While transformers certainly present a breakthrough in the NLP community and certainly stir up the state-of-the-art again, I don't really see how you go from that to the "computers will understand us" conclusion to be honest. People said that during the word2vec stir up and what we got out of that was incremental results (which is not bad, it's in the nature of things really). We can already build models that do everything you describe as single tasks, while that's exciting, it's not going to lead to the singularity. We've got a long way to go in terms of understanding models, making them computationally tractable, and making them do what we want in the first place without resorting to hoping that our unsupervised model learns something useful. It's likely that the attention mechanisms we see today will be a large part of that but I'm honestly a bit baffled at the "People are vastly underestimating the changes that are about to come from NLP." part. They're not, people already think that today's AI is magic, I don't think it is benefitial to reinforce that. Speech is nuanced, we're making good progress in many areas but we're not on the cusp of any revolutionary change in computational understanding really.
skocznymroczny 7y ago
Perhaps, but people overestimate the progress in AI also. Remember Google's demo of AI reserving a spot at a barber shop? Yeah... that never happened, even though it was supposed to be any day now.
dlkf 7y ago
> So the model is bounded by the context window, which is currently mostly at 512 tokens, and is thus, bounded in how much it can understand. This is begging the question as to whether the model "understands" anything at all. And once you adopt a definition of "understanding" that isn't equivalent to "got a high score on some pointless academic challenge" the answer is a resounding "no." The whole enterprise of AGI hype is based on this equivocation of words like "understanding" and "intelligence." We use a very restricted definition in proving that the tech is smart, and then switch out our restricted definition for the colloquial one when the audience isn't looking. > This will unlock better conversational abilities Shouldn't be hard given that as it stands there are none, except for creating a human-sounding slurry that is devoid of real content. > but also, better ways to understand how different pieces of textual information relate Is this a real need? What problem does this solve that forums + wikipedia + arxiv + google + a literate human hasn't already?
ThinkCritically 7y ago
To any of the readers out there, I am currently looking into better understanding the potential business applications of this NLP technology. It would be especially interesting to hear from people who work extensively in social media as to how "Automated reading" could one day help them with their daily job. If you have time to discuss, DM me on twitter - I am @ralphbrooks.
perl4ever 7y ago
Let's say you had a problem stated as follows: You have several thousand documents in HTML, with roughly similar content (to a human) but not entirely consistent in the ordering of the sections, or the formatting, or the names, or the language and structure. But each document describes an entity of the same class (to a human) and very nearly all of them have a section that summarizes the document. They have a lot of parts in common, but they are fundamentally not designed to line up with a structure for data processing. Is there any practical way to find that summary? Sure, something obvious that takes no time to script is to look for something like "Overview" but you quickly get bogged down in exceptions. Maybe this seems very mundane and simpleminded, but it is the sort of thing that many people would assume is best done by a human. Is there anything current or in the near future that would be significantly easier than a human reading all of them?

gzu 7y ago

I can't wait for the day we see "deep-dream" styled literary works.

jandrese 7y ago
There have been plenty of "AI written" textual works, like the AI Dungeon for a D&D game, or the AI Recipe Generator that attempts to make something that looks like a cooking recipe. For the most part they aren't successful because the "AI" isn't smart enough to have a goal in mind, so they end up just monkey-cheesing everything. Pasting together snippets in ways that are usually grammatically correct but make no sense.
- okigan 7y ago
  any books/papers that address "goal" oriented NLP or hybrid based discussion?
  - jandrese 7y ago
    The ability to set your own goals and task yourself to achieve them is the essence of AI. Not "AI" as we know it today, but Sci-fi AI where it's a machine person.
    - joe_the_user 7y ago
      Indeed, but that kind of describes the gulf between current language processing and present AI. Present AI generates tokens that seem to have meaning or seem like an appropriate response to a statement but where it become evident after 2-3 paragraphs, there's no substantial relation to either underlying meaning or underlying goals. Part of this is "underlying meaning" is an intuitive way to describe things but whatever is underlying here is more tenuous than a classical logic/GOFAI model of the world but more "solid" than a long, clever stream of associations.
    - sgt101 7y ago
      Is it? It's pretty clear that this kind of activity isn't that common in humans - mostly we pick up our goals from cues and drivers in our environment and society.
    - okigan 7y ago

K0SM0S 7y ago

If what you claim is true, it's not just the internet that will be changed by NLP but all of civilization!

m0zg 7y ago

I think you're the one who overestimates how much this will affect NLP. I'd say bulk of what was possible to deliver with this is already here, the subsequent changes will be incremental. The cold hard truth about statistical (and by extension, deep) NLP is that it's just a fancy way of counting numbers mostly. The only way to get to _real_ language understanding is AGI, and _nobody_ is working on that. You fundamentally cannot interact comfortably with a human if your system does not have probabilistic, contextualized cognition, and can't incorporate knowledge about the world.

corporateslave5 7y ago
I respectfully disagree. I’ve been in the field full time for a few years, I watch the state of the art closely. It’s hard to see the thought/theoretical progression of deep transformer models by just reading posts here and there. I’m not saying these NLP methods will be some kind of AI, just that they will produce products, content, and ways of interacting with the world that are categorically different from what we have seen in the past. For instance, question and answering tasks have only recently been able to: Find an answer in a text document that spans multiple non contiguous paragraphs Understand context across a whole book. The context window of current nlp is stuck at 512 tokens, mostly because of computational complexity. This has been broken just recently by the reformer model. Which is a primitive, early way to get around the computation costs of attention mechanisms. Just wait. The ideas are there. They just take time to refine.
- skywhopper 7y ago
  Since you seem to be observing this closely, I'm curious what steps are being taken to identify and avoid bias in the results generated by these systems?
- p1esk 7y ago
  What exactly do you expect from these Transformer-based models? This particular one is underwhelming because it provides a tiny improvement over the previous largest one (from Nvidia) with more than double the size. The fundamental limitation of these "optimization tools" as you call them - they don't have any common sense, and any way to query an external source of information (e.g. wikipedia), or ask a human to clarify. Another big problem is we don't have any way to do quality filtering on the outputs. From my experiments with GPT-2, it produces one interesting paragraph of text out of 20 - if you squint at it really hard. And most of those 20 don't make much sense at all. So no, the existing ideas are definitely not enough. Maybe some novel hybrid of symbolic AI with statistical optimization will lead to a breakthrough. This one does not strike me as anything other than "let's use moar weights!!"

lopmotr 7y ago

Can you explain your reasoning? It's easy to imagine a new invention or idea will revolutionize everything because it's never been done before and feels powerful, but even if it works, it might not. It sounds like you believe NLP will enable more general voice control of computers than Siri/Alexa/etc. But will that really be much more significant than people expect? Google is already pretty good at understanding ambiguous queries. It's eliminated the need to know where to look for information. Or are you talking about the computer writing code or doing business itself?

codingslave 7y ago
Replying on my laptop. Basically, transformer models are the best for NLP. They use something called attention based mechanisms, which allows the model to draw correlations between pieces of text/tokens that are far apart. The issue is that this is an O(n^2) operation. So the model is bounded by the context window, which is currently mostly at 512 tokens, and is thus, bounded in how much it can understand. Recent innovations, and further study, will broaden the context window, and thus unlock better reading comprehension and context understanding. For instance, the ability to answer a question using a piece of text is mostly stuck at just finding one paragraph. The future will see models that can find multiple different paragraphs, understand how they relate, pull the relevant information, and synthesize it. This sounds like a minor step forwards, but its important. This will unlock better conversational abilities, but also, better ways to understand how different pieces of textual information relate. The scattershot of information across the internet can go away. Computers can better understand context to act on human intention through language, unlocking the ability to handle ambiguity. This will change the internet.
- skywhopper 7y ago
  If this is really where the researchers think these tools are headed (and I don't really doubt you on that point), then this is incredibly dangerous stuff. No matter how good your system is, the impact of implicit, unintentional, and non-targeted bias is huge on the sorts of content these systems will produce. But expose it to the levels of intentional manipulation present on the Internet of today, and these models don't stand a chance of producing something that safely does what you claim. I'm happy to be wrong about this, but I'm not seeing any discussion about the safety and security of using these systems. And if it's not even being discussed, we can be sure nothing's actually being done about it. Selling promises of active-agent computers interpreting human intent and summarizing information from the Internet without addressing this concern is irresponsible at this point.

skywhopper 7y ago

Can you explain more about what you mean by "able to handle the ambiguity of human language"? Such a claim goes far beyond language processing itself. Such a capability would have to involve cultural and interpersonal dynamics, domain knowledge, and definitive agency on the part of the computer. And all of _that_ is just to interpret sincere, honest attempts at communication. Can it safely and appropriately handle humor, irony, and sarcasm? What about coordinated malicious attacks?

heavyarms 7y ago

I would agree with you to a certain extent, but I still think there is a big missing component to make this a reality. Larger and more accurate general language models are great, but to enable use cases other than categorization, translation, summarization, etc., there will almost certainly have to a be contextualized knowledge graph layer. This is basically what I assume you mean when you say "the ability to create, synthesize, and understand content." The way I see it, transformer-based general language models will be the first and last step of a NLP system. I other words, they will do the raw processing of the input and will do the "take this output and put it in natural language based on this context" portion. Automating the part in the middle, the part that actually understands what the text means and can do logic with it, there's still no progress on that AFAIK. This is similar to how computer vision models work great in most cases, but to build a self-driving car you still need all the other components that do path planning, predicting what the cars around you will do based on the state of the environment, etc.

codingslave 7y ago
Replying from my laptop. I agree, there needs to be a way to represent relationships between information. I personally don't think knowledge graphs will be the ones to do it, not because they dont work, but because of how imperfect they are, in the data quality sense. See this paper here: "DIFFERENTIABLE REASONING OVER A VIRTUAL KNOWLEDGE BASE" https://openreview.net/pdf?id=SJxstlHFPH https://openreview.net/pdf?id=SJxstlHFPH Which is a recent effort, among many, by google research to build a model that can view a document as a knowledge graph, instead of explicitly tying pieces of the document to the graph, the idea to is to create a graph from the document. This is paper is a bit different from that, they do input a knowledge graph for training, but I think the idea and track of where they are headed has a ton of room to evolve. The trick is that transformer models have unlocked the ability to understand the text, so all of this "quasi knowledge graph extraction" that i was just explaining, is only recently possible! There's no research on it, because the baseline understanding of tokens has been too primitive. This is why there is so much room to grow, BERT has unlocked new methods, it can be used as a base for a ton of new NLP. Just to emphasize again, I'm not saying what I outlined above will be a good way to do it, just that ideas like this could only be tested recently. There's a million new ways to spin this problem.

the8472 7y ago

codingslave 7y ago
Yes, but duplex doesn't work and was too early.

deleted 7y ago

[deleted]

Tenoke 7y ago

> mostly at 512 token GPT uses 1024 tokens context window which does work out to a fair amount (given the massive vocab of 50k+ which means a token can be more than a word), though of course it's pretty limited. Google's recent Reformer[0] allows you to do attention much more cheaply and I'm currently training a Reformer model that isn't quite as big but has a context of ~64k tokens (though a much smaller vocab). I'm not completely sure if this is the solution but it looks like a step in that direction and so far the model is doing pretty good (I also plan to post the weights when I'm finished, though I am not sure if Google don't just plan to do that themselves). I am somewhat disappointed they went with just 1024 for this model, too though. 0. https://ai.googleblog.com/2020/01/reformer-efficient-transformer.html https://ai.googleblog.com/2020/01/reformer-efficient-transfo...

corporateslave5 7y ago
Yep, the number has been expanding. I just picked 512 because Bert uses that. But yeah 50k and beyond unlocks new ways of using context. I can’t wait to see how they optimize and improve the reformer model. I expect there to be an improvement like there was for BERT -> Albert

joe_the_user 7y ago

Computers will be able to handle the ambiguity of human language, transcending their rigid “only do exactly what you tell them” models of the world. So, are reasonable examples now of these models allowing semantic context? So, far, what I have seen is generated text where the lack of understanding takes three paragraphs to become obvious rather than one. Human language is this marvelous framework involving symbols associating with other symbols as well as to well-known and vaguely-guessed facts about the world. Human relations are very robust and, for example, two people can have a longish conversations where at the end, they realize they're talking about two different people (or different days or events). But in those circumstances, they can correct and adjust. "Solid" understanding is there but it's under a lot of layers of social cues and protocols and multiple meanings.

mumblemumble 7y ago
> So, are reasonable examples now of these models allowing semantic context? This is about where I am stuck. I'll start believing that we truly are on the cusp of a revolution as soon as I see Google Translate reliably knowing when to translate "home" into French as "domicile", "foyer", something those lines, or as "accueil." Right now it seems to very frequently choose "accueil", which is generally wrong, except when you're talking about websites and software user interfaces. That it's biased so strongly toward that error speaks volumes about how critical semantics are to sorting out natural language, and also about how bad current NLP systems are at dealing with semantics.
- joe_the_user 7y ago
  Syntax and semantics were developed for human language, yet it's much easier to puzzle the difference in a computer language than in human language. With syntax and semantics so wrapped together, however, it kind of seems like you can go a long way with just capturing syntax, rhythm, word choice and etc. Which is to say the semantic side can be even worse than it seems, ie, nonexistent.
  - deleted 7y ago
    [deleted]

lostcolony 7y ago

>More than that, I think NLP will unlock new ways of interacting with computers. Computers will be able to handle the ambiguity of human language, transcending their rigid “only do exactly what you tell them” models of the world. As exciting as this sounds, I can't help but feel that given -we- haven't figured out how to handle the ambiguity of human language, I'm not convinced a computer attempting to is really markedly better for many use cases than requiring exactness. But operating at a human level of 'understanding', and being broadly accessible, may be enough to change the world. Hopefully for the better.

KKKKkkkk1 7y ago

Feels like I’ve heard this tune before. Actually, I think it was first played in the 1950s.

empath75 7y ago

I wonder if you could have like a higher order transformer — you first generate a series of prompts and then expand on the prompts one at a time.

Someone 7y ago

”People are vastly underestimating the changes that are about to come from NLP.” Maybe, maybe not. IMO, we don’t know what problem we have to solve to get what you describe. We also don’t have a metric as to how far we are along the path towards that goal (do we need 100B parameters? A trillion?), nor do we have any idea as to whether the current approach can get us there.

jaggednad 7y ago

I think you’re right that we are on the cusp or real natural language understanding. It’s an incredible moment. But we are also kinda far from these huge models being in every home because of the enormous amount of compute they require. Even running inference with these things is kinda expensive

tastroder 7y ago

> Basically, transformer models are the best for NLP. Yes, this year. While transformers certainly present a breakthrough in the NLP community and certainly stir up the state-of-the-art again, I don't really see how you go from that to the "computers will understand us" conclusion to be honest. People said that during the word2vec stir up and what we got out of that was incremental results (which is not bad, it's in the nature of things really). We can already build models that do everything you describe as single tasks, while that's exciting, it's not going to lead to the singularity. We've got a long way to go in terms of understanding models, making them computationally tractable, and making them do what we want in the first place without resorting to hoping that our unsupervised model learns something useful. It's likely that the attention mechanisms we see today will be a large part of that but I'm honestly a bit baffled at the "People are vastly underestimating the changes that are about to come from NLP." part. They're not, people already think that today's AI is magic, I don't think it is benefitial to reinforce that. Speech is nuanced, we're making good progress in many areas but we're not on the cusp of any revolutionary change in computational understanding really.

skocznymroczny 7y ago

Perhaps, but people overestimate the progress in AI also. Remember Google's demo of AI reserving a spot at a barber shop? Yeah... that never happened, even though it was supposed to be any day now.

dlkf 7y ago

> So the model is bounded by the context window, which is currently mostly at 512 tokens, and is thus, bounded in how much it can understand. This is begging the question as to whether the model "understands" anything at all. And once you adopt a definition of "understanding" that isn't equivalent to "got a high score on some pointless academic challenge" the answer is a resounding "no." The whole enterprise of AGI hype is based on this equivocation of words like "understanding" and "intelligence." We use a very restricted definition in proving that the tech is smart, and then switch out our restricted definition for the colloquial one when the audience isn't looking. > This will unlock better conversational abilities Shouldn't be hard given that as it stands there are none, except for creating a human-sounding slurry that is devoid of real content. > but also, better ways to understand how different pieces of textual information relate Is this a real need? What problem does this solve that forums + wikipedia + arxiv + google + a literate human hasn't already?

ThinkCritically 7y ago

To any of the readers out there, I am currently looking into better understanding the potential business applications of this NLP technology. It would be especially interesting to hear from people who work extensively in social media as to how "Automated reading" could one day help them with their daily job. If you have time to discuss, DM me on twitter - I am @ralphbrooks.

perl4ever 7y ago

Let's say you had a problem stated as follows: You have several thousand documents in HTML, with roughly similar content (to a human) but not entirely consistent in the ordering of the sections, or the formatting, or the names, or the language and structure. But each document describes an entity of the same class (to a human) and very nearly all of them have a section that summarizes the document. They have a lot of parts in common, but they are fundamentally not designed to line up with a structure for data processing. Is there any practical way to find that summary? Sure, something obvious that takes no time to script is to look for something like "Overview" but you quickly get bogged down in exceptions. Maybe this seems very mundane and simpleminded, but it is the sort of thing that many people would assume is best done by a human. Is there anything current or in the near future that would be significantly easier than a human reading all of them?