7 ms·

Natural language generation: The commercial state of the art in 2020

90 points by polm23 6y ago

polm23 6y ago
This is a short overview of the state of NLG by Robert Dale, co-author of "Building Natural Language Generation Systems", which is basically the book for NLG. He give a list of commercial providers and concludes that most of them just offer smart templates. This is the important part: > To the extent that you can tell from the clues to functionality that are surfaced by these various products, all the tools are ultimately very similar in terms of how they work, which might be referred to as ‘smart template’ mechanisms. There is a recognition that, at least for the kinds of use cases we see today, much of the text in any given output can be predetermined and provided as boilerplate, with gaps to be filled dynamically based on per-record variations in the underlying data source. Add conditional inclusion of text components and maybe some kind of looping control construct, and the resulting NLG toolkit, as in the case of humans and chimpanzees, shares 99% of its DNA with the legal document automation and assembly tools of the 1990s, like HotDocs (https://www.hotdocs.com https://www.hotdocs.com). As far as I can tell, linguistic knowledge, and other refined ingredients of the NLG systems built in research laboratories, is sparse and generally limited to morphology for number agreement (one stock dropped in value vs. three stocks dropped in value). That sounds pretty negative, but he emphasizes that putting an easy-to-use UI on well understood technology is meeting real business needs. At the end he briefly touches on GPT2 and related technology.
- PaulHoule 6y ago
  Smart templates arent bad at all given that for most uses you want to enforce style and attributes. Generation is easier than understanding. Systems like GPT-3 are not capable of respecting constraints, which is this basis for practical creativity.
picodguyo 6y ago
ML and NLP surveys are outdated basically the minute the author hits send nowadays. With GPT-3 all of these old school NLG companies seem quaint. Much of their functionality can be replicated in minutes with GPT-3, even by non-technical users familiar with prompt design. As soon as OpenAI offer fine tuning OOTB (coming soon) NLP/G will be all but solved for 99% of use cases.
- PedroBatista 6y ago
  Flying cars buddy! you just wait..
  - picodguyo 6y ago
    I know, I know. But I don't say this lightly. I've worked in NLP/G for nearly 15 years and after spending the last 6 months working with GPT-3 I feel the writing is on the wall.
    - Saad_M 6y ago
      But it isn’t. I am still actively involved in NLG both professionally and academically and neural NLG systems have great promise but are still far from actively delivering tangible solutions to areas such data-to-text NLG. Inaccuracy, hallucinations are still highly problematic.
- polm23 6y ago
  I don't want to suggest you shot off a random comment without actually reading the linked survey. But it spends a whole section discussing that, and why it doesn't see commercial use. Here's a particularly relevant bit: > Sometimes the results produced by GPT-2 and its ilk are quite startling in their apparent authenticity. More often than not they are just a bit off. And sometimes they are just gibberish. As is widely acknowledged, neural text generation as it stands today has a significant problem: driven as it is by information that is ultimately about language use, rather than directly about the real world, it roams untethered to the truth. While the output of such a process might be good enough for the presidential teleprompter, it would not cut it if you want the hard facts about how your pension fund is performing. So, at least for the time being, nobody who develops commercial applications of NLG technology is going to rely on this particular form of that technology.
pdevr 6y ago
Is there any place where NLG guys hang around? Most groups seem to have NLP as the focus.
- Saad_M 6y ago
  On the academic side the SIGGEN mailing list is where a lot of the activity in the NLG community goes on: https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=SIGGEN https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=SIGGEN
- ghenson2 6y ago
  Wonder the same thing or just an active discord for these topics...
- visarga 6y ago
  EleutherAI on discord. https://discord.gg/vtRgjbM https://discord.gg/vtRgjbM
is_true 6y ago
I think that most use cases for NLG are unethical. I provide raw data for some news services and that data could be shown as tables or simple dashboards that are faster to read for users, but my customers insist in generating as much text as possible from, sometimes, 2 values. The incentives are totally misaligned, news services try to get as much time from it's readers as possible. It's quite depressing.
- mkl95 6y ago
  Usage of these algorithms and heuristics is starting to become quite obvious in Spanish sports journalism. I stumble into articles that are basically AI gibberish generated from a short quote every other day.
- coding123 6y ago
  I tend to agree. While technologically its a marvel. But I do have the feeling the Robocallers and NLG customers are hanging out in the same place.
- tyingq 6y ago
  Google is probably going to struggle with SEO content/link mills that now produce something better than spun gibberish.
  - lumost 6y ago
    I'd expect the only rational decision would be to ban the content mill domains from the crawler. Is there really that much value for the search customer in being directed to a generated page?
    - tyingq 6y ago
      That's the dilemma. NLG of sufficient quality[1] won't be easily detectable as machine generated. [1] "Quality" here meaning how it looks to Google's crawler, not actual quality content.
      - 6y ago
ghenson2 6y ago
Why is Google's T5 not mentioned? https://github.com/google-research/text-to-text-transfer-transformer https://github.com/google-research/text-to-text-transfer-tra...
- riku_iki 6y ago
  curious if there any examples of using it for text generation, since it has different model structure from GPT.
  - ghenson2 6y ago
    Check out the T5 models at huggingface for this. Main NLG use cases with T5 are translations, summarization and question generation. Latter is sophisticated + nothing trivial and "NLG", so yeah.
  - FL33TW00D 6y ago
    From my experience T5 is currently the best publicly available model for NLG.
    - ghenson2 6y ago
      Did you use it already in some production environment?
      - FL33TW00D 6y ago
        Not yet but I am working on it!
        ghenson2 6y ago
        Nice, will you use it via huggingface?
        FL33TW00D 6y ago
        Yeah currently experimenting with T5 large in FP16. Will then use the ONNX conversion script to decrease inference time and deploy it!
wodenokoto 6y ago
What are the commercial usage of GPT-3 API? I must completely lack imagination, because I don’t know what to use it for if it doesn’t give me access to the weights.
visarga 6y ago
The state of NLG as of June 2020. A long time ago. Apparently it was published just one day before GPT-3.
jaxwerk 6y ago
As an engineer who has worked briefly with one of the start-ups focused on here on their core product, Robert's critique of the the technology driving the current generation of template-based commercial solutions is absolutely spot-on. The results are useful but the template based approach along with the really lackluster tooling both my employer and competitors in the space had cobbled together made self-service nigh impossible. His assessment that these companies are really professional services companies dealing in bespoke one-off software solutions despite their efforts to market as robust AI products that magically turn some arbitrary JSON schema into prose is very fair I think. I am eager to see if neural approaches develop to better handle constraints, but part of me thinks the level of control over generated prose that template based approaches provide is essential for customer-facing text and that part of the market will persist even when every other recipe in Google search results is GPT-3 generated rubbish around the actual damn list of ingredients.

90 points by polm23 6y ago

This is a short overview of the state of NLG by Robert Dale, co-author of "Building Natural Language Generation Systems", which is basically the book for NLG. He give a list of commercial providers and concludes that most of them just offer smart templates. This is the important part: > To the extent that you can tell from the clues to functionality that are surfaced by these various products, all the tools are ultimately very similar in terms of how they work, which might be referred to as ‘smart template’ mechanisms. There is a recognition that, at least for the kinds of use cases we see today, much of the text in any given output can be predetermined and provided as boilerplate, with gaps to be filled dynamically based on per-record variations in the underlying data source. Add conditional inclusion of text components and maybe some kind of looping control construct, and the resulting NLG toolkit, as in the case of humans and chimpanzees, shares 99% of its DNA with the legal document automation and assembly tools of the 1990s, like HotDocs (https://www.hotdocs.com https://www.hotdocs.com). As far as I can tell, linguistic knowledge, and other refined ingredients of the NLG systems built in research laboratories, is sparse and generally limited to morphology for number agreement (one stock dropped in value vs. three stocks dropped in value). That sounds pretty negative, but he emphasizes that putting an easy-to-use UI on well understood technology is meeting real business needs. At the end he briefly touches on GPT2 and related technology.

PaulHoule 6y ago
Smart templates arent bad at all given that for most uses you want to enforce style and attributes. Generation is easier than understanding. Systems like GPT-3 are not capable of respecting constraints, which is this basis for practical creativity.

picodguyo 6y ago

ML and NLP surveys are outdated basically the minute the author hits send nowadays. With GPT-3 all of these old school NLG companies seem quaint. Much of their functionality can be replicated in minutes with GPT-3, even by non-technical users familiar with prompt design. As soon as OpenAI offer fine tuning OOTB (coming soon) NLP/G will be all but solved for 99% of use cases.

PedroBatista 6y ago
Flying cars buddy! you just wait..
- picodguyo 6y ago
  I know, I know. But I don't say this lightly. I've worked in NLP/G for nearly 15 years and after spending the last 6 months working with GPT-3 I feel the writing is on the wall.
  - Saad_M 6y ago
    But it isn’t. I am still actively involved in NLG both professionally and academically and neural NLG systems have great promise but are still far from actively delivering tangible solutions to areas such data-to-text NLG. Inaccuracy, hallucinations are still highly problematic.
polm23 6y ago
I don't want to suggest you shot off a random comment without actually reading the linked survey. But it spends a whole section discussing that, and why it doesn't see commercial use. Here's a particularly relevant bit: > Sometimes the results produced by GPT-2 and its ilk are quite startling in their apparent authenticity. More often than not they are just a bit off. And sometimes they are just gibberish. As is widely acknowledged, neural text generation as it stands today has a significant problem: driven as it is by information that is ultimately about language use, rather than directly about the real world, it roams untethered to the truth. While the output of such a process might be good enough for the presidential teleprompter, it would not cut it if you want the hard facts about how your pension fund is performing. So, at least for the time being, nobody who develops commercial applications of NLG technology is going to rely on this particular form of that technology.

pdevr 6y ago

Is there any place where NLG guys hang around? Most groups seem to have NLP as the focus.

Saad_M 6y ago
On the academic side the SIGGEN mailing list is where a lot of the activity in the NLG community goes on: https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=SIGGEN https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=SIGGEN
ghenson2 6y ago
Wonder the same thing or just an active discord for these topics...
visarga 6y ago
EleutherAI on discord. https://discord.gg/vtRgjbM https://discord.gg/vtRgjbM

is_true 6y ago

I think that most use cases for NLG are unethical. I provide raw data for some news services and that data could be shown as tables or simple dashboards that are faster to read for users, but my customers insist in generating as much text as possible from, sometimes, 2 values. The incentives are totally misaligned, news services try to get as much time from it's readers as possible. It's quite depressing.

mkl95 6y ago
Usage of these algorithms and heuristics is starting to become quite obvious in Spanish sports journalism. I stumble into articles that are basically AI gibberish generated from a short quote every other day.
coding123 6y ago
I tend to agree. While technologically its a marvel. But I do have the feeling the Robocallers and NLG customers are hanging out in the same place.
tyingq 6y ago
Google is probably going to struggle with SEO content/link mills that now produce something better than spun gibberish.
- lumost 6y ago
  I'd expect the only rational decision would be to ban the content mill domains from the crawler. Is there really that much value for the search customer in being directed to a generated page?
  - tyingq 6y ago
    That's the dilemma. NLG of sufficient quality[1] won't be easily detectable as machine generated. [1] "Quality" here meaning how it looks to Google's crawler, not actual quality content.
    - 6y ago

ghenson2 6y ago

Why is Google's T5 not mentioned? https://github.com/google-research/text-to-text-transfer-transformer https://github.com/google-research/text-to-text-transfer-tra...

riku_iki 6y ago
curious if there any examples of using it for text generation, since it has different model structure from GPT.
- ghenson2 6y ago
  Check out the T5 models at huggingface for this. Main NLG use cases with T5 are translations, summarization and question generation. Latter is sophisticated + nothing trivial and "NLG", so yeah.
- FL33TW00D 6y ago
  From my experience T5 is currently the best publicly available model for NLG.
  - ghenson2 6y ago
    Did you use it already in some production environment?
    - FL33TW00D 6y ago
      Not yet but I am working on it!
      - ghenson2 6y ago
        Nice, will you use it via huggingface?
        FL33TW00D 6y ago
        Yeah currently experimenting with T5 large in FP16. Will then use the ONNX conversion script to decrease inference time and deploy it!

wodenokoto 6y ago

What are the commercial usage of GPT-3 API? I must completely lack imagination, because I don’t know what to use it for if it doesn’t give me access to the weights.

visarga 6y ago

The state of NLG as of June 2020. A long time ago. Apparently it was published just one day before GPT-3.

jaxwerk 6y ago

As an engineer who has worked briefly with one of the start-ups focused on here on their core product, Robert's critique of the the technology driving the current generation of template-based commercial solutions is absolutely spot-on. The results are useful but the template based approach along with the really lackluster tooling both my employer and competitors in the space had cobbled together made self-service nigh impossible. His assessment that these companies are really professional services companies dealing in bespoke one-off software solutions despite their efforts to market as robust AI products that magically turn some arbitrary JSON schema into prose is very fair I think. I am eager to see if neural approaches develop to better handle constraints, but part of me thinks the level of control over generated prose that template based approaches provide is essential for customer-facing text and that part of the market will persist even when every other recipe in Google search results is GPT-3 generated rubbish around the actual damn list of ingredients.