5 ms·

Jeremy here (co-author of this paper). Let me know if you have any questions!

by jph00 8y ago

woliveirajr 8y ago
Portuguese (specifically, Brazilian Portuguese) isn't being studied yet ?
- jph00 8y ago
  Doesn't look like it. Hope to see someone take it up!
groovybits 8y ago
Is this the same classification project that was presented at this past Bsides NoVA?
- jph00 8y ago
  No I don't know what that is
derdude 8y ago
Hey Jeremy, thanks for sharing this awesome article! Do you think this is also applicable to classify 1. readability and 2. entertainment/fun of a text? Thanks!
- jph00 8y ago
  Yes I think it would work great for both of those applications. For readability, you'd ideally want a dataset showing how long someone took to read each document. For entertainment, some kind of review dataset would be fine (it's basically a type of sentiment analysis, which we've already tested on 3 datasets.)
juliango 8y ago
Does the (future) sequence labeling project refers to Part Of Speech tagging?
- jph00 8y ago
  That would certainly be one of them.
Abundnce10 8y ago
This method dramatically improves over previous approaches to text classification, and the code and pre-trained models allow anyone to leverage this new approach to better solve problems such as: Finding documents relevant to a legal case; Identifying spam, bots, and offensive comments; Classifying positive and negative reviews of a product; Grouping articles by political orientation; I'm starting a new project where I'm given many recipes and I need to take in a free form text of recipe ingredients (e.g. "1/2 cup diced onions", "two potatoes, cut into 1-inch cubes", etc.) and build a program that identifies the ingredient (e.g. onion, potato), as well as the quantity (e.g. 0.5 cup, 2.0 units). Could I use something like Fast.ai to tackle this problem?
- jph00 8y ago
  I'm not sure - what you're describing is information extraction. I haven't tried that yet, but I'm certainly interested in doing so (especially for medical data).
  - state_less 8y ago
    Similarly, could something like this be useful to extract out a command that a user wants to run from a transcription? For example, "Add a user named Jenny to our client list.", which results in the command, 'create user Jenny'. Or, "Could you add Jenny to our client list?", which results in the same command, 'create user Jenny'. Perhaps instead of outputting the next word, output the expected command from a set of commands?
- timClicks 8y ago
  If you have sufficient labelled data, conditional random fields works well for this kind of problem. A technical team from NY Times have a great piece on it https://open.blogs.nytimes.com/2015/04/09/extracting-structured-data-from-recipes-using-conditional-random-fields/ https://open.blogs.nytimes.com/2015/04/09/extracting-structu...
- ioot17 8y ago
  CRF works quite well, it's actually what I utilize right now to approach recipe parsing on https://cookalo.com/ https://cookalo.com/. It's based on CRFsuite with Python bindings for data training on already labeled recipes. If you build your own app and want to do some comparison, feel free to run some benchmarks against it.
Eridrus 8y ago
When I think of state of the art in this area, I think of the Deep Contextualized Word Vectors/ELMo paper from Peters et al, which you cite, but you don't have any comparisons to. The only point of reference between the two papers I see is the CoVe models, which you guys beat pretty handily, but the ELMo model also beats the CoVe model handily, just on different datasets, so not clear how they stack up. Any chance you could do some more direct comparisons? You do say that it's a more complex architecture, and the tokenized char convolution stuff is a bit of a pain to do, but if that actually helps, it's not that bad to do once. From an engineering perspective, not changing the LM weights is kind of nice because then you can train multiple separate models on top of the embeddings without needing to retrain everything (and deal with the associated "noise" when retraining models) and it gives some nice modularity. It would be nice to know how much of the performance is lost from having embeddings that can be shared across a lot of tasks. Random note: it seems like in Table 7, you have bolded "Freez + discr + stlr" in the IMDb column which has a value of 5.00, whereas "Full + discr" has a value of 4.57, and so should probably be the bolded number.
- jph00 8y ago
  Very interesting comments. Yes absolutely want to do comparisons to ELMo. It's a little tricky to do so on our datasets, since ELMo isn't really a complete method on its own, but more an addendum to existing methods. In the future we hope to do seq2seq and sequence labeling studies, and we can then ensure we pick datasets that the ELMo paper covered. Using char tokens can definitely be helpful, as can sub-words. It's something we've been working on too, and hope to show results of this in the future. I mainly disagree with your view of end-to-end training. In computer vision we pretty much gave up on trying to re-use hyper-columns without fine-tuning, because the fine-tuning just helps so much. It's really no trouble doing the fine-tuning - in fact the consistency of using a single model form across so many different datasets is really convenient and helpful for doing additional levels of transfer learning. Thanks for the note about table 7 - it's actually an error (it should be 5.57, not 4.57; Sebastian is in the process of uploading a corrected version).
  - madisonmay 8y ago
deleted 8y ago
[deleted]
timClicks 8y ago
Hi Jeremy (& Sebastian of course), this is an amazing contribution to the state of the art! I haven't read things in depth, but am curious - how do these models cope with out of vocabulary terms?
- madisonmay 8y ago
  Initially embeddings of OOV terms are initialized to mean embeddings, and then the language model fine-tuning step allows the model to adjust these values. See [18] here: https://github.com/fastai/fastai/blob/master/courses/dl2/imdb.ipynb https://github.com/fastai/fastai/blob/master/courses/dl2/imd....

Jeremy here (co-author of this paper). Let me know if you have any questions!

by jph00 8y ago

Jeremy here (co-author of this paper). Let me know if you have any questions!

woliveirajr 8y ago

Portuguese (specifically, Brazilian Portuguese) isn't being studied yet ?

jph00 8y ago
Doesn't look like it. Hope to see someone take it up!

groovybits 8y ago

Is this the same classification project that was presented at this past Bsides NoVA?

jph00 8y ago
No I don't know what that is

derdude 8y ago

Hey Jeremy, thanks for sharing this awesome article! Do you think this is also applicable to classify 1. readability and 2. entertainment/fun of a text? Thanks!

jph00 8y ago
Yes I think it would work great for both of those applications. For readability, you'd ideally want a dataset showing how long someone took to read each document. For entertainment, some kind of review dataset would be fine (it's basically a type of sentiment analysis, which we've already tested on 3 datasets.)

juliango 8y ago

Does the (future) sequence labeling project refers to Part Of Speech tagging?

jph00 8y ago
That would certainly be one of them.

Abundnce10 8y ago

This method dramatically improves over previous approaches to text classification, and the code and pre-trained models allow anyone to leverage this new approach to better solve problems such as: Finding documents relevant to a legal case; Identifying spam, bots, and offensive comments; Classifying positive and negative reviews of a product; Grouping articles by political orientation; I'm starting a new project where I'm given many recipes and I need to take in a free form text of recipe ingredients (e.g. "1/2 cup diced onions", "two potatoes, cut into 1-inch cubes", etc.) and build a program that identifies the ingredient (e.g. onion, potato), as well as the quantity (e.g. 0.5 cup, 2.0 units). Could I use something like Fast.ai to tackle this problem?

jph00 8y ago
I'm not sure - what you're describing is information extraction. I haven't tried that yet, but I'm certainly interested in doing so (especially for medical data).
- state_less 8y ago
  Similarly, could something like this be useful to extract out a command that a user wants to run from a transcription? For example, "Add a user named Jenny to our client list.", which results in the command, 'create user Jenny'. Or, "Could you add Jenny to our client list?", which results in the same command, 'create user Jenny'. Perhaps instead of outputting the next word, output the expected command from a set of commands?
timClicks 8y ago
If you have sufficient labelled data, conditional random fields works well for this kind of problem. A technical team from NY Times have a great piece on it https://open.blogs.nytimes.com/2015/04/09/extracting-structured-data-from-recipes-using-conditional-random-fields/ https://open.blogs.nytimes.com/2015/04/09/extracting-structu...
ioot17 8y ago
CRF works quite well, it's actually what I utilize right now to approach recipe parsing on https://cookalo.com/ https://cookalo.com/. It's based on CRFsuite with Python bindings for data training on already labeled recipes. If you build your own app and want to do some comparison, feel free to run some benchmarks against it.

Eridrus 8y ago

When I think of state of the art in this area, I think of the Deep Contextualized Word Vectors/ELMo paper from Peters et al, which you cite, but you don't have any comparisons to. The only point of reference between the two papers I see is the CoVe models, which you guys beat pretty handily, but the ELMo model also beats the CoVe model handily, just on different datasets, so not clear how they stack up. Any chance you could do some more direct comparisons? You do say that it's a more complex architecture, and the tokenized char convolution stuff is a bit of a pain to do, but if that actually helps, it's not that bad to do once. From an engineering perspective, not changing the LM weights is kind of nice because then you can train multiple separate models on top of the embeddings without needing to retrain everything (and deal with the associated "noise" when retraining models) and it gives some nice modularity. It would be nice to know how much of the performance is lost from having embeddings that can be shared across a lot of tasks. Random note: it seems like in Table 7, you have bolded "Freez + discr + stlr" in the IMDb column which has a value of 5.00, whereas "Full + discr" has a value of 4.57, and so should probably be the bolded number.

jph00 8y ago
Very interesting comments. Yes absolutely want to do comparisons to ELMo. It's a little tricky to do so on our datasets, since ELMo isn't really a complete method on its own, but more an addendum to existing methods. In the future we hope to do seq2seq and sequence labeling studies, and we can then ensure we pick datasets that the ELMo paper covered. Using char tokens can definitely be helpful, as can sub-words. It's something we've been working on too, and hope to show results of this in the future. I mainly disagree with your view of end-to-end training. In computer vision we pretty much gave up on trying to re-use hyper-columns without fine-tuning, because the fine-tuning just helps so much. It's really no trouble doing the fine-tuning - in fact the consistency of using a single model form across so many different datasets is really convenient and helpful for doing additional levels of transfer learning. Thanks for the note about table 7 - it's actually an error (it should be 5.57, not 4.57; Sebastian is in the process of uploading a corrected version).
- madisonmay 8y ago

deleted 8y ago

[deleted]

timClicks 8y ago

Hi Jeremy (& Sebastian of course), this is an amazing contribution to the state of the art! I haven't read things in depth, but am curious - how do these models cope with out of vocabulary terms?

madisonmay 8y ago
Initially embeddings of OOV terms are initialized to mean embeddings, and then the language model fine-tuning step allows the model to adjust these values. See [18] here: https://github.com/fastai/fastai/blob/master/courses/dl2/imdb.ipynb https://github.com/fastai/fastai/blob/master/courses/dl2/imd....