4 ms·

The Reddit comment corpus is an awesome dataset. There's relatively little mark-up to scrub out, low duplication, good metadata, and a variety of topics. We us

The Reddit comment corpus is an awesome dataset. There's relatively little mark-up to scrub out, low duplication, good metadata, and a variety of topics. We used it to train a syntax-enriched word2vec model. Write up and demo: https://explosion.ai/blog/sense2vec-with-spacy https://explosion.ai/blog/sense2vec-with-spacy Btw, the above was run on CPU in a couple of days, because spaCy doesn't use GPUs yet. I've applied for a grant from NVidia so I can fix that. If anyone from NVidia is reading, email me? :)

wyldfire 10y ago
> But when we ran the model on more data, and it was gone and soon forgotten. Just like Carrot Top. "it was gone" meaning the association between Carrot Top and Kate Mara? So after better training, who is now most_similar(['Carrot_Top|PERSON'])? EDIT: RTFA, used the interactive demo. most_similar() is now a category I would describe as "actors/comedians popular in the 90's: "Bill Murray, Gary Busey, David Spade, Charlie Sheen, Ashton Kutcher, Chris Farley".
- syllogism 10y ago
  https://demos.explosion.ai/sense2vec/?word=carrot%20top&sense=auto https://demos.explosion.ai/sense2vec/?word=carrot%20top&sens... Weirdly there's a bug that's dropped PERSON from the sense list. Fixing. Edit: Fixed. Edit2: Ah this is super misleading atm. I'll have a think about how to do this better. auto is case insensitive, but if you specify a sense, it's case sensitive. So you need to do "Carrot Top" and set PERSON. Btw, contrast with carrot top "NOUN".
  - deleted 10y ago
    [deleted]
    - deleted 10y ago
      [deleted]
AlexCoventry 10y ago
> I've applied for a grant from NVidia so I can fix that. A g2.xlarge is 65c/hour on AWS, FWIW. https://aws.amazon.com/ec2/pricing/on-demand/ https://aws.amazon.com/ec2/pricing/on-demand/
- samstave 10y ago
  Yeah, for OD? thats 468/month... I'd use a spot instance and stop it whenever possible.
  - syllogism 10y ago
    Spot instances are pretty painful for training. It's annoying to have the machine randomly shut down.
    - viksit 10y ago
      ^ That. For all that people say about spot instances, there's no infrastructure I know if to manage jobs and have them migrate to higher priced instances without losing state.
      - RBerenguel 10y ago
        You can always snapshot and keep track of state as you go (a little bit tricky with Spark, though). We use spot instances for training we know is not vital (as in, has to be done, but rather run it twice and save money anyway that run it for sure). Also, once you know what availability specific instances have you can always choose better (i.e. maybe c3.xlarge is slightly more expensive as spot than large, you can do with large... but xlarge has almost no shutdowns)
  - pc86 10y ago
    Presumably that's a decent chunk of what the grant is for?
- syllogism 10y ago
  That's honestly no way to get anything done... Unless you're replicating someone else's thing exactly, You can't really get by with one training process. You want to be trying different things, and running a few samples of each configuration to account for random variation. I'm not even talking about decadent hyper-parameter sweeps to fine-tune. I'm talking like, how wide do I need my layers to be, what optimizers are good, how deep should I make the network, etc. I want to be training 5-10 models at a time minimum. 20-30 would be much more productive. If I can only train one model at once, it's not really worth the effort — it's better to work on one of the other tickets for the library.

wyldfire 10y ago

> But when we ran the model on more data, and it was gone and soon forgotten. Just like Carrot Top. "it was gone" meaning the association between Carrot Top and Kate Mara? So after better training, who is now most_similar(['Carrot_Top|PERSON'])? EDIT: RTFA, used the interactive demo. most_similar() is now a category I would describe as "actors/comedians popular in the 90's: "Bill Murray, Gary Busey, David Spade, Charlie Sheen, Ashton Kutcher, Chris Farley".

syllogism 10y ago
https://demos.explosion.ai/sense2vec/?word=carrot%20top&sense=auto https://demos.explosion.ai/sense2vec/?word=carrot%20top&sens... Weirdly there's a bug that's dropped PERSON from the sense list. Fixing. Edit: Fixed. Edit2: Ah this is super misleading atm. I'll have a think about how to do this better. auto is case insensitive, but if you specify a sense, it's case sensitive. So you need to do "Carrot Top" and set PERSON. Btw, contrast with carrot top "NOUN".
- deleted 10y ago
  [deleted]
  - deleted 10y ago
    [deleted]

AlexCoventry 10y ago

> I've applied for a grant from NVidia so I can fix that. A g2.xlarge is 65c/hour on AWS, FWIW. https://aws.amazon.com/ec2/pricing/on-demand/ https://aws.amazon.com/ec2/pricing/on-demand/

samstave 10y ago
Yeah, for OD? thats 468/month... I'd use a spot instance and stop it whenever possible.
- syllogism 10y ago
  Spot instances are pretty painful for training. It's annoying to have the machine randomly shut down.
  - viksit 10y ago
    ^ That. For all that people say about spot instances, there's no infrastructure I know if to manage jobs and have them migrate to higher priced instances without losing state.
    - RBerenguel 10y ago
      You can always snapshot and keep track of state as you go (a little bit tricky with Spark, though). We use spot instances for training we know is not vital (as in, has to be done, but rather run it twice and save money anyway that run it for sure). Also, once you know what availability specific instances have you can always choose better (i.e. maybe c3.xlarge is slightly more expensive as spot than large, you can do with large... but xlarge has almost no shutdowns)
- pc86 10y ago
  Presumably that's a decent chunk of what the grant is for?
syllogism 10y ago
That's honestly no way to get anything done... Unless you're replicating someone else's thing exactly, You can't really get by with one training process. You want to be trying different things, and running a few samples of each configuration to account for random variation. I'm not even talking about decadent hyper-parameter sweeps to fine-tune. I'm talking like, how wide do I need my layers to be, what optimizers are good, how deep should I make the network, etc. I want to be training 5-10 models at a time minimum. 20-30 would be much more productive. If I can only train one model at once, it's not really worth the effort — it's better to work on one of the other tickets for the library.