6 ms·

Voyages in sentence space

90 points by bkudria 9y ago

TuringTest 9y ago
The voyage looks somewhat random, yet inspiring. I can imagine using it to grab ideas for the overarching story arc in some storytelling. > I am waiting at the entrance of the fantasy kingdom. ... > I have a revelation about reality. Main problem is, it doesn't look at all robust. A small change in a single word at the start or end sentence produces a whole different result. That doesn't look like a "space" at all, as insights about closeness do not work in it.
- TuringTest 9y ago
  Ok, I've played with the "Look around" example and it does behave as a space, with the "Closer to home" option producing somewhat similar results to the starting point, and the slider producing concepts that look farther away from it. Maybe the "Gimme a gradient" option step is too large for sentences that are far away, and an option to produce more steps, closer between them, would produce more intuitive results?
- OscarCunningham 9y ago
  I guess that can happen sometimes in actual spaces too. The shortest path from a point near the north pole to the south pole can change dramatically if you slightly alter its endpoints. Given a high-dimensional space it could be that most possible paths behave in this weird way.
  - jimmytidey 9y ago
    How useful is the spatial analogy if the space in question defies all our spatial intuitions? I can see it has mathematical meaning, but isn't the whole post about having intuitive understanding of sentence space?
    - gugagore 9y ago
      I think it's less about intuitive understanding, and more about having a flat, numerical representation of a sentence that (qualitatively) satisfies some properties.
- ppod 9y ago
  This is essentially the reason that differentiable methods alone don't work that well for NLP. Language is a discrete arbitrary symbol system, so small changes may have large effects at many levels. ("Dog bites man" is not like "Man bites dog", discrete is not like discreet), etc.
- maljx 9y ago
  I feel like the "middle" sentence is not intuitively what you'd normally think of something in between. Even for sentences that are quite close. "Eve walked home slowly" -> "Eve walked home quickly" gives a middle result of "As white hand smiled."
- rockostrich 9y ago
  Robin is an author that dabbles in machine learning and farming olives for making olive oil. It's not surprising that his examples aren't robust, but I always enjoy reading about his endeavors because I end up in a rabbit hole on whatever subject he's writing about.
xtiansimon 9y ago
_"You can also increase or decrease the distance you peer into sentence space from your initial location; as you increase it, the results get more diverse. [...] What you’re seeing is the transition from the richness of arbitrary text to the regularity of this particular sentence space."_ There's a huge gap in here. Mr Sloan has worked on the idea of a gradient in language. He's worked to realate the craft of colors to sentence creativity. But color craft on a computer is just an abstraction for picking colors out of the gammut of screen displays. And this gammut is a subset of the visible spectrum, which is a subset of the electromatnetic spectrum. He didn't work at all on how this project relates to language itself. I guess I have to read the papers. What exactly is 'sentence space'?
- YeGoblynQueenne 9y ago
  >> What exactly is 'sentence space'? It's a multi-dimensional cartesian space where each sentence occupies a single point (i.e. a set of coordinates). The point is that once you have such a sentence space, you can then do geometry over it. Like calculate distance between sentences etc. The hope is of course that the sentence space is constrtucted so as to encode the semantic relation between sentences, so for instance sentences that are close to each other encode similar meaning etc.
  - xtiansimon 9y ago
    Sorry if I wasn't clear--What's in there? You've mentioned 'each sentence' and a 'multi-dimensional cartesian space', what _constitutes_ the 'space'? Each sentence from where? What determines a set of words is a sentence? Is it a WordNet space? Is it a sampling from a corpus? A neural network trained on what? the internet? One tree? What's the forest? Old growth? Jungle? Alpine above the tree line?!
    - deleted 9y ago
      [deleted]
    - YeGoblynQueenne 9y ago
      Ah, OK, you were joking. Sorry for missing it, I had my humour removed at birth :)
atrilumen 9y ago
Can this be used for style transfer? I'm looking for a solution to create a consistent personality for a chatbot, where the content all is created by different team members.
gojomo 9y ago
For a similar tool in the somewhat simpler-to-understand ‘word space’, see Travis Hoppe’s ‘Transorthogonal Linquistics’: https://github.com/thoppe/transorthogonal-linguistics?files=1 https://github.com/thoppe/transorthogonal-linguistics?files=... Shayne Miel has observed that its results for the space from ‘sociology’ to ‘mathematics’ closely match an earlier XKCD comic: https://groups.google.com/d/msg/gensim/a8PF8CInRKk/6XWcp87WB https://groups.google.com/d/msg/gensim/a8PF8CInRKk/6XWcp87WB
- xtiansimon 9y ago
  Warmer. _"English dump of Wikipedia that was sentence and word tokenized by NLTK."_ Wikipedia would certainly cover the morphology of a 'space'. I'm curious how the linguistic relations are derived from 'context', which I gather is what word2vec does. Although NLTK does have an interface to WordNet.

90 points by bkudria 9y ago

The voyage looks somewhat random, yet inspiring. I can imagine using it to grab ideas for the overarching story arc in some storytelling. > I am waiting at the entrance of the fantasy kingdom. ... > I have a revelation about reality. Main problem is, it doesn't look at all robust. A small change in a single word at the start or end sentence produces a whole different result. That doesn't look like a "space" at all, as insights about closeness do not work in it.

TuringTest 9y ago
Ok, I've played with the "Look around" example and it does behave as a space, with the "Closer to home" option producing somewhat similar results to the starting point, and the slider producing concepts that look farther away from it. Maybe the "Gimme a gradient" option step is too large for sentences that are far away, and an option to produce more steps, closer between them, would produce more intuitive results?
OscarCunningham 9y ago
I guess that can happen sometimes in actual spaces too. The shortest path from a point near the north pole to the south pole can change dramatically if you slightly alter its endpoints. Given a high-dimensional space it could be that most possible paths behave in this weird way.
- jimmytidey 9y ago
  How useful is the spatial analogy if the space in question defies all our spatial intuitions? I can see it has mathematical meaning, but isn't the whole post about having intuitive understanding of sentence space?
  - gugagore 9y ago
    I think it's less about intuitive understanding, and more about having a flat, numerical representation of a sentence that (qualitatively) satisfies some properties.
ppod 9y ago
This is essentially the reason that differentiable methods alone don't work that well for NLP. Language is a discrete arbitrary symbol system, so small changes may have large effects at many levels. ("Dog bites man" is not like "Man bites dog", discrete is not like discreet), etc.
maljx 9y ago
I feel like the "middle" sentence is not intuitively what you'd normally think of something in between. Even for sentences that are quite close. "Eve walked home slowly" -> "Eve walked home quickly" gives a middle result of "As white hand smiled."
rockostrich 9y ago
Robin is an author that dabbles in machine learning and farming olives for making olive oil. It's not surprising that his examples aren't robust, but I always enjoy reading about his endeavors because I end up in a rabbit hole on whatever subject he's writing about.

xtiansimon 9y ago

_"You can also increase or decrease the distance you peer into sentence space from your initial location; as you increase it, the results get more diverse. [...] What you’re seeing is the transition from the richness of arbitrary text to the regularity of this particular sentence space."_ There's a huge gap in here. Mr Sloan has worked on the idea of a gradient in language. He's worked to realate the craft of colors to sentence creativity. But color craft on a computer is just an abstraction for picking colors out of the gammut of screen displays. And this gammut is a subset of the visible spectrum, which is a subset of the electromatnetic spectrum. He didn't work at all on how this project relates to language itself. I guess I have to read the papers. What exactly is 'sentence space'?

YeGoblynQueenne 9y ago
>> What exactly is 'sentence space'? It's a multi-dimensional cartesian space where each sentence occupies a single point (i.e. a set of coordinates). The point is that once you have such a sentence space, you can then do geometry over it. Like calculate distance between sentences etc. The hope is of course that the sentence space is constrtucted so as to encode the semantic relation between sentences, so for instance sentences that are close to each other encode similar meaning etc.
- xtiansimon 9y ago
  Sorry if I wasn't clear--What's in there? You've mentioned 'each sentence' and a 'multi-dimensional cartesian space', what _constitutes_ the 'space'? Each sentence from where? What determines a set of words is a sentence? Is it a WordNet space? Is it a sampling from a corpus? A neural network trained on what? the internet? One tree? What's the forest? Old growth? Jungle? Alpine above the tree line?!
  - deleted 9y ago
    [deleted]
  - YeGoblynQueenne 9y ago
    Ah, OK, you were joking. Sorry for missing it, I had my humour removed at birth :)

atrilumen 9y ago

Can this be used for style transfer? I'm looking for a solution to create a consistent personality for a chatbot, where the content all is created by different team members.

gojomo 9y ago

For a similar tool in the somewhat simpler-to-understand ‘word space’, see Travis Hoppe’s ‘Transorthogonal Linquistics’: https://github.com/thoppe/transorthogonal-linguistics?files=1 https://github.com/thoppe/transorthogonal-linguistics?files=... Shayne Miel has observed that its results for the space from ‘sociology’ to ‘mathematics’ closely match an earlier XKCD comic: https://groups.google.com/d/msg/gensim/a8PF8CInRKk/6XWcp87WB https://groups.google.com/d/msg/gensim/a8PF8CInRKk/6XWcp87WB

xtiansimon 9y ago
Warmer. _"English dump of Wikipedia that was sentence and word tokenized by NLTK."_ Wikipedia would certainly cover the morphology of a 'space'. I'm curious how the linguistic relations are derived from 'context', which I gather is what word2vec does. Although NLTK does have an interface to WordNet.