3 ms·
In a recent interview with Communications, Hinton said his own research on word vectors goes back to the mid-1980s, when he, David Rumelhart, and Ronald William
by YeGoblynQueenne 11y ago
In a recent interview with Communications, Hinton said his own research on word
vectors goes back to the mid-1980s, when he, David Rumelhart, and Ronald
Williams published work in Nature that demonstrated family relationships as
vectors. "The vectors were only six components long because computers were very
small then, but it took a long time for it to catch on," Hinton said.
Yeah, I know the work he's talking about. It's the one related to this dataset:
https://archive.ics.uci.edu/ml/datasets/Kinship https://archive.ics.uci.edu/ml/datasets/Kinship
From that page:
Creator:
Geoff Hinton
Donor:
J. Ross Quinlan
Data Set Information:
This relational database consists of 24 unique names in two families (they have
equivalent structures). Hinton used one unique output unit for each person and
was interested in predicting the following relations: wife, husband, mother,
father, daughter, son, sister, brother, aunt, uncle, niece, and nephew. Hinton
used 104 input-output vector pairs (from a space of 12x24=288 possible pairs).
The prediction task is as follows: given a name and a relation, have the outputs
be on for only those individuals (among the 24) that satisfy the relation. The
outputs for all other individuals should be off.
Hinton's results: Using 100 vectors as input and 4 for testing, his results on
two passes yielded 7 correct responses out of 8. His network of 36 input units,
3 layers of hidden units, and 24 output units used 500 sweeps of the training
set during training.
Quinlan's results: Using FOIL, he repeated the experiment 20 times (rather than
Hinton's 2 times). FOIL was correct 78 out of 80 times on the test cases.
And yet, if you have a wee look at Hinton's publication on Rexa, there's 43
citations, while there's a single one on Quinlan's (from Muggleton, duh).
So, you know, maybe it's not logic and reasoning that's the problem here, rather
a certain tendency to drum up results of neural models even when they don't do
any better than other techniques.
But, really, it doesn't matter. Google has the airwaves (so to speak). No matter
what happens anywhere else, in academia or business, their stuff is going to be
publicised the most and that's what we all have to deal with.
- romaniv 11y ago> ut, really, it doesn't matter. Google has the airwaves (so to speak). No matter what happens anywhere else, in academia or business, their stuff is going to be publicised the most and that's what we all have to deal with. One thing that bothers me about ANN hype is that those models are very resource and data hungry. Which suits Google perfectly, but will keep that kind of AI out of people's hands/computers for decades to come.
- wcrichton 11y agoThis is checked back by the advent of cloud computing and public datasets, though. It's super easy for anyone to spin up a thousand nodes and run TensorFlow. There's a huge number of open-source image and text datasets. The limiting factor is then money, not hardware.
- yongjik 11y agoWell, I just clicked through your link and the "data" is very nicely organized like this (yes, literally): father(Christopher, Arthur) father(Christopher, Victoria) father(Andrew, James) father(Andrew, Jennifer) ... mother(Penelope, Arthur) mother(Penelope, Victoria) mother(Christine, James) ... This is about as close to a "logical input" as one can possibly get. One would naturally expect a logic-based system to handle this very well, so the fact that it indeed does is not that interesting. Moreover, IMHO, it is not clear how we could extend a logic-based system to handle the messiness of natural language. (Well, perhaps it was also unclear how to do that with word vectors, when Hinton wrote that paper, but now it's pretty clear.) Not saying word vectors are the "ultimate" solution, but logic isn't even at the level of word vectors.
- YeGoblynQueenne 11y agoThe data is in that format because Quinlan pre-processed it for use in FOIL, which expects first-order logic terms as input, but I don't see what that's a problem? Turning natural language to word embeddings is a very similar process: you might as well argue that the fact that a statistical system does well with word vectors is not that interesting (it is, it's a brilliant idea). >> it is not clear how we could extend a logic-based system to handle the messiness of natural language. There's a lot of recent stuff in Muggleton's Latest Advances in Inductive Logic Programming and a ton of work from others, before and after that. There's no problem in handling "messiness" with logic. Like I note elsewhere, your computer is a logic machine and it handles messiness just fine (no offense meant).