8 ms·
It's fun to find the sources to see how GPT output correlates with the input data: Input texts: Go, rate thy minions, proud insulting boy! Hither to London,
by xiphias2 6y ago
It's fun to find the sources to see how GPT output correlates with the input data:
Input texts:
Go, rate thy minions, proud insulting boy!
Hither to London, to be crown'd our king. / Welcome, sweet prince, to London, to your chamber.
/ Post you to London, and you will find it so /
Now to London,
To see these honours in possession.
How will the country for these woful chances
Misthink the king and not be satisfied!
Output:
Go, rating to London, with all these woful chances
Misthink the king and not be satisfied!
- master_yoda_1 6y agoDid you understand the correlation? And what part of the source code is doing it? Or you had just fun without any understanding?
- xiphias2 6y agoGPT is using multi-headed attention, so of course it's not as simple as putting a few texts together, but I was still interested in finding some similar texts (that can be done because the training data is only 1MB).
- master_yoda_1 6y agowhy don't use TF-IDF and use just elastic search?
- sillysaurusx 6y agoThat's a really interesting idea. Could you go into detail about how you're searching for similar texts using GPT? It's true that the probability distribution is a sort of "edit distance". And GPT has already been used for text compression: https://bellard.org/nncp/gpt2tc.html https://bellard.org/nncp/gpt2tc.html so it seems not too far of a stretch to use it for similarity matching. (Sure, perhaps there are more efficient or more effective techniques than using GPT for this, but I like the idea and am curious how it works.)
- xiphias2 6y agoI was using Ctrl-F in Chrome for the words on the training data (Shakespeare texts). With Teansformer models it's quite easy to print out top Query-Key pairs to debug what happened, but that was not my intention.