4 ms·
I would be curious to know how much when we write, how much of it is self-attention and how much of it is our fore-brain actually trying to make sense? My gues
by blazespin 7y ago
I would be curious to know how much when we write, how much of it is self-attention and how much of it is our fore-brain actually trying to make sense? My guess is that the more tired / rushed / burned out you are, the % of self attention increases.
Sometimes watching the news, it seems like 90% of what they say when they are 'vamping' is just self-attention.
Has anyone posted any GPT / Hacker News generated text yet? Wisdom of the crowds, indeed. It'd be interesting to post using it with light editing, especially something that uses upvotes for training.
One of the things I was thinking about was training on your favorite novel, so you could have a sort of conversation with it / ask it questions. A kind of interactive cliff notes. However, as looked into it I realized it was still too much of a markov chain like thing to be functionally useful. Fun idea though.
The real win, in all of this, of course is auto completion in different mediums. Code completion demos are pretty wild - https://tabnine.com/blog/deep/ https://tabnine.com/blog/deep/ Come to think about it, you could probably use it for writing academic papers as well assuming you know the content well.
Self-Attention and Human/Computer interaction is a very brave new world. I don't think people really yet know the potential for seismic shift here.
- minimaxir 7y agoI have a very large dump of GPT-2 generated Hacker News titles here: https://github.com/minimaxir/hacker-news-gpt-2 https://github.com/minimaxir/hacker-news-gpt-2 That's just with the smallest 124M model though; on short form content especially, I'm not convinced of the value of larger models.
- leod 7y agoI've trained a Transformer encoder-decoder model (this was slightly before GPT2 came out) to generate HN comments from titles. There is a demo running at https://hncynic.leod.org https://hncynic.leod.org
- rahimnathwani 7y agoThis is cool. If you were to cache the results and generate a unique URL for each, people could easily share the funniest ones.
- leod 7y agoThanks! I actually planned to make results shareable at the start, but, knowing the internet, I did not like the idea of being held responsible for whatever content (say offensive or even illegal things) people would put into the titles.
- CDSlice 7y agoIt doesn't seem very accurate, there isn't close to enough Electron hate whenever it is in the title. This is pure gold though: > How does one make a web app using a standard framework? I've never used it, but it sounds like someone has been able to put together something like a Web app with only one app. Edit: This is even better. > Rewriting a Linux kernel in Rust, by hand, is definitely the right thing to do as a beginner/intermediate programmer.
- nekopa 7y agoMy favorite, even sounds like it would work: Title: My Emacs Productivity Tricks/Hacks hncynic 1 minute ago I used this for some time and never looked back. In my .emacs.d file, the arrow keys, a key with a cursor keys (which are the key bindings for the .emacs.d file above) and then a shortcut to switch to the command that makes use of those. But I now have a full screen keyboard and mouse. Here's another way to do it: M-x { C-c } You go in the current directory, move up the left arrow key, press escape and hit the backspace key.
- 7y ago