float-trip

4 ms·

1.
by float-trip 2y ago
Reddit's caches are set up to only ever return the last 1,000 of anything. So for example - you can't scroll past 1k items on /new, and if you save more than 1k posts then you'll have to unsave some to retrieve the other
2.
by float-trip 3y ago
Related comment from gwern: https://news.ycombinator.com/item?id=38438859 . Can't find the docs now - I think they were the old GPT 3 ones - but they suggested a low value somewhere around 0.01 and 0.1. Also - why qlora
3.
by float-trip 3y ago
That's what I ended up doing (`[Author] username [Title] post title...`) > Adding new tokens needs a ton of data to train what the token means. But how much? 300M tokens is fine for a simple version of ChatML with ~4 tokens. Not for
4.
by float-trip 3y ago
I tried adding special tokens for a reddit-style dataset once. The format was: `<|post_author|>username<|post_title|>title here...` The resulting model was so much worse than just formatting everything plaintext. This was with M
5.
by float-trip 3y ago
Thanks for writing up. Rather than zeroing out the loss for the prompt, did you also try using weighted loss with Axolotl? At one point, Microsoft's GPT 3 docs suggested this was beneficial when the responses are short (like you have w
6.
by float-trip 3y ago
There's a breakdown here for anyone interested (ctrl+f "weight flops for") https://medium.com/@dzmitrybahdanau/the-flops-calculus-of-la...
7.
by float-trip 3y ago
The prompt for Bing Chat was previously reproduced by the same person as here, using the same trick. The Bing lead disclaimed it as inaccurate, though: https://twitter.com/MParakhin/status/1627491603731423232
8.
by float-trip 3y ago
Two other recent literature reviews worth reading: "Transformer Taxonomy" - https://kipp.ly/blog/transformer-taxonomy/ "Five years of progress in GPTs" - https://finbarrtimbers.subst