3 ms·

I love this. I first tried it with Grover in 2019 and ran into similar token issues: https://twitter.com/jonathanfly/status/1143773270451396608?lang=en https://

I love this. I first tried it with Grover in 2019 and ran into similar token issues: https://twitter.com/jonathanfly/status/1143773270451396608?lang=en https://twitter.com/jonathanfly/status/1143773270451396608?l... Looks like you don't have a 'most common/simple words' vocab restriction? That's a pretty fun one too. https://twitter.com/jonathanfly/status/1144345934857474048 https://twitter.com/jonathanfly/status/1144345934857474048

Der_Einzige 4y ago
I feel bad for not being active on Twitter, specifically because I missed out on stuff like this (which predates my paper). I should have cited this thread since it's evidence that someone else had utilized this technique in my paper! I think this vocabulary filtering technique will pick up steam again now that good quality open source LLMs are proliferating. I've always wanted to try these techniques with ChatGPT, and the inability to do so has caused me to somewhat publicly complain about their lack of access to ChatGPT's vocabulary probability distribution (they don't give more than the top 10 most likely continuations).
- kastnerkyle 4y ago
  You might be interested in this thread from 2019 here on HN as well, I linked to a line of work on constrained sequence generation from Sony CSL that is pretty nifty https://news.ycombinator.com/item?id=21334639 https://news.ycombinator.com/item?id=21334639 As an aside - do you have any academic references for masking / constraining based next step transitions? It's a trick I've been messing with for years, but I probably picked this up from some older poetry generation work. Haven't been able to find any direct reference for simple 1 step masking, let alone more involved things like constraining based on the unrolled, modified markov sequence or things bigger than 1 step.
  - Der_Einzige 4y ago
    I think that this is what you will be looking for https://huggingface.co/blog/constrained-beam-search https://huggingface.co/blog/constrained-beam-search This technique has been quietly available within HF for awhile, and has a lot of real world use-cases.
    - kastnerkyle 4y ago
      For any models where you get access to the code (counting HF in this), definitely a lot of tooling available - and this tool looks really nice hadn't seen it before! Hope similar levels of control can become available for the bigger, less accessible systems, even though people don't want to expose token log-likelihoods there should be ways. I've also been reading your paper on these topics, based on previous comments on HN. So thanks for that! Always interested in work about constrained generation. I'd really like to see some version of the techniques from this palindrome generation https://www.ijcai.org/Proceedings/15/Papers/353.pdf https://www.ijcai.org/Proceedings/15/Papers/353.pdf revisited in modern contexts. Or some of the other Sony CSL work on this topic (e.g. https://www.francoispachet.fr/wp-content/uploads/2021/01/papadopoulos-15b.pdf https://www.francoispachet.fr/wp-content/uploads/2021/01/pap...) which I will admit I (still) do not understand yet.

Der_Einzige 4y ago

I feel bad for not being active on Twitter, specifically because I missed out on stuff like this (which predates my paper). I should have cited this thread since it's evidence that someone else had utilized this technique in my paper! I think this vocabulary filtering technique will pick up steam again now that good quality open source LLMs are proliferating. I've always wanted to try these techniques with ChatGPT, and the inability to do so has caused me to somewhat publicly complain about their lack of access to ChatGPT's vocabulary probability distribution (they don't give more than the top 10 most likely continuations).

kastnerkyle 4y ago
You might be interested in this thread from 2019 here on HN as well, I linked to a line of work on constrained sequence generation from Sony CSL that is pretty nifty https://news.ycombinator.com/item?id=21334639 https://news.ycombinator.com/item?id=21334639 As an aside - do you have any academic references for masking / constraining based next step transitions? It's a trick I've been messing with for years, but I probably picked this up from some older poetry generation work. Haven't been able to find any direct reference for simple 1 step masking, let alone more involved things like constraining based on the unrolled, modified markov sequence or things bigger than 1 step.
- Der_Einzige 4y ago
  I think that this is what you will be looking for https://huggingface.co/blog/constrained-beam-search https://huggingface.co/blog/constrained-beam-search This technique has been quietly available within HF for awhile, and has a lot of real world use-cases.
  - kastnerkyle 4y ago
    For any models where you get access to the code (counting HF in this), definitely a lot of tooling available - and this tool looks really nice hadn't seen it before! Hope similar levels of control can become available for the bigger, less accessible systems, even though people don't want to expose token log-likelihoods there should be ways. I've also been reading your paper on these topics, based on previous comments on HN. So thanks for that! Always interested in work about constrained generation. I'd really like to see some version of the techniques from this palindrome generation https://www.ijcai.org/Proceedings/15/Papers/353.pdf https://www.ijcai.org/Proceedings/15/Papers/353.pdf revisited in modern contexts. Or some of the other Sony CSL work on this topic (e.g. https://www.francoispachet.fr/wp-content/uploads/2021/01/papadopoulos-15b.pdf https://www.francoispachet.fr/wp-content/uploads/2021/01/pap...) which I will admit I (still) do not understand yet.