4 ms·

In the GPT-3 era, why not allow the algorithm to change the author’s wording (reordering or phrase rewrites) to more nicely distribute long and short words and

In the GPT-3 era, why not allow the algorithm to change the author’s wording (reordering or phrase rewrites) to more nicely distribute long and short words and give the line breaker more raw material to work with? /s

ericwaller 4y ago
I like it! I've definitely found myself doing this while manually breaking lines in code comments
gwern 4y ago
This is something I've thought about, inspired by the old textfiles where the authors manually space or rewrite to avoid hyphenation and fit exactly into 80col. You don't want to use GPT-3 raw because the number of candidate versions would bankrupt you if you do it naively: there are way too many synonyms for each word and combinatorially many. If you do it in a single pass, then it's not too much different from greedy linebreaking, since it can't 'look forward' to minimize badness of linebreaks & synonym. The system I have in mind would do dynamic programming like usual to find linebreaks, and each line-end word would have a list of synonyms with a distance given by a standard word embedding like word2vec. Then you have a standard DP which tries to minimize the sum of # of hyphens, characters left/right of the ideal position, and distance for a synonym rewrite multiplied by a configurable parameter weight. This will give you a linebreaking algorithm which can be tuned to do pure linebreaking (set the weight of synonyms to ∞) to perfectly preserve meaning even at the cost of appearance, or treat synonyms as free (set to 0) and spent like a drunken sailor to wedge the text in regardless of meaning distortion. You need to generate a lot of synonyms, but you can easily cache that and word embedding NNs are extremely small/fast/easy to run locally, unlike GPT-3. So, won't break the bank. Now, you might say that rewriting words individually, even if the meaning is very close thanks to word2vec, can still lead to weirdness in global meaning, coherency, or text style. True! Then you can run it through GPT-3 as a final phase: take several different candidates (generated by setting the weight to a few different values), feed them through the GPT-3 Search API to find the closest (or the embedding API to do inner-product distance), and take the best one (again, with a weighting factor for how faithful to the original semantics you want to force the rewrites to be). My guess is that given a large dictionary of synonyms, and the final GPT-3 pass to score for naturalness, creating many possible rewrites, the DP search will be able to find extremely subtle rewrites which eliminate the the need for hyphenation or ragged margins while still reading completely naturally. Text run through it will look identical (because we can be careless readers - did you notice the 'the the' in the previous sentence?), but just magically fit perfectly in its column.

ericwaller 4y ago

I like it! I've definitely found myself doing this while manually breaking lines in code comments

gwern 4y ago

This is something I've thought about, inspired by the old textfiles where the authors manually space or rewrite to avoid hyphenation and fit exactly into 80col. You don't want to use GPT-3 raw because the number of candidate versions would bankrupt you if you do it naively: there are way too many synonyms for each word and combinatorially many. If you do it in a single pass, then it's not too much different from greedy linebreaking, since it can't 'look forward' to minimize badness of linebreaks & synonym. The system I have in mind would do dynamic programming like usual to find linebreaks, and each line-end word would have a list of synonyms with a distance given by a standard word embedding like word2vec. Then you have a standard DP which tries to minimize the sum of # of hyphens, characters left/right of the ideal position, and distance for a synonym rewrite multiplied by a configurable parameter weight. This will give you a linebreaking algorithm which can be tuned to do pure linebreaking (set the weight of synonyms to ∞) to perfectly preserve meaning even at the cost of appearance, or treat synonyms as free (set to 0) and spent like a drunken sailor to wedge the text in regardless of meaning distortion. You need to generate a lot of synonyms, but you can easily cache that and word embedding NNs are extremely small/fast/easy to run locally, unlike GPT-3. So, won't break the bank. Now, you might say that rewriting words individually, even if the meaning is very close thanks to word2vec, can still lead to weirdness in global meaning, coherency, or text style. True! Then you can run it through GPT-3 as a final phase: take several different candidates (generated by setting the weight to a few different values), feed them through the GPT-3 Search API to find the closest (or the embedding API to do inner-product distance), and take the best one (again, with a weighting factor for how faithful to the original semantics you want to force the rewrites to be). My guess is that given a large dictionary of synonyms, and the final GPT-3 pass to score for naturalness, creating many possible rewrites, the DP search will be able to find extremely subtle rewrites which eliminate the the need for hyphenation or ragged margins while still reading completely naturally. Text run through it will look identical (because we can be careless readers - did you notice the 'the the' in the previous sentence?), but just magically fit perfectly in its column.