2 ms·

I love the concept and execution but it really is so sad that the results weren’t more compelling. I do wonder whether the problem is that training on multiple

I love the concept and execution but it really is so sad that the results weren’t more compelling. I do wonder whether the problem is that training on multiple fonts just doesn’t actually add more data points to the dataset. Fundamentally, you’re going to get a model that knows how to uppercase or lowercase each canonical Latin letter. This model might actually be quite good at generating appropriate lowercase forms for a font given the uppercase glyphs, but it’s not learning an abstract concept of ‘lowercaseness’. Instead of extending the training set with more fonts, the only real source of additional training data would be more alphabets. Cyrillic and Greek both have case systems so just adding those would have more than doubled the number of training cases - I think the trick would be to start from the Unicode case mapping tables to generate your example data, to give your model more variety of upper/lowercase pairs to get its teeth into and really make it possible to ask it to uppercase an arbitrary letter form.

gwern 6y ago
The approach he used didn't really make sense. I get the impression Tom is so dedicated to his old custom hackedup NN framework that he tailors the approach to what he could easily do in it. If you asked someone else, "I want to automatically turn characters into 'more uppercase' or 'more lowercase' versions, and it has to involve neural networks; what do?", they would say something like "Easy enough! dump character maps into a standard StyleGAN2-ADA, train for a few days, then find the latent direction corresponding to uppercasing & lowercasing, and edit whatever font you please." (People have been generating fonts with RNNs or GANs for ages.) You could do this over a week or so, the tooling has gotten pretty easy to use. And if you asked someone on the cutting-edge, they'd suggesting using OA CLIP through Aleph/BigSleep/etc to automatically edit images using a text input prompt of "a lowercase letter" or "an uppercase letter", or one of the hybrids like StyleCLIP https://github.com/orpatashnik/StyleCLIP https://github.com/orpatashnik/StyleCLIP . This approach might take all of an hour or two (but results probably would be worse).

gwern 6y ago

The approach he used didn't really make sense. I get the impression Tom is so dedicated to his old custom hackedup NN framework that he tailors the approach to what he could easily do in it. If you asked someone else, "I want to automatically turn characters into 'more uppercase' or 'more lowercase' versions, and it has to involve neural networks; what do?", they would say something like "Easy enough! dump character maps into a standard StyleGAN2-ADA, train for a few days, then find the latent direction corresponding to uppercasing & lowercasing, and edit whatever font you please." (People have been generating fonts with RNNs or GANs for ages.) You could do this over a week or so, the tooling has gotten pretty easy to use. And if you asked someone on the cutting-edge, they'd suggesting using OA CLIP through Aleph/BigSleep/etc to automatically edit images using a text input prompt of "a lowercase letter" or "an uppercase letter", or one of the hybrids like StyleCLIP https://github.com/orpatashnik/StyleCLIP https://github.com/orpatashnik/StyleCLIP . This approach might take all of an hour or two (but results probably would be worse).